| Status | ||||
|---|---|---|---|---|
|
| Table of Contents |
|---|
What is the
...
PIPEFORCE HTTP API?
The Pipeline PIPEFORCE HTTP API allows you to manage and execute pipelines via a HTTP requests, similar to a REST call. any aspect of the PIPEFORCE platform via RESTful HTTP calls. You can setup automation and data integration apps. Execute commands and pipelines. Manage users, webhooks, configs, gateway endpoints and much more using this API.
Here is a list of of available the most important entry-endpoints:
Path | Method | Description | ||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| Execute adhoc
See: https://pipeforce.github.io/api#/Pipeline%20API
| ||||||||||||||||||||||||||||||||||||||
|
| Store | ||||||||||||||||||||||||||||||||||||||
|
| Load and execute
| ||||||||||||||||||||||||||||||||||||||
|
| Return | ||||||||||||||||||||||||||||||||||||||
|
| Delete | ||||||||||||||||||||||||||||||||||||||
|
| Create, read, update and delete persisted JSON documents, attachments and other data types in the persistence DB, called the Property Store. | ||||||||||||||||||||||||||||||||||||||
|
| Execute
See here for a full list of available commands: |
For more details see the API Documentation.
Execute
...
pipeline (adhoc)
| Info |
|---|
For security reasons, executing adhoc pipelines is by default restricted to admin, support and developer users in DEV stage only. For production stage you should always consider to use persisted pipelines instead. |
In order to execute an adhoc pipeline (= execute a pipeline without persisting it), you can send a HTTP POST request to this endpoint url with the pipeline script as YAML or JSON in the body:
...
| Info |
|---|
|
The most common usage scenario is to send a pipeline YAML script to the server and execute it there without storing it.
...
You can also send and execute a pipeline in JSON format this way. In this case you have to set the Content-Type header to application/json instead.
HTTP Methods
In order to execute a pipeline script on the backend, you need to send a HTTP POST request to the endpoint /api/v3/pipeline by using one of these HTTP options:
...
HTTP Method
...
Content-Type
...
Request Body
...
Message Body
...
Execution
...
is set to
...
is set to
...
is set to
...
then will become
...
is
...
POST
...
application/json
...
A JSON pipeline document.
...
null or the value set by the body section of the pipeline.
...
A pipeline set as JSON from the request body.
...
POST
...
application/yaml
...
A YAML pipeline document.
...
null or the value set by the body section of the pipeline.
...
A pipeline set as YAML from the request body. See examples with no body and embedded body.
...
POST
...
application/x-www-form-urlencoded
...
The pipeline as URL encoded request query string whereas the key of each parameter is the command name and the value defines the parameters to the command. Key and value must be separated by a colon :. Multiple command parameters are separated by semicolon ;. Example (before url encoding): log=message:'Hello World!';level:INFO
...
null
...
A pipeline set as URL encoded query string from the request body. See example with url encoded query.
...
POST
...
None
...
.
...
null or the value set by the body section of the pipeline.
...
A pipeline set as YAML from the request body. Any other format in the body will throw an 400 bad request error.
...
POST
...
multipart/form-data
...
One part with name="pipeline" with a YAML pipeline and one or more parts with name="file" in the content disposition header. See how HTTP multipart works.
...
Content Collection referencing all fileparts.
...
A pipeline set as YAML from the pipeline part and with a content collection in the body created from all file parts of the request body. See Example.
Using the HTTP method GET with the /pipeline endpoint is not possible and will cause an error.
Example
...
: No body
In case you would like to execute a pipeline YAML script without a message body, you can run this:
| Code Block | ||
|---|---|---|
| ||
curl -u 'username:password' \
-X POST "https://hub-try.pipeforce.org/api/v3/pipeline" \
-H "Content-Type: application/yaml" \
--data-binary @- << EOF
pipeline:
- drive.read:
path: /my.pdf
- pdf.stamp:
text: "Hello World!"
- drive.save:
path: /my-stamped.pdf
EOF |
Example
...
: With body embedded in the pipeline
In case you would like to execute a pipeline YAML script with message body, which is embedded inside the YAML, you can run it like this example shows:
| Code Block | ||
|---|---|---|
| ||
curl -u 'username:password' \
-X POST "https://hub-try.pipeforce.org/api/v3/pipeline" \
-H "Content-Type: application/yaml" \
--data-binary @- << EOF
pipeline:
- log:
message: "BODY: ${body.text}"
body: {"text": "Hello World!"}
EOF |
Example 7: With url encoded pipeline
In this example, a simple adhoc pipeline is called using an url encoded query string.
Suppose you have a pipeline that looks like this:
| Code Block | ||
|---|---|---|
| ||
pipeline:
- datetime:
format: dd.MM.YY
- set.body:
value: "Today is: ${body}" |
You can rewrite it to a query string like this (not url encoded):
| Code Block |
|---|
datetime=format:dd.MM.YY&set.body=value:"Today is: ${body}" |
The query parameter key becomes the command name (for example
datetime).The query parameter value specifies the parameters to the command in the format of
<key>:<value>pairs. The value part of such a parameter can optionally be put into 'single' or "double" quotes. Examples:log=message:HELLO WORLDlog='HELLO WORLD'log="HELLO WORLD"
Multiple
<key>:<value>pairs must be separated by;. Example:log='HELLO WORLD';level=INFO
Multiple commands must be separated by
&. Examples:log='HELLO WORLD';level=INFO&datetimelog='HELLO WORLD';level=INFO&datetime=format:dd.MM.YY
Finally, values need to be URL encoded. Example:
datetime=format:dd.MM.YYencodes todatetime=format%3Add.MM.YY
For more details about the application/x-www-url-encoded content type in the HTTP standard see for example: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST
After the values have been url encoded, the query string looks like this:
| Code Block |
|---|
datetime=format%3Add.MM.YY&set.body=value%3A%20%22Today%20is%3A%20%23%7Bbody%7D%2 |
Below you can find the execution example using this url encoded query string:
| Code Block | ||
|---|---|---|
| ||
curl -u 'username:password' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-X POST 'https://hub-<your-domain>/api/v3/pipeline' \
-d 'datetime=format%3Add.MM.YY&set.body=value%3A%20%22Today%20is%3A%20%23%7Bbody%7D%22' |
Example
...
: With multipart body
Let's assume you would like to execute a pipeline script and additionally processing one or more files in this same pipeline which must be placed in the body.
...
Note that the pipeline has the part name pipeline and one or more files must all have the name file.
...
Execute pipeline (persisted)
A persisted pipeline is one which is stored in the property store.
It is also possible to load and execute such a persisted pipeline stored in from the property store by sending a HTTP POST request toIn order to execute an adhoc pipeline (= with a Pipeline YAML script uploaded to the server), you have to send a HTTP POST request to this endpoint url:
| Code Block |
|---|
https://hub-NS.pipeforce.net/api/v3/pipeline:<path>{path} |
| Info |
|---|
|
...
| Code Block |
|---|
https://hub-NS.pipeforce.net/api/v3/pipeline:uuid:<uuid>{uuid} |
| Info |
|---|
|
This uuid-approach has the advantage that it will work even if the pipeline has been moved / renamed to a different path.
Any request parameter given will be set as variables vars to the pipeline.
In case there is a body in the request, it will be set as initial body to the pipeline.
If header ContentType is set to application/json the body will be parsed to a JSON object and provided as initial body value. Otherwise, the initial body will be provided as content object or it will be null in case no request body was set.
HTTP Methods
In order to execute a persisted pipeline on the backend, you need to send a HTTP POST request to the endpoint /api/v3/pipeline:<path> or /api/v3/pipeline:uuid:<uuid> by using one of these HTTP options:
...
HTTP Method
...
Content-Type
...
Request Body
...
Message Body
...
Execution
...
is set to
...
is set to
...
is set to
...
then will become
...
is
...
POST
...
application/json
...
A JSON data document.
...
The JSON data document parsed to a JSON instance.
...
The persisted pipeline loaded from the property store using its path or uuid.
...
POST
...
Any
...
Any data.
...
The data from the body provided as a content object
...
The persisted pipeline loaded from the property store using its key or uuid.
...
POST
...
None
...
None
...
null
...
The persisted pipeline loaded from the property store using its path or uuid.
...
PUT
...
application/json
...
A JSON pipeline.
...
None
...
The pipeline given as JSON in the request body will be converted and stored as YAML property with path given by <path> from request path. In case a pipeline with such path already exists, updates the existing one.
...
PUT
...
application/yaml
...
A YAML pipeline.
...
None
...
The pipeline given as YAML in the request body will be stored as YAML property with path given by <path> from request path. In case a pipeline with such path already exists, updates the existing one.
...
PUT
...
application/x-www-form-urlencoded
...
An url encoded pipeline URI.
...
None
...
The pipeline given as url encoded pipeline URI in the request body will be converted and stored as YAML property with path given by <path> from request path. In case a pipeline with such path already exists, updates the existing one.
...
For more details about managing persisted pipelines, see the API documentation.
Example: Load and Execute persisted pipeline
In this example, a pipeline persisted under path global/app/myapp/pipeline/mypipeline will be loaded and executed at the backend without any body and or request parameters:
| Code Block | ||
|---|---|---|
| ||
curl -u 'username:password' \ -X POST 'https://<hostname>hub-NS.pipeforce.net/api/v3/pipeline:global/app/myapp/pipeline/mypipeline' |
Example
...
:
...
Persist a pipeline (
...
save it in property store)
In this example a new persisted pipeline will be created under path global/app/myapp/pipeline/newpipeline.:
| Code Block | ||
|---|---|---|
| ||
curl -u 'username:password' \
-h 'Content-Type: application/yaml' \
-X PUT 'https://<hostname>/api/v3/pipeline:global/app/myapp/pipeline/newpipeline' \
--data-raw 'pipeline:
- log:
message: "HELLO WORLD!"' |
...