SINCE VERSION 1.0
What is the Pipeline HTTP API?
The Pipeline HTTP API allows you to manage and execute pipelines via a HTTP requests, similar to a REST call. Here is a list of of available endpoints:
Path | Method | Description |
---|---|---|
| POST | Execute adhoc |
| PUT | Store |
| POST | Load and execute
|
| GET | Return |
| DELETE | Delete |
For more details see the API Documentation.
Execute adhoc pipeline
In order to execute an adhoc pipeline (= execute a pipeline without persisting it), you can send a HTTP POST
request to this endpoint url with the pipeline script as YAML or JSON in the body:
https://hub-NS.pipeforce.net/api/v3/pipeline
Replace
<NS>
by your own namespace.
The most common usage scenario is to send a pipeline YAML script to the server and execute it there without storing it.
Because of security, adhoc pipelines are limited in the features what they can do. If you need full functionality, consider to execute a persisted pipeline instead.
Example
Let's for example assume an adhoc pipeline like this to be executed at server side:
pipeline: - body.set: value: "Today is: ${@date.now()}"
This pipeline can be executed at server side by sending it in the body of a HTTP POST
request with a header of Content-Type: application/yaml
, like this HTTP message log shows:
POST /api/v3/pipeline HTTP/1.1 Host: https://hub-NS.pipeforce.net Content-Type: application/yaml Authorization: Basic bXl1c2VybmFtZTptcGFzc3dvcmQ= Content-Length: 63 pipeline: - body.set: value: "Today is: ${@date.now()}"
As a cURL command, this could be executed in your terminal like this:
curl --location 'https://hub-NS.pipeforce.net/api/v3/pipeline' \ --user 'myusername:mypassword' \ --header 'Content-Type: application/yaml' \ --data-raw 'pipeline: - body.set: value: "Today is: ${@date.now()}"'
And here you can see a Postman snippet as well (replace {{hubBaseUrl}}
by your own hub url):
The response result of all of these requests would be something like this:
Today is: 31.07.2023, 15:54:16
You can also send and execute a pipeline in JSON format this way. In this case you have to set the Content-Type
header to application/json
instead.
HTTP Methods
In order to execute a pipeline script on the backend, you need to send a HTTP POST request to the endpoint /api/v3/pipeline
by using one of these HTTP options:
HTTP Method |
| Request Body | Message Body | Execution |
---|---|---|---|---|
is set to | is set to | is set to | then will become | is |
|
| A JSON pipeline document. |
| A pipeline set as JSON from the request body. |
|
| A YAML pipeline document. |
| A pipeline set as YAML from the request body. See examples with no body and embedded body. |
|
| The pipeline as URL encoded request query string whereas the key of each parameter is the command name and the value defines the parameters to the command. Key and value must be separated by a colon |
| A pipeline set as URL encoded query string from the request body. See example with url encoded query. |
| None | A YAML pipeline document (same as with |
| A pipeline set as YAML from the request body. Any other format in the body will throw an |
|
| One part with | Content Collection referencing all | A pipeline set as YAML from the |
Using the HTTP method GET
with the /pipeline
endpoint is not possible and will cause an error.
Example 5: No body
In case you would like to execute a pipeline YAML script without a message body, you can run this:
curl -u 'username:password' \ -X POST "https://hub-try.pipeforce.org/api/v3/pipeline" \ -H "Content-Type: application/yaml" \ --data-binary @- << EOF pipeline: - drive.read: path: /my.pdf - pdf.stamp: text: "Hello World!" - drive.save: path: /my-stamped.pdf EOF
Example 6: With body embedded
In case you would like to execute a pipeline YAML script with message body, which is embedded inside the YAML, you can run this:
curl -u 'username:password' \ -X POST "https://hub-try.pipeforce.org/api/v3/pipeline" \ -H "Content-Type: application/yaml" \ --data-binary @- << EOF pipeline: - log: message: "BODY: ${body.text}" body: {"text": "Hello World!"} EOF
Example 7: With url encoded pipeline
In this example, a simple adhoc pipeline is called using an url encoded query string.
Suppose you have a pipeline that looks like this:
pipeline: - datetime: format: dd.MM.YY - set.body: value: "Today is: ${body}"
You can rewrite it to a query string like this (not url encoded):
datetime=format:dd.MM.YY&set.body=value:"Today is: ${body}"
The query parameter key becomes the command name (for example
datetime
).The query parameter value specifies the parameters to the command in the format of
<key>:<value>
pairs. The value part of such a parameter can optionally be put into 'single' or "double" quotes. Examples:log=message:HELLO WORLD
log='HELLO WORLD'
log="HELLO WORLD"
Multiple
<key>:<value>
pairs must be separated by;
. Example:log='HELLO WORLD';level=INFO
Multiple commands must be separated by
&
. Examples:log='HELLO WORLD';level=INFO&datetime
log='HELLO WORLD';level=INFO&datetime=format:dd.MM.YY
Finally, values need to be URL encoded. Example:
datetime=format:dd.MM.YY
encodes todatetime=format%3Add.MM.YY
For more details about the application/x-www-url-encoded
content type in the HTTP standard see for example: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST
After the values have been url encoded, the query string looks like this:
datetime=format%3Add.MM.YY&set.body=value%3A%20%22Today%20is%3A%20%23%7Bbody%7D%2
Below you can find the execution example using this url encoded query string:
curl -u 'username:password' \ -H 'Content-Type: application/x-www-form-urlencoded' \ -X POST 'https://hub-<your-domain>/api/v3/pipeline' \ -d 'datetime=format%3Add.MM.YY&set.body=value%3A%20%22Today%20is%3A%20%23%7Bbody%7D%22'
Example 8: With multipart body
Let's assume you would like to execute a pipeline script and additionally processing one or more files in this same pipeline which must be placed in the body.
Here’s an example of an HTTP multipart request as defined by the HTTP specification. This example does multiple steps in one request: It uploads a PDF, puts a new text on this PDF and finally stores it at server side:
curl -u 'username:password' \ -X POST 'https://hub-try.pipeforce.org/api/v3/pipeline' \ -F 'pipeline="pipeline: - pdf.stamp: text: \"Hello World!\" - drive.save: path: /my-stamped.pdf "' \ -F 'file=@/my1.pdf' \
Note that the pipeline has the part name pipeline
and one or more files must all have the name file
.
HTTP API for persisted pipeline calls
It is also possible to load and execute a pipeline stored in the property store by sending a HTTP POST request to
In order to execute an adhoc pipeline (= with a Pipeline YAML script uploaded to the server), you have to send a HTTP POST
request to this endpoint url:
https://hub-NS.pipeforce.net/api/v3/pipeline:<path>
Replace
<NS>
by your own namespace.Replace <path> by the path of the pipeline in the property store.
Instead of using the property path, you can also set the uuid of the pipeline property:
https://hub-NS.pipeforce.net/api/v3/pipeline:uuid:<uuid>
Replace
<NS>
by your own namespace.Replace
<uuid>
by the uuid of the pipeline property to be loaded and executed.
This approach has the advantage that it will work even if the pipeline has been moved / renamed to a different path.
Any request parameter given will be set as variables to the pipeline.
In case there is a body in the request, it will be set as initial body to the pipeline.
If header ContentType
is set to application/json
the body will be parsed to a JSON object and provided initial body value. Otherwise, the initial body will be provided as content object or it will be null in case no request body was set.
HTTP Methods
In order to execute a persisted pipeline on the backend, you need to send a HTTP POST request to the endpoint /api/v3/pipeline:<path>
or /api/v3/pipeline:uuid:<uuid>
by using one of these HTTP options:
HTTP Method |
| Request Body | Message Body | Execution |
---|---|---|---|---|
is set to | is set to | is set to | then will become | is |
|
| A JSON data document. | The JSON data document parsed to a JSON instance. | The persisted pipeline loaded from the property store using its path or uuid. |
| Any | Any data. | The data from the body provided as a content object | The persisted pipeline loaded from the property store using its key or uuid. |
| None | None |
| The persisted pipeline loaded from the property store using its path or uuid. |
|
| A JSON pipeline. | None | The pipeline given as JSON in the request body will be converted and stored as YAML property with path given by |
|
| A YAML pipeline. | None | The pipeline given as YAML in the request body will be stored as YAML property with path given by |
|
| An url encoded pipeline URI. | None | The pipeline given as url encoded pipeline URI in the request body will be converted and stored as YAML property with path given by |
Example 9: Execute persisted pipeline
In this example, a pipeline persisted under path global/app/myapp/pipeline/mypipeline
will be loaded and executed at the backend without any body and request parameters:
curl -u 'username:password' \ -X POST 'https://<hostname>/api/v3/pipeline:global/app/myapp/pipeline/mypipeline'
Example 10: Store a pipeline (persist it)
In this example a new persisted pipeline will be created under path global/app/myapp/pipeline/newpipeline
.
curl -u 'username:password' \ -h 'Content-Type: application/yaml' \ -X PUT 'https://<hostname>/api/v3/pipeline:global/app/myapp/pipeline/newpipeline' \ --data-raw 'pipeline: - log: message: "HELLO WORLD!"'
Add Comment