...

Pipelines and Commands are Functions as a Service (FaaS). Their main purpose is to create and maintain data integrations between different systems and implement business logic in a fairly flexible and easy way.

Overview

...

Introduction

The main purpose of Pipelines and Commands is to create and maintain data integrations between different systems and to implement business logic in a fairly flexible and easy way:

• A pipeline is written in an executable YAML format.

• Each pipeline consists of one or more commands.

• A command is a component which can create, edit, send data and execute logic.

• The output of the first command will become the input of the second, aso.

• A pipeline can be executed manually or by a trigger.

• Commands are sometimes also referred to as Pipes.

...

• The pipeline can be created in your preferred local editor or in the Online Workbench (since version 7.0only

  Status
  colour Blue title ENTERPRISE

  ,

  Status
  colour Purple title CORPORATE

  ).

• To manually execute a pipeline locally you can use the CLI (we will cover these possibilities of the CLI later in this chapter):

  • pi pipeline file /path/to/your/local/pipeline.pi.yaml

  • pi pipeline remote key/of/your/pipeline/on/server

  • pi pipeline uri “inline-pipeline”

• But the easiest way to ad-hoc execute a pipeline is by using the Online Workbench of your portal (available since version 7.0 for

  Status
  colour Blue title ENTERPRISE

  ,

  Status
  colour Purple title CORPORATE

  licenses).

What is a Command?

The most important part of a pipeline is a command. Easily spoken, a command is a single server side task or function which can be executed from outside by its unique name. It is based on the command pattern and function as a service software architecture patterns. It supports input parameters, an input message and can optionally produce an output message:

...

Execution of a single command

There are many possibilities to execute a single command. This makes sure that for any purpose you can use the best matching approach.

Via Online Workbench

Only:

Since version 7.0 of PIPEFORCE, the Online Workbench is available. This is an advanced online editor with full code completion support where you can write pipelines and commands to be executed and then run them online. This is the easiest and most preferred way to ad-hoc execute a command or commands in pipelines. Here you can see the log command as an example to be executed there:

...

Via Command Line Interface (

...

CLI)

Another approach to execute a single command is by using the Command Line Interface (CLI) (CLI). For example there is a built-in command datetime which returns the current date time. A call of this command using the CLI would look like this:

...

Via HTTP GET and POST

...

You can execute any command also as a rest-like endpoint using HTTP GET or POST calls. See this example using curl to call the datetime command:

...

Code Block
curl -H "apitoken: <YOUR_TOKEN>" \ -X GET "https://hub-acme.pipeforce.net/api/v3/command/log?message=HELLO"

Via online documentation

Only:

There is a web UI available in the portal in order to document any command enabled for your instance. Furthermore you can also execute commands for testing purposes using this web interface. By default the url to the command form looks like this:

...

Whereas <NAME> needs to be replaced by the name of the command. Here’s an example of the default command form for the datetime command:

...

Documentation via CLI

An alternative to see the documentation of all available commands is to use the CLI tool:

...

log:
  type: "object"
  description: "Logs the given input message without changing it. Sets the log message\
    \ in the body in case body is empty. Doesn't overwrite any existing content in\
    \ the body."
  inputType: "JsonNode"
  outputType: "JsonNode"
  properties:
    message:
      type: "number"
      description: "The message to log. Can be a string or a pipe expression. If null\
        \ or empty, the full pipe message will be logged."
      default: null
    level:
      type: "string"
      description: "The log level. Can be one of DEBUG, TRACE, INFO, WARN, ERROR.\
        \ If null or empty, INFO will be used."
      default: "INFO"
  required:
  - "message"

Summary

...

What is a Pipeline?

Two or more commands can be combined to a pipeline. If such a pipeline gets executed, the commands in it will be executed by their serial ordering within the pipeline. The output message of the first command will become the input message of the next command and so on. By default, such a pipeline is written in a YAML format.

...

The PDF file my.pdf is the output from the command drive.read and will become the input for pdf.stamp. After pdf.stamp was executed and has put the stamp on the file, it will send it as output to drive.save which stores it to the data room. Instead of a PDF file, the message body can also be of any other format like JSON, string or a like.

{
  "pipeline": [
    {
      "drive.read": {
        "path": "/my.pdf"
      }
    },
    {
      "pdf.stamp": {
        "text": "Hello World!"
      }
    },
    {
      "drive.save": {
        "path": "/my-stamped.pdf"
      }
    }
  ]
}

Execution of a pipeline

Similar to a single command, a pipeline of commands is executed by sending it to server. Its a more advanced version of the “Function as a Service” approach since it can execute a bunch of commands in a predefined order.

Via Online Workbench

Only:

Since version 7.0 of PIPEFORCE, the Online Workbench is available. This is an advanced online editor with full code completion support where you can write pipelines and commands to be executed and then run them online. This is the easiest and most preferred way to ad-hoc execute a command or pipelines. Here you can see a simple pipeline after its ad-hoc execution as an example:

...

Via CLI

Another approach to execute a pipeline is by using the CLI: Command Line Interface (CLI).

Local file

Lets assume you have a local pipeline file stored at src/global/app/myapp/pipeline/test.pi.yaml inside of your PIPEFORCE workspace, then you can execute it via this CLI call:

...

Remote

In case you have stored your pipeline at server side in the property store (Property Store), then you can execute it using this call:

...

This command searches for a property in the property store with key global/app/myapp/pipeline/test and executes it plus sends any results back to your terminal.

Pipeline URI

A third option to execute a pipeline is by using a pipeline uri which is an inline version of a pipeline. You can rewrite any pipeline YAML fromat also as a pipeline uri. Lets assume this example:

...

This is handy especially for smaller pipelines which you want to execute ad-hoc for example.

Via HTTP

You can execute a pipeline also by sending it via HTTP POST to the server. See this example:

...

With this flexibility you can for example define a bash script and store it locally to execute this pipe with a single command and not much configuration, setup or coding required.

Pipeline scopes

Every pipeline script may consist of four main sections, called scopes:

...

All scopes except pipeline are optional in a pipeline script. Even if not explicitly defined in the pipeline script, each scope exists implicitly. That means you can access it and read / set values from / on it without declaring it in the pipeline. For example by using a pipeline expression (PE).

headers

The headers section is optional. A header is a name value pair to define "global configuration" hints and configurations for the given pipeline. Only text is allowed as content, no complex objects. Not meant to be changed during pipeline processing.

...

You can set values in the headers scope using the Pipeline Expression Language (PEL). See here: Pipeline Expression Language (PEL).

vars

The vars section is optional and contains transient variables as name value pairs. It is meant as a transient scope for states during the pipeline processing.

...

You can access values in the vars scope using the Pipeline Expression Language (PEL). See here: Pipeline Expression Language (PEL).

pipeline

The pipeline section is mandatory and lists all commands which must be executed in given order.

...

You can set dynamic parameter values on commands using the Pipeline Expression Language (PEL). See here: Pipeline Expression Language (PEL).

body

The body section is optional. It defines a single object to be used as “data pool” or transformation data during the pipeline processing.

...

You can access values in the body scope using the Pipeline Expression Language (PEL). See here: Pipeline Expression Language (PEL).

Content Object

In order to process documents and other files using a pipeline, you first need to load such a file into the pipeline. After loaded, the file is automatically converted into a so called content object format. This is a wrapper around a document which provides all required information for such a document like its name size, mime type aso., for easier processing inside the pipeline. The content object provides these attributes:

...

pipeline:
  # Load document from drive and set it as content object in the body
  - drive.read:
      path: "invoice.pdf"
  # Access the attributes of the content object in the body
  - log:
      message: "Name: #{body.name}, Size: #{body.size}" 

Content Object Collection

In case multiple documents are loaded into a pipeline, such documents are grouped together in a so called content object collection. Such a collection has a similar meaning like a folder in a local file system.

...

A Content Collection is also a Content Object and therefore it also has all attributes of the Content Object.

Uploading a file

In order to upload a file and use it inside a command or pipeline you have different possibilities.

Upload a single file to a single command

In case a command expects a file as input message in its body, you can execute the command from the client using HTTP POST and put the file content in the body of the request. It will be loaded from the body and provided as input to the command’s body. Here’s an example request using the command transform.word2pdf which takes a .docx document as input converts it to PDF and sends back the result as response to the client:

POST /api/v3/pipe:transform.word2pdf HTTP/1.1 
Host: hub-acme.pipefore.net

<BINARY DATA OF my.docx>

Upload one or more files to a pipeline

In order to upload one or multiple files to be executed by a pipeline, you can make a HTTP POST request with header Content-Type: multipart/form-data to the pipeline endpoint. This will create a HTTP request with multiple parts in the HTTP request body, whereas the very first part is the pipeline YAML and all other parts are one or more files to be uploaded and used inside this pipeline.

...

Uploading a file base64 encoded

Another way to upload a file to be used inside a pipeline is to “embed” the file base64 encoded inside the pipeline and upload this pipeline using HTTP POST:

...

Auto-completion support for pipeline scripts

Note: This is experimental.

In order to enable auto-completion support for pipeline configuration files in your local development editor, you need to add the pipeline schema file to your IDE.

...

Support in IntelliJ

To enable auto-completion in IntelliJ, open preferences and navigate to JSON Schema Mappings:

...

Support in Visual Studio Code

To enable auto-completion in Visual Studio Code, open Preferences → Settings and search for section json.schemas and add a new mapping entry like this:

...