The Pipeline Expression Language (PEL) or just PE (Pipeline Expression) is a powerful expression language that is used inside a pipeline for data mapping and to dynamically calculate, set and change values at processing time of a pipeline.

Licenses	COMMUNITY, ENTERPRISE, CORPORATE
Since	VERSION 6.0

What is the PEL?

The Pipeline Expression Language (PEL) or just PE (Pipeline Expression) is an expression language that can optionally be used inside a pipeline to do data mapping and to dynamically calculate, set and change values at processing time of the pipeline. This gives you a huge flexibility in your pipeline.

Typically a PE is embedded between #{ and } and is placed in the value part of headers, variables or command parameters. It uses late binding: It will be executed only in case the according entry (header, variable or command parameter) is referenced somewhere.

Here is a simple example of a PE, placed inside the value of a command parameter:

pipeline:
  - log:
      message: "#{1 + 1}"

Output:

It also supports interpolation in order to use the PEL like a template language inside a text string. So string concatenation is done for you:

pipeline:
  - log:
      message: "Result: #{1 + 1}"

Output:

Result: 2

Its also possible to access the values from one of the pipeline scopes (headers, vars and body), like shown in this example for vars:

vars:
  counter: 0
pipeline:
  - log:
      message: "Counter = #{vars.counter + 1}"

Output:

Counter = 1

You can also set values using a PEL in combination with the set command:

vars:
  counter: 0
pipeline:
  - set:
      value: "#{vars.counter + 1}"
      output: "#{vars.counter}"
  - log:
      message: "Counter = #{vars.counter}"

Output:

Counter: 1

Operators

The PEL supports all common operators as also known for most programming languages. It is similar to the expressions in Microsoft Excel, although the syntax is slightly different and aligned a bit more with the traditional programming syntax.

Relational operators

Is equal (==)

Example 1

pipeline:
  - log:
      message: "#{2 == 1}"

Output:

false

Is not equal (!=)

Example 1

pipeline:
  - log:
      message: "#{2 != 1}"

Output:

true

Less than (<)

Example 1

pipeline:
  - log:
      message: "#{1 < 5}"

Output:

true

Example 2

pipeline:
  - log:
      message: "#{0.5 < 1}"

Output:

true

Less or equal than (<=)

Example 1

pipeline:
  - log:
      message: "#{1 <= 5}"

Output:

true

Greater than (>)

Example 1

pipeline:
  - log:
      message: "#{1 > 5}"

Output:

false

Greater or equal than (>=)

Example 1

pipeline:
  - log:
      message: "#{5 >= 5}"

Output:

true

Detect alphabetical order with <, >, <=, >=

Example 1

pipeline:
  - log:
      message: "#{'Adam' < 'Zacharias'}"

Output:

true

Regular expression matching (matches)

Example 1

pipeline:
  - log:
      message: "#{'5.0067' matches '^-?\\d+(\\.\\d{2})?$'}"

Output:

false

Logical operators

and

Example 1

pipeline:
  - log:
      message: "#{true and false}"

Output:

false

or

Example 1

pipeline:
  - log:
      message: "#{true or false}"

Output:

true

not (!)

Example 1

pipeline:
  - log:
      message: "#{!true}"

Output:

false

Mathematical operators

Addition and subtraction

Example 1 - Addition

pipeline:
  - log:
      message: "#{1 + 1}"

Output:

Example 2 - Subtraction

pipeline:
  - log:
      message: "#{10 - 1}"

Output:

Example 3 - Addition an subtraction

pipeline:
  - log:
      message: "#{25 - 5 + 10}"

Output:

Example 4 - String concatenation

pipeline:
  - log:
      message: "#{'Hello ' + 'World!'}"

Output:

Hello World!

Multiplication and division

Example 1 - Multiplication

pipeline:
  - log:
      message: "#{3 * 5}"

Output:

Example 2 - Negative multiplication

pipeline:
  - log:
      message: "#{-1 * 5}"

Output:

-5

Example 3 - Division

pipeline:
  - log:
      message: "#{20 / 5}"

Output:

Example 4 - Modulus

pipeline:
  - log:
      message: "#{7 % 4}"

Output:

Example 5 - Operator precedence

pipeline:
  - log:
      message: "#{5 + 4 - 1 * 2}"

Output:

Example 6 - Brackets

pipeline:
  - log:
      message: "#{(5 + 4 - 1) * 2}"

Output:

Assignment

Example 1

pipeline:
  - set:
      value: "1"
      output: "#{vars.counter}"
  - log:
      message: "#{vars.counter}"

Output:

Example 2

vars:
  counter: 12
pipeline:
  - set:
      value: "#{vars.counter + 1}"
      output: "#{vars.counter}"
  - log:
      message: "#{vars.counter}"

Output:

Working with lists and maps / dictionaries

Creating a new list

Example 1 - A new empty list

vars:
  numbers: "#{{}}"
pipeline:
  - log:
      message: "#{vars.numbers}"

Output:

[]

Example 2 - A new list with default content

vars:
  numbers: "#{{1, 2, 4}}"
pipeline:
  - log:
      message: "#{vars.numbers}"

Output:

[1, 2, 4]

Example 3 - A new, nested list

vars:
  scores: "#{{ {1, 3}, {5, 8} }}"
pipeline:
  - log:
      message: "#{vars.scores}"

Output:

[[1, 3], [5, 8]]

Accessing lists and arrays

Example 1

vars:
  numbers: "#{{1, 2, 4}}"
pipeline:
  - log:
      message: "#{vars.numbers[1]}"

Output:

Creating a new map / dictionary

Example 1 - A new empty map

vars:
  persons: "#{ {:} }"
pipeline:
  - log:
      message: "#{vars.persons}"

Output:

{}

Example 2 - A new map with default values

vars:
  persons: "#{ {hanna:'burger', max:'hotdog', julie:'salad'} }"
pipeline:
  - log:
      message: "#{vars.persons}"

Output:

{hanna=burger, max=hotdog, julie=salad}

Example 3 - A new map with later binding

vars:
  persons: "#{ {:} }"
pipeline:
  - log:
      message: "#{vars.persons['Hanna'] = 23}"

Output:

{Hanna=23}

Accessing maps/dictionaries

Example 1

vars:
  persons: "#{ {hanna:'burger', max:'hotdog', julie:'salad'} }"
pipeline:
  - log:
      message: "#{vars.persons['max']}"

Output:

hotdog

Navigating objects

A PE can point to values inside an object (or nested data structure), like this JSON for example:

{
  "person": {
    "name": "Bart Simpson",
    "age": 12,
    "hobbies": [
      "skateboard",
      "tv",
      "pranks"
    ]
  }
}

You can navigate any structured object available inside a vars or body scope using the dot operator. For example:

#{person.name}

To access a list/array, you can use the index operator []:

#{person.hobbies[0]}

Example 1

In this more advanced example, there are different things to mention:

We create an inline map as initial vars value.
Multi line values are done by using ' (this trims any new line and space character at the beginning and end of the string, which is important to interpret it here as map instead of a string).
We set the body using the initial vars value using the command set.body also with a multi line value.
Multiple lines can also be set using |. Differently to ' in this case new lines will be kept so that the output of the body will look exactly as formatted in the value parameter. This is perfect if you want to write a template for example with exact format output as the value looks like.
There are comments in the configuration. A comment line starts with #.

See the official YAML documentation about how to deal with multi-line values. Here is a good summary: https://yaml-multiline.info/

# Set inline map as initial body value
vars: 
   data: '#{ 
      {
        person: {
          name: "Bart Simpson",
          age: 12,
          hobbies: {
            "skateboard",
            "tv",
            "pranks"
          }
        }
      }
    }'
pipeline:
  # Set in the body a multiline string
  - set.body:
      value: | 
        Name:  #{vars.data.person.name}
        Age:   #{vars.data.person.age}
        Hobby: #{vars.data.person.hobbies[0]}

Formatted output:

Name:  Bart Simpson
Age:   12
Hobby: skateboard

Selection Expression

With the selection syntax you can select a subset of items from a given collection to be returned as new collection by specifying a selection expression.

Similar to the WHERE part of an SQL query.

The syntax is like this:

collectionName.?[selectionExpression]

Whereas collectionName is the variable name of the collection (can be an array, map, list, aso.) and selectionExpression is the expression which selects the items to be returned from the list.

Example 1

Lets assume we have a collection of entities like this stored in the body:

[
  {
    "person": {
      "name": "Bart Simpson",
      "age": 12,
      "hobbies": [
        "skateboard",
        "tv",
        "pranks"
      ]
    }
  },
  {
    "person": {
      "name": "Maggie Simpson",
      "age": 1,
      "hobbies": [
        "drinking milk",
        "crawling",
        "crying"
      ]
    }
  }
]

Then, we can select a subset of the entries using a selection like this:

pipeline:
  - log:
      message: #{body.?[person.name == 'Maggie Simpson']}

Output would be a sublist with the entries matching the criteria:

[
  {
    "person": {
      "name": "Maggie Simpson",
      "age": 1,
      "hobbies": [
        "drinking milk",
        "crawling",
        "crying"
      ]
    }
  }
]

Here is the same example but with the data set embedded into the pipeline in the vars scope:

vars: 
   data: '#{{
        {
          person: {
            name: "Bart Simpson",
            age: 12,
            hobbies: {
              "skateboard",
              "tv",
              "pranks"
            }
          }
        },
        {
          person: {
            name: "Maggie Simpson",
            age: 1,
            hobbies: {
              "drinking milk",
              "crawling",
              "crying"
            }
          }
        }
      }}'
pipeline:
  - log:
      message: "#{vars.data.?[person.name == 'Maggie Simpson']}"

Projection Expression

With the projection syntax you can select specific property values out from a collection of objects.

Similar to the SELECT part of an SQL query.

The syntax is like this:

collectionName.![projectionExpression]

Whereas collectionName is the variable name of the collection (can be an array, map, list, aso.) and projectionExpression is the expression which selects the properties to be returned from each object in the list.

Example 1

Lets assume we have a collection of entities like this stored in the body:

[
  {
    "person": {
      "name": "Bart Simpson",
      "age": 12,
      "hobbies": [
        "skateboard",
        "tv",
        "pranks"
      ]
    }
  },
  {
    "person": {
      "name": "Maggie Simpson",
      "age": 1,
      "hobbies": [
        "drinking milk",
        "crawling",
        "crying"
      ]
    }
  }
]

Then, we can select properties from this collection like this:

pipeline:
  - log:
      message: "#{body.![person.name]}"

Output:

['Bart Simpson', 'Maggie Simpson']

And here the example with embedded data set in the vars scope:

vars: 
   data: '#{{
        {
          person: {
            name: "Bart Simpson",
            age: 12,
            hobbies: {
              "skateboard",
              "tv",
              "pranks"
            }
          }
        },
        {
          person: {
            name: "Maggie Simpson",
            age: 1,
            hobbies: {
              "drinking milk",
              "crawling",
              "crying"
            }
          }
        }
      }}'
pipeline:
  - log:
      message: "#{vars.data.![person.name]}"

PEL Utils

Inside a PE you can use built-in Utils as helpers to simplify work.

A refecerence about available PEL Utils can be found in this section: PEL Utils Reference.

Example 1

vars:
  cities: "Munich, Bratislava, New York"
pipeline:
  - log: 
      message: "#{@list.size(vars.cities)}"

Output:

Example 2

pipeline:
  - log: 
      message: "#{@text.lang('Hallo, Herr Meier!')}"

Output:

GERMAN