SINCE VERSION 1.0

What is the Pipeline Expression Language (PEL)?

By default, the parameters and variables of a pipeline YAML script are static values. In some use cases this is not sufficient enough since such these values must be calculated, validated, re-wired or prepared before they can be passed to a command. This is what a Pipeline Expression can be used for.

The Pipeline Expression Language (PEL) or just PE (Pipeline Expression) is a powerful and easy to learn expression language based on Spring EL and adds additional features to this widely used "defacto-standard" expression language. It can be used inside a pipeline to access a specific portion of data and dynamically calculate, convert or set values.

The Pipeline Expression Language is optional in a pipeline YAML. You do not need it for basic tasks working with PIPEFORCE. But it gives you additional flexibility and power to automate and integrate nearly any process without the need to learn a complex programming language. So it is wise to learn at least the basics of it.

It depends on your basic knowledge, but in average it is a matter of about 15-30 minutes to understand and use the basic concepts of this powerful expression language.

Basic usage​

Typically a PE (Pipeline Expression) starts with ${ and ends with }. Here is a simple example of a PE, placed inside the value of a command parameter:

pipeline:
  - body.set:
      value: ${1 + 1}

Optionally, you can also use #{ to start a PE, but this must be placed inside quotes " and ":

pipeline:
  - body.set:
      value: "#{1 + 1}"

Output:

2

A PE can be placed in the value part of pipeline headers and variables, in parameter values of most commands and in the body of a pipeline.

It uses late binding: It will be executed only in case the according entry (header, variable, command parameter, ...) is referenced somewhere.

It also supports interpolation in order to use the PEL like a template language inside a text string. So string concatenation can be done like this:

pipeline:
  - body.set:
      value: "Result: ${1 + 1}"

Output:

Result: 2

Accessing Implicite Pipeline Objects​

Using a Pipeline Expression, you can access the values (= attributes) inside pipeline scopes like headers, vars, and body in order to read and write them.

These scopes are provided as implicit objects and therefore they're always available inside any pipeline expression even if no section like headers:, vars: or body: or was declared or any other allocation was done.

Also see: Implicite Pipeline Objects Reference .

vars (variables)

This object gives you access to all variables of the current pipeline.

Also see https://logabit.atlassian.net/wiki/spaces/PA/pages/2552856577#vars .

Let's assume, you have defined a variable counter and you would like to access this counter in your expression, then you could write an expression like this:

${vars.counter}

Here is an example of a pipeline which declares this variable and uses the Pipeline Expression to output the value of counter to the body:

vars:
  counter: 12

pipeline:
  - body.set:
      value: "The counter is: ${vars.counter}"

This will result in an output like this:

The counter is: 12

headers

This object gives you access to all pipeline headers of the current pipeline.

Also see https://logabit.atlassian.net/wiki/spaces/PA/pages/2552856577#headers .

Here is an example of a pipeline which accesses the header attribute contentType and writes it to the body:

headers:
  contentType: "text/plain"

pipeline:
  - body.set:
      value: "The type is: ${headers.contentType}"

This will result in an output like this:

The type is: text/plain

body

This object gives you access to the current body of the current pipeline.

Also see https://logabit.atlassian.net/wiki/spaces/PA/pages/2552856577#body .

Here is an example which defines an initial body value and replaces this with another text in the pipeline:

body: "Hello World"

pipeline:
  - body.set:
      value: "The text from body is: ${body}"

This will result in an output like this:

The text from body is: Hello World

Other implicit objects (request, response, exception)

Beside the scopes vars, headers and body there are more implicit objects available:request, response and exception.

Here is an example to access the value of a request query string like ?someKey=someValue of a pipeline which was triggered by an HTTP request using the request object:

pipeline:
  - body.set: "Param value is: ${request.params['someKey']}"

This will result in an output like this:

Param value is: someValue

For more information about all available implicit objects and their attributes, see:
Implicite Pipeline Objects Reference

Navigating nested data structures​

A Pipeline Expression can also point to nested attributes inside an object or array, like this JSON for example:

{
  "person": {
    "name": "Bart Simpson",
    "age": 12,
    "hobbies": [
      "skateboard",
      "tv",
      "pranks"
    ]
  }
}

You can navigate any structured object available inside the scopes using the dot operator. For example:

${person.name}

or using the associative array navigator:

${person['name']}

And to access a list/array, you can use the index operator []:

${person.hobbies[0]}

Example​

In this more advanced example, there are different things to mention:

1. We create an an initial body and set an inline JSON document there

2. In the pipeline we convert the JSON from the initial body in order to output the JSON values as text lines, overwriting the initial body.

3. Multiple lines can also be set using |. Differently to ' in this case new lines will be kept so that the output of the body will look exactly as formatted in the value parameter. This is perfect if you want to write a template for example with exact format output as the value looks like.

4. There are comments in the configuration. A comment line starts with #.

See the official YAML documentation about how to deal with multi-line values. Here is a good summary: https://yaml-multiline.info/

# Set initial body value
body: { 
        "person": {
          "name": "Bart Simpson",
          "age": 12,
          "hobbies": [
            "skateboard",
            "tv",
            "pranks"
          ]
        }
      }
      
pipeline:
  # Transform body to multiline string
  - body.set:
      value: | 
        Name:  ${body.person.name}
        Age:   ${body.person['age']}
        Hobby: ${body.person.hobbies[0]}

Formatted output:

Name:  Bart Simpson
Age:   12
Hobby: skateboard

Accessing attributes with special name

In order to make it as easy as possible, the name of attributes you would like to access via PEL should follow the rules of common variable naming. So it should not be a number or contain any special characters like for example ; , - # + * ? ! § % & .

Here are some example of valid standard names:

• myVariable

• my_variable

• myvariable

And here are some example of special names which can cause problems when used inside a Pipeline Expression:

• my:Variable

• my-variable

• my.variable

• 123

Sometimes it is not possible to manage and change the attribute names since they are given by external inputs. In this case you can access the attributes using the associative array notation. Example:

vars:
  foo: {"non:standard.name", "bar"}
  
  
pipeline:
  - body.set
      value: ${vars.foo['non:standard.name']}

Relational operators​

Is equal ==​

Example 1​

pipeline:
  - body.set:
      value: "${2 == 1}"

Output:

false

Is not equal !=​

Example 1​

pipeline:
  - body.set:
      value: ${2 != 1}

Output:

true

Less than <​

Example 1​

pipeline:
  - body.set:
      value: ${1 < 5}

Output:

true

Example 2​

pipeline:
  - body.set:
      value: ${0.5 < 1}

Output:

true

Less or equal than <=​

Example 1​

pipeline:
  - body.set:
      value: ${1 <= 5}

Output:

true

Greater than >​

Example 1​

pipeline:
  - body.set:
      value: ${1 > 5}

Output:

false

Greater or equal than >=​

Example 1​

pipeline:
  - body.set:
      value: ${5 >= 5}

Output:

true

Detect alphabetical order with <, >, <=, >=​

Example 1​

pipeline:
  - body.set:
      value: ${'Adam' < 'Zacharias'}

Output:

true

Regular expression matching matches​

Example 1​

pipeline:
  - body.set
      value: ${'5.0067' matches '^-?\\d+(\\.\\d{2})?$'}

Output:

false

Logical operators​

and​

Example 1​

pipeline:
  - body.set
      value: ${true and false}

Output:

false

or​

Example 1​

pipeline:
  - body.set:
      value: #{true or false}

Output:

true

not, !​

Example 1​

pipeline:
  - body.set:
      value: ${!true}

Output:

false

Example 2​

pipeline:
  - body.set:
      value: ${not true}

Output:

false

Mathematical operators​

Addition + and subtraction -​

Example 1 - Addition​

pipeline:
  - body.set:
      value: ${1 + 1}

Output:

2

Example 2 - Subtraction​

pipeline:
  - body.set:
      value: ${10 - 1}

Output:

9

Example 3 - Addition an subtraction​

pipeline:
  - body.set:
      value: ${25 - 5 + 10}

Output:

30

Example 4 - String concatenation​

pipeline:
  - body.set:
      value: ${'Hello ' + 'World!'}

Output:

Hello World!

Multiplication * and division /, %​

Example 1 - Multiplication​

pipeline:
  - body.set:
      value: ${3 * 5}

Output:

15

Example 2 - Negative multiplication​

pipeline:
  - body.set:
      value: ${-1 * 5}

Output:

-5

Example 3 - Division​

pipeline:
  - body.set:
      value: ${20 / 5}

Output:

4

Example 4 - Modulus​

pipeline:
  - body.set:
      value: ${7 % 4}

Output:

3

Example 5 - Operator precedence​

pipeline:
  - body.set:
      value: ${5 + 4 - 1 * 2}

Output:

7

Example 6 - Brackets​

pipeline:
  - body.set:
      value: ${(5 + 4 - 1) * 2}

Output:

16

Assignment​

Example 1​

pipeline:
  - set:
      value: "1"
      output: ${vars.counter}
  - body.set:
      value: ${vars.counter}

Output:

1

Example 2​

vars:
  counter: 12
pipeline:
  - set:
      value: ${vars.counter + 1}
      output: ${vars.counter}
  - body.set:
      value: ${vars.counter}

Output:

13

Lists and Objects

Creating a list (JSON Array)​

Example 1 - A new empty list​

vars:
  numbers: []
pipeline:
  - body.set:
      value: ${vars.numbers}

Output:

[]

As an alternative you could also use the internal notation, but this is deprecated and is only documented here for backwards compatibility:

vars:
  numbers: ${{}}
pipeline:
  - body.set:
      value: ${vars.numbers}

Output:

[]

Example 2 - A new list with default content​

vars:
  numbers: [1, 2, 4]
pipeline:
  - body.set:
      value: ${vars.numbers}

Output:

[1, 2, 4]

Example 3 - A new, nested list​

vars:
  scores: [[1, 3], [5, 8] ]
pipeline:
  - body.set
      value: ${vars.scores}

Output:

[[1, 3], [5, 8]]

Accessing lists​

Example 1​

vars:
  numbers: [1, 2, 4]
pipeline:
  - body.set
      value: ${vars.numbers[1]}

Output:

2

Creating a new object (JSON object)​

Example 1 - A new empty map​

vars:
  persons: {}
pipeline:
  - body.set
      value: ${vars.persons}

Output:

{}

Example 2 - A new map with default values​

vars:
  persons: {"hanna":"burger", "max":"hotdog", "julie":"salad"}
pipeline:
  - body.set
      value: ${vars.persons}

Output:

{
	"hanna": "burger",
	"max": "hotdog",
	"julie": "salad"
}

Example 3 - A new map with later binding

Later binding means that a value is set on the object after it was declared.​

vars:
  persons: {}
pipeline:
  - body.set:
      value: ${vars.persons['Hanna'] = 23}

Output:

{
  "Hanna": 23
}

Accessing maps​

Example 1​

vars:
  persons: {"hanna":"burger", "max":"hotdog", "julie":"salad"}
pipeline:
  - body.set
      value: ${vars.persons.max}

Output:

hotdog

You can also use the associative array notation to access the max attribute:

vars:
  persons: {"hanna":"burger", "max":"hotdog", "julie":"salad"}
pipeline:
  - body.set
      value: ${vars.persons['max']'}

Save navigation - Avoid NotExists- and NullPointerException

In case you would like to access a property of an object which doesn't exist, a NullPointerException or AttributeNotExistsException will be thrown, indicating that the object you're trying to access doesn't exist. Let's see this example which will result in such an exception:

vars:
  data: null
pipeline:
  - body.set:
      value: ${vars.data.name}

This will throw an exception (= error) since the object data is null and therefore the attribute name cannot be found at all.

In this chapter we will discuss the options you have to avoid such errors.

Elvis operator for save navigation

SINCE VERSION 10 ​

In order to return null instead of throwing an exception, you can use the safe navigation operator ?.. This example will not throw an exception. Instead, it will simply put null into the body:

vars:
  data: null
pipeline:
  - body.set:
      value: ${vars.data?.name}

It is also possible to use this operator on nested properties:

vars:
  data: null
pipeline:
  - body.set:
      value: ${vars.data?.deep?.deeper?.value}

This example will also put null into the body instead of throwing a NullPointerException.

But this works only if the level you’re trying to access doesn’t exist at all (is null). It doesn't work if there is an element but this element just doesn't contain an attribute with given name. This for example will not work here:

vars:
  data: {"deeper": null}
pipeline:
  - body.set:
      value: ${vars.data?.notexisting?.value}

Check if object has attribute before accessing

SINCE VERSION 9.0

As an alternative, you can use the @data.has util to check whether an attribute exists before you're trying to access it. This will also work for non-existing attributes on existing elements:

vars:
  data: {"deeper": null}
pipeline:
  - body.set:
     if: ${@data.has(#this, 'vars.data.notexisting')}
     value: ${vars.data.notexisting}

You can also use it inline combined with a ternary operator:

vars:
  data: {"deeper": null}
pipeline:
  - body.set:
     value: "${@data.has(#this, 'vars.data.notexisting') ? vars.data.notexisting : null}"

Use an the associative array notation

SINCE VERSION 9.0

Another option is to use the associative array notation to access an attribute which could potentially not exists. Example:

vars:
  data: null
pipeline:
  - body.set:
      value: ${vars.data.['deep']}    

But this works only for one level.

Filtering​

The PEL can be used to filter lists in an easy way.

Selection Expression .?​ (filter for objects)

With the selection syntax you can select a subset of items from a given collection to be returned as new collection by specifying a selection expression.

Similar to the WHERE part of an SQL query.

The syntax is like this:

collectionName.?[selectionExpression]

Whereas collectionName is the variable name of the collection (can be an array, map, list, aso.) and selectionExpression is the expression which selects the items to be returned from the list.

Example 1​

Lets assume we have a collection of entities like this stored in the body:

[
  {
    "person": {
      "name": "Bart Simpson",
      "age": 12,
      "hobbies": [
        "skateboard",
        "tv",
        "pranks"
      ]
    }
  },
  {
    "person": {
      "name": "Maggie Simpson",
      "age": 1,
      "hobbies": [
        "drinking milk",
        "crawling",
        "crying"
      ]
    }
  }
]

Then, we can select a subset of the entries using a selection like this:

pipeline:
  - body.set:
      value: ${body.?[person.name == 'Maggie Simpson']}

Output would be a sublist with the entries matching the criteria:

[
  {
    "person": {
      "name": "Maggie Simpson",
      "age": 1,
      "hobbies": [
        "drinking milk",
        "crawling",
        "crying"
      ]
    }
  }
]

Here is the complete example in one pipeline:

# Set initial body value
body: [
        {
          "person": {
            "name": "Bart Simpson",
            "age": 12,
            "hobbies": [
              "skateboard",
              "tv",
              "pranks"
            ]
          }
        },
        {
          "person": {
            "name": "Maggie Simpson",
            "age": 1,
            "hobbies": [
              "drinking milk",
              "crawling",
              "crying"
            ]
          }
        }
      ]
      
pipeline:
  # Overwrite initial body with this selection result
  - body.set:
      value: ${body.?[person.name == 'Maggie Simpson']}

Projection Expression .!​ (filter for attributes)

With the projection syntax you can select specific attribute values out from a collection of objects.

Similar to the SELECT part of an SQL query.

The syntax is like this:

collectionName.![projectionExpression]

Whereas collectionName is the variable name of the collection (can be an array, map, list, aso.) and projectionExpression is the expression which selects the properties to be returned from each object in the list.

Example 1​

Lets assume we have a collection of entities like this stored in the body:

[
  {
    "person": {
      "name": "Bart Simpson",
      "age": 12,
      "hobbies": [
        "skateboard",
        "tv",
        "pranks"
      ]
    }
  },
  {
    "person": {
      "name": "Maggie Simpson",
      "age": 1,
      "hobbies": [
        "drinking milk",
        "crawling",
        "crying"
      ]
    }
  }
]

Then, we can select properties from this collection like this:

pipeline:
  - body.set:
      value: ${body.![person.name]}

Output:

['Bart Simpson', 'Maggie Simpson']

And here is the complete example:

# Set initial body value
body: [
        {
          "person": {
            "name": "Bart Simpson",
            "age": 12,
            "hobbies": [
              "skateboard",
              "tv",
              "pranks"
            ]
          }
        },
        {
          "person": {
            "name": "Maggie Simpson",
            "age": 1,
            "hobbies": [
              "drinking milk",
              "crawling",
              "crying"
            ]
          }
        }
      ]
      
pipeline:
  # Overwrite initial body with this projection result
  - body.set:
      value: ${body.![person.name]}

Ternary Operator (If-Then-Else)​

The ternary operator can be used to define an if-then-else condition in your expression in a single line.

pipeline:
  - body.set:
      value: ${10 > 100 ? 'condition is true':'condition is false'}

This example would output condition is false in the body. As you can see, the part left of the question mark ? defines the condition to be evaluated. If this condition evaluates to true, then the part left of the colon : is executed. Otherwise the right part:

condition ? trueExpression : falseExpression

Elvis Operator (Default Value)

You can shorten the ternary operator in case you would like to check a value for null or false and if given, fallback to a default value in this case. You can do so by using the so called "Elvis" operator. The syntax is this:

VALUE?:DEFAULT_VALUE

Whereas VALUE must be a PEL value or variable. If VALUE is false or null, then the DEFAULT_VALUE will be returned. Otherwise VALUE will be returned.

For example if VALUE is null:

vars:
  someName: null
pipeline:
  - body.set:
      value: ${vars.someName?:'myDefaultValue'}

Output will be in this case:

"myDefaultValue"

Another example, now with VALUE not null and not false (it contains a string value):

vars:
  someName: "Bart Simpson"
pipeline:
  - body.set:
      value: ${vars.someName?:'myDefaultValue'}

Output will be in this case:

"Bart Simpson"

Handling non-existent fields (check if field exists)

Sometimes it is necessary to check whether a given field (attribute) on an object exists before trying to access it. To do so, you have different options:

Example 1: With associative array and null check

body: {"foo": "someValue"}

pipeline:
  - body.set:
      value: |
        Field 'foo' exists: ${body["foo"] != null}
        Field 'bar' exists: ${body["bar"] != null}

This will return:

Field 'foo' exists: true
Field 'bar' exists: false

The caveat of this approach is that it will treat a null value of the attribute the same way like this attribute doesn't exist:

body: {"foo": "someValue", "bar": null}

pipeline:
  - body.set:
      value: |
        Field 'foo' exists: ${body["foo"] != null}
        Field 'bar' exists: ${body["bar"] != null}

This will return (which has wrong semantics in this case, since field bar exists but has null value):

Field 'foo' exists: true
Field 'bar' exists: false

Example 2: Use @data.has(..) utility

A better approach is to use the PEL utility @data.has(...) which will check wether the field exists:

body: {"foo": "someValue", "bar": null}

pipeline:
  - body.set:
      value: |
        Field 'foo' exists: ${@data.has(body, "foo")}
        Field 'bar' exists: ${@data.has(body, "bar")}

This will return:

Field 'foo' exists: true
Field 'bar' exists: true

#this and #root variables

The special variables #this and #root are available in any PEL whereas #root always points to the root of the evaluation context which is by default usually the pipeline message with headers, vars and body scope. So if you are in an iteration loop or inside a selection or projection, you can always use #root to access the evaluation context. Example:

vars:
  data: {"deeper": null}
pipeline:
  - body.set:
     if: ${@data.has(#root, 'vars.data')}
     value: ${vars.data}

As you can see in this example, #root is used here in order to put the whole root context into the utility in order to check for attribute existence by path.

The variable #this has a similar meaning but while #root will never change for a given PEL evaluation, the value of #this can change for example in case you are in an iteration loop.