Page Comparison

...

This approach is sometimes also called Iterator. Looping over a given set of data objects is also called iterating over the items.

Iterate with command data.

...

mapping

You can use the command data.list.iterate command mapping with parameter iterate set to true in order to iterate over a given list of data and apply transformation patterns at the same time.

NOTE

This command is optimized for huge data iteration cycles and it doesn't add command execution counts for each cycle. So you should prefer this approach whenever possible.

Here is an example:

pipeline:
    - data.list.iterate:
        listA: [{"name": "Max", "allowed": false}, {"name": "Hennah", "allowed": false}]
        listB:  [{"name": "Max", "age": 12}, {"name": "Hennah", "age": 23}]
        where: "itemA.name == itemB.name and itemB.age > 18"
        do: "itemA.allowed = true"

As you can see, in this example there are two lists: listA and listB. For every item in listA, the listB is also iterated. In the where parameter you can define a PEL expression. In case this expression returns true, the expression in do is executed. In this example this means for every entry in listA it is checked whether there is the same name entry in listB and if so, the age is checked. If this value is > 18, the origin listA will be changed and the value of allowed set to true. The result will look like this:

[
    {
        "name": "Max",
        "allowed": false
    },
    {
        "name": "Hennah",
        "allowed": true
    }
]

It is also possible to define multiple do-expressions to be executed on each iteration cycle. See this example, where additionally a new attribute approved with the current timestamp will be added on each "where-matching" entry:

pipeline:
    - data.list.iterate:
        listA: [{"name": "Max", "allowed": false}, {"name": "Hennah", "allowed": false}]
        listB:  [{"name": "Max", "age": 12}, {"name": "Hennah", "age": 23}]
        where: "itemA.name == itemB.name and itemB.age > 18"
        do: |
          itemA.allowed = true;
          itemA.approved = @date.timestamp();

As you can see, multiple do-expressions will be separated by a semicolon ;. You can write them in one single line, or in multiple lines using the pipe symbol |. The output will look like this:

[
    {
        "name": "Max",
        "allowed": false
    },
    {
        "name": "Hennah",
        "allowed": true,
        "approved":  1659266178365
    }
]

You can also iterate only a single listA without any where condition, like this example shows:

pipeline:
    - data.list.iterate:
        listA: [{"name": "Max", "allowed": false}, {"name": "Hennah", "allowed": false}]
        do: "itemA.allowed = true"

If the where parameter is missing, the do expression will be executed on any iteration item. In this example the result would be:

[
    {
        "name": "Max",
        "allowed": true
    },
    {
        "name": "Hennah",
        "allowed": true
    }
]

If-Then-Else conditions inside a do expression can be implemented using the ternary operator (condition ? whenTrueAction : elseAction). Let's rewrite the example from above and replace the where parameter by a ternary operator inside the do parameter:

pipeline:
    - data.list.iterate:
        listA: [{"name": "Max", "allowed": false}, {"name": "Hennah", "allowed": false}]
        listB:  [{"name": "Max", "age": 12}, {"name": "Hennah", "age": 23}]
        do: "(itemA.name == itemB.name and itemB.age > 18) ? itemA.allowed = true : ''"

...

and apply calculations and / or mappings on each iteration items.

For more information how to do this, see: JSON Data Mapping .

Iterate with command foreach​

The foreach command can also be used for iterations: For every item in the list, a given set of commands will be executed until all items are processed.

For more information how to do this, see: https://logabit.atlassian.net/wiki/spaces/PA/pages/2543714420/Controlling+Pipeline+Flow#Foreach-(Iterator)%E2%80%8B.

Iterate with PEL​

In some situations it is also handy to use directly the PEL selection or PEL projection features of the Pipeline Expression Language (PEL) on a given list in order to iterate it.

For more information how to do this, see: https://logabit.atlassian.net/wiki/spaces/PA/pages/2543026496/Pipeline+Expression+Language+PEL#Filtering%E2%80%8B

Iterate with custom function​

For very complex data iteration tasks, you could also use the function.run command and write a serverless function which iterates over the data. Since this approach requires knowledge about the scripting language and is usually not the best performing option, you should choose it only if there is no other option available to solve your iteration taskto solve your iteration task.

For more information how to do this, see: Python Functions .

Iterate with custom script​

You can also use a an embedded script to iterate. See

For more information how to do this, see: /wiki/spaces/PA/pages/2603319334

Iterate with custom microservice​

And if a script (serverless function / lambda) is also not working all the approaches mentioned before do not work for you, you can write a custom microservice and run it inside PIEPFORCE. But this approach is outside of the scope of this data transformation section. See section Microservices for more details

For more information how to do this, see: Microservices Framework.

...