What is Testing?​

In any system, testing workflows and integrations is a very complex task because of many states, data conversions and different interfaces involved.

PIPEFORCE has many toolings and best practises to simplify testing of apps, automations and integrations.

Testing Functions​

...

When you call the command test.run, it will automatically pick up all functions starting with test_ and execute them. More details about test.run see below.

Testing Web User Interfaces​

...

In order to use this service, you have to create a (paid) BrowserStack account first.

Setup credentials​

Then you have to setup the credentials for your test.

As PIPEFORCE secret​

In case your tests must run inside of PIPEFORCE, you have to setup the BrowserStack credentials as secrets there.

...

Create another secret with name PIPEFORCE_TEST_BROWSERSTACK_ACCESS_KEY and of type secret-text and paste your BROWSERSTACK_ACCESS_KEY as value (you can find it in your BrowserStack account as well).

As environment variables​

In case you would like to execute the tests locally in your IDE before you deploy and execute them to PIPEFORCE, you have to set the credentials as these environment variables: PIPEFORCE_TEST_BROWSERSTACK_USERNAME and PIPEFORCE_TEST_BROWSERSTACK_ACCESS_KEY locally.

Write a test script​

Now, you can write functions with BrowserStack tests. Here is an example:

...

For more details about what and how you can test using Selenium, see the Official Online Documentation.

Executing tests​

You have multiple options to run tests stored in PIPEFORCE.

Using a command​

You can use the test.run command to run the tests:

pipeline:
  - test.run

Using the CLI​

In order to execute a test run using the CLI use this line in your terminal:

pi command test.run

Test Run Report​

When executing via CLI or command, the result will always be a test run report in JSON format which has a structure similar to this example:

...

To switch the output format of the test result, set the command parameter reportFormat accordingly. For more details, see the command documentation for test.run.

Testing Pipelines

...

This approach also supports mock objects to replace real commands depending on given criteria. This way you can change the input data of your pipeline-under-test and test fully disconnected from any external systems for example.

Create a Pipeline-Under-Test

Let’s assume your pipeline-under-test (= original pipeline) is stored in the property store under path global/app/my.domain.myapp/pipeline/highest-ratings and contains this YAML:

...

[
  {
    "id": 11,
    "title": "Silicon Power ...",
    "price": 109,
    "description": "3D NAND flash are applied to ...",
    "category": "electronics",
    "rating": {
      "rate": 4.8,
      "count": 319
    }
  },
  {
    "id": 12,
    "title": "WD 4TB Gaming Drive ...",
    "price": 114,
    "description": "Expand your PS4 gaming experience, ...",
    "category": "electronics",
    "rating": {
      "rate": 4.8,
      "count": 400
    }
  }
]

Create a Testing-Pipeline

In order to execute this pipeline-under-test and make sure that exactly the expected data is returned, you can run a pipeline like this in the playground:

...

Error in [command: assert:v1, pos: 2]: Expected: 13 (Integer), but is: 12 (Integer). Difference: 2

Naming and storage of Testing-Pipeline

In order to automate your tests, you must store the testing-pipeline always under path

...

The test folder of an app should contain any test scripts required for this certain app.

Automatically run Testing-Pipelines

Any time testing is required, PIPEFORCE will automatically pickup all the tests from the testing folder which do have a suffix of -test and executes them. This can be for example when system restarts or when a new app was installed. This can be configured in the admin and app settings.

Manually run Testing Pipelines

In order to run a single testing pipeline, you can call it directly using the pipeline.run command or in the online workbench. Also debugging is possible here. This is mainly useful when you are developing.

...

Another option to execute and visualize the test results is by using the Online Test Console, see below.

Mocking Commands and Data

In some cases it is required that data and command calls of the pipeline-under-test are not the real, production ones. But instead, you would like to call “dummy endpoints” and return “dummy data” just for the purpose of testing. This is called mocking or mock objects. The Pipeline Testing Framework allows you to mock any command and define its behaviour and returned data based on given conditions. You can do so by using the command mock.command before you run the pipeline-under-test. Using this command you can define the name of the command to be mocked, the when condition when this mock should become effective and the body and variable values (the “dummy data”) to be set on execution.

...

As you can see, we set the command parameter to http.get. Furthermore, we make sure the mock gets activated only in case the url matches by setting an boolean expression here. In case of a match, we return the dummy data in the body as defined by thenSetBody.

Testing BPMN Workflows

...

Here is an example, how you could write and run such a workflow test in PIPEFORCE:

Step 1: The BPMN under test

This is the BPMN under test. It has a user task and a service task. For this example we will keep it simple:

...

Create this BPMN and its pipeline in PIPEFORCE and deploy it.

Step 2: Write the test

In the next step you need to write a testing pipeline which executes this workflow, configures the required mock, executes the required tasks and verifies that the BPMN workflow behaves as expected. Here is an example, how such a testing pipeline could look like:

...

• 1: Registers a new mock for any subsequent mail.send command call of current test run. You can also define the variables and the body to be returned in case the mock is called. In this example the mock is doing nothing. It simply disables the command. Furthermore, you can mock a whole pipeline if required. For more details see the API docs of the mock command.

• 2: Starts the workflow under test and auto-assigns the first user task to myusername. This is optional and simplifies the way how to retrieve the taskId required for the next step.

• 3: “Simulate” the submit of an external workflow form by completing the MyUserTask using the workflow.task.complete command and passing the form field values as process variables to the task.

• 4: Make sure the tasks MyUserTask and MyServiceTask has been already processed and finished using the command workflow.assert. See the API doc of this command since it provides more options to apply workflow asserts.

• 5 - 7: Load the final process variables and make sure they contain the expected final values using the workflow.variables.get and assert command.

Step 3: Execute the test

After you have stored your pipeline, you can run it directly. In case your test has been passed, the pipeline will be executed without any error. Otherwise an error is shown targeting you to failed testing assert.

A much better approach to execute your test pipeline would be to use the test.run command or the Online Test Console since in this case also long running tests can be covered and details about the test executions are shown.

Changing workflow settings for a test run

By using the testing framework of PIPEFORCE you can setup, mock, execute and assert your test results in order to test workflows. In case you need to adjust your BPMN workflow settings like a timer setting or similar inside the workflow engine, you can do so by using dynamic values here for example ${piwf_timerDuration} and change this value by passing a process variable with this name whenever you start your workflow under test. Example:

...

Testing Messaging Queues

PIPEFORCE apps can also create and use messaging queues (usually RabbitMQ Queues). See section Messaging for more details on it. The creation and setup of such queues and attaching consumers to it is usually done in an automated way by PIPEFORCE for you. But in order to make sure the queue setup of your app works as expected, it is good practise to also have tests in your pipeline verifying the correct setup of these queues.

...

vars:
  queue: null
  
pipeline:

  - message.queue.find:
      includeBindings: true
      
  # Select the "queue-under-test" from the result
  - var.set:
      key: queue
      value: ${body.?[name == 'pipeforce_default_dlq'][0]}
      
  - assert:
      message: "There must be exactly one binding"
      true: ${@data.size(vars.queue.bindings) == 1}
      
  - assert:
      message: "There must be at least one consumer" 
      true: ${vars.queue.consumers > 0}

Online Test Console​

You can run all of your remote tests also online using the Tests view. To do so, login to PIPEFORCE with your developer account and then in the LOW CODE section click on Tests and then Run Tests. The test result is finally shown as a test report like this example shows:

...