What is Testing?

In any system, testing workflows and integrations is a very complex task because of many states, data conversions and different interfaces involved.

PIPEFORCE has many toolings and best practises to simplify testing of apps, automations and integrations.

Testing Functions

Since Version: 9.0

You can test your function by creating another function starting with name test_. Inside this function you can define your test asserts. In case such a test assert has been failed, throw an exception.

Example:

def  helloworld():
    return "Hello World!"

def test_helloworld():

    # Your test goes here...

    result = helloworld()
    if result != "Hello World!":
        raise Exception("Expected 'Hello World!' but was: " + result)

When you call the command test.run, it will automatically pick up all functions starting with test_ and execute them. More details about test.run see below.

Testing Web User Interfaces

Since Version: 9.0

Using Functions, you can automate the testing of Web interfaces using a remote Selenium service.

PIPEFORCE has built-in support for the UI testing service BrowserStack.

In order to use this service, you have to create a (paid) BrowserStack account first.

Setup credentials

Then you have to setup the credentials for your test.

As PIPEFORCE secret

In case your tests must run inside of PIPEFORCE, you have to setup the BrowserStack credentials as secrets there.

To do so, login to the PIPEFORCE portal and create a new secret PIPEFORCE_TEST_BROWSERSTACK_USERNAME of type secret-text and paste your BROWSERSTACK_USERNAME as value (you can find it in your BrowserStack account).

Create another secret with name PIPEFORCE_TEST_BROWSERSTACK_ACCESS_KEY and of type secret-text and paste your BROWSERSTACK_ACCESS_KEY as value (you can find it in your BrowserStack account as well).

As environment variables

In case you would like to execute the tests locally in your IDE before you deploy and execute them to PIPEFORCE, you have to set the credentials as these environment variables: PIPEFORCE_TEST_BROWSERSTACK_USERNAME and PIPEFORCE_TEST_BROWSERSTACK_ACCESS_KEY locally.

Write a test script

Now, you can write functions with BrowserStack tests. Here is an example:


from selenium.webdriver.remote.webdriver import WebDriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

from pipeforce_sdk import BrowserStackTest, PipeforceClient


def test_portal_login(pipeforce: PipeforceClient):

    bs = BrowserStackTest(pipeforce)
    driver: WebDriver = bs.driver

    driver.get(pipeforce.config.PIPEFORCE_URL)
    WebDriverWait(driver, 10).until(EC.title_contains(pipeforce.config.PIPEFORCE_NAMESPACE))

    driver.quit()

Make sure that the test function always starts with prefix test_ since only in this case it will be picked-up and executed automatically.

Note that the selenium and pipeforce_sdk packages are already part of the FaaS backend. So no need for you to install them via on_requirements. For more details see: https://logabit.atlassian.net/wiki/spaces/PA/pages/2548367361

In this example you can see that at first a BrowserStackTest class is created. This encapsulates all login and setup steps for you.

Then, the Selenium driver is retrieved and a the current portal URL is loaded.

After some wait it is checked, whether the loaded page contains the PIPEFORCE namespace as title.

Finally, the driver is stopped.

After you have saved your script in the property store, it will be automatically deployed to the FaaS backend.

Then, you can run the test using the Test section in the portal or by using the test.run command (see above).

After the test run, you can see the result also in your BrowserStack Dashboard.

For more details about what and how you can test using Selenium, see the Official Online Documentation.

Executing tests

You have multiple options to run tests stored in PIPEFORCE.

Using a command

You can use the test.run command to run the tests:

pipeline:
  - test.run

Using the CLI

In order to execute a test run using the CLI use this line in your terminal:

Test Run Report

When executing via CLI or command, the result will always be a test run report in JSON format which has a structure similar to this example:

testUnits = An array of all test units (= test scripts)
location = The location of the test script in the property editor.
type = The script type. Is usually always js = JavaScript.
unitResult = The final result of the test unit (= PASSED if all passed, FAILED if at least one has been failed). One of PASSED, FAILED, ERROR or IGNORED.
- tests = An array containing all tests found in the location script and their result.
  - testResult = The test result. One of
    - PASSED = Test run was successful.
    - FAILED = Test run was not successful because of an assert has been failed.
    - ERROR = Test run was not successful because an exception has been thrown.
    - IGNORED = The test was not executed because if was marked as ignore.
  - testName = The name of the test (method) inside the test script.
  - testStartLine = The line number where the test (method) is located or null in case it could not be detected.
  - testDuration = Start time and duration information of the test run.
  - exception = The exception message in case of FAILED or ERROR. This value is null in any other cases.
- unitSummary = A summary of all tests of this unit.
- unitDuration = Start time and duration information of all tests in this unit
locationpatterns = The patterns passed to the test.run command.
overallSummary = A summary of all tests.
overallDuration = Start time and duration information of the overall test run.
overallResult = The final result of the full test run. One of PASSED, FAILED, ERROR, IGNORED or NO_TESTS.

To switch the output format of the test result, set the command parameter reportFormat accordingly. For more details, see the command documentation for test.run.

Testing Pipelines

Since version 9.5

In order to test pipelines you need two pipelines:

The pipeline-under-test = The original pipeline which will be used in production and must be tested beforehand.
The testing-pipeline = A pipeline which calls the pipeline under test and then checks the result whether it complies with the expectations.

The testing-pipeline calls the pipeline-under-test and checks its result using the assert commands.

This approach also supports mock objects to replace real commands depending on given criteria. This way you can change the input data of your pipeline-under-test and test fully disconnected from any external systems for example.

Create a Pipeline-Under-Test

Let’s assume your pipeline-under-test (= original pipeline) is stored in the property store under path global/app/my.domain.myapp/pipeline/highest-ratings and contains this YAML:

It loads all products from the external products REST endpoint https://fakestoreapi.com/products and filters the by ratings >= 4.8. After executing this pipeline, the result looks similar to this:

Create a Testing-Pipeline

In order to execute this pipeline-under-test and make sure that exactly the expected data is returned, you can run a pipeline like this in the playground:

As you can see, the command pipeline.run is used to run the persisted pipeline-under-test and the assert command is used to check the result of the called pipeline.

If all asserts are successful, that means that the pipeline-under-test works as expected. As soon as at least one assert fails, the testing pipeline also fails and lists the failed assert. Here is an example of such an error message. Lets change the expect value from 12 to 13 and see what happens:

Naming and storage of Testing-Pipeline

In order to automate your tests, you must store the testing-pipeline always under path

for example

as it is in this example. This naming schema is important so PIPEFORCE can find all testing pipelines.

The test folder of an app should contain any test scripts required for this certain app.

Automatically run Testing-Pipelines

Any time testing is required, PIPEFORCE will automatically pickup all the tests from the testing folder which do have a suffix of -test and executes them. This can be for example when system restarts or when a new app was installed. This can be configured in the admin and app settings.

Manually run Testing Pipelines

In order to run a single testing pipeline, you can call it directly using the pipeline.run command or in the online workbench. Also debugging is possible here. This is mainly useful when you are developing.

The downside of this approach is that it doesn’t return a testing protocol and the pipeline execution time is limited to a few minutes. Therefore, long running tests cannot be executed using this approach. You should use the command test.run instead, since this command will collect all tests matching the given location patterns and executes them in the background. This way the tests can also run up to multiple hours if required. In order to run all the testing-pipelines inside a given app, you can call it for example like this:

This will pick-up all testing-pipelines inside the app my.domain.myapp and executes all of them. Finally it will return the overall testing protocol in JSON or Junit format so you can integrate it in any supported testing tool.

Optionally, you can trigger such a test run by your CD/CI tool like Jenkins or GitHub Actions by executing a GET to this Command RESTful Endpoint:

Note: In order to wait for the response on longer running tests via auto-polling, you have to set the parameter pollingRedirectEnabled=true. For more details how this works, see: https://logabit.atlassian.net/wiki/spaces/DEV/pages/2497118230.

Another option to execute and visualize the test results is by using the Online Test Console, see below.

Mocking Commands and Data

In some cases it is required that data and command calls of the pipeline-under-test are not the real, production ones. But instead, you would like to call “dummy endpoints” and return “dummy data” just for the purpose of testing. This is called mocking or mock objects. The Pipeline Testing Framework allows you to mock any command and define its behaviour and returned data based on given conditions. You can do so by using the command mock.command before you run the pipeline-under-test. Using this command you can define the name of the command to be mocked, the when condition when this mock should become effective and the body and variable values (the “dummy data”) to be set on execution.

Let’s use the endpoint https://fakestoreapi.com/products from the example above and let’s assume we are not allowed to call this endpoint from inside a testing environment. Instead, we need to return mock data whenever this endpoint gets called in our test run. To do so, we change the testing-pipeline and add the mock.command before we execute the pipeline-under-test:

As you can see, we set the command parameter to http.get. Furthermore, we make sure the mock gets activated only in case the url matches by setting an boolean expression here. In case of a match, we return the dummy data in the body as defined by thenSetBody.

Testing BPMN Workflows

Since version 10

In order to test BPMN workflows in PIPEFORCE, you can also use the pipeline testing tools like the testing-pipeline approach and the mock.command for example. Any mock is made available across multiple request hops even across multiple workflow tasks and microservices. Just make sure the mock is created using the parameter contextId set to CURRENT (which is the default). In this case the mock contextId will be shared with any workflow and microservice so a mock can attach to it when required.

Note: It is a good practise to start with writing the BPMN workflow test first before you start implementing and weaving it. You should mock any external form or pipeline first until the pipelines, forms and external system and data is ready to be used. Later, you can then more easily use these mocks as durable part of your tests.

Here is an example, how you could write and run such a workflow test in PIPEFORCE:

Step 1: The BPMN under test

This is the BPMN under test. It has a user task and a service task. For this example we will keep it simple:

Lets assume the MyUserTask will be finished by a submit of an external form and the MyServiceTask will call a pipeline and return the result in the body as process variable back to the task:

PIPEFORCE provides test tools so we can “simulate” the external task call by a form and also to mock the mail.send command in the pipeline so no real email is sent whenever we do a test run. This means, in our automated test instead of opening and clicking a real form, we simply want to complete the MyUserTask with the form data passed to it and in the pipeline assigned to the MyServiceTask we only want to "mock away" the mail.send command:

Create this BPMN and its pipeline in PIPEFORCE and deploy it.

Step 2: Write the test

In the next step you need to write a testing pipeline which executes this workflow, configures the required mock, executes the required tasks and verifies that the BPMN workflow behaves as expected. Here is an example, how such a testing pipeline could look like:

The steps in this test explained in detail:

1: Registers a new mock for any subsequent mail.send command call of current test run. You can also define the variables and the body to be returned in case the mock is called. In this example the mock is doing nothing. It simply disables the command. Furthermore, you can mock a whole pipeline if required. For more details see the API docs of the mock command.
2: Starts the workflow under test and auto-assigns the first user task to myusername. This is optional and simplifies the way how to retrieve the taskId required for the next step.
3: “Simulate” the submit of an external workflow form by completing the MyUserTask using the workflow.task.complete command and passing the form field values as process variables to the task.
4: Make sure the tasks MyUserTask and MyServiceTask has been already processed and finished using the command workflow.assert. See the API doc of this command since it provides more options to apply workflow asserts.
5 - 7: Load the final process variables and make sure they contain the expected final values using the workflow.variables.get and assert command.

Step 3: Execute the test

After you have stored your pipeline, you can run it directly. In case your test has been passed, the pipeline will be executed without any error. Otherwise an error is shown targeting you to failed testing assert.

A much better approach to execute your test pipeline would be to use the test.run command or the Online Test Console since in this case also long running tests can be covered and details about the test executions are shown.

Changing workflow settings for a test run

By using the testing framework of PIPEFORCE you can setup, mock, execute and assert your test results in order to test workflows. In case you need to adjust your BPMN workflow settings like a timer setting or similar inside the workflow engine, you can do so by using dynamic values here for example ${piwf_timerDuration} and change this value by passing a process variable with this name whenever you start your workflow under test. Example:

Note: The prefix piwf_ on the process variable makes sure that it is not treated as business variable and is filtered out from any pipeline calls.

Testing Messaging Queues

PIPEFORCE apps can also create and use messaging queues (usually RabbitMQ Queues). See section Messaging for more details on it. The creation and setup of such queues and attaching consumers to it is usually done in an automated way by PIPEFORCE for you. But in order to make sure the queue setup of your app works as expected, it is good practise to also have tests in your pipeline verifying the correct setup of these queues.

In order to do so, you can use the command message.queue.find which will return you all required information of the existing queues. Based on this information you can then create your test asserts.

Here is a simple example of a testing pipeline which makes sure that a queue of given name exists and has at least one consumer and exactly one binding attached to it:

Online Test Console

You can run all of your remote tests also online using the Tests view. To do so, login to PIPEFORCE with your developer account and then in the LOW CODE section click on Tests and then Run Tests. The test result is finally shown as a test report like this example shows:

What is Testing?​

Testing Functions​

Testing Web User Interfaces​

Setup credentials​

As PIPEFORCE secret​

As environment variables​

Write a test script​

Executing tests​

Using a command​

Using the CLI​

Test Run Report​