Testing an Automation App
What is Testing?
In any system, testing workflows and integrations is a very complex task because of many states, data conversions and different interfaces involved.
PIPEFORCE has many toolings and best practises to simplify testing of apps, automations and integrations.
Testing Functions
Since Version: 9.0
You can test your function by creating another function starting with name test_
. Inside this function you can define your test asserts. In case such a test assert has been failed, throw an exception.
Example:
def helloworld():
return "Hello World!"
def test_helloworld():
# Your test goes here...
result = helloworld()
if result != "Hello World!":
raise Exception("Expected 'Hello World!' but was: " + result)
When you call the command test.run
, it will automatically pick up all functions starting with test_
and execute them. More details about test.run
see below.
Testing Web User Interfaces
Since Version: 9.0
Using Functions, you can automate the testing of Web interfaces using a remote Selenium service.
PIPEFORCE has built-in support for the UI testing service BrowserStack.
In order to use this service, you have to create a (paid) BrowserStack account first.
Setup credentials
Then you have to setup the credentials for your test.
As PIPEFORCE secret
In case your tests must run inside of PIPEFORCE, you have to setup the BrowserStack credentials as secrets there.
To do so, login to the PIPEFORCE portal and create a new secret PIPEFORCE_TEST_BROWSERSTACK_USERNAME
of type secret-text
and paste your BROWSERSTACK_USERNAME
as value (you can find it in your BrowserStack account).
Create another secret with name PIPEFORCE_TEST_BROWSERSTACK_ACCESS_KEY
and of type secret-text
and paste your BROWSERSTACK_ACCESS_KEY
as value (you can find it in your BrowserStack account as well).
As environment variables
In case you would like to execute the tests locally in your IDE before you deploy and execute them to PIPEFORCE, you have to set the credentials as these environment variables: PIPEFORCE_TEST_BROWSERSTACK_USERNAME
and PIPEFORCE_TEST_BROWSERSTACK_ACCESS_KEY
locally.
Write a test script
Now, you can write functions with BrowserStack tests. Here is an example:
from selenium.webdriver.remote.webdriver import WebDriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from pipeforce_sdk import BrowserStackTest, PipeforceClient
def test_portal_login(pipeforce: PipeforceClient):
bs = BrowserStackTest(pipeforce)
driver: WebDriver = bs.driver
driver.get(pipeforce.config.PIPEFORCE_URL)
WebDriverWait(driver, 10).until(EC.title_contains(pipeforce.config.PIPEFORCE_NAMESPACE))
driver.quit()
Make sure that the test function always starts with prefix test_
since only in this case it will be picked-up and executed automatically.
Note that the selenium
and pipeforce_sdk
packages are already part of the FaaS backend. So no need for you to install them via on_requirements
. For more details see: Python Functions
In this example you can see that at first a BrowserStackTest
class is created. This encapsulates all login and setup steps for you.
Then, the Selenium driver is retrieved and a the current portal URL is loaded.
After some wait it is checked, whether the loaded page contains the PIPEFORCE namespace as title.
Finally, the driver is stopped.
After you have saved your script in the property store, it will be automatically deployed to the FaaS backend.
Then, you can run the test using the Test
section in the portal or by using the test.run
command (see above).
After the test run, you can see the result also in your BrowserStack Dashboard.
For more details about what and how you can test using Selenium, see the Official Online Documentation.
Executing tests
You have multiple options to run tests stored in PIPEFORCE.
Using a command
You can use the test.run
command to run the tests:
pipeline:
- test.run
Using the CLI
In order to execute a test run using the CLI use this line in your terminal:
Test Run Report
When executing via CLI or command, the result will always be a test run report in JSON format which has a structure similar to this example:
testUnits
= An array of all test units (= test scripts)location
= The location of the test script in the property editor.type
= The script type. Is usually always js = JavaScript.unitResult
= The final result of the test unit (=PASSED
if all passed,FAILED
if at least one has been failed). One ofPASSED
,FAILED
,ERROR
orIGNORED
.tests
= An array containing all tests found in the location script and their result.testResult
= The test result. One ofPASSED
= Test run was successful.FAILED
= Test run was not successful because of an assert has been failed.ERROR
= Test run was not successful because an exception has been thrown.IGNORED
= The test was not executed because if was marked as ignore.
testName
= The name of the test (method) inside the test script.testStartLine
= The line number where the test (method) is located or null in case it could not be detected.testDuration
= Start time and duration information of the test run.exception
= The exception message in case of FAILED or ERROR. This value is null in any other cases.
unitSummary
= A summary of all tests of this unit.unitDuration
= Start time and duration information of all tests in this unit
locationpatterns
= The patterns passed to the test.run command.overallSummary
= A summary of all tests.overallDuration
= Start time and duration information of the overall test run.overallResult
= The final result of the full test run. One ofPASSED
,FAILED
,ERROR
,IGNORED
orNO_TESTS
.
To switch the output format of the test result, set the command parameter reportFormat
accordingly. For more details, see the command documentation for test.run.
Testing Pipelines
Since version 9.5
In order to test pipelines you need two pipelines:
The pipeline-under-test = The original pipeline which will be used in production and must be tested beforehand.
The testing-pipeline = A pipeline which calls the pipeline under test and then checks the result whether it complies with the expectations.
The testing-pipeline calls the pipeline-under-test and checks its result using the assert commands.
This approach also supports mock objects to replace real commands depending on given criteria. This way you can change the input data of your pipeline-under-test and test fully disconnected from any external systems for example.
Create a Pipeline-Under-Test
Let’s assume your pipeline-under-test (= original pipeline) is stored in the property store under path global/app/my.domain.myapp/pipeline/highest-ratings
and contains this YAML:
It loads all products from the external products REST endpoint https://fakestoreapi.com/products
and filters the by ratings >= 4.8. After executing this pipeline, the result looks similar to this:
Create a Testing-Pipeline
In order to execute this pipeline-under-test and make sure that exactly the expected data is returned, you can run a pipeline like this in the playground:
As you can see, the command pipeline.run is used to run the persisted pipeline-under-test and the assert command is used to check the result of the called pipeline.
If all asserts are successful, that means that the pipeline-under-test works as expected. As soon as at least one assert fails, the testing pipeline also fails and lists the failed assert. Here is an example of such an error message. Lets change the expect
value from 12
to 13
and see what happens:
Naming and storage of Testing-Pipeline
In order to automate your tests, you must store the testing-pipeline always under path
for example
as it is in this example. This naming schema is important so PIPEFORCE can find all testing pipelines.
The test
folder of an app should contain any test scripts required for this certain app.
Automatically run Testing-Pipelines
Any time testing is required, PIPEFORCE will automatically pickup all the tests from the testing folder which do have a suffix of -test
and executes them. This can be for example when system restarts or when a new app was installed. This can be configured in the admin and app settings.
Manually run Testing Pipelines
In order to run a single testing pipeline, you can call it directly using the pipeline.run command or in the online workbench. Also debugging is possible here. This is mainly useful when you are developing.
The downside of this approach is that it doesn’t return a testing protocol and the pipeline execution time is limited to a few minutes. Therefore, long running tests cannot be executed using this approach. You should use the command test.run instead, since this command will collect all tests matching the given location patterns and executes them in the background. This way the tests can also run up to multiple hours if required. In order to run all the testing-pipelines inside a given app, you can call it for example like this:
This will pick-up all testing-pipelines inside the app my.domain.myapp
and executes all of them. Finally it will return the overall testing protocol in JSON or Junit format so you can integrate it in any supported testing tool.
Optionally, you can trigger such a test run by your CD/CI tool like Jenkins or GitHub Actions by executing a GET to this Command RESTful Endpoint:
Note: In order to wait for the response on longer running tests via auto-polling, you have to set the parameter pollingRedirectEnabled=true
. For more details how this works, see: https://logabit.atlassian.net/wiki/spaces/DEV/pages/2497118230.
Another option to execute and visualize the test results is by using the Online Test Console, see below.
Mocking Commands and Data
In some cases it is required that data and command calls of the pipeline-under-test are not the real, production ones. But instead, you would like to call “dummy endpoints” and return “dummy data” just for the purpose of testing. This is called mocking or mock objects. The Pipeline Testing Framework allows you to mock any command and define its behaviour and returned data based on given conditions. You can do so by using the command mock.command before you run the pipeline-under-test. Using this command you can define the name of the command
to be mocked, the when
condition when this mock should become effective and the body and variable values (the “dummy data”) to be set on execution.
Let’s use the endpoint https://fakestoreapi.com/products
from the example above and let’s assume we are not allowed to call this endpoint from inside a testing environment. Instead, we need to return mock data whenever this endpoint gets called in our test run. To do so, we change the testing-pipeline and add the mock.command
before we execute the pipeline-under-test:
As you can see, we set the command
parameter to http.get
. Furthermore, we make sure the mock gets activated only in case the url matches by setting an boolean expression here. In case of a match, we return the dummy data in the body as defined by thenSetBody
.
Testing BPMN Workflows
Since version 10
In order to test BPMN workflows in PIPEFORCE, you can also use the pipeline testing tools like the testing-pipeline approach and the mock.command
for example. Any mock is made available across multiple request hops even across multiple workflow tasks and microservices. Just make sure the mock is created using the parameter contextId
set to CURRENT
(which is the default). In this case the mock contextId will be shared with any workflow and microservice so a mock can attach to it when required.
Note: It is a good practise to start with writing the BPMN workflow test first before you start implementing and weaving it. You should mock any external form or pipeline first until the pipelines, forms and external system and data is ready to be used. Later, you can then more easily use these mocks as durable part of your tests.
Here is an example, how you could write and run such a workflow test in PIPEFORCE:
Step 1: The BPMN under test
This is the BPMN under test. It has a user task and a service task. For this example we will keep it simple:
Lets assume the MyUserTask
will be finished by a submit of an external form and the MyServiceTask
will call a pipeline and return the result in the body as process variable back to the task:
PIPEFORCE provides test tools so we can “simulate” the external task call by a form and also to mock the mail.send
command in the pipeline so no real email is sent whenever we do a test run. This means, in our automated test instead of opening and clicking a real form, we simply want to complete the MyUserTask
with the form data passed to it and in the pipeline assigned to the MyServiceTask
we only want to "mock away" the mail.send
command:
Create this BPMN and its pipeline in PIPEFORCE and deploy it.
Step 2: Write the test
In the next step you need to write a testing pipeline which executes this workflow, configures the required mock, executes the required tasks and verifies that the BPMN workflow behaves as expected. Here is an example, how such a testing pipeline could look like:
The steps in this test explained in detail:
1: Registers a new mock for any subsequent
mail.send
command call of current test run. You can also define the variables and the body to be returned in case the mock is called. In this example the mock is doing nothing. It simply disables the command. Furthermore, you can mock a whole pipeline if required. For more details see the API docs of the mock command.2: Starts the workflow under test and auto-assigns the first user task to
myusername
. This is optional and simplifies the way how to retrieve thetaskId
required for the next step.3: “Simulate” the submit of an external workflow form by completing the
MyUserTask
using theworkflow.task.complete
command and passing the form field values as processvariables
to the task.4: Make sure the tasks
MyUserTask
andMyServiceTask
has been already processed and finished using the command workflow.assert. See the API doc of this command since it provides more options to apply workflow asserts.5 - 7: Load the final process variables and make sure they contain the expected final values using the workflow.variables.get and assert command.
Step 3: Execute the test
After you have stored your pipeline, you can run it directly. In case your test has been passed, the pipeline will be executed without any error. Otherwise an error is shown targeting you to failed testing assert.
A much better approach to execute your test pipeline would be to use the test.run
command or the Online Test Console since in this case also long running tests can be covered and details about the test executions are shown.
Changing workflow settings for a test run
By using the testing framework of PIPEFORCE you can setup, mock, execute and assert your test results in order to test workflows. In case you need to adjust your BPMN workflow settings like a timer setting or similar inside the workflow engine, you can do so by using dynamic values here for example ${piwf_timerDuration}
and change this value by passing a process variable with this name whenever you start your workflow under test. Example:
Note: The prefix piwf_
on the process variable makes sure that it is not treated as business variable and is filtered out from any pipeline calls.
Testing Messaging Queues
PIPEFORCE apps can also create and use messaging queues (usually RabbitMQ Queues). See section Messaging for more details on it. The creation and setup of such queues and attaching consumers to it is usually done in an automated way by PIPEFORCE for you. But in order to make sure the queue setup of your app works as expected, it is good practise to also have tests in your pipeline verifying the correct setup of these queues.
In order to do so, you can use the command message.queue.find which will return you all required information of the existing queues. Based on this information you can then create your test asserts.
Here is a simple example of a testing pipeline which makes sure that a queue of given name exists and has at least one consumer and exactly one binding attached to it:
Online Test Console
You can run all of your remote tests also online using the Tests view. To do so, login to PIPEFORCE with your developer account and then in the LOW CODE
section click on Tests
and then Run Tests
. The test result is finally shown as a test report like this example shows: