In PIPEFORCE pipelines you can execute Python functions as part of a pipeline execution. This way you can use the full power of this popular scripting language inside your pipelines without the need of maintaining the Python interpreter setup, container builds and container deployments. After the function has been deployed, it can be executed from inside the pipeline by a single command call:
The Python functions will be executed by a backend service inside PIPEFORCE. This approach is also known as Function as a Service (FaaS) or Lambda: You just send a Python function to the service and receive the calculated result. You do not care about any interpreter, image deployment, scalability issues or any other task related to the execution side.
This approach opens a lot of new possibilities to pipelines and your applications, such as for example:
Create a set of libraries of functions for your custom needs and re-use them from anywhere inside your app pipelines.
Write advanced tests using a Python testing framework.
Do advanced data transformations and mappings with Python.
And many more...
Since all FaaS services by default are stateless inside PIPEFORCE, it is possible to scale the execution of the Python scripts easily automatically and nearly unlimited. Since it is possible to run multiple of such FaaS execution services. Only the resources available to your cluster set the limit.
The easiest way is to let PIPEFORCE manage the deployment of function scripts for you.
To do so, simply create a new script property inside the function folder of your app. For example: global/app/myapp/function/helloworld. Set the mime type of the property to application/python; type=script.
Then, place your Python code inside the script property. Make sure you place all of your code always inside a function like this:
return "Hello World"
A function with name function is considered as the default function and will get picked-up automatically in case no concrete function name was specified. More about this later.
After you have saved this property in the property editor, it automatically gets deployed to the FaaS backend. This is also true in case you edit or rename the property. If you delete the property, it will also automatically be undeployed from the FaaS backend for you.
In case a function is called using the command function.run and the function could not be found in the FaaS backend (for example because the backend did auto-rescale), it will be automatically tried to install this script from the property store. Therefore, you should store the script code always in the property store. More information can be found in the section about executing a function below.
Make sure to always define an app prefix to your function name like myapp: which will specify to which app the function script belongs to. By default functions without this prefix will be rejected.
Be aware that scripts deployed manually using function.put must also be fully managed manually. In case the FaaS container in the backend automatically re-scales, it could be that your functions deployed there are gone. So you have to re-deploy them also manually. Therefore, if possible, instead of doing a manual deployment using function.putprefer to save your scripts in the property store and let PIPEFORCE automatically manage the deployment for you.
The name must always be in format APP_NAME:SCRIPT_PATH whereas APP_NAME must be replaced by the name of the app, the function belongs to and SCRIPT_PATH must be replaced by the dot-based path of the script inside the function folder. For example for a name of io.pipeforce.myapp:utils.date one would assume that the script resides in this property path: global/app/io.pipeforce.myapp/function/utils/date.
The result of such a call is always a JSON in the PIPEFORCE result format, which looks like this:
This will call the function myfunction() inside the script myscipt located in the function folder of app myapp. All parameters (except the command default parameters) passed here will be passed as arguments to the function. In this example, there is a function myfunction(someArg) expected.
The function path must always be fully qualified. Meaning, it must consist always of the three parts appName:scriptPath:functionName.
In case function.run is called and the function was not found in the FaaS backend, it will be tried to automatically deploy it from the property store. Since the path in the property store is derived from the function script name, it is important to keep the source of the functions in the property store always under the path global/app/myapp/function/ whereas myapp must be replaced by the prefix of your function call.
The schema to find the property for a given function is like this:
First part of the name (the part before the first colon :) is the app name.
Second part of the name (after the first colon :) is the sub path inside the /function folder. Any period . will be replaced by a forward slash /.
Any implicit function name will be ignored (everything starting from the second colon : if exists).
Here are some example mappings from function script name to function properties in the property store:
In this case the name of an argument of the function will be mappped to the name of the attribute in the first level of the JSON Object. This way, the order of the attributes and arguments doesn't matter as long as the names match (for example firstName -> firstName). Therefore, a call like this would also work in order to call a function with this signature: function(firstName, lastName) (order of arguments is different compared to the order in the JSON):
You can pass for example a text string to this function as a byte array, by using this call:
args: "This is a simple text"
The value This is a simple text will be passed to the argument my_data as byte array. So make sure to treat it inside the function like this. Refer to the Python docs in order to see how to handle byte arrays inside a Python script.
By default it is not good practise to add "hardcoded" environment variables or secrets in your source code since in this case they will become part of your repo in your version control system like GitHub so everyone with access to your sources can see these values.
Instead, it is better to store environment variables in an extra configuration store like the property store and secrets encrypted in the secret store.
Then, you can configure your scripts so these environment variables and secrets will be automatically passed to it whenever required at runtime.
For security and performance reasons, this can only be done on deployment on the function script.
In order to set environment variables on the Python FaaS service, define the keyword faasConfig: in the comment in the script head, followed by a YAML style listing of the env variables required to be passed along to the Python script service. Example:
On deployment of the script, the faasConfig section will be parsed and the given env variables will be additionally deployed to the Python FaaS service.
Once the environment variables have been passed by a script this way, they will be visible to all other scripts in the same Python FaaS service! This is also true for secrets passed as environment variables.
On deployment, the secret PIPEFORCE_TEST_SECRET will be looked-up in the secret store and then passed along as value of the env variable. In case no such secret exists in the secret store, the value defaultSecretText will be set instead as fallback. This is optional. If no default value is set and also no secret exists, an exception will be raised instead and deployment will fail.
The secret can then be accessed like any other env variable inside the script.
For security reasons only secrets starting with prefix PIPEFORCE_TEST_ and PIPEFORCE_SHARED_ can be passed this way. Any other secret cannot be passed to the function service. In this case an exception will be raised on deployment.
This call will load the script helloworld and will implicitly call the function function() inside of it (since no function name is given).
In case you have function names inside your script with names differently to function, then you need to specify them by passing the suffix :my_function_name to the name parameter of the function.run command, whereas my_function_name must be replaced with the name of the function you'd like to call.
Let's assume, you have a script deployed under name myapp:helloworld with a custom function name in it like this:
In case the suffix is missing, the default function name function will be expected to exist inside the code.
Calling Commands and Pipelines from inside a script
In some cases it is necessary to callback PIPEFORCE hub and execute commands or pipelines from inside a Python function. For example if you would like to lookup some data from the property store, trigger automations or send messages to name just a few use cases.
This can be done, by simply defining the named argument pipeforce in your function signature. In case such an argument exists, the FaaS service automatically injects a new instance of PipeforceClient with it so it can be used inside your function. This client is already setup with current authentication and tracing features so no need for you to configure this.
In this example we will use the auto-injected pipeforce client in order to load a property value from the property store:
For performance reasons function.get without any parameters will return a list which doesn't contain the code of the functions. In order to see the code, you have to query for a single function using command function.get and parameter name set to the function you would like to return:
You can also install dependencies for your Python scripts from the PyPi package index. To do so, declare a function on_requirements() without any args and return a list of requirements to be installed. For example:
You can develop your Python functions in many different ways and you should choose the way, which works best for you. In this section we would like to show you some of our best practises how to write Python Functions as a Service very effectively. Pick the ideas you like here for your own workflow.
Create a new project folder and initialize it as a PIPEFORCE app by calling the command pi init on your terminal using the PIPEFORCE CLI tool. This will create the required folder structure for you.
Then inside this folder create a new app by calling pi new app.
Then open the project folder with the IDE of your choice. We suggest to develop the Python functions locally using an advanced IDE of your choice like Microsoft Visual Studio Code or IntelliJ for example.
Go to your newly created app folder and create a new sub folder function in it.
Then create a Python script inside this folder. For example hello.py with a script code like this:
Finally, you can deploy your function to the FaaS backend by calling pi publish. This will install your function in the PIPEFORCE backend and makes it available to the other components in PIPEFORCE.
You can execute your function from inside any pipeline then by using the command function.run.
We highly recommend to always develop your functions first locally and write tests for them as part of the development process. Only after all local test runs haven been passed, deploy your function to the backend.
What works here best for us, is putting the test functions also into the same FaaS Python script.
See this example in order to add some test functions:
result = function()
assert result == "Hello", "I expect Hello here"
result = function()
assert result == "World", "This probably will fail"
if __name__ == "__main__":
As you can see, we added two test functions here. Each of it starts with prefix test_. Additionally we added an advice in order to run these test functions whenever the script is directly executed. This way you can run an debug your script locally in your IDE or by calling it directly from your local terminal:
Instead of calling each test function inside the __main__ advice (which is sometimes the only possible way in case you cannot install additional packages), we recommend to execute a unit testing framework like pytest for example in order to pick-up and execute all of your test functions automatically for you:
pip install pytest
After you have installed pytest you can run all your test functions using a command in your terminal like this:
The tool pytest will automatically execute all functions starting with prefix test_. In this case, you do not need the __main__ section any longer.
Once the function has been deployed to PIPEFORCE, you can run the test functions online by using the command test.run or in the Test section of the PIPEFORCE WebUI.
One last thing we would like to recommend to you is using a source code control system like GitHub in order to manage different versions of your scripts and share them with other developers in your team easily.
To do so, you can create a new repository for your whole project folder and commit anything into this repo.