SINCE VERSION 9.0

What are Python Functions?

In PIPEFORCE pipelines you can execute Python functions as part of a pipeline execution. This way you can use the full power of this popular scripting language inside your pipelines without the need of maintaining the Python interpreter setup, container builds and container deployments. After the function has been deployed, it can be executed from inside the pipeline by a single command call:

The Python functions will be executed by a backend service inside PIPEFORCE. This approach is also known as Function as a Service (FaaS) or Lambda: You just send a Python function to the service and receive the calculated result. You do not care about any interpreter, image deployment, scalability issues or any other task related to the execution side.

This approach opens a lot of new possibilities to pipelines and your applications, such as for example:

• Create a set of libraries of functions for your custom needs and re-use them from anywhere inside your app pipelines.

• Write advanced tests using a Python testing framework.

• Do advanced data transformations and mappings with Python.

• And many more...

Since all FaaS services by default are stateless inside PIPEFORCE, it is possible to scale the execution of the Python scripts easily automatically and nearly unlimited. Since it is possible to run multiple of such FaaS execution services. Only the resources available to your cluster set the limit.

Here are some documentation references to Python:

• Official Python Documentation

• Intro to Python Functions

Deploy a function​

The first step is to declare and deploy the function.

Auto deployment​

The easiest way is to let PIPEFORCE manage the deployment of function scripts for you.

To do so, simply create a new script property inside the function folder of your app. For example: global/app/myapp/function/helloworld. Set the mime type of the property to application/python; type=script.

Then, place your Python code inside the script property. Make sure you place all of your code always inside a function like this:

A function with name function is considered as the default function and will get picked-up automatically in case no concrete function name was specified. More about this later.

After you have saved this property in the property editor, it automatically gets deployed to the FaaS backend. This is also true in case you edit or rename the property. If you delete the property, it will also automatically be undeployed from the FaaS backend for you.

Skip auto-deployment​

In some situations you dont want to auto-deploy a function script from inside the /function property folder. To do so, add the faasConfig: header in the top line comments of the script. For example:

In this case, this script will be excluded by any auto-deployment approaches. Manual deployment is still possible.

Manual deployment​

Alternatively, you can use the command function.put in order to declare and deploy a Python function manually. See this example:

For parameter code you can also set a custom uri pointing to the script to be deployed. Example:

Undeploy a function​

In order to undeploy a function, you can use the command function.delete. For example:

Execute a function​

Once a function has been deployed, it can be called from inside any pipeline using the command function.run as this example shows:

The name must always be in format APP_NAME:SCRIPT_PATH whereas APP_NAME must be replaced by the name of the app, the function belongs to and SCRIPT_PATH must be replaced by the dot-based path of the script inside the function folder. For example for a name of io.pipeforce.myapp:utils.date one would assume that the script resides in this property path: global/app/io.pipeforce.myapp/function/utils/date.

The result of such a call is always a JSON in the PIPEFORCE result format, which looks like this:

• value - This is the return value of given type. In case the function has no return value, this value is set to null and valueType is set to null.

• There could be also some other attributes in the result JSON as well depending on the execution types but usually they can be ignored since they contain only metadata required by the framework. You should not use these additional data in your app.

You can then use the returned value for further processing inside your pipeline.

Execute via util​

Another option to execute a function is using the util @function.run(name, args). It is similar to the command function.run. For example:

Function as Command​

The third option to execute a function is by calling it similar to a command directly or in a pipeline. For example:

This will call the function myfunction() inside the script myscipt located in the function folder of app myapp. All parameters (except the command default parameters) passed here will be passed as arguments to the function. In this example, there is a function myfunction(someArg) expected.

Auto deploy on execution​

In case function.run is called and the function was not found in the FaaS backend, it will be tried to automatically deploy it from the property store. Since the path in the property store is derived from the function script name, it is important to keep the source of the functions in the property store always under the path global/app/myapp/function/ whereas myapp must be replaced by the prefix of your function call.

The schema to find the property for a given function is like this:

1. First part of the name (the part before the first colon :) is the app name.

2. Second part of the name (after the first colon :) is the sub path inside the /function folder. Any period . will be replaced by a forward slash /.

3. Any implicit function name will be ignored (everything starting from the second colon : if exists).

Here are some example mappings from function script name to function properties in the property store:

• myapp:util.hello -> global/app/myapp/function/util/hello

• myapp:util.hello:my_func -> global/app/myapp/function/util/hello

• io.pipeforce.myapp:foo -> global/app/io.pipeforce.myapp/function/foo

• hello = Invalid, since no app name part exists.

Passing arguments​

You can also pass arguments to a function. These arguments must be passed as a JSON object, JSON array or as simple byte array to the command function.run using the parameter args.

JSON object argument​

Let's assume you have a function like this deployed under name myapp:helloworld:

The arguments can be passed to the function as JSON object by using the parameter args of the command function.run:

Or like this in full YAML:

In this case the name of an argument of the function will be mappped to the name of the attribute in the first level of the JSON Object. This way, the order of the attributes and arguments doesn't matter as long as the names match (for example firstName -> firstName). Therefore, a call like this would also work in order to call a function with this signature: function(firstName, lastName) (order of arguments is different compared to the order in the JSON):

Dynamic arguments​

In case you have dynamic arguments or entries in the JSON not known beforehand, you can use the variable keyword arguments symbol **kwargs of Python inside your script. See this example:

See the official Python documentation about **kwargs for more details.

JSON array argument​

Another option to pass arguments to a function is by using a JSON array.

Let's assume again you have a function like this deployed under name myapp:hello:

Then, you can call this function with arguments using a JSON array like this:

Or like this in full YAML:

The entries of the JSON array will be mapped to the arguments of the function from left to right. So entry [0] will map to firstName and entry [1] will map to lastName.

Dynamic arguments​

In case you have dynamic arguments or the number of entries in the JSON array is not known beforehand, you can use the variable arguments symbol *arg of Python inside your script. See this example:

See the official Python documentation about *args for more details.

Byte array argument​

It is also possible to pass a byte array to a function. This is handy in case you would like to send binary data or single arguments in an easy way.

Let's assume you have a function like this deployed under name myapp:helloworld:

You can pass for example a text string to this function as a byte array, by using this call:

The value This is a simple text will be passed to the argument my_data as byte array. So make sure to treat it inside the function like this. Refer to the Python docs in order to see how to handle byte arrays inside a Python script.

No argument​

These values passed to the args parameter of the command function.run will all be interpreted as calls to functions having no argument:

• args: null

• args: []

• args: {}

• args:

And if no args parameter is given at all.

Example:

Custom URI argument​

It is also possible to pass a custom URI to the args parameter of command function.run, pointing to the value to be passed as argument. Example:

The given URI will be resolved and it's content will be passed to the function by applying the rules mentioned above.

Passing secrets and ENV variables​

By default it is not good practise to add "hardcoded" environment variables or secrets in your source code since in this case they will become part of your repo in your version control system like GitHub so everyone with access to your sources can see these values.

Instead, it is better to store environment variables in an extra configuration store like the Property Store and secrets encrypted in the Secrets store.

Then, you can configure your scripts so these environment variables and secrets will be automatically passed to it whenever required at runtime.

For security and performance reasons, this can only be done on deployment on the function script.

env​

In order to set environment variables on the Python FaaS service, define the keyword faasConfig: in the comment in the script head, followed by a YAML style listing of the env variables required to be passed along to the Python script service. Example:

On deployment of the script, the faasConfig section will be parsed and the given env variables will be additionally deployed to the Python FaaS service.

secret​

If you would like to pass secrets from the secret store this way, you can use the uri prefix $uri:secret: in order to point to the secret to be passed. Here is an example:

On deployment, the secret PIPEFORCE_TEST_SECRET will be looked-up in the secret store and then passed along as value of the env variable. In case no such secret exists in the secret store, the value defaultSecretText will be set instead as fallback. This is optional. If no default value is set and also no secret exists, an exception will be raised instead and deployment will fail.

The secret can then be accessed like any other env variable inside the script.

Returning values

Make sure the type of your return value of your function is something, which can be converted to a JSON. This is true for all primitives like integer, boolean or string or any dict or array containing such primitives. Also nested.

The returned value is embedded into a result JSON with a structure like this:

• value - This is the return value of given type. In case the function has no return value, this value is set to null.

You can then use the returned value from the result JSON for further processing inside your pipeline.

Use a custom function name​

By default the name of the function inside the script must be function, for example, lets assume you have this function script deployed under name myapp:helloworld:

In order to call this function, you can execute the command function.run like this:

This call will load the script helloworld and will implicitly call the function function() inside of it (since no function name is given).

In case you have function names inside your script with names differently to function, then you need to specify them by passing the suffix :my_function_name to the name parameter of the function.run command, whereas my_function_name must be replaced with the name of the function you'd like to call.

Let's assume, you have a script deployed under name myapp:helloworld with a custom function name in it like this:

Then, you can call this function using this:

For sure it is also possible to have multiple functions with different names inside a single script. Let's see this example script deployed under myapp:utils:

In order to call the specific function hello, you can use this command call:

In case the suffix is missing, the default function name function will be expected to exist inside the code.

Calling Commands and Pipelines from inside a script​

In some cases it is necessary to callback PIPEFORCE hub and execute commands or pipelines from inside a Python function. For example if you would like to lookup some data from the property store, trigger automations or send messages to name just a few use cases.

This can be done, by simply defining the named argument pipeforce in your function signature. In case such an argument exists, the FaaS service automatically injects a new instance of PipeforceClient with it so it can be used inside your function. This client is already setup with current authentication and tracing features so no need for you to configure this.

In this example we will use the auto-injected pipeforce client in order to load a property value from the property store:

The PipeforceClient injected here is part of the official Python SDK library for PIPEFORCE.

See here for the developer API documentation of the SDK.

You can call this function from inside your pipeline as usual:

Note: We did not specify any args in the function.run command since the pipeforce argument will be automatically set by the FaaS service.

List deployed functions​

Return all functions​

In order to list all deployed functions, you can use the command functions.get without any parameter:

This will return a list of all functions with additional metadata. For example:

Return a single function​

For performance reasons function.get without any parameters will return a list which doesn't contain the code of the functions. In order to see the code, you have to query for a single function using command function.get and parameter name set to the function you would like to return:

Which will return the information about the function like this example:

Hooks​

Hooks are functions with a reserved name. In case such a function is defined in a script, it will be called whenever the according action happened.

on_deploy​

This function will be called, whenever a script was deployed to the FaaS backend. This is true for new deployments, but also for updates.

Example:

on_undeploy​

This function will be called, whenever a script is about to be undeployed from the FaaS backend.

Example:

Install Packages​

on_requirements​

You can also install dependencies for your Python scripts from the PyPi package index. To do so, declare a function on_requirements() without any args and return a list of requirements to be installed. For example:

On deployment, this function will be auto-executed and each requirement from the list will be installed using pip.

Best Practises​

You can develop your Python functions in many different ways and you should choose the way, which works best for you. In this section we would like to show you some of our best practises how to write Python Functions as a Service very effectively. Pick the ideas you like here for your own workflow.

Development​

Create a new project folder and initialize it as a PIPEFORCE app by calling the command pi init on your terminal using the PIPEFORCE CLI tool. This will create the required folder structure for you.

Then inside this folder create a new app by calling pi new app.

Then open the project folder with the IDE of your choice. We suggest to develop the Python functions locally using an advanced IDE of your choice like Microsoft Visual Studio Code or IntelliJ for example.

Go to your newly created app folder and create a new sub folder function in it.

Then create a Python script inside this folder. For example hello.py with a script code like this:

Finally, you can deploy your function to the FaaS backend by calling pi publish. This will install your function in the PIPEFORCE backend and makes it available to the other components in PIPEFORCE.

You can execute your function from inside any pipeline then by using the command function.run.

Testing​

We highly recommend to always develop your functions first locally and write tests for them as part of the development process. Only after all local test runs haven been passed, deploy your function to the backend.

What works here best for us, is putting the test functions also into the same FaaS Python script.

See this example in order to add some test functions:

As you can see, we added two test functions here. Each of it starts with prefix test_. Additionally we added an advice in order to run these test functions whenever the script is directly executed. This way you can run an debug your script locally in your IDE or by calling it directly from your local terminal:

Instead of calling each test function inside the __main__ advice (which is sometimes the only possible way in case you cannot install additional packages), we recommend to execute a unit testing framework like pytest for example in order to pick-up and execute all of your test functions automatically for you:

After you have installed pytest you can run all your test functions using a command in your terminal like this:

The tool pytest will automatically execute all functions starting with prefix test_. In this case, you do not need the __main__ section any longer.

Once the function has been deployed to PIPEFORCE, you can run the test functions online by using the command test.run or in the Test section of the PIPEFORCE WebUI.

Source Code Control​

One last thing we would like to recommend to you is using a source code control system like GitHub in order to manage different versions of your scripts and share them with other developers in your team easily.

To do so, you can create a new repository for your whole project folder and commit anything into this repo.

Hint: PIPEFORCE allows to install apps directly from GitHub using the Marketplace. This way you can easily distribute and/or rollout your applications then. See: App Marketplace .