Property Store
What is the Property Store?
The property store is a key-value database inside PIPEFORCE to save your application data like sources, attachments or JSON data for example. You can create, update, query and delete data on the property store using the property.* commands.
What is a Property?
Each data (for example plain text, JSON, binary data, ...) stored in the property store is wrapped into an envelope called a property.
Property Attributes
Each property has multiple attributes (the envelope data). The most important ones are:
Attribute | Description |
|---|---|
| The unique, absolute path of the property. This value can change over time, for example, if a property was moved to another virtual location. Example value: Note: Since version 9.0 this attribute was renamed from |
| This attribute contains the payload of the property serialized as string. Example value: Note: If you need to store a huge amount of binary data, consider to use property attachments instead. |
| A unqiue identificator of the property. Differently to Example value: |
| This attribute contains the mime type of the value. If this attribute is Example value: |
| A unix epoch timestamp in millis when this property was created. Example value: |
| A unix epoch timestamp in millis when this property was updated last or Example value: |
| The time to live (ttl) in minutes of this property since creation. If the time to live has been expired, the property is eligable to be deleted. Usually one of the next cleanup jobs will then delete this property. There is no guarantee that the property is deleted exactly after this time has expired. If the value of timeToLive is Example value: |
| A boolean value indicating whether this property has a lock assigned. In case a lock is assigned, the property can only be altered by the user or group, this lock is exclusive to. See section Property Locking for more details. |
| A boolean value indicating whether this property has been moved to the trash bin. See section Trash Bin for more details. |
Property Path
Every property has a unique path. Such a property path is structured in an hierarchical way and has this base structure:
/pipeforce/<namespace>/<localPath>For example:
/pipeforce/mynamespace/global/app/myapp/config/appEvery property belongs to a certain namespace. If such a path starts with a slash / at the beginning, it is interpreted as absolute. Meaning, it must start with /pipeforce followed by the namespace, this property belongs to. Properties are always stored with their absolute paths in the property store.
At runtime, property paths can be defined and interpreted relatively. Here is an example of an relative path of a property:
global/app/myapp/config/appRelative property paths do not start with a slash. They do not contain information about the current namespace. This information is automatically added when a property is loaded or stored.
WHY PROPERTY PATH?
It defines the unique path of a property in a tree structure.
It makes it easier to query a nested subset of properties "recursively" using path patterns.
It allows to group properties together under a "path-like" structure.
It is also used for declaring nested read and write permissions.
It defines the storage location of the property for performance optimization (for example archive is stored on a different location)
Property Path Pattern
Since the path of a property is similar to a path in a file tree, it is possible to use path matching techniques in order to easily find and load a nested subset of properties "recursively" from the store. By default, PIPEFORCE uses the so called "ant-style" path matcher. This matcher uses the following rules, applied to a path:
The double asterisk
**matches any character in the path at given position.The single asterisk
*matches zero or more charachters inside a "directory" of the path (up to the next slash/).
The asterisk * is often also called wildcard. A property path containing wildcards are called a path pattern.
Example 1
This path pattern will select all properties having a path starting with global/app/myapp at any sublevel. So it will match these paths:
But it won't mach these paths:
Example 2
This path pattern will select all properties having a path starting with global/app/myapp inside this level (meaning inside the "directory" myapp) but not at any sublevel. So it will match these paths:
But it won't match these paths:
Example 3
It is also possible to combine multiple wildcards inside a pattern.
This path pattern will match any path starting with prefix global/app, followed by any app name (without any slash), followed by /data/, followed by any further charachters at any level. It will match these paths:
But it won't match these paths:
Example 4
This pattern would match any relative path at top level, but not any sub-level. So it will match these paths:
But it won't match these paths:
Example 5
This pattern will match anything at any level. So it will match these paths:
There is no path, this pattern will not match.
Since the catch-all pattern ** (without any additional path information) will match anything, it should be used very rarely. It can cause huge performance impacts. Because of this reason, the catch-all is not everywhere allowed. So think twice before using it.
Property Value and Type
The value attribute of a property defines the payload of a property. Such a payload could be a JSON document or a plain text for example. It is always stored as string (text) value. Which type the text value represents, is defined by the type attribute which could be of any supported mime type e.g. application/json or text/plain. The latter is the default, if no type is specified. Important mime types, used by PIPEFORCE are:
Mime Type | Description |
|---|---|
| A JavaScript script (deprecated, use |
| JSON document. |
| YAML document (not IANA listed). |
| XML document as part of an application. |
| HTML document. |
| A JavaScript script. |
| Plain text document (string). This is the default, if no |
| XML document. |
See the http://IANA.org websites for a full list of official mime types: https://www.iana.org/assignments/media-types/media-types.xhtml
Property types can additionally contain parameters. Example:
Multiple parameters are separated by a colon ;. See the IANA specification for details. PIPEFORCE uses custom type parameters in some situations. Some examples:
Type | Description |
|---|---|
| Indicates, that the property value is a base64 encoded string. |
| The property value is a pipeline script, written in YAML. |
| The property value is an XML document defining a workflow (for example BPMN). |
| The property value is a JSON document defining a PIPEFORCE form config. |
| The property value is a JSON document defining a PIPEFORCE list config. |
| The property value is a JSON document defining a JSON schema which complies with the definition from json-schema.org. |
| The property value is a FreeMarker template . |
| The property value is a Velocity template . |
Binary Value
The value attribute contains always values of type string. In order to store binary data there, it must be base64 encoded before it gets persisted. In this case, the property must be marked as a base64 value. This is done in the type attribute using the mime type parameter encoding=base64. Example:
In this case the property can be automatically converted back into a "binary" file when loaded.
The base64 encoding approach is only meant for small files, not bigger than a few KB and not that many. In case you want to store more and bigger files, consider storing them as property attachments instead, since they do have a chunked storage handling concept and can store a huge amount of binary data more effectively.
Create a Property
To create a new property in the property store, you need to define a property schema first using the command property.schema.put. Example:
This schema defines the base envelope data for the new property like its path, uuid and type. After the schema has been created, the property payload can be added and edited.
To simplify these steps, you can create the schema and add the payload value in one single step:
Property created event
Whenever you create a new a property, an event with key property.created is fired with the created property stored in payload.target of the event object. This way you can listen in your pipelines for properties created newly. See the reference documentation for details. Here is an example how to listen to such an event in a persisted pipeline:
This persisted pipeline gets automatically executed whever a new property with a path containing global/app/myapp/data/mydata has been created.
Update a Property
To change the value of the property, you can use the command property.put. The property must already exist beforehand. Example:
In order to change the envelope data (attributes) of a property, you need to use the property.schema.put command:
If you would like to change the path of an existing property, you can use the property.move command:
Why are there two different commands to create a property and set its value?
Because there are two different security levels with permissions. Those users who are able to change the value of a property with property.put, not always have the permission to create a new property or changing its other attributes using property.schema.put. Having two commands allows it to differentiate in a pipeline what a user can do, since in PIPEFORCE any command can have assigned different permissions.
Property changed event
Whenever you update a property value or one of its other attributes, an event with key property.updated is fired with the origin property stored in origin of the event object and the final version of the property in target. This way you can listen in your pipelines for property changes. See the reference documentation for details. Here is an example how to listen to such an event in a persisted pipeline:
Property moved event
Whenever you move a property by changing its origin path, an event with key property.moved is fired with the origin path of the property stored in origin of the event object and the final path of the property in target. This way you can listen in your pipelines for property movements. See the reference documentation for details. Here is an example how to listen to such an event in a persisted pipeline:
Delete a Property
In order to delete a property from the property store, you can use the command property.schema.delete, which deletes the property value and all metadata that belongs to this property. Example:
You can also delete multiple properties at once using a pattern. Example:
This example will delete all properties of the app myapp recursively.
Move to trash bin
Instead of deleting a property, you can also move it to a trash bin. In this case the property wont be deleted immediately, instead it will be deleted in case the trash bin is cleaned up or in case the time to live in the trash bin has been expired. Example:
In this example, the property wont be deleted. Instead it will be moved to the trash bin for 30 days and then deleted after this time.
For more information see: TODO
Property deleted event
Whenever you delete a property, an event with key property.deleted is fired with the origin property stored in payload.origin of the event object. This way you can listen in your pipelines for property deletions. See the reference documentation for details. Here is an example how to listen to such an event in a persisted pipeline:
Loading Properties
If you want to query for property data, you have multiple options in PIPEFORCE.
Listing properties
One of the simplest approaches to query for multiple properties is by using the command property.list. Example:
This command will return a result similar to the result shown below, which describes all important attributes of the property:
You can also use a path pattern in order to load multiple properties matching this path pattern. This example will load all properties inside the pipeline folder:
This will return "recursively" all properties below the path global/app/myapp/.... The properties attributes including the value will be returned.
Getting property value
In case you would like to load only the value (payload) of a property but not its metadata, you can use the command property.value.get. Example:
Advanced / Deep Querying
In case the base commands for loading properties are not sufficient for your case, you can use JSON Property Querying which supports also complex search operations on JSON documents.