What is the Property Store?​

The property store is a key-value database inside PIPEFORCE to save your application data like sources, attachments or JSON data for example. You can create, update, query and delete data on the property store using the property.* commands.

What is a Property?​

Each data (for example plain text, JSON, binary data, ...) stored in the property store is wrapped into an envelope called a property.

Property Attributes​

Each property has multiple attributes (the envelope data). The most important ones are:

Property Path​

Every property has a unique path. Such a property path is structured in an hierarchical way and has this base structure:

/pipeforce/<namespace>/<localPath>

For example:

/pipeforce/mynamespace/global/app/myapp/config/app

Every property belongs to a certain namespace. If such a path starts with a slash / at the beginning, it is interpreted as absolute. Meaning, it must start with /pipeforce followed by the namespace, this property belongs to. Properties are always stored with their absolute paths in the property store.

At runtime, property paths can be defined and interpreted relatively. Here is an example of an relative path of a property:

global/app/myapp/config/app

Relative property paths do not start with a slash. They do not contain information about the current namespace. This information is automatically added when a property is loaded or stored.

WHY PROPERTY PATH?

• It defines the unique path of a property in a tree structure.

• It makes it easier to query a nested subset of properties "recursively" using path patterns.

• It allows to group properties together under a "path-like" structure.

• It is also used for declaring nested read and write permissions.

• It defines the storage location of the property for performance optimization (for example archive is stored on a different location)

Property Path Pattern​

Since the path of a property is similar to a path in a file tree, it is possible to use path matching techniques in order to easily find and load a nested subset of properties "recursively" from the store. By default, PIPEFORCE uses the so called "ant-style" path matcher. This matcher uses the following rules, applied to a path:

• The double asterisk ** matches any character in the path at given position.

• The single asterisk * matches zero or more charachters inside a "directory" of the path (up to the next slash /).

The asterisk * is often also called wildcard. A property path containing wildcards are called a path pattern.

Example 1

global/app/myapp/**

This path pattern will select all properties having a path starting with global/app/myapp at any sublevel. So it will match these paths:

global/app/myapp/config/app
global/app/myapp/pipeline/mypipeline
global/app/myapp/sub1/sub2/sub3/sub4/sub5/sub6/file

But it won't mach these paths:

global/app/ontherapp/config/app
global/something
tmp/data

Example 2

global/app/myapp/*

This path pattern will select all properties having a path starting with global/app/myapp inside this level (meaning inside the "directory" myapp) but not at any sublevel. So it will match these paths:

global/app/myapp/file1
global/app/myapp/file2
global/app/myapp/another

But it won't match these paths:

global/app/myapp/config/app
global/app/ontherapp/config/app
tmp/data

Example 3

It is also possible to combine multiple wildcards inside a pattern.

global/app/*/data/**

This path pattern will match any path starting with prefix global/app, followed by any app name (without any slash), followed by /data/, followed by any further charachters at any level. It will match these paths:

global/app/myapp/data/user
global/app/ontherapp/data/user
global/app/third/data/folder/file

But it won't match these paths:

global/app/myapp/config/app
tmp/data

Example 4

*

This pattern would match any relative path at top level, but not any sub-level. So it will match these paths:

resourceA
resourceB

But it won't match these paths:

folder/file1
folder/folder2/file2

Example 5

**

This pattern will match anything at any level. So it will match these paths:

resourceA
resourceB
folder/file1
folder/folder2/file2

There is no path, this pattern will not match.

Property Value and Type​

The value attribute of a property defines the payload of a property. Such a payload could be a JSON document or a plain text for example. It is always stored as string (text) value. Which type the text value represents, is defined by the type attribute which could be of any supported mime type e.g. application/json or text/plain. The latter is the default, if no type is specified. Important mime types, used by PIPEFORCE are:

See the http://IANA.org websites for a full list of official mime types: https://www.iana.org/assignments/media-types/media-types.xhtml

Property types can additionally contain parameters. Example:

text/plain; charset=UTF8

Multiple parameters are separated by a colon ;. See the IANA specification for details. PIPEFORCE uses custom type parameters in some situations. Some examples:

Binary Value​

The value attribute contains always values of type string. In order to store binary data there, it must be base64 encoded before it gets persisted. In this case, the property must be marked as a base64 value. This is done in the type attribute using the mime type parameter encoding=base64. Example:

value: "<some base64 encoded value>"
type: "application/pdf;encoding=base64"

In this case the property can be automatically converted back into a "binary" file when loaded.

Create a Property​

To create a new property in the property store, you need to define a property schema first using the command property.schema.put. Example:

pipeline:
  - property.schema.put:
      path: "global/app/myapp/data/mydata"
      type: "application/json"

This schema defines the base envelope data for the new property like its path, uuid and type. After the schema has been created, the property payload can be added and edited.

To simplify these steps, you can create the schema and add the payload value in one single step:

pipeline:
  - property.schema.put:
      path: "global/app/myapp/data/mydata"
      type: "application/json"
      value: {"hello": "world"}

Property created event​

Whenever you create a new a property, an event with key property.created is fired with the created property stored in payload.target of the event object. This way you can listen in your pipelines for properties created newly. See the reference documentation for details. Here is an example how to listen to such an event in a persisted pipeline:

pipeline:
  - event.listen:
      key: "property.created"
      filter: "#{body.payload.target.path.contains('global/app/myapp/data/mydata')}"

  # Do something here

This persisted pipeline gets automatically executed whever a new property with a path containing global/app/myapp/data/mydata has been created.

Update a Property​

To change the value of the property, you can use the command property.put. The property must already exist beforehand. Example:

pipeline:
 - property.put: 
     path: "global/app/myapp/data/mydata"
     value: {"new": "value"}

In order to change the envelope data (attributes) of a property, you need to use the property.schema.put command:

pipeline:
 - property.schema.put: 
     path: "global/app/myapp/data/mydata"
     timeToLive: 5

If you would like to change the path of an existing property, you can use the property.move command:

pipeline:
 - property.move: 
     path: "global/app/myapp/data/mydata"
     to: "global/app/myapp/data/newlocation"

Property changed event​

Whenever you update a property value or one of its other attributes, an event with key property.updated is fired with the origin property stored in origin of the event object and the final version of the property in target. This way you can listen in your pipelines for property changes. See the reference documentation for details. Here is an example how to listen to such an event in a persisted pipeline:

pipeline:
  - event.listen:
      key: property.updated
      filter: ${body.payload.origin.path.contains('global/app/myapp/data/mydata')}

  # Do something here

Property moved event​

Whenever you move a property by changing its origin path, an event with key property.moved is fired with the origin path of the property stored in origin of the event object and the final path of the property in target. This way you can listen in your pipelines for property movements. See the reference documentation for details. Here is an example how to listen to such an event in a persisted pipeline:

pipeline:
  - event.listen:
      key: property.moved
      filter: ${body.payload.origin.contains('global/app/myapp/data/mydata')}

  # Do something here

Delete a Property​

In order to delete a property from the property store, you can use the command property.schema.delete, which deletes the property value and all metadata that belongs to this property. Example:

pipeline:
 - property.delete:
     path: "global/app/myapp/data/mydata"

You can also delete multiple properties at once using a pattern. Example:

pipeline:
 - property.delete:
     pattern: "global/app/myapp/**"

This example will delete all properties of the app myapp recursively.

Move to trash bin

Instead of deleting a property, you can also move it to a trash bin. In this case the property wont be deleted immediately, instead it will be deleted in case the trash bin is cleaned up or in case the time to live in the trash bin has been expired. Example:

pipeline:
 - property.delete:
     path: "global/app/myapp/data/mydata"
     trashBin: true
     trashBinTimeToLive: 2592000000

In this example, the property wont be deleted. Instead it will be moved to the trash bin for 30 days and then deleted after this time.

For more information see: TODO

Property deleted event​

Whenever you delete a property, an event with key property.deleted is fired with the origin property stored in payload.origin of the event object. This way you can listen in your pipelines for property deletions. See the reference documentation for details. Here is an example how to listen to such an event in a persisted pipeline:

pipeline:
  - event.listen:
      key: property.deleted
      filter: ${body.payload.origin.path.contains('global/app/myapp/data/mydata')}

  # Do something here

Loading Properties​

If you want to query for property data, you have multiple options in PIPEFORCE.

Listing properties​

One of the simplest approaches to query for multiple properties is by using the command property.list. Example:

pipeline:
  - property.list:
      pattern: global/app/myapp/pipeline/helloworld

This command will return a result similar to the result shown below, which describes all important attributes of the property:

[
  {
    "path": "/pipeforce/enterprise/global/app/myapp/pipeline/helloworld",
    "uuid": "529a38e7-9188-4135-b87b-3d890f6764f3",
    "value": "pipeline:\n  - log:        \n      message: \"Hello World\"",
    "defaultValue": null,
    "type": "application/yaml;type=pipeline",
    "created": 1613397114448,
    "updated": null,
    "timeToLive": null,
  }
]

You can also use a path pattern in order to load multiple properties matching this path pattern. This example will load all properties inside the pipeline folder:

pipeline:
  - property.list:
      pattern: global/app/myapp/**

This will return "recursively" all properties below the path global/app/myapp/.... The properties attributes including the value will be returned.

[
  {
    "path": "...",
    "uuid": "...",
    "value": "...",
    "defaultValue": null,
    "type": "...",
    "created": 123,
    "updated": null,
    "timeToLive": null,
  },
  {
    "path": "...",
    "uuid": "...",
    "value": "...",
    "defaultValue": null,
    "type": "...",
    "created": 123,
    "updated": null,
    "timeToLive": null,
  }
]

Getting property value​

In case you would like to load only the value (payload) of a property but not its metadata, you can use the command property.value.get. Example:

pipeline:
  - property.value.get:
      path: global/app/myapp/pipeline/helloworld

Advanced / Deep Querying​

In case the base commands for loading properties are not sufficient for your case, you can use JSON Property Querying which supports also complex search operations on JSON documents.