Property Attachments

Owned by PIPEFORCE
Last updated: Oct 20, 2024
4 min read

What is a Property Attachment?

Properties in the Property Store are typically smaller documents like JSON data for example. Sometimes, it is necessary to enrich this metadata and link it with “binary attachments” like images for example. This is where Property Attachments come in place: They contain binary block data and are always linked to exactly one Property. Property Attachments are similar to email attachments with the difference that their size can be theoretically unlimited.

The storage of the binary data of Attachments can scale well. Each Property can have 0..n Attachments assigned.

Even if it is possible to store binary data as Base64 encoded metadata value to a Property, you should consider to add them as Attachment to a Property instead. Especially if the size of this data is > 100KB, since data in Attachments can be organized in a much more effective way in the backend so performance is better.

Create an Attachment (<=10MB)​

Before you can add binary data to an Attachment, you have to create it first and link it to an existing Property, using the command property.attachment.put.

Let's consider an example of a an job applicant, which is modelled as a Property and exists inside the Property Store under key global/app/jobs/applicant/1. For this applicant now a CV given as PDF local file my-cv.pdf must be added as Attachment. You can do so, by executing the command property.attachment.put:

curl -u 'username:password' \ -H 'Content-Type: application/pdf' \ -X POST 'https://hub-<your-domain>/api/v3/command/property.attachment.put?key=global/app/jobs/applicant/1&name=my-cv.pdf' \ --data-binary "@/local/path/my-cv.pdf"

Such a call will do two things at once: It creates a new Attachment object with name my-cv.pdf and attaches it to the Property with key global/app/jobs/applicant/1. Then, it uploads and stores the PDF data at server side by reading the content from the body.

This approach is fast and easy for files with size up to 10 MB. For files bigger than 10 MB, you need chunked uploads.

Create an Attachment (>10MB)​

Since the size of a single request is limited to max. 10 MB, in case you need to add Attachments with content size bigger than this, you must use chunked uploads instead.

In the backend, the binary data of an Attachment is always split up into blocks, called chunks. Each Attachment will have one to many chunks assigned. Each chunk is by default up to 10 MB. In case an Attachment has no chunk assigned, it's size is 0:

Each property can have 0..N attachments. Attachments can have 0..N chunks.

Where content is finally saved, depends on the backend implementation. This picture should give you just a basic understanding about the relation between these core objects of the Property Store.

When it comes to uploading Attachments of size > 10 MB, you need to implement in your client code a routine similar to this:

CHUNKED UPLOAD

  1. Create an empty attachment using the command property.attachment.put.

  2. Read the local file to be uploaded, extract a chunk up to 10 MB from it and upload it using the command property.attachment.chunk.put.

  3. Repeat 2. until all chunks are uploaded.

Let's use the job applicant example from above and let's assume, the applicant wants to add an introduction movie as Attachment which is around 100 MB big in size. Creating an empty Attachment for this could look like this example:

curl -u 'username:password' \ -X POST 'https://hub-<your-domain>/api/v3/command/property.attachment.put?key=global/app/jobs/applicant/1&name=my-cv.mp4&contentType=video/mp4'

Instead of sending this as HTTP POST, you could also use an HTTP GET. See HTTP API for more options. After the empty Attachment has been created, sending a single chunk could look like this example:

curl -u 'username:password' \ -H 'Content-Type: application/octet-stream' \ -X POST 'https://hub-<your-domain>/api/v3/command/property.attachment.chunk.put?key=global/app/jobs/applicant/1&name=my-cv.pdf' \ --data-binary "@/local/path/chunk01"

This needs to be repeated until all content has been uploaded. In this example, the parameter index is omitted. In this case, the chunk index is automatically calculated. Also note to set the Content-Type to application/octet-stream.

Download Attachment​

In order to download the content of an Attachment, you can use the command property.attachment.content which will stream the whole content of the Attachment as as one single file to the client. The chunks will be concatenated automatically in the backend.

Here is an example to download the file my-cv.pdf from the upload example above as one single content stream:

Download Attachment (chunked)​

For big Attachments > ~20 MB it makes sense to use chunked download, if possible in order to reduce system load and increase download speed. Furthermore you can easily implement pause and resume and do integrity checks per chunk. In this approach you first have to load the number of chunks of the Attachment using the command property.attachment.get and then you can call the command property.attachment.chunk.content for each index. At client side you have to merge the chunks finally into a single file. Here is an example how to download a the chunk at index 0:

Integrity check​

In order to do an integrity check of the chunks downloaded, you can use response header Digest which is sent with the response of the command property.attachment.chunk.content and contains the MD5 checksum of the downloaded chunk in format md5=<hexadecimal>, for example Digest: md5=3ead0df0186e96d94d5ab025c282ed1f.

You can then calculate the same checksum of your downloaded chunk and compare it with the digest value from the header. If they do not match, delete the local chunk and download again.