SINCE VERSION 8.0

What is a Webhook?

A webhook is a unique HTTP URL in PIPEFORCE which can be called by an external system. In case the external system calls the webhook url, this will trigger any custom action inside PIPEFORCE.

This is a very effective and straight forward possibility to integrate external systems into PIPEFORCE using a push / listening approach, with a very low requirements barrier to the caller system. Nearly every modern system can be integrated this way.

The URL of such an webhook, which gets called by the external system, has a format similar to this:

You can create and manage multiple custom webhooks and URLs each with individual settings. When called, a webhook will be validated first and after this an internal message gets produced. Pipelines can listen to this message using event.listen or message.receive to consume and process the incoming webhook finally. This way you can make sure, webhooks comply with a given set of rules before they will be passed across multiple pipelines, queues and optional microservices:

Create a Webhook endpoint​

Before some external service can call your webhook, you have to create and endpoint for it. You can create such a webhook using the command webhook.put. Here is an example using it in a pipeline:

Or you can use the CLI to create the Webhook:

The possible parameters to create a webhook are:

• eventKey = The key, the pipelines will listen to (required).

• payloadType = (optional) What to do with the payload, before it is send to the messaging queues. Possible values are:

  • raw = The payload will be send without any conversion.

  • base64 = The payload will be encoded to base64 (default).

  • ignore = No payload will be send with the webhook message even if specified in the request.

• maxPayloadLength = (optional) The max allowed number of bytes in the payload (body) of the webhook. If bigger, the webhook will be rejected. Default value is 512000 (500KB). The maximum possible value for this is 4194304 (4MB).

As eventKey define the internal unique name of the webhook. It's good practise that this name is lower case, grouped by periods and starts with prefix webhook.. The result after executing the webhook.put command is a JSON document like this:

The uuid (also called token) is the unique identifier of your webhook. Since it is very hard to guess this token, it is used to secure your webhook. The external services can use the webhookUrl in order to call your webhook.

In case you're using the PIPEFORCE Developer Portal, you can create a Webhook with a few clicks:

Open

Define an event key in the creation dialog and click "Add":

Open

Finally, the Webhook gets listed and you can get its token from the list:

Calling a Webhook​

After you have setup the Webhook successfully, it can be triggered (called) from external. To do so, send a GET or POST HTTP request to the webhook url which was returned when you created it. For example:

Result JSON

Any successful webhook call will respond with a JSON in the response body.

Processing response

The default JSON in response body of a webhook request is of this format:

If you just want to send a message to the server and you’re not interested in any response, you can ignore this JSON in the response body. Otherwise, it contains information about the task and how to retrieve the result.

As you can see, in this example status is set to "processing" since the webhook has triggered a task in the background which is still processing ( = task running longer than the request process). Therefore, no result value is set: "value": null.

Additionally, pollingRedirectEnabled is set to false, which is the default. So no auto-redirect to the polling endpoint is done. See below for more details on this.

The correlationId can be used to do polling for the final result, once the background task has been finished. Also see below for more details on this.

OK response

In case the webhook was calling a very short-running task which could finish during the non-blocking request time, it is returned with the response. In this case, the result JSON in the body will look similar to this:

Polling for result​

Calling a webhook is by default executed async. This means a webhook call like this will start the execution of the webhook but will not wait for a response of it. Instead, it will return immediately with a 200 OK status code but without any result.

In case you expect a result, you can retrieve it using long polling: The webook call returns the header pipeforce-result-correlationid. You can use this id from the header in order to poll for the result of the webhook at the /api/v3/result endpoint:

Whereas <ID> must be replaced by the pipeforce-result-correlationid from the header.

This endpoint will return a 302 MOVED status code with a Location header as long as the result is not ready. In this case, another request can be executed after a few seconds. If the HTTP client can follow redirects, it will be automatically redirected again to this endpoint after a few seconds. Otherwise, you need to call this endpoint manually after a few seconds.

If the result is ready and can be retrieved, a 200 OK status code is returned with the result in the body and redirects will stop.

Auto-redirect to result endpoint

When calling the webhook endpoint, it is by default always returning 200 OK and the final result value may or may not be in the response JSON (depending on whether the background process takes longer than the request process). If the caller client supports redirects, you can set the parameter pollingRedirectEnabled to true on the webhook call:

In this case the webhook endpoint will return a redirect to the /api/v3/result endpoint and will use the polling approach as described above. It will do as many redirects as required and return with the final result when finished. So there is no extra work on client side necessary.

You can also set the request parameter pollingRedirectEnabledon the /api/v3/result endpoint to stop or start the auto-redirect for polling at any time:

Polling maximum and wait time

There is a dynamic wait time for polls. The more often you poll, the longer the wait time will be between two polls. So do not poll too frequently. Poll at a maximum of every 5 seconds.

If you poll too frequently, a status 429 TOO_MANY_REQUESTS is returned.

Result not found

In case the polling result doesn't exists or was cleaned-up in the cache already, a status code 410 GONE is returned.

Consume a webhook call (trigger pipeline by a webhook​)

After you have successfully setup the webhook, any time the webhook url is triggered (called) from the outside, a new event + message is produced inside PIPEFORCE, which can then be consumed by any pipeline. To do so, use the event.listen or message.receive command to listen for such new event messages.

Using event.listen

Here’s an example which sends an email whenever a new lead was created using a webhook with the eventKey =webhook.lead.created. The pipeline uses the event.listen command to handle the incoming webhook call:

The input body of the event.listen command is the payload of the event message submitted from the outside caller.

In case the sender has sent some payload in the body of the webhook request, this payload is made available for you by default as base64 encoded string in the origin field of the event. To access this data, you have to convert this value as shown in this example:

In case the payload is a serializable format like a string or a JSON document for example, you can set payloadHandling to raw for the webhook. In this case, it is not needed to convert the payload from base64, so you can use it directly:

For security reasons, by default, the webhook pipeline is executed with very limited anonymousUser privileges. So, make sure that you use only commands in your pipeline which can be executed by this user. In case you need more privileges, you can use the iam.run.as command as shown in this example to switch to the privileges of the given user before executing the command. See the IAM portal for the permissions (or roles) of a given user. Also see Groups, Roles, and Permissions for more details on user privileges / permissions.

Using message.receive

Any incoming webhook call is in the first place validated in the backend and then produces an internal event with name webhook.<name>. In your pipeline you can directly fetch this event using the command event.receive at this point.

Additionally, any webhook event is also forwarded to the messaging system as a common PIPEFORCE system message so it can also be handled as notification: Notifications. The key of this forwarded message is the event key prefixed with pipeforce.event. For example if the webhook event key is webhook.lead.created, then the message key will become pipeforce.event.webhook.lead.created.

See the drawing from the beginning again here to visualize the difference of event.listen and message.receive:

Listening to a webhook this way works similar to event.listen. Just create a pipeline and put the command message.receive at the very top but now listening to key pipeforce.event.webhook.lead.created compared to the example above:

List existing Webhooks​

To list all existing webhooks, you can use the webhook.get command:

You will see a JSON / YAML list with details about all existing webhooks.

In order to get the details of a single webhook, use the webhook.get with the param uuid, where uuid is the token of the webhook you want to list:

You can also list all existing Webhooks in the Portal:

Edit a Webhook​

In order to edit an existing webhook, you can use the webhook.put command, and set the uuid (= token) of the webhook to edit. For example:

Delete a Webhook​

In order to delete an existing webhook, you can use the command webhook.delete:

And then, set the uuid of the webhook you want to delete.

You can also delete a Webhook by using the Portal in the "Webhooks" section.

Receiving uploaded files via Webhook​

It is also possible to send files as a playload with a webhook. To do so, execute the request as multipart POST with the body formatted as multipart/form-data. For example:

More information about multipart POST requests can be found here: POST - HTTP | MDN

Logging and Tracing of Webhooks​

Each webhook execution will be logged using the last 6 chars of the uuid/token (shortened for for security and performance reasons) in combination with a random traceId. For example, a webhook with a uuid/token like this:

Will result in a log entry like this:

The left part before the colon : is the shortened webhook uuid/token. The right part is a random id generated newly for each webhook call. So multiple calls of same webhook can be monitored separately.

This traceId is also returned to the caller of the webhook by using the command webhook.receive so he can refer to this exact call when required:

Additionally, it will also be set to the initial message sent by the webhook call:

This way you can trace a webhook all the way across different queues, pipelines and microservices.

For example, if you would like to follow all traces of a given webhook, search for the last 6 characters of the webhook uuid/token like 6936bd in all logs and you should find all traces of a given webbhook ordered by the time of their executions.

If you would like to follow only a certain webhook call instead, use the fully qualified traceId like 6936bd:efg35ee for example and search for it. You should find all traces related to exactly this single webhook call.