...

You can create and manage multiple custom webhooks and URLs each with individual settings. When called, a webhook will be validated first and then after this an internal message gets produced, pipelines . Pipelines can listen to this message using event.listen or message.receive to consume and process the incoming webhook finally. This way you can make sure, webhooks comply with a given set of rules before they will be passed across multiple pipelines, queues and optional microservices:

...

Create a Webhook endpoint​

Before some external service can call your webhook, you have to create and endpoint for it. You can create such a webhook using the command webhook.put. Here is an example using it in a pipeline:

pipeline:
  - webhook.put:
      eventKey: "webhook.lead.created"

Or you can use the CLI to create the Webhook:

pi command webhook.put eventKey=webhook.lead.created

The possible parameters to create a webhook are:

• eventKey = The key, the pipelines will listen to (required).

• payloadType = (optional) What to do with the payload, before it is send to the messaging queues. Possible values are:

  • raw = The payload will be send without any conversion.

  • base64 = The payload will be encoded to base64 (default).

  • ignore = No payload will be send with the webhook message even if specified in the request.

• maxPayloadLength = (optional) The max allowed number of bytes in the payload (body) of the webhook. If bigger, the webhook will be rejected. Default value is 512000 (500KB). The maximum possible value for this is 4194304 (4MB).

As eventKey define the internal unique name of the webhook. It's good practise that this name is lower case, grouped by periods and starts with prefix webhook.. The result after executing the webhook.put command is a JSON document like this:

{
  "eventKey": "webhook.lead.created",
  "webhookUrl": "https://hub-try.pipeforce.org/api/v3/webhook.receive?token=a29a4f16-989d-48c8-ab54-7b6150733ba1",
  "uuid": "a29a4f16-989d-48c8-ab54-7b6150733ba1",
  "payloadType": "base64",
  ...
}

The uuid (also called token) is the unique identifier of

Create a Webhook endpoint​

Before some external service can call your webhook, you have to create and endpoint for it. You can create such a webhook using the command webhook.put. Here is an example using it in a pipeline:

pipeline:
  - webhook.put:
      eventKey: "webhook.lead.created"

Or you can use the CLI to create the Webhook:

pi command webhook.put eventKey=webhook.lead.created

The possible parameters to create a webhook are:

• eventKey = The key, the pipelines will listen to (required).

• payloadType = (optional) What to do with the payload, before it is send to the messaging queues. Possible values are:

  • raw = The payload will be send without any conversion.

  • base64 = The payload will be encoded to base64 (default).

  • ignore = No payload will be send with the webhook message even if specified in the request.

• maxPayloadLength = (optional) The max allowed number of bytes in the payload (body) of the webhook. If bigger, the webhook will be rejected. Default value is 512000 (500KB). The maximum possible value for this is 4194304 (4MB).

As eventKey define the internal unique name of the webhook. It's good practise that this name is lower case, grouped by periods and starts with prefix webhook.. The result after executing the webhook.put command is a JSON document like this:

{
  "eventKey": "webhook.lead.created",
  "webhookUrl": "https://hub-try.pipeforce.org/api/v3/webhook.receive?token=a29a4f16-989d-48c8-ab54-7b6150733ba1",
  "uuid": "a29a4f16-989d-48c8-ab54-7b6150733ba1",
  "payloadType": "base64",
  ...
}

The uuid (also called token) is the unique identifier of your webhook. Since it is very hard to guess this token, it is used to secure your webhook. The external services can use the webhookUrl in order to call your webhook.

...

After you have setup the Webhook successfully, it can be triggered (called) from external. To do so, send a GET or POST HTTP request to the webhook url which was returned when you created it. For example:

https://hub-try.pipeforce.org/api/v3/command/webhook.receive?token=abcdef

Polling for result​

Calling a webhook is by default executed async. This means a webhook call like this will start the execution of the webhook but will not wait for a response of it. Instead, it will return immediately with a 200 OK status code but without any result.

In case you expect a result, you can retrieve it using long polling: The webook call returns the header pipeforce-result-correlationid. You can use this id in the header in order to poll for the result of the webhook at the /api/v3/result endpoint:

https://main-try.pipeforce.net/api/v3/result?correlationId=<ID>

Whereas <ID> must be replaced by the pipeforce-result-correlationid from the header.

This endpoint will return a 302 MOVED status code with a Location header as long as the result is not ready. In this case, another request can be executed after a few seconds. If the HTTP client can follow redirects, it will be automatically redirected again to this endpoint after a few seconds. Otherwise, you need to call this endpoint manually after a few seconds.

If the result is ready and can be retrieved, a 200 OK status code is returned with the result in the body and redirects will stop.

Auto-redirect to result endpoint

When calling the webhook endpoint, it is by default always returning 200 OK without any result in the body. If the caller client supports redirects, you can set the parameter pollingRedirectEnabled to true on the webhook call. In this case the webhook endpoint will return a redirect to the /result endpoint and will use the polling approach as described above:

https://hub-try.pipeforce.org/api/v3/command/webhook.receive?token=abcdef&pollingRedirectEnabled=true

Polling maximum and wait time

There is a dynamic wait time for polls. The more often you poll, the longer the wait time will be between two polls. So do not poll too frequently. Poll at a maximum of every 5 seconds.

If you poll too frequently, a status 429 TOO_MANY_REQUESTS is returned.

Result not found

In case the polling result doesn't exists or was cleaned-up in the cache, a status code 410 GONE is returned.

Trigger pipeline by a Webhook call​

After you have successfully setup the webhook, any time the webhook url is triggered (called) from the outside, a new message is produced inside PIPEFORCE, which can then be consumed by any pipeline. To do so, use the event.listen or message.receive command to listen for such new event messages. Here’s an example which sends an email whenever a new lead was created using a webhook with the eventKey =webhook.lead.created:

pipeline:

 - event.listen:
     eventKey: webhook.lead.created
     
 - iam.run.as: systemuser

 - mail.send:
     to: name@company.tld
     subject: "New lead was created!"
     body: ${@convert.fromBase64(body.payload.origin)}

The input body of the event.listen command is the payload of the event message submitted from the outside caller.

In case the sender has sent some payload in the body of the webhook request, this payload is made available for you by default as base64 encoded string in the origin field of the event. To access this data, you have to convert this value as shown in this example:

${@convert.fromBase64(body.payload.origin)}

In case the payload is a serializable format like a string or a JSON document for example, you can set payloadHandling to raw for the webhook. In this case, it is not needed to convert the payload from base64, so you can use it directly:

pipeline:

 - event.listen:
     eventKey: webhook.lead.created

 - iam.run.as: systemuser

 - mail.send:
     to: name@company.tld
     subject: "New lead was created!"
     body: ${body.payload}

For security reasons, by default, the webhook pipeline is executed with very limited anonymousUser privileges. So, make sure that you use only commands in your pipeline which can be executed by this user. In case you need more privileges, you can use the iam.run.as command as shown in this example to switch to the privileges of the given user before executing the command. See the IAM portal for the permissions (or roles) of a given user. Also see Groups, Roles, and Permissions for more details on user privileges / permissions.

...

Result JSON

Any successful webhook call will respond with a JSON in the response body.

Processing response

The default JSON in response body of a webhook request is of this format:

{
    "statusCode": 200,
    "status": "processing",
    "value": null,
    "pollingRedirectEnabled": false,
    "correlationId": "467609e2-cd8a-48ef-922a-8b107b523b35",
    ...
}

If you just want to send a message to the server and you’re not interested in any response, you can ignore this JSON in the response body. Otherwise, it contains information about the task and how to retrieve the result.

As you can see, in this example status is set to "processing" since the webhook has triggered a task in the background which is still processing ( = task running longer than the request process). Therefore, no result value is set: "value": null.

Additionally, pollingRedirectEnabled is set to false, which is the default. So no auto-redirect to the polling endpoint is done. See below for more details on this.

The correlationId can be used to do polling for the final result, once the background task has been finished. Also see below for more details on this.

OK response

In case the webhook was calling a very short-running task which could finish during the non-blocking request time, it is returned with the response. In this case, the result JSON in the body will look similar to this:

{
    "statusCode": 200,
    "status": "ok",
    "value": "HELLO WORLD",
    "pollingRedirectEnabled": false,
    "correlationId": "617609e2-cd3a-50ef-921a-3b107b523b30",
    ...
}

Polling for result​

Calling a webhook is by default executed async. This means a webhook call like this will start the execution of the webhook but will not wait for a response of it. Instead, it will return immediately with a 200 OK status code but without any result.

In case you expect a result, you can retrieve it using long polling: The webook call returns the header pipeforce-result-correlationid. You can use this id from the header in order to poll for the result of the webhook at the /api/v3/result endpoint:

https://main-try.pipeforce.net/api/v3/result?pipeforce-result-correlationid=<ID>

Whereas <ID> must be replaced by the pipeforce-result-correlationid from the header.

This endpoint will return a 302 MOVED status code with a Location header as long as the result is not ready. In this case, another request can be executed after a few seconds. If the HTTP client can follow redirects, it will be automatically redirected again to this endpoint after a few seconds. Otherwise, you need to call this endpoint manually after a few seconds.

If the result is ready and can be retrieved, a 200 OK status code is returned with the result in the body and redirects will stop.

Auto-redirect to result endpoint

When calling the webhook endpoint, it is by default always returning 200 OK and the final result value may or may not be in the response JSON (depending on whether the background process takes longer than the request process). If the caller client supports redirects, you can set the parameter pollingRedirectEnabled to true on the webhook call:

https://hub-try.pipeforce.org/api/v3/command/webhook.receive?token=abcdef&pollingRedirectEnabled=true

In this case the webhook endpoint will return a redirect to the /api/v3/result endpoint and will use the polling approach as described above. It will do as many redirects as required and return with the final result when finished. So there is no extra work on client side necessary.

You can also set the request parameter pollingRedirectEnabledon the /api/v3/result endpoint to stop or start the auto-redirect for polling at any time:

https://main-try.pipeforce.net/api/v3/result?pipeforce-result-correlationid=<ID>&pollingRedirectEnabled=false

Polling maximum and wait time

There is a dynamic wait time for polls. The more often you poll, the longer the wait time will be between two polls. So do not poll too frequently. Poll at a maximum of every 5 seconds.

If you poll too frequently, a status 429 TOO_MANY_REQUESTS is returned.

Result not found

In case the polling result doesn't exists or was cleaned-up in the cache already, a status code 410 GONE is returned.

Consume a webhook call (trigger pipeline by a webhook​)

After you have successfully setup the webhook, any time the webhook url is triggered (called) from the outside, a new event + message is produced inside PIPEFORCE, which can then be consumed by any pipeline. To do so, use the event.listen or message.receive command to listen for such new event messages.

Using event.listen

Here’s an example which sends an email whenever a new lead was created using a webhook with the eventKey =webhook.lead.created. The pipeline uses the event.listen command to handle the incoming webhook call:

pipeline:

 - event.listen:
     eventKey: webhook.lead.created

 - mail.send:
     to: name@company.tld
     subject: "New lead was created!"
     body: ${@convert.fromBase64(body.payload.origin)}

The input body of the event.listen command is the payload of the event message submitted from the outside caller.

In case the sender has sent some payload in the body of the webhook request, this payload is made available for you by default as base64 encoded string in the origin field of the event. To access this data, you have to convert this value as shown in this example:

${@convert.fromBase64(body.payload.origin)}

In case the payload is a serializable format like a string or a JSON document for example, you can set payloadHandling to raw for the webhook. In this case, it is not needed to convert the payload from base64, so you can use it directly:

pipeline:

 - event.listen:
     eventKey: webhook.lead.created

 - mail.send:
     to: name@company.tld
     subject: "New lead was created!"
     body: ${body.payload}

For security reasons, by default, the webhook pipeline is executed with very limited anonymousUser privileges. So, make sure that you use only commands in your pipeline which can be executed by this user. In case you need more privileges, you can use the iam.run.as command as shown in this example to switch to the privileges of the given user before executing the command. See the IAM portal for the permissions (or roles) of a given user. Also see Groups, Roles, and Permissions for more details on user privileges / permissions.

Using message.receive

Any incoming webhook call is in the first place validated in the backend and then produces an internal event with name webhook.<name>. In your pipeline you can directly fetch this event using the command event.receive at this point.

Additionally, any webhook event is also forwarded to the messaging system as a common PIPEFORCE system message so it can also be handled as notification: Notifications. The key of this forwarded message is the event key prefixed with pipeforce.event. For example if the webhook event key is webhook.lead.created, then the message key will become pipeforce.event.webhook.lead.created.

See the drawing from the beginning again here to visualize the difference of event.listen and message.receive:

Listening to a webhook this way works similar to event.listen. Just create a pipeline and put the command message.receive at the very top but now listening to key pipeforce.event.webhook.lead.created compared to the example above:

pipeline:

 - message.receive:
     key: pipeforce.event.webhook.lead.created

 - mail.send:
     to: name@company.tld
     subject: "New lead was created!"
     body: ${body.payload}
     

List existing Webhooks​

To list all existing webhooks, you can use the webhook.get command:

...