Webhooks
SINCE VERSION 8.0
- 1 What is a Webhook?
- 2 Create a Webhook endpoint
- 3 Calling a Webhook
- 3.1 Result JSON
- 3.1.1 Processing response
- 3.1.2 OK response
- 3.2 Polling for result
- 3.1 Result JSON
- 4 Trigger pipeline by a Webhook call
- 5 List existing Webhooks
- 6 Edit a Webhook
- 7 Delete a Webhook
- 8 Receiving uploaded files via Webhook
- 9 Logging and Tracing of Webhooks
What is a Webhook?
A webhook is a unique HTTP URL in PIPEFORCE which can be called by an external system. In case the external system calls the webhook url, this will trigger any custom action inside PIPEFORCE.
This is a very effective and straight forward possibility to integrate external systems into PIPEFORCE using a push / listening approach, with a very low requirements barrier to the caller system. Nearly every modern system can be integrated this way.
The URL of such an webhook, which gets called by the external system, has a format similar to this:
https://hub-<your-domain>/api/v3/command/webhook.receive?token=<token>
Replace
<token>
by the token of your webhook. See below to get such an token.The token is sometimes also referred to as the uuid, the unique id, of a webhook.
It's also possible to place the
token
param as request header instead (recommended, because it is more secure).
You can create and manage multiple custom webhooks and URLs each with individual settings. When called, a webhook will be validated first and then an internal message gets produced, pipelines can listen to using event.listen
or message.receive
. This way you can make sure, webhooks comply with a given set of rules before they will be passed across multiple pipelines, queues and optional microservices:
Webhooks are used to trigger actions on the backend. They are designed as async message senders from external. If you need more advanced requirements like direct responses, specific response headers or similar, consider to use the API Gateway instead.
Create a Webhook endpoint
Before some external service can call your webhook, you have to create and endpoint for it. You can create such a webhook using the command webhook.put
. Here is an example using it in a pipeline:
pipeline:
- webhook.put:
eventKey: "webhook.lead.created"
Or you can use the CLI to create the Webhook:
pi command webhook.put eventKey=webhook.lead.created
The possible parameters to create a webhook are:
eventKey
= The key, the pipelines will listen to (required).payloadType
= (optional) What to do with the payload, before it is send to the messaging queues. Possible values are:raw
= The payload will be send without any conversion.base64
= The payload will be encoded to base64 (default).ignore
= No payload will be send with the webhook message even if specified in the request.
maxPayloadLength
= (optional) The max allowed number of bytes in the payload (body) of the webhook. If bigger, the webhook will be rejected. Default value is512000
(500KB). The maximum possible value for this is4194304
(4MB).
As eventKey
define the internal unique name of the webhook. It's good practise that this name is lower case, grouped by periods and starts with prefix webhook.
. The result after executing the webhook.put
command is a JSON document like this:
The uuid
(also called token) is the unique identifier of your webhook. Since it is very hard to guess this token, it is used to secure your webhook. The external services can use the webhookUrl
in order to call your webhook.
Make sure you keep the
uuid
(token) secure and only communicate it to the external partners via a secure channel.Once created, the uuid cannot be changed afterwards since it is the link of external services to your internal actions.
In case you're using the PIPEFORCE Developer Portal, you can create a Webhook with a few clicks:
Define an event key in the creation dialog and click "Add":
Finally, the Webhook gets listed and you can get its token from the list:
Calling a Webhook
After you have setup the Webhook successfully, it can be triggered (called) from external. To do so, send a GET or POST HTTP request to the webhook url which was returned when you created it. For example:
Result JSON
Any successful webhook call will respond with a JSON in the response body.
Processing response
The default JSON in response body of a webhook request is of this format:
If you just want to send a message to the server and you’re not interested in any response, you can ignore this JSON in the response body. Otherwise, it contains information about the task and how to retrieve the result.
As you can see, in this example status
is set to "processing"
since the webhook has triggered a task in the background which is still processing ( = task running longer than the request process). Therefore, no result value is set: "value": null
.
Additionally, pollingRedirectEnabled
is set to false
, which is the default. So no auto-redirect to the polling endpoint is done. See below for more details on this.
The correlationId
can be used to do polling for the final result, once the background task has been finished. Also see below for more details on this.
OK response
In case the webhook was calling a very short-running task which could finish during the non-blocking request time, it is returned with the response. In this case, the result JSON in the body will look similar to this:
Polling for result
Calling a webhook is by default executed async. This means a webhook call like this will start the execution of the webhook but will not wait for a response of it. Instead, it will return immediately with a 200 OK status code but without any result.
In case you expect a result, you can retrieve it using long polling: The webook call returns the header pipeforce-result-correlationid
. You can use this id from the header in order to poll for the result of the webhook at the /api/v3/result
endpoint:
Whereas <ID>
must be replaced by the pipeforce-result-correlationid
from the header.
This endpoint will return a 302 MOVED status code with a Location
header as long as the result is not ready. In this case, another request can be executed after a few seconds. If the HTTP client can follow redirects, it will be automatically redirected again to this endpoint after a few seconds. Otherwise, you need to call this endpoint manually after a few seconds.
If the result is ready and can be retrieved, a 200 OK status code is returned with the result in the body and redirects will stop.
Auto-redirect to result endpoint
When calling the webhook endpoint, it is by default always returning 200 OK and the final result value may or may not be in the response JSON (depending on whether the background process takes longer than the request process). If the caller client supports redirects, you can set the parameter pollingRedirectEnabled
to true
on the webhook call:
In this case the webhook endpoint will return a redirect to the /api/v3/result
endpoint and will use the polling approach as described above. It will do as many redirects as required and return with the final result when finished. So there is no extra work on client side necessary.
You can also set the request parameter pollingRedirectEnabled
on the /api/v3/result
endpoint to stop or start the auto-redirect for polling at any time:
Polling maximum and wait time
There is a dynamic wait time for polls. The more often you poll, the longer the wait time will be between two polls. So do not poll too frequently. Poll at a maximum of every 5 seconds.
If you poll too frequently, a status 429 TOO_MANY_REQUESTS is returned.
Result not found
In case the polling result doesn't exists or was cleaned-up in the cache already, a status code 410 GONE is returned.
Trigger pipeline by a Webhook call
After you have successfully setup the webhook, any time the webhook url is triggered (called) from the outside, a new message is produced inside PIPEFORCE, which can then be consumed by any pipeline. To do so, use the event.listen
or message.receive
command to listen for such new event messages. Here’s an example which sends an email whenever a new lead was created using a webhook with the eventKey
=webhook.lead.created
:
The input body of the event.listen
command is the payload of the event message submitted from the outside caller.
In case the sender has sent some payload in the body of the webhook request, this payload is made available for you by default as base64 encoded string in the origin
field of the event. To access this data, you have to convert this value as shown in this example:
In case the payload is a serializable format like a string or a JSON document for example, you can set payloadHandling
to raw
for the webhook. In this case, it is not needed to convert the payload from base64, so you can use it directly:
For security reasons, by default, the webhook pipeline is executed with very limited anonymousUser
privileges. So, make sure that you use only commands in your pipeline which can be executed by this user. In case you need more privileges, you can use the iam.run.as
command as shown in this example to switch to the privileges of the given user before executing the command. See the IAM portal for the permissions (or roles) of a given user. Also see Groups, Roles, and Permissions for more details on user privileges / permissions.
List existing Webhooks
To list all existing webhooks, you can use the webhook.get
command:
You will see a JSON / YAML list with details about all existing webhooks.
In order to get the details of a single webhook, use the webhook.get
with the param uuid
, where uuid is the token of the webhook you want to list:
You can also list all existing Webhooks in the Portal:
Edit a Webhook
In order to edit an existing webhook, you can use the webhook.put
command, and set the uuid (= token) of the webhook to edit. For example:
Delete a Webhook
In order to delete an existing webhook, you can use the command webhook.delete
:
And then, set the uuid of the webhook you want to delete.
You can also delete a Webhook by using the Portal in the "Webhooks" section.
Receiving uploaded files via Webhook
It is also possible to send files as a playload with a webhook. To do so, execute the request as multipart POST with the body formatted as multipart/form-data
. For example:
More information about multipart POST requests can be found here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST
Logging and Tracing of Webhooks
Each webhook execution will be logged using the last 6 chars of the uuid/token (shortened for for security and performance reasons) in combination with a random traceId. For example, a webhook with a uuid/token like this:
Will result in a log entry like this:
The left part before the colon :
is the shortened webhook uuid/token. The right part is a random id generated newly for each webhook call. So multiple calls of same webhook can be monitored separately.
This traceId is also returned to the caller of the webhook by using the command webhook.receive
so he can refer to this exact call when required:
Additionally, it will also be set to the initial message sent by the webhook call:
This way you can trace a webhook all the way across different queues, pipelines and microservices.
For example, if you would like to follow all traces of a given webhook, search for the last 6 characters of the webhook uuid/token like 6936bd
in all logs and you should find all traces of a given webbhook ordered by the time of their executions.
If you would like to follow only a certain webhook call instead, use the fully qualified traceId like 6936bd:efg35ee
for example and search for it. You should find all traces related to exactly this single webhook call.