How to use Synthetiq's webhooks to receive data extraction results.
Overview of Synthetiq Webhooks
When documents are processed using the Data Extraction API, Synthetiq synchronously responds to the initial request (to confirm that the process has started), and then sends the results asynchronously using webhooks. This is because the data extraction process can take anywhere from a few seconds to a few minutes, depending on the length and complexity of the document. Synthetiq uses HTTPS to send these results to your endpoint as a JSON payload.
Settings Up Webhooks
To begin receiving data extraction results from Synthetiq, follow these steps:
- Create a webhook endpoint as an HTTP endpoint (i.e. URL) on your local server.
- Parse event objects sent by Synthetiq and return 2xx response status codes.
- Deploy your webhook endpoint to a publicly accessible HTTPS URL.
- Connect your HTTPS URL to Synthetiq's platform by setting up a webhook in Synthetiq's platform (Developers -> Webhooks).
Step 1: Create A Webhook Endpoint On Your Local Server
Set up an HTTP endpoint on your local machine that can accept unauthenticated webhook requests with a POST method.
For example, this is a route written in Express (Node.js). The /webhooks route is configured to accept only POST requests and expects data to be delivered in a JSON payload.
const express = require('express');
const app = express();
app.post('/webhooks', (req, res) => {
res.status(200).send({});
});
app.listen(3000, () => {
console.log(`Example app listening at http://localhost:3000`)
});
Step 2: Parse Event Objects And Return 2xx Responses
Your endpoint must be configured to read event objects for the type of event notifications you want to receive. Synthetiq sends events to your webhook endpoint as part of a POST request with a JSON payload.
Check Event Objects
Your endpoint must process the events and quickly (within 6 seconds) return a successful status code (2xx) prior to any complex logic that could cause a timeout.
Built-In Retries
Synthetiq webhooks have built-in retry methods for failed requests:
| Retry Number | Timing |
|---|---|
| 1 | 5 minutes after the actual request |
| 2 | 15 minutes after the prior retry |
| 3 | 30 minutes after the prior retry |
| 4 | 1 hour after the prior retry |
| 5 | 2 hours after the prior retry |
| 6 | 4 hours after the prior retry |
| 7 | 8 hours after the prior retry |
After several hours, we will:
- Email all users with permission level of Owner, as well as the users the created and most recently updated the endpoint in Synthetiq's platform.
- Automatically disable the endpoint (if you haven’t addressed the issue).
Handling Duplicate Events
Your webhook endpoints might occasionally receive the same event more than once. You can guard against duplicate events by logging the eventId of events you’ve processed, and then checking future events against this.
Step 3: Deploy Your Endpoint To A Public HTTPS URL
To begin receiving events from Synthetiq, you must deploy your endpoint to a publicly accessible HTTPS URL.
Receive Events With An HTTPS Server
If you use an HTTPS URL for your webhook endpoint, Synthetiq validates that the connection to your server is secure before sending your webhook data. For this to work, your server must be correctly configured to support HTTPS with a valid SSL certificate.
Step 4: Register Your Public URL In Synthetiq's Webhooks Dashboard
Once you've deployed your endpoint to a public HTTPS URL, you can add the endpoint to Synthetiq's platform through our webhooks dashboard.
To add an endpoint to Synthetiq's platform:
- Navigate to Developers > Webhooks
- Click the button
This will open a form with the following information:
| Field | Description |
|---|---|
| URL | The public HTTPS URL your endpoint was deployed to. |
| Description | A brief description of your webhook. |
| Listening For | The event(s) that your endpoint will listen for. |
| Status | A toggle to decide whether your endpoint should immediately be activated (if not, you can activated it later). |
Once you have completed the form, click . Your endpoint will be created and you'll begin receiving events immediately if your endpoint was activated.
You will be able to see all events related to this endpoint in the "Events" tab, and the "Integration" tab provides your Webhook Secret Key, which you can use to further secure your endpoint (suggested).
Secure Your Webhooks Endpoint
To let you verify that events were sent by Synthetiq (not by a third party), Synthetiq can optionally sign the webhook events it sends to your endpoints by including a signature in each event’s x-synthetiq-signature header.
You can verify signatures either using Synthetiq's Official Node.js library, or using your own solution See the "Verifying Signatures Manually" section in this article to find more details.
Before you can verify signatures, you need to retrieve your endpoint’s secret from within the Synthetiq platform by navigating to this specific webhook and clicking on the "Integration" tab.
Synthetiq generates a unique secret key for each endpoint. Additionally, if you use multiple endpoints, you must obtain a secret for each one you want to verify signatures on.
Highly Recommended To Increase Security!
Security your endpoint provides increased security for your overall application. It is highly recommended that all organizations who implement Synthetiq webhooks secure their endpoints.
Verify Signatures Using Synthetiq's Node.js Library
When using Synthetiq's library to verify signatures, you perform the verification by providing:
- the event payload,
- the x-synthetiq-signature header,
- the x-synthetiq-event-timestamp, and
- the endpoint’s secret.
If verification fails, Synthetiq returns an error.
Preventing Replay Attacks
A replay attack is when an attacker intercepts a valid payload and its signature, then re-transmits them. To mitigate such attacks, Synthetiq includes a timestamp in the x-synthetiq-event-timestamp header. Because this timestamp is part of the signed payload, it is also verified by the signature, so an attacker cannot change the timestamp without invalidating the signature. If the signature is valid but the timestamp is too old, you can have your application reject the payload.
Our libraries have a default tolerance of five minutes between the timestamp and the current time. You can change this tolerance by providing an additional parameter when verifying signatures. Use Network Time Protocol (NTP) to ensure that your server’s clock is accurate and synchronizes with the time on Synthetiq's servers.
Synthetiq generates the timestamp and signature each time an event is sent to your endpoint. If Synthetiq retries an event (e.g., your endpoint previously replied with a non-2xx status code), then a new signature and timestamp is generated for the new delivery attempt.
Verifying Signatures Manually
Synthetiq generates signatures using a hash-based message authentication code (HMAC) with SHA-256.
Although it’s recommended to use our official libraries to verify webhook event signatures, you can create a custom solution by following these steps.
Step 1: Extract The Timestamp And Signatures From The Header
Retrieve the x-synthetiq-event-timestamp and x-synthetiq-signature from request headers.
Step 2: Prepare The signed_payload String
The signed_payload string is created by concatenating:
- The timestamp (as a string),
- The character ".", and
- The actual JSON payload (i.e., the request body)
Step 3: Determine The Expected Signature
Compute an HMAC with the SHA256 hash function. Use the endpoint’s signing secret as the key, and use the signed_payload string as the message.
Step 4: Compare The Signatures
Compare the signature (or signatures) in the header to the expected signature. For an equality match, compute the difference between the current timestamp and the received timestamp, then decide if the difference is within your tolerance.
To protect against timing attacks, use a constant-time string comparison to compare the expected signature to each of the received signatures.