Receive Video Analysis Callbacks
Introduction
Sightengine uses callbacks to let your application know when things happen asynchronously, outside of an API request cycle.
For instance, when a live-stream or a long-running video is analyzed, callbacks will be used to inform you when the status of the job changes, or when specific content is detected.
Please use callbacks to track your jobs rather than polling the API. Callbacks are much more efficient for both you and Sightengine. Polling requests are rate-limited so polling would not scale and might introduce unwanted delays.
Callback URL
There are two ways to tell Sightengine where to send callbacks:
- Account-level callback URL (recommended): set it once on the callbacks page of your dashboard. It is used for every video request that has no callback_url parameter. These callbacks are signed, so that you can verify that they come from Sightengine.
- Per-request callback URL: set the callback_url parameter when submitting the video. It overrides the account-level URL for that request. These callbacks are not signed, as there is no signing secret associated with them.
If neither is set, no callbacks are sent, and you will need to poll the API to get the results.
Receiving webhook callbacks
Once you have submitted a video stream and received a 200 HTTP status code, the engine will start consuming the stream and will assign scores to relevant frames and send detailed results to your callback URL.
To avoid sending too many callback events, the engine will send callbacks only when one of the following happens:
- the video has been completely read and the analysis is done
- an "interesting event" has been detected, such as the presence of nudity or weapons
- the job was stopped, or an error was encountered after the initial request was closed
What counts as an interesting event depends on whether the request uses a workflow:
- Without a workflow, a callback is sent when a frame has a probability of 0.5 or more for a concept that is rejected by default (such as explicit nudity, weapons or gore). This rule is fixed and cannot be customized. Concepts that are accepted by default never trigger a callback on their own, but their scores are still included in every callback.
- With a workflow, a callback is sent when a frame leads to a REJECT or REVIEW action, according to the rules and thresholds you defined in the workflow.
Server-side code for callback receipt
Sightengine will submit a POST request to your configured callback URL. You can listen for this callback and process it like this:
# callbacks cannot be received through the command line
# please choose a server-side language to receive and process callbacks
# or use an approach that does not require callbacks (such as synchronous video analysis)
import json
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt
# Using Django
@csrf_exempt
def my_callback_view(request):
payload = request.body
content = json.loads(payload)
# You now have access to the following
# the media id => content['media']['id']
# the moderation data => content['data']
# the moderation status => content['data']['status']
# Acknowledge receipt
return HttpResponse(status=200)
$payload = @file_get_contents('php://input');
$content = json_decode($payload, true);
// You now have access to the following:
// the media id => $content['media']['id']
// the moderation data => $content['data']
// the moderation status => $content['data']['status']
// Acknowledge receipt
http_response_code(200);
// This example uses Express to receive callbacks
const app = require('express')();
// Use body-parser to retrieve the raw body as a buffer
const bodyParser = require('body-parser');
// Match the raw body to content type application/json
app.post('/webhook', bodyParser.raw({type: 'application/json'}), (request, response) => {
let event;
content = JSON.parse(request.body);
// You now have access to the following:
// the media id => content.media.id
// the moderation data => content.data
// the moderation status => content.data.status
// Acknowledge receipt
response.json({received: true});
});
app.listen(8000, () => console.log('Running on port 8000'));
You can verify that the callback is legitimate and coming from Sightengine by verifying the callback signature. Only callbacks sent to your account-level callback URL are signed: callbacks sent to a URL set with the callback_url parameter have no signature.
Callback content
The callback is a JSON object containing information on the job along with the analysis results.
Callback type
The callback_type field tells you what the callback is about. For video analysis, it matches the job status:
- video.analysis.ongoing
- video.analysis.finished
- video.analysis.stopped
- video.analysis.failure
Job status
| ongoing | The job is still on-going. Callbacks with this status are usually triggered because relevant content has been detected and needs to be reported to you immediately, before the video ends. The data you are receiving is preliminary and you should expect to receive more updates. |
| finished | The job has finished. This is usually because the stream has ended. Results are final. |
| stopped | The job has been stopped before it reached its end. This is typically a consequence of stop requests having been sent to the API. |
| failure | The job encountered an error and failed. The data.error object gives more details, with the same type, code and message fields as API errors. Frames analyzed before the error are still included. |
Results
Results are accessible on a per-frame basis under the data.frames key. The structure of the results depends on your choice of models. Please head to our Model reference for more details.
Each callback contains all the frames analyzed so far, not only the new ones. Callbacks for long videos and live streams therefore get larger over time. The last callback contains the complete results.
If the request uses a workflow, the callback also contains a summary object at the top level, with the action (accept, review or reject), the reject_prob and the reject_reason for the video so far.
Test callbacks
You can send a test callback to your endpoint from the callbacks page of your dashboard, for any type of event. Test callbacks are signed and structured like the callbacks sent for your API requests, with sample data. They can be told apart with the live_mode field, set to false on test callbacks and to true otherwise.
Job progress
If the duration of the video is known beforehand, the progress of the job will be provided in the progress field. The value will be between 0 and 1, with a value of 1 corresponding to completion of the job. As an example, a value of 0.5 means that 50% of the job has been completed.
In many cases such as live-streams, the duration of the video is not known in advance. In such cases, the progress cannot be provided and a null value is returned.
Example
{
"callback_type": "video.analysis.finished",
"live_mode": true,
"media": {
"id": "med_9urgHE38fdLs0R85qa7MX",
"uri": "https://example.domain.com/path/to/video.mp4"
},
"request": "req_9urg32dJhsENyFeehczYi",
"data": {
"status": "finished",
"started": 1618946523.492469,
"last_update": 1618946527.68613,
"progress": 1.0,
"operations": 6,
"frames": [
{
"info": {
"id": "med_9urgHE38fdLs0R85qa7MX_1",
"position": 0
},
"nudity": {
"sexual_activity": 0.01,
"sexual_display": 0.01,
"erotica": 0.01,
"sextoy": 0.01,
"suggestive": 0.01,
"suggestive_classes": {
"bikini": 0.01,
"cleavage": 0.01,
"cleavage_categories": {
"very_revealing": 0.01,
"revealing": 0.01,
"none": 0.99
},
"male_chest": 0.01,
"male_chest_categories": {
"very_revealing": 0.01,
"revealing": 0.01,
"slightly_revealing": 0.01,
"none": 0.99
},
"male_underwear": 0.01,
"lingerie": 0.01,
"miniskirt": 0.01,
"other": 0.01
},
"none": 0.99,
"context": {
"sea_lake_pool": 0.01,
"outdoor_other": 0.99,
"indoor_other": 0.01
}
}
},
{
"info": {
"id": "med_9urgHE38fdLs0R85qa7MX_2",
"position": 1000
},
"nudity": {
"sexual_activity": 0.01,
"sexual_display": 0.01,
"erotica": 0.01,
"sextoy": 0.01,
"suggestive": 0.01,
"suggestive_classes": {
"bikini": 0.01,
"cleavage": 0.01,
"cleavage_categories": {
"very_revealing": 0.01,
"revealing": 0.01,
"none": 0.99
},
"male_chest": 0.01,
"male_chest_categories": {
"very_revealing": 0.01,
"revealing": 0.01,
"slightly_revealing": 0.01,
"none": 0.99
},
"male_underwear": 0.01,
"lingerie": 0.01,
"miniskirt": 0.01,
"other": 0.01
},
"none": 0.99,
"context": {
"sea_lake_pool": 0.01,
"outdoor_other": 0.99,
"indoor_other": 0.01
}
}
},
...
]
}
}
Callback recommendations
Acknowledging receipt
To acknowledge receipt of a callback, your endpoint should return a 2xx HTTP status code. Any other information returned in the request headers or request body is ignored. All response codes outside this range, including 3xx codes, will indicate to Sightengine that you did not receive the callback. This does mean that a URL redirection or a "Not Modified" response will be treated as a failure.
Your endpoint has 20 seconds to respond. If your callback script performs complex logic, or makes network calls, it's possible the script would timeout before Sightengine sees its complete execution. For that reason, you may want to have your callback endpoint immediately acknowledge receipt by returning a 2xx HTTP status code, and then perform the rest of its duties.
No retries
Failed callbacks (error status code, timeout, connection or TLS error) are not retried. If your endpoint misses a callback, and in particular the final one, retrieve the results by polling the API with the media id. You can review the latest callbacks sent to your endpoint, along with the status code they received, on the callbacks page of your dashboard.
CSRF
If you are using a web framework such as Rails or Django, your site might be expecting a CSRF token to be set for all incoming POST requests. CSRFs are important to prevent cross site request forgery, but they will prevent you from receiving legitimate callbacks from Sightengine. You should therefore exempt the callback route from CSRF protection.
Verifying callbacks
As an extra security measure, you should check that the media id value returned in the callback matches the media id returned in the initial API response. This can be helpful if you have several streams analyzed at the same time and need to verify what media each callback refers to.
HTTPS
If you use an HTTPS URL for your callback endpoint, please make sure that your server is correctly configured to support HTTPS with a valid server certificate. Sightengine will validate that the connection to your server is secure before sending the data.
Idempotency
Keep in mind that your callback may occasionally receive the same event more than once. We advise you to guard against duplicated event receipts by making your event processing idempotent.