Scraping Products Instructions
Scraper API
Webhooks

Webhooks

Webhooks allow you to receive real-time notifications when scraping operations complete, fail, or start. Instead of polling for results, configure a webhook endpoint to receive instant updates about your scraping jobs.

Overview

Webhooks are HTTP callbacks that deliver event notifications to your configured URL. Use webhooks to:

  • Get notified when scrapes complete — Receive results automatically
  • Handle failures gracefully — React to errors in real-time
  • Integrate with chat platforms — Send notifications to Discord or Slack
  • Build event-driven workflows — Trigger downstream processes automatically

Supported Platforms

Platform Description
Discord Send rich embeds to Discord channels via webhook URLs
Slack Post messages to Slack channels using incoming webhooks
Custom Send JSON payloads to any HTTP endpoint with HMAC signature verification

Webhook Configuration

Parameters

Parameter Type Required Description
url string Yes Your webhook endpoint URL
type string Yes Webhook type: "discord", "slack", or "custom"
events array Yes List of events to subscribe to
secret string No Secret key for HMAC signature (custom webhooks only)

Available Events

Event Description
started Job has been queued and started
completed Job finished successfully
failed Job encountered an error and failed

Usage Examples

With Scrape Requests 

curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "mode": "browser",
    "webhook": {
      "url": "https://your-server.com/webhook",
      "type": "custom",
      "events": ["completed", "failed"],
      "secret": "your-secret-key"
    }
  }'

With Crawl Requests 

curl -X POST "https://scrape.evomi.com/api/v1/scraper/crawl" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "max_urls": 100,
    "webhook": {
      "url": "https://discord.com/api/webhooks/...",
      "type": "discord",
      "events": ["completed"]
    }
  }'

With Map Requests 

curl -X POST "https://scrape.evomi.com/api/v1/scraper/map" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "sources": ["sitemap", "commoncrawl"],
    "webhook": {
      "url": "https://hooks.slack.com/services/...",
      "type": "slack",
      "events": ["completed", "failed"]
    }
  }'

With Search Requests 

curl -X POST "https://scrape.evomi.com/api/v1/scraper/search" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "e-commerce platforms",
    "max_urls": 20,
    "webhook": {
      "url": "https://your-server.com/webhook",
      "type": "custom",
      "events": ["completed"]
    }
  }'

With Schedules 

curl -X POST "https://scrape.evomi.com/api/v1/account/schedule" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Daily Price Monitor",
    "config_id": "cfg_abc123",
    "interval_minutes": 1440,
    "webhook": {
      "url": "https://discord.com/api/webhooks/...",
      "type": "discord",
      "events": ["completed", "failed"]
    }
  }'

Platform-Specific Integration

Discord Webhooks

Discord webhooks post rich embeds to a channel. Get your webhook URL from channel settings:

  1. Go to Channel Settings → Integrations → Webhooks
  2. Click New Webhook and copy the URL
# Discord webhook example
"webhook": {
  "url": "https://discord.com/api/webhooks/123456789/abc123...",
  "type": "discord",
  "events": ["completed", "failed"]
}

Discord embeds include:

  • Job status with color coding (green for success, red for failure)
  • URL scraped
  • Credits used
  • Timestamp

Slack Webhooks

Slack webhooks post messages using incoming webhook URLs. Create one in your Slack app:

  1. Create a Slack App or use an existing one
  2. Enable Incoming Webhooks
  3. Create a new webhook URL
# Slack webhook example
"webhook": {
  "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXX",
  "type": "slack",
  "events": ["completed"]
}

Slack messages include:

  • Status emoji (✅ or ❌)
  • Job details in attachment format
  • Link to view results

Custom Webhooks

Custom webhooks send JSON payloads to any HTTP endpoint. Use the secret parameter for HMAC signature verification:

# Custom webhook with HMAC signature
"webhook": {
  "url": "https://your-server.com/api/webhooks/evomi",
  "type": "custom",
  "events": ["started", "completed", "failed"],
  "secret": "your-webhook-secret"
}

Payload Structure

Custom Webhook Payload

When using custom webhooks, Evomi sends a POST request with the following JSON structure:

{
  "event": "completed",
  "timestamp": "2026-03-09T12:00:00Z",
  "data": {
    "task_id": "task_abc123",
    "task_type": "scrape",
    "url": "https://example.com",
    "status": "success",
    "credits_used": 5,
    "result": {
      "content": "...",
      "metadata": {}
    }
  }
}

Payload Fields

Field Type Description
event string Event type: started, completed, or failed
timestamp string ISO 8601 timestamp of the event
data.task_id string Unique identifier for the task
data.task_type string Type: scrape, crawl, map, or search
data.url string URL that was scraped (for scrape tasks)
data.status string success or failed
data.credits_used number Credits consumed by the operation
data.result object Scraping results (for completed events)
data.error string Error message (for failed events)

HMAC Signature Verification

For custom webhooks, verify the request authenticity using HMAC-SHA256 signatures.

How It Works

  1. Evomi creates a signature using your secret and the request body
  2. The signature is sent in the X-Evomi-Signature header
  3. Your server verifies the signature before processing

Verification Example (Node.js) 

const crypto = require('crypto');

function verifyWebhook(req, secret) {
  const signature = req.headers['x-evomi-signature'];
  const payload = JSON.stringify(req.body);
  
  const expectedSignature = 'sha256=' + 
    crypto.createHmac('sha256', secret)
      .update(payload)
      .digest('hex');
  
  return signature === expectedSignature;
}

// Usage
app.post('/webhooks/evomi', (req, res) => {
  if (!verifyWebhook(req, 'your-webhook-secret')) {
    return res.status(401).send('Invalid signature');
  }
  
  // Process webhook
  console.log('Event:', req.body.event);
  res.status(200).send('OK');
});

Verification Example (Python) 

import hmac
import hashlib
import json

def verify_webhook(headers, body, secret):
    signature = headers.get('X-Evomi-Signature', '')
    expected = 'sha256=' + hmac.new(
        secret.encode(),
        json.dumps(body).encode(),
        hashlib.sha256
    ).hexdigest()
    
    return hmac.compare_digest(signature, expected)

# Usage in Flask
@app.route('/webhooks/evomi', methods=['POST'])
def handle_webhook():
    if not verify_webhook(request.headers, request.json, 'your-webhook-secret'):
        return 'Invalid signature', 401
    
    # Process webhook
    event = request.json.get('event')
    print(f'Event: {event}')
    return 'OK', 200

Best Practices

1. Respond Quickly

Return a 200 OK response within 5 seconds. Long-running processes should be handled asynchronously:

@app.route('/webhook', methods=['POST'])
def webhook():
    # Queue the work for background processing
    process_webhook_async.delay(request.json)
    return 'OK', 200

2. Implement Retry Logic

If your endpoint is temporarily unavailable, Evomi will retry:

  • Up to 5 attempts
  • Exponential backoff
  • 24-hour retention

3. Verify Signatures

Always verify HMAC signatures for custom webhooks to ensure requests come from Evomi.

4. Handle Duplicate Events

Events may be delivered more than once. Use the task_id to deduplicate:

processed_tasks = set()

def handle_webhook(payload):
    task_id = payload['data']['task_id']
    if task_id in processed_tasks:
        return  # Already processed
    processed_tasks.add(task_id)
    # Process the event

5. Use HTTPS

Webhook URLs must use HTTPS with a valid SSL certificate.

Troubleshooting

Webhooks Not Received

  1. Check firewall settings — Ensure your server accepts connections from Evomi IPs
  2. Verify SSL certificate — HTTPS is required with a valid certificate
  3. Review server logs — Look for incoming requests from Evomi
  4. Test manually — Use curl to test your endpoint

Test Your Endpoint 

curl -X POST "https://your-server.com/webhook" \
  -H "Content-Type: application/json" \
  -H "X-Evomi-Signature: sha256=..." \
  -d '{
    "event": "completed",
    "timestamp": "2026-03-09T12:00:00Z",
    "data": {
      "task_id": "test_123",
      "task_type": "scrape",
      "status": "success"
    }
  }'

What’s Next?

Rate LimitsError Handling