Network Capture

Scraper API

Network Capture

Network Capture allows you to intercept and capture HTTP responses made by the browser during page load. This is powerful for extracting API responses, JSON data, and other network traffic that wouldn’t be visible in the page HTML.

Overview

When scraping JavaScript-heavy applications, the data you need often comes from XHR/Fetch API calls rather than the initial HTML. Network Capture lets you define filters to capture these responses automatically.

ℹ️

Network Capture only works with mode=browser or mode=auto. It is not available in mode=request since there is no browser to intercept network traffic.

Basic Usage

Parameter	Type	Default	Description
`networkCapture`	array	-	Array of filter objects (max 10 filters)

Simple Example:

{
  "url": "https://example.com/products",
  "mode": "browser",
  "networkCapture": [
    {
      "filterType": "resourceType",
      "resourceType": "fetch",
      "httpResponseBody": true
    }
  ]
}

This captures all Fetch API responses and includes the response body.

Filter Types

Network Capture supports six filter types, plus a composite type for combining filters.

Resource Type Filter

Capture responses by resource type.

Field	Type	Required	Description
`filterType`	string	Yes	Must be `"resourceType"`
`resourceType`	string	Yes	One of: `document`, `xhr`, `fetch`
`httpResponseBody`	boolean	No	Include response body (default: `false`)

Example - Capture all XHR requests:

{
  "filterType": "resourceType",
  "resourceType": "xhr",
  "httpResponseBody": true
}

Available Resource Types:

Type	Description
`document`	Main HTML document requests
`xhr`	XMLHttpRequest (AJAX) calls
`fetch`	Fetch API requests

URL Filter

Capture responses matching a URL pattern.

Field	Type	Required	Description
`filterType`	string	Yes	Must be `"url"`
`value`	string	Yes	URL pattern to match (3-8192 characters)
`matchType`	string	No	How to match: `contains` (default), `startsWith`, `endsWith`, `exact`
`httpResponseBody`	boolean	No	Include response body (default: `false`)

Example - Capture API endpoints:

{
  "filterType": "url",
  "value": "/api/v1/products",
  "matchType": "contains",
  "httpResponseBody": true
}

Match Types:

Type	Description	Example Pattern	Matches
`contains`	URL contains the value	`/api/`	`https://site.com/api/users`
`startsWith`	URL starts with the value	`https://api.`	`https://api.site.com/data`
`endsWith`	URL ends with the value	`.json`	`https://site.com/data.json`
`exact`	URL exactly matches	`https://site.com/api`	Only `https://site.com/api`

Status Code Filter

Capture responses with specific HTTP status codes.

Field	Type	Required	Description
`filterType`	string	Yes	Must be `"statusCode"`
`statusCodes`	array	Yes	Array of status codes to capture
`httpResponseBody`	boolean	No	Include response body (default: `false`)

Example - Capture successful responses:

{
  "filterType": "statusCode",
  "statusCodes": [200, 201],
  "httpResponseBody": true
}

Example - Capture errors for debugging:

{
  "filterType": "statusCode",
  "statusCodes": [400, 401, 403, 404, 500],
  "httpResponseBody": true
}

Content Type Filter

Capture responses by MIME type.

Field	Type	Required	Description
`filterType`	string	Yes	Must be `"contentType"`
`contentTypes`	array	Yes	Array of content types to match
`matchType`	string	No	How to match: `contains` (default), `startsWith`, `endsWith`, `exact`
`httpResponseBody`	boolean	No	Include response body (default: `false`)

Example - Capture JSON responses:

{
  "filterType": "contentType",
  "contentTypes": ["application/json"],
  "matchType": "contains",
  "httpResponseBody": true
}

Example - Capture multiple content types:

{
  "filterType": "contentType",
  "contentTypes": ["application/json", "text/javascript", "application/xml"],
  "httpResponseBody": true
}

HTTP Method Filter

Capture responses by HTTP method.

Field	Type	Required	Description
`filterType`	string	Yes	Must be `"method"`
`methods`	array	Yes	Array of HTTP methods
`httpResponseBody`	boolean	No	Include response body (default: `false`)

Example - Capture POST responses:

{
  "filterType": "method",
  "methods": ["POST"],
  "httpResponseBody": true
}

Example - Capture data-modifying requests:

{
  "filterType": "method",
  "methods": ["POST", "PUT", "PATCH"],
  "httpResponseBody": true
}

Header Filter

Capture responses containing a specific header with a matching value.

Field	Type	Required	Description
`filterType`	string	Yes	Must be `"header"`
`headerName`	string	Yes	Name of the header to check
`headerValue`	string	Yes	Value to match
`matchType`	string	No	How to match: `contains` (default), `startsWith`, `endsWith`, `exact`
`httpResponseBody`	boolean	No	Include response body (default: `false`)

Example - Capture responses with custom header:

{
  "filterType": "header",
  "headerName": "X-Api-Version",
  "headerValue": "v2",
  "matchType": "exact",
  "httpResponseBody": true
}

Composite Filter

Combine multiple filters with AND logic. A response must match ALL conditions to be captured.

Field	Type	Required	Description
`filterType`	string	Yes	Must be `"composite"`
`conditions`	array	Yes	Array of simple filters (2-10 conditions)
`httpResponseBody`	boolean	No	Include response body (default: `false`)

⚠️

Composite filters cannot be nested. The conditions array must contain only simple filter types (not other composite filters).

Example - Capture JSON API responses:

{
  "filterType": "composite",
  "conditions": [
    {
      "filterType": "url",
      "value": "/api/",
      "matchType": "contains"
    },
    {
      "filterType": "contentType",
      "contentTypes": ["application/json"]
    },
    {
      "filterType": "statusCode",
      "statusCodes": [200]
    }
  ],
  "httpResponseBody": true
}

This captures responses that:

Have /api/ in the URL AND
Return application/json content type AND
Have a 200 status code

Response Format

Captured network responses are returned in the networkCapture field of the API response.

Response Structure

{
  "success": true,
  "url": "https://example.com/products",
  "networkCapture": [
    {
      "interceptionStatus": {
        "status": "success",
        "error": null
      },
      "statusCode": 200,
      "url": "https://example.com/api/v1/products",
      "headers": {
        "content-type": "application/json",
        "cache-control": "no-cache"
      },
      "httpResponseBody": "{\"products\":[...]}",
      "filter": {
        "filterType": "url",
        "value": "/api/",
        "matchType": "contains",
        "httpResponseBody": true
      },
      "request": {
        "url": "https://example.com/api/v1/products",
        "method": "GET",
        "headers": {
          "accept": "application/json"
        },
        "body": null
      }
    }
  ]
}

Response Fields

Field	Type	Description
`interceptionStatus`	object	Status of the capture operation
`interceptionStatus.status`	string	`"success"` or `"error"`
`interceptionStatus.error`	string	Error message if status is `"error"`
`statusCode`	integer	HTTP status code of the response
`url`	string	Full URL of the captured request
`headers`	object	Response headers as key-value pairs
`httpResponseBody`	string	Response body (only if `httpResponseBody: true` in filter)
`filter`	object	The filter that matched this response
`request`	object	Details about the original request
`request.url`	string	Request URL
`request.method`	string	HTTP method (GET, POST, etc.)
`request.headers`	object	Request headers
`request.body`	string	Request body (for POST/PUT requests)

Practical Examples

Capture Product API Data

Extract product data from an e-commerce site that loads products via API:

{
  "url": "https://shop.example.com/category/electronics",
  "mode": "browser",
  "wait_until": "networkidle",
  "networkCapture": [
    {
      "filterType": "composite",
      "conditions": [
        {
          "filterType": "url",
          "value": "/products",
          "matchType": "contains"
        },
        {
          "filterType": "resourceType",
          "resourceType": "fetch"
        }
      ],
      "httpResponseBody": true
    }
  ]
}

Capture GraphQL Responses

Capture GraphQL API responses:

{
  "url": "https://app.example.com/dashboard",
  "mode": "browser",
  "networkCapture": [
    {
      "filterType": "composite",
      "conditions": [
        {
          "filterType": "url",
          "value": "/graphql",
          "matchType": "endsWith"
        },
        {
          "filterType": "method",
          "methods": ["POST"]
        }
      ],
      "httpResponseBody": true
    }
  ]
}

Debug Failed Requests

Capture failed API calls for debugging:

{
  "url": "https://example.com/app",
  "mode": "browser",
  "networkCapture": [
    {
      "filterType": "statusCode",
      "statusCodes": [400, 401, 403, 404, 500, 502, 503],
      "httpResponseBody": true
    }
  ]
}

Capture Multiple Endpoints

Use multiple filters to capture different types of responses:

{
  "url": "https://example.com/dashboard",
  "mode": "browser",
  "networkCapture": [
    {
      "filterType": "url",
      "value": "/api/users",
      "matchType": "contains",
      "httpResponseBody": true
    },
    {
      "filterType": "url",
      "value": "/api/orders",
      "matchType": "contains",
      "httpResponseBody": true
    },
    {
      "filterType": "url",
      "value": "/api/products",
      "matchType": "contains",
      "httpResponseBody": true
    }
  ]
}

Capture JSON with Size Filtering

Capture only JSON responses with specific characteristics:

{
  "url": "https://example.com/search",
  "mode": "browser",
  "wait_until": "networkidle",
  "networkCapture": [
    {
      "filterType": "composite",
      "conditions": [
        {
          "filterType": "contentType",
          "contentTypes": ["application/json"],
          "matchType": "contains"
        },
        {
          "filterType": "statusCode",
          "statusCodes": [200]
        },
        {
          "filterType": "resourceType",
          "resourceType": "xhr"
        }
      ],
      "httpResponseBody": true
    }
  ]
}

Best Practices

Use Specific Filters

Be as specific as possible with your filters to avoid capturing unnecessary data:

// Good - specific URL pattern
{
  "filterType": "url",
  "value": "/api/v2/products/search",
  "matchType": "contains"
}

// Less ideal - too broad
{
  "filterType": "resourceType",
  "resourceType": "fetch"
}

Combine with wait_until

Use wait_until: "networkidle" to ensure all API calls complete before the response is returned:

{
  "url": "https://example.com",
  "mode": "browser",
  "wait_until": "networkidle",
  "networkCapture": [...]
}

Only Request Body When Needed

Setting httpResponseBody: true increases response size. Only enable it when you need the actual response content:

// When you need response bodies
{
  "filterType": "url",
  "value": "/api/data",
  "httpResponseBody": true
}

// When you only need to know what requests were made
{
  "filterType": "url",
  "value": "/api/data",
  "httpResponseBody": false
}

Use Composite for Precision

Composite filters help avoid capturing unwanted responses:

{
  "filterType": "composite",
  "conditions": [
    {"filterType": "url", "value": "/api/"},
    {"filterType": "statusCode", "statusCodes": [200]},
    {"filterType": "contentType", "contentTypes": ["application/json"]}
  ],
  "httpResponseBody": true
}

Test Examples

Test Network Capture with these reliable sites. Copy and run immediately.

HTTPBin - Basic JSON Response

HTTPBin is a simple HTTP testing service:

curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://httpbin.org/",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "url",
        "value": "httpbin.org",
        "matchType": "contains",
        "httpResponseBody": true
      }
    ]
  }'

What you’ll capture: All requests made to httpbin.org.

JSONPlaceholder - Fake REST API

JSONPlaceholder provides a free fake REST API:

curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://jsonplaceholder.typicode.com/",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "composite",
        "conditions": [
          {
            "filterType": "url",
            "value": "jsonplaceholder.typicode.com",
            "matchType": "contains"
          },
          {
            "filterType": "contentType",
            "contentTypes": ["application/json"]
          }
        ],
        "httpResponseBody": true
      }
    ]
  }'

What you’ll capture: JSON responses from the fake API.

DummyJSON - Products API

DummyJSON provides fake product data:

curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://dummyjson.com/",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "composite",
        "conditions": [
          {
            "filterType": "url",
            "value": "dummyjson.com",
            "matchType": "contains"
          },
          {
            "filterType": "statusCode",
            "statusCodes": [200]
          }
        ],
        "httpResponseBody": true
      }
    ]
  }'

What you’ll capture: Product data, user info, and cart data as JSON.

GitHub - Capture Repository API

GitHub’s web interface makes API calls to load repository data:

curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://github.com/facebook/react",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "composite",
        "conditions": [
          {
            "filterType": "url",
            "value": "latest-commit/main",
            "matchType": "contains"
          },
          {
            "filterType": "contentType",
            "contentTypes": ["application/json"]
          }
        ],
        "httpResponseBody": true
      }
    ]
  }'

What you’ll capture: Repository stats, contributor data, and more.

Capture All XHR/Fetch Requests (Any Site)

A generic example that captures all XHR/Fetch requests from any site:

curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "YOUR_TARGET_URL",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "resourceType",
        "resourceType": "xhr",
        "httpResponseBody": true
      },
      {
        "filterType": "resourceType",
        "resourceType": "fetch",
        "httpResponseBody": true
      }
    ]
  }'

What you’ll capture: All XHR and Fetch API calls made by the page.

ℹ️

HTTPBin, JSONPlaceholder, and DummyJSON are free public test APIs - perfect for learning how Network Capture works.

Limitations

Maximum of 10 filters per request or 15 MB whichever first occurs
Composite filters support 2-10 conditions
Only available in mode=browser or mode=auto
URL pattern length: 3-8192 characters

JavaScript Automation AI Enhancement