# Network Capture

Source: https://docs.evomi.com/scraping-products-instructions/scraper-api/parameters/network-capture/

Network Capture allows you to intercept and capture HTTP responses made by the browser during page load. This is powerful for extracting API responses, JSON data, and other network traffic that wouldn't be visible in the page HTML.

## Overview

When scraping JavaScript-heavy applications, the data you need often comes from XHR/Fetch API calls rather than the initial HTML. Network Capture lets you define filters to capture these responses automatically.

Network Capture only works with `mode=browser` or `mode=auto`. It is not available in `mode=request` since there is no browser to intercept network traffic.

## Basic Usage

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `networkCapture` | array | - | Array of filter objects (max 10 filters) |

**Simple Example:**
```json
{
  "url": "https://example.com/products",
  "mode": "browser",
  "networkCapture": [
    {
      "filterType": "resourceType",
      "resourceType": "fetch",
      "httpResponseBody": true
    }
  ]
}
```

This captures all Fetch API responses and includes the response body.

## Filter Types

Network Capture supports six filter types, plus a composite type for combining filters.

### Resource Type Filter

Capture responses by resource type.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `filterType` | string | Yes | Must be `"resourceType"` |
| `resourceType` | string | Yes | One of: `document`, `xhr`, `fetch` |
| `httpResponseBody` | boolean | No | Include response body (default: `false`) |

**Example - Capture all XHR requests:**
```json
{
  "filterType": "resourceType",
  "resourceType": "xhr",
  "httpResponseBody": true
}
```

**Available Resource Types:**

| Type | Description |
|------|-------------|
| `document` | Main HTML document requests |
| `xhr` | XMLHttpRequest (AJAX) calls |
| `fetch` | Fetch API requests |

### URL Filter

Capture responses matching a URL pattern.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `filterType` | string | Yes | Must be `"url"` |
| `value` | string | Yes | URL pattern to match (3-8192 characters) |
| `matchType` | string | No | How to match: `contains` (default), `startsWith`, `endsWith`, `exact` |
| `httpResponseBody` | boolean | No | Include response body (default: `false`) |

**Example - Capture API endpoints:**
```json
{
  "filterType": "url",
  "value": "/api/v1/products",
  "matchType": "contains",
  "httpResponseBody": true
}
```

**Match Types:**

| Type | Description | Example Pattern | Matches |
|------|-------------|-----------------|---------|
| `contains` | URL contains the value | `/api/` | `https://site.com/api/users` |
| `startsWith` | URL starts with the value | `https://api.` | `https://api.site.com/data` |
| `endsWith` | URL ends with the value | `.json` | `https://site.com/data.json` |
| `exact` | URL exactly matches | `https://site.com/api` | Only `https://site.com/api` |

### Status Code Filter

Capture responses with specific HTTP status codes.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `filterType` | string | Yes | Must be `"statusCode"` |
| `statusCodes` | array | Yes | Array of status codes to capture |
| `httpResponseBody` | boolean | No | Include response body (default: `false`) |

**Example - Capture successful responses:**
```json
{
  "filterType": "statusCode",
  "statusCodes": [200, 201],
  "httpResponseBody": true
}
```

**Example - Capture errors for debugging:**
```json
{
  "filterType": "statusCode",
  "statusCodes": [400, 401, 403, 404, 500],
  "httpResponseBody": true
}
```

### Content Type Filter

Capture responses by MIME type.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `filterType` | string | Yes | Must be `"contentType"` |
| `contentTypes` | array | Yes | Array of content types to match |
| `matchType` | string | No | How to match: `contains` (default), `startsWith`, `endsWith`, `exact` |
| `httpResponseBody` | boolean | No | Include response body (default: `false`) |

**Example - Capture JSON responses:**
```json
{
  "filterType": "contentType",
  "contentTypes": ["application/json"],
  "matchType": "contains",
  "httpResponseBody": true
}
```

**Example - Capture multiple content types:**
```json
{
  "filterType": "contentType",
  "contentTypes": ["application/json", "text/javascript", "application/xml"],
  "httpResponseBody": true
}
```

### HTTP Method Filter

Capture responses by HTTP method.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `filterType` | string | Yes | Must be `"method"` |
| `methods` | array | Yes | Array of HTTP methods |
| `httpResponseBody` | boolean | No | Include response body (default: `false`) |

**Example - Capture POST responses:**
```json
{
  "filterType": "method",
  "methods": ["POST"],
  "httpResponseBody": true
}
```

**Example - Capture data-modifying requests:**
```json
{
  "filterType": "method",
  "methods": ["POST", "PUT", "PATCH"],
  "httpResponseBody": true
}
```

### Header Filter

Capture responses containing a specific header with a matching value.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `filterType` | string | Yes | Must be `"header"` |
| `headerName` | string | Yes | Name of the header to check |
| `headerValue` | string | Yes | Value to match |
| `matchType` | string | No | How to match: `contains` (default), `startsWith`, `endsWith`, `exact` |
| `httpResponseBody` | boolean | No | Include response body (default: `false`) |

**Example - Capture responses with custom header:**
```json
{
  "filterType": "header",
  "headerName": "X-Api-Version",
  "headerValue": "v2",
  "matchType": "exact",
  "httpResponseBody": true
}
```

### Composite Filter

Combine multiple filters with AND logic. A response must match ALL conditions to be captured.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `filterType` | string | Yes | Must be `"composite"` |
| `conditions` | array | Yes | Array of simple filters (2-10 conditions) |
| `httpResponseBody` | boolean | No | Include response body (default: `false`) |

Composite filters cannot be nested. The `conditions` array must contain only simple filter types (not other composite filters).

**Example - Capture JSON API responses:**
```json
{
  "filterType": "composite",
  "conditions": [
    {
      "filterType": "url",
      "value": "/api/",
      "matchType": "contains"
    },
    {
      "filterType": "contentType",
      "contentTypes": ["application/json"]
    },
    {
      "filterType": "statusCode",
      "statusCodes": [200]
    }
  ],
  "httpResponseBody": true
}
```

This captures responses that:
- Have `/api/` in the URL **AND**
- Return `application/json` content type **AND**
- Have a 200 status code

## Response Format

Captured network responses are returned in the `networkCapture` field of the API response.

### Response Structure

```json
{
  "success": true,
  "url": "https://example.com/products",
  "networkCapture": [
    {
      "interceptionStatus": {
        "status": "success",
        "error": null
      },
      "statusCode": 200,
      "url": "https://example.com/api/v1/products",
      "headers": {
        "content-type": "application/json",
        "cache-control": "no-cache"
      },
      "httpResponseBody": "{\"products\":[...]}",
      "filter": {
        "filterType": "url",
        "value": "/api/",
        "matchType": "contains",
        "httpResponseBody": true
      },
      "request": {
        "url": "https://example.com/api/v1/products",
        "method": "GET",
        "headers": {
          "accept": "application/json"
        },
        "body": null
      }
    }
  ]
}
```

### Response Fields

| Field | Type | Description |
|-------|------|-------------|
| `interceptionStatus` | object | Status of the capture operation |
| `interceptionStatus.status` | string | `"success"` or `"error"` |
| `interceptionStatus.error` | string | Error message if status is `"error"` |
| `statusCode` | integer | HTTP status code of the response |
| `url` | string | Full URL of the captured request |
| `headers` | object | Response headers as key-value pairs |
| `httpResponseBody` | string | Response body (only if `httpResponseBody: true` in filter) |
| `filter` | object | The filter that matched this response |
| `request` | object | Details about the original request |
| `request.url` | string | Request URL |
| `request.method` | string | HTTP method (GET, POST, etc.) |
| `request.headers` | object | Request headers |
| `request.body` | string | Request body (for POST/PUT requests) |

## Practical Examples

### Capture Product API Data

Extract product data from an e-commerce site that loads products via API:

```json
{
  "url": "https://shop.example.com/category/electronics",
  "mode": "browser",
  "wait_until": "networkidle",
  "networkCapture": [
    {
      "filterType": "composite",
      "conditions": [
        {
          "filterType": "url",
          "value": "/products",
          "matchType": "contains"
        },
        {
          "filterType": "resourceType",
          "resourceType": "fetch"
        }
      ],
      "httpResponseBody": true
    }
  ]
}
```

### Capture GraphQL Responses

Capture GraphQL API responses:

```json
{
  "url": "https://app.example.com/dashboard",
  "mode": "browser",
  "networkCapture": [
    {
      "filterType": "composite",
      "conditions": [
        {
          "filterType": "url",
          "value": "/graphql",
          "matchType": "endsWith"
        },
        {
          "filterType": "method",
          "methods": ["POST"]
        }
      ],
      "httpResponseBody": true
    }
  ]
}
```

### Debug Failed Requests

Capture failed API calls for debugging:

```json
{
  "url": "https://example.com/app",
  "mode": "browser",
  "networkCapture": [
    {
      "filterType": "statusCode",
      "statusCodes": [400, 401, 403, 404, 500, 502, 503],
      "httpResponseBody": true
    }
  ]
}
```

### Capture Multiple Endpoints

Use multiple filters to capture different types of responses:

```json
{
  "url": "https://example.com/dashboard",
  "mode": "browser",
  "networkCapture": [
    {
      "filterType": "url",
      "value": "/api/users",
      "matchType": "contains",
      "httpResponseBody": true
    },
    {
      "filterType": "url",
      "value": "/api/orders",
      "matchType": "contains",
      "httpResponseBody": true
    },
    {
      "filterType": "url",
      "value": "/api/products",
      "matchType": "contains",
      "httpResponseBody": true
    }
  ]
}
```

### Capture JSON with Size Filtering

Capture only JSON responses with specific characteristics:

```json
{
  "url": "https://example.com/search",
  "mode": "browser",
  "wait_until": "networkidle",
  "networkCapture": [
    {
      "filterType": "composite",
      "conditions": [
        {
          "filterType": "contentType",
          "contentTypes": ["application/json"],
          "matchType": "contains"
        },
        {
          "filterType": "statusCode",
          "statusCodes": [200]
        },
        {
          "filterType": "resourceType",
          "resourceType": "xhr"
        }
      ],
      "httpResponseBody": true
    }
  ]
}
```

## Best Practices

### Use Specific Filters

Be as specific as possible with your filters to avoid capturing unnecessary data:

```json
// Good - specific URL pattern
{
  "filterType": "url",
  "value": "/api/v2/products/search",
  "matchType": "contains"
}

// Less ideal - too broad
{
  "filterType": "resourceType",
  "resourceType": "fetch"
}
```

### Combine with wait_until

Use `wait_until: "networkidle"` to ensure all API calls complete before the response is returned:

```json
{
  "url": "https://example.com",
  "mode": "browser",
  "wait_until": "networkidle",
  "networkCapture": [...]
}
```

### Only Request Body When Needed

Setting `httpResponseBody: true` increases response size. Only enable it when you need the actual response content:

```json
// When you need response bodies
{
  "filterType": "url",
  "value": "/api/data",
  "httpResponseBody": true
}

// When you only need to know what requests were made
{
  "filterType": "url",
  "value": "/api/data",
  "httpResponseBody": false
}
```

### Use Composite for Precision

Composite filters help avoid capturing unwanted responses:

```json
{
  "filterType": "composite",
  "conditions": [
    {"filterType": "url", "value": "/api/"},
    {"filterType": "statusCode", "statusCodes": [200]},
    {"filterType": "contentType", "contentTypes": ["application/json"]}
  ],
  "httpResponseBody": true
}
```

## Test Examples

Test Network Capture with these reliable sites. Copy and run immediately.

### HTTPBin - Basic JSON Response

HTTPBin is a simple HTTP testing service:

```bash
curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://httpbin.org/",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "url",
        "value": "httpbin.org",
        "matchType": "contains",
        "httpResponseBody": true
      }
    ]
  }'
```

**What you'll capture:** All requests made to httpbin.org.

---

### JSONPlaceholder - Fake REST API

JSONPlaceholder provides a free fake REST API:

```bash
curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://jsonplaceholder.typicode.com/",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "composite",
        "conditions": [
          {
            "filterType": "url",
            "value": "jsonplaceholder.typicode.com",
            "matchType": "contains"
          },
          {
            "filterType": "contentType",
            "contentTypes": ["application/json"]
          }
        ],
        "httpResponseBody": true
      }
    ]
  }'
```

**What you'll capture:** JSON responses from the fake API.

---

### DummyJSON - Products API

DummyJSON provides fake product data:

```bash
curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://dummyjson.com/",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "composite",
        "conditions": [
          {
            "filterType": "url",
            "value": "dummyjson.com",
            "matchType": "contains"
          },
          {
            "filterType": "statusCode",
            "statusCodes": [200]
          }
        ],
        "httpResponseBody": true
      }
    ]
  }'
```

**What you'll capture:** Product data, user info, and cart data as JSON.

---

### GitHub - Capture Repository API

GitHub's web interface makes API calls to load repository data:

```bash
curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://github.com/facebook/react",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "composite",
        "conditions": [
          {
            "filterType": "url",
            "value": "latest-commit/main",
            "matchType": "contains"
          },
          {
            "filterType": "contentType",
            "contentTypes": ["application/json"]
          }
        ],
        "httpResponseBody": true
      }
    ]
  }'
```

**What you'll capture:** Repository stats, contributor data, and more.

---

### Capture All XHR/Fetch Requests (Any Site)

A generic example that captures all XHR/Fetch requests from any site:

```bash
curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "YOUR_TARGET_URL",
    "mode": "browser",
    "delivery": "json",
    "wait_until": "networkidle",
    "networkCapture": [
      {
        "filterType": "resourceType",
        "resourceType": "xhr",
        "httpResponseBody": true
      },
      {
        "filterType": "resourceType",
        "resourceType": "fetch",
        "httpResponseBody": true
      }
    ]
  }'
```

**What you'll capture:** All XHR and Fetch API calls made by the page.

HTTPBin, JSONPlaceholder, and DummyJSON are free public test APIs - perfect for learning how Network Capture works.

## Limitations

- Maximum of **10 filters** per request or **15 MB** whichever first occurs
- Composite filters support **2-10 conditions**
- Only available in `mode=browser` or `mode=auto`
- URL pattern length: 3-8192 characters