Scraping Products Instructions
Scraper API
Scraper Configs
AI Creation

AI Creation

Generate complete scraper configurations automatically using natural language prompts. Our AI agent analyzes your request, discovers URLs, inspects pages, and builds optimized extraction schemas—all without manual configuration.

Our config generator will:

  1. Parse your natural language request
  2. Discover relevant URLs if needed
  3. Scrape pages to understand the structure
  4. Generate optimized extraction schemas
  5. Test and validate the configuration
  6. Return a ready-to-use config

Endpoint Reference

Generate Config

Config generation is always asynchronous. Submit your request and poll for results.

POST https://scrape.evomi.com/api/v1/account/configs/generate

Request:

curl -X POST "https://scrape.evomi.com/api/v1/account/configs/generate" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Blog Scraper",
    "prompt": "Scrape the title, author, date and content from blog posts on evomi.com"
  }'

Response (202 Accepted):

{
  "success": true,
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "processing",
  "message": "Config generation submitted for background processing",
  "check_url": "/api/v1/account/configs/generate/tasks/550e8400-e29b-41d4-a716-446655440000",
  "credits_reserved": 150
}

Check Task Status 

GET https://scrape.evomi.com/api/v1/account/configs/generate/tasks/{task_id}
curl "https://scrape.evomi.com/api/v1/account/configs/generate/tasks/550e8400-e29b-41d4-a716-446655440000" \
  -H "x-api-key: YOUR_API_KEY"

Response (Still Processing):

{
  "success": true,
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "processing",
  "message": "Task is still processing"
}

Response (Completed):

{
  "success": true,
  "id": "cfg_xyz789",
  "name": "Product Scraper",
  "config": { ... },
  "created_at": "2026-02-19T12:05:00",
  "message": "Scrape config generated and saved successfully",
  "ai_steps_used": 4,
  "credits_used": 120
}

Credit Costs

Action Cost
AI step (LLM call) 30 credits each
Max steps per request 8 steps (240 credits max)
Internal API calls for discovery & scraping Charged at normal rates (2-10 credits estimate)
ℹ️
Credits are reserved upfront and unused credits are automatically refunded. If the AI fails to generate a valid config, you’re only charged for steps actually used. Average requests are 1-3 steps.

Example Prompts

Blog & News Content 

{
  "name": "Evomi Blog Posts",
  "prompt": "Scrape blog posts from evomi.com - find the blog section and extract title, date and content"
}
{
  "name": "BBC News Headlines",
  "prompt": "Scrape headlines and summaries from https://www.bbc.com/news"
}

E-Commerce Products 

{
  "name": "Product Price Tracker",
  "prompt": "Scrape product name, price, availability and image URL from amazon.com product pages"
}
{
  "name": "Shopify Product Scraper",
  "prompt": "I need a config for Shopify e-commerce product pages with price and title extraction, heres my link ..."
}

Anti-Block & Protected Sites 

{
  "name": "Amazon Product Scraper",
  "prompt": "Scrape product title, price, and reviews from Amazon product pages, heres the link ..., I dont want to be blocked"
}
{
  "name": " Job Listings",
  "prompt": "Scrape job title, company, location, and description from ...com job posting pages, heres my cookies ..., I dont want to be blocked "
}
{
  "name": "Glassdoor Reviews",
  "prompt": "Scrape company reviews from https://www.glassdoor.com/Reviews/Google-Reviews-E9079.htm, I dont want to be blocked"
}

Performance & Filtering 

{
  "name": "Clean Content Config",
  "prompt": "Scrape https://example.com but exclude all ads, navigation, footers and sidebars"
}
{
  "name": "Fast Scraping Config",
  "prompt": "Create a fast scraping config"
}
{
  "name": "Article Content Only",
  "prompt": "Scrape only the main article content from news sites, stripping out all navigation, ads, and related articles sections"
}

Parameters

Parameter Type Required Description
name string Yes Unique name for the generated config
prompt string Yes Natural language description of what to scrape

Prompt Tips

Got a bad result? Being specific can help!

  • Target URL or domain or search pattern — Helps the AI find the right pages
  • Data fields names — List exactly what you want extracted
  • Site characteristics — Mention if it’s an SPA, has anti-bot protection, etc.

Examples:

❌ Vague ✅ Specific
“scrape products” “Scrape product name, price, and availability from amazon.com product pages”
“get news” “Scrape headline, author, date, and summary from BBC News articles”
“make a config” “Create a browser config for scraping JavaScript-rendered content "
“get all the info from … " “go to …com and interact with the sliding cards using javascript and …”
⚠️
If the AI cannot fulfill your request (e.g., site requires login, paywall, or no matching content found), it will return a deny action with an explanation. You’ll only be charged for steps used.

Error Responses

AI Denied Request 

{
  "success": false,
  "action": "deny",
  "reason": "The requested content requires user login credentials. Please provide authentication cookies if allowed.",
  "ai_steps_used": 2,
  "credits_used": 60
}

Generation Failed 

{
  "success": false,
  "action": "error",
  "reason": "Could not find product urls on amazon",
  "ai_steps_used": 2,
  "credits_used": 150
}

Insufficient Credits 

{
  "success": false,
  "error": "Insufficient credits",
  "error_code": "INSUFFICIENT_CREDITS",
  "message": "Your account has insufficient credits. Required: 180, Available: 50"
}

Full Workflow Example

Step 1: Submit Generation Request 

curl -X POST "https://scrape.evomi.com/api/v1/account/configs/generate" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "News Scraper",
    "prompt": "Scrape headlines, summaries and links from the BBC News homepage"
  }'

Step 2: Poll for Status 

curl "https://scrape.evomi.com/api/v1/account/configs/generate/tasks/abc123-uuid-here" \
  -H "x-api-key: YOUR_API_KEY"

Repeat until status is success or failure.

Step 3: Use the Generated Config 

curl -X POST "https://scrape.evomi.com/api/v1/scraper/realtime" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "config_id": "cfg_abc123"
  }'

Best Practices

  1. Be specific — Detailed prompts yield better configs
  2. Poll for results — Generation is async; poll the task endpoint until complete
  3. Test before scaling — Run a test scrape to verify the config works
  4. Iterate — If results aren’t perfect, refine your prompt and regenerate