Beautiful Soup

This guide explains how to use Evomi proxies with Beautiful Soup and the requests library for web scraping in Python. Beautiful Soup handles HTML parsing while requests manages the HTTP connections through the proxy.

Prerequisites

• Python 3.8+ installed
• Your Evomi proxy credentials (username and password)

Installation

pip install beautifulsoup4 requests

For SOCKS5 proxy support, also install the SOCKS extra:

pip install "requests[socks]"

Configuration

import requests
from bs4 import BeautifulSoup

proxy_host = "rp.evomi.com"
proxy_port = "1000"
proxy_user = "your_username"
proxy_pass = "your_password_session-anychars_mode-speed"

proxy_url = f"http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get("https://ip.evomi.com/s", proxies=proxies, timeout=15)
response.raise_for_status()

soup = BeautifulSoup(response.content, "html.parser")
print("Proxy IP:", soup.get_text().strip())

Replace your_username and your_password with your actual Evomi credentials (keep the _session-anychars_mode-speed parameters).

Explanation

1. Proxy URL: The format http://user:pass@host:port embeds authentication directly in the URL. The requests library extracts the credentials and sends them as a Proxy-Authorization header.
2. proxies dictionary: Maps both http and https traffic through the same proxy. Even though the proxy URL uses http://, it correctly tunnels HTTPS traffic via the HTTP CONNECT method.
3. timeout: Always set a timeout to prevent the script from hanging if the proxy is unreachable.
4. raise_for_status(): Raises an exception for 4xx/5xx status codes, making error handling cleaner.

Scraping HTML Elements

A more realistic example that fetches and parses actual page content:

import requests
from bs4 import BeautifulSoup

proxy_user = "your_username"
proxy_pass = "your_password_session-anychars_mode-speed"
proxy_url = f"http://{proxy_user}:{proxy_pass}@rp.evomi.com:1000"
proxies = {"http": proxy_url, "https": proxy_url}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
}

response = requests.get(
    "https://example.com",
    proxies=proxies,
    headers=headers,
    timeout=15,
)
response.raise_for_status()

soup = BeautifulSoup(response.content, "html.parser")

for heading in soup.find_all("h1"):
    print(heading.get_text(strip=True))

Using SOCKS5

After installing requests[socks], change the proxy URL scheme to socks5h://:

proxy_url = f"socks5h://{proxy_user}:{proxy_pass}@rp.evomi.com:1002"
proxies = {"http": proxy_url, "https": proxy_url}

The socks5h:// scheme routes DNS resolution through the proxy, which prevents DNS leaks. Use socks5:// if you want local DNS resolution instead.

Using Sessions

For multiple requests, use a requests.Session to reuse the underlying TCP connection:

session = requests.Session()
session.proxies.update({"http": proxy_url, "https": proxy_url})
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
})

response = session.get("https://example.com", timeout=15)
soup = BeautifulSoup(response.content, "html.parser")

Evomi Proxy Endpoints

Tips and Troubleshooting

• Credentials: Double-check your username, password, host, and port. The password format is your_password_session-anychars_mode-speed.
• Special characters in credentials: If your password contains special characters (@, :, etc.), URL-encode them using urllib.parse.quote().
• SSL errors: If you encounter SSL certificate errors from the target site (not the proxy), ensure your system’s CA certificates are up to date. Avoid verify=False in production.
• Error handling: Wrap requests in try/except blocks to catch requests.exceptions.ProxyError, requests.exceptions.Timeout, and requests.exceptions.HTTPError.
• Rate limiting: Be respectful of target websites. Add delays between requests with time.sleep() and respect robots.txt.
• Dynamic content: Beautiful Soup and requests only fetch the initial HTML. They do not execute JavaScript. For JS-heavy sites, use Selenium or Playwright instead.