# Beautiful Soup

Source: https://docs.evomi.com/proxy-instructions/integration-guides/beautiful-soup/

This guide explains how to use Evomi proxies with Beautiful Soup and the `requests` library for web scraping in Python. Beautiful Soup handles HTML parsing while `requests` manages the HTTP connections through the proxy.

## Prerequisites

- Python 3.8+ installed
- Your Evomi proxy credentials (username and password)

## Installation

```bash
pip install beautifulsoup4 requests
```

For SOCKS5 proxy support, also install the SOCKS extra:

```bash
pip install "requests[socks]"
```

## Configuration

```python
import requests
from bs4 import BeautifulSoup

proxy_host = "rp.evomi.com"
proxy_port = "1000"
proxy_user = "your_username"
proxy_pass = "your_password_session-anychars_mode-speed"

proxy_url = f"http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get("https://ip.evomi.com/s", proxies=proxies, timeout=15)
response.raise_for_status()

soup = BeautifulSoup(response.content, "html.parser")
print("Proxy IP:", soup.get_text().strip())
```

Replace `your_username` and `your_password` with your actual Evomi credentials (keep the `_session-anychars_mode-speed` parameters).

## Explanation

1. **Proxy URL**: The format `http://user:pass@host:port` embeds authentication directly in the URL. The `requests` library extracts the credentials and sends them as a `Proxy-Authorization` header.
2. **`proxies` dictionary**: Maps both `http` and `https` traffic through the same proxy. Even though the proxy URL uses `http://`, it correctly tunnels HTTPS traffic via the HTTP `CONNECT` method.
3. **`timeout`**: Always set a timeout to prevent the script from hanging if the proxy is unreachable.
4. **`raise_for_status()`**: Raises an exception for 4xx/5xx status codes, making error handling cleaner.

## Scraping HTML Elements

A more realistic example that fetches and parses actual page content:

```python
import requests
from bs4 import BeautifulSoup

proxy_user = "your_username"
proxy_pass = "your_password_session-anychars_mode-speed"
proxy_url = f"http://{proxy_user}:{proxy_pass}@rp.evomi.com:1000"
proxies = {"http": proxy_url, "https": proxy_url}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
}

response = requests.get(
    "https://example.com",
    proxies=proxies,
    headers=headers,
    timeout=15,
)
response.raise_for_status()

soup = BeautifulSoup(response.content, "html.parser")

for heading in soup.find_all("h1"):
    print(heading.get_text(strip=True))
```

Always set a realistic `User-Agent` header. Many websites block requests that use Python's default User-Agent string.

## Using SOCKS5

After installing `requests[socks]`, change the proxy URL scheme to `socks5h://`:

```python
proxy_url = f"socks5h://{proxy_user}:{proxy_pass}@rp.evomi.com:1002"
proxies = {"http": proxy_url, "https": proxy_url}
```

The `socks5h://` scheme routes DNS resolution through the proxy, which prevents DNS leaks. Use `socks5://` if you want local DNS resolution instead.

## Using Sessions

For multiple requests, use a `requests.Session` to reuse the underlying TCP connection:

```python
session = requests.Session()
session.proxies.update({"http": proxy_url, "https": proxy_url})
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
})

response = session.get("https://example.com", timeout=15)
soup = BeautifulSoup(response.content, "html.parser")
```

## Evomi Proxy Endpoints

| Proxy Type | HTTP | HTTPS | SOCKS5 |
|---|---|---|---|
| **Residential** | `rp.evomi.com:1000` | `rp.evomi-proxy.com:1001` | `rp.evomi.com:1002` |
| **Mobile** | `mp.evomi.com:3000` | `mp.evomi-proxy.com:3001` | `mp.evomi.com:3002` |
| **Datacenter** | `dcp.evomi.com:2000` | `dcp.evomi-proxy.com:2001` | `dcp.evomi.com:2002` |

HTTPS endpoints use the `evomi-proxy.com` domain — that is the hostname our TLS certificate covers, and it reaches the same servers. A client that verifies certificates will reject the `evomi.com` form on the HTTPS port.

## Tips and Troubleshooting

- **Credentials**: Double-check your username, password, host, and port. The password format is `your_password_session-anychars_mode-speed`.
- **Special characters in credentials**: If your password contains special characters (`@`, `:`, etc.), URL-encode them using `urllib.parse.quote()`.
- **SSL errors**: If you encounter SSL certificate errors from the target site (not the proxy), ensure your system's CA certificates are up to date. Avoid `verify=False` in production.
- **Error handling**: Wrap requests in `try/except` blocks to catch `requests.exceptions.ProxyError`, `requests.exceptions.Timeout`, and `requests.exceptions.HTTPError`.
- **Rate limiting**: Be respectful of target websites. Add delays between requests with `time.sleep()` and respect `robots.txt`.
- **Dynamic content**: Beautiful Soup and `requests` only fetch the initial HTML. They do not execute JavaScript. For JS-heavy sites, use Selenium or Playwright instead.