Rate Limits
Request limits, timeouts, and overflow queue behavior.
Crawly applies rate limits and timeouts to ensure fair usage and reliable performance for all users. When the server is at capacity, requests are automatically queued instead of being rejected.
Rate Limit
This limit applies across all endpoints combined. Exceeding the limit returns a 429 status with error type RATE_LIMITED.
Timeouts per Endpoint
| Endpoint | Timeout | Notes |
|---|---|---|
/v1/scrape | 60 seconds | Single page extraction with headless browser |
/v1/transcript | 30 seconds | YouTube transcript extraction |
Overflow Queue
When the server is at full capacity, requests are automatically queued instead of being rejected. Queued requests return a 202 status with a jobId and pollUrl that you can use to poll for results via GET /v1/result/:jobId.
Timeout Errors
When a request exceeds the timeout, the API returns a 504 status with error type TIMEOUT. Timeout errors are automatically refunded. No credits are charged for failed requests.