We updated the status code for "scrape failed" event from 429 to 503

/ proxycurl

We updated the status code for "scrape failed" event from 429 to 503

Subscribe to our newsletter

Get the latest news from Proxycurl

We updated the API today with a new 503 error status code.

Context -- there are two reasons why you will get an error with the status code of 429. And it has been confusing for users because users assume they made a mistake.

The first reason is that your account has hit an account-level rate limit of 300 requests/minute. However, most users do not hit that limit.

The second reason is that scraping failed. And this is a limitation on our end—a backend error. And backend errors should not return a 4xx error code, given that 4xx errors signify a client-side error.

Today, we deployed an update to the API, which will return a status code of 503 if a scrape failed instead of 429. The way you should handle it is the same as is with error 429. If you get a status code of 503, do retry the request immediately, and you're good to go.

Subscribe to our newsletter

Get the latest news from Proxycurl

Latest Articles

Here’s what we’ve been up to recently.