More often or not, Proxycurl is not the bottleneck for your application.
On a regular account, every account has a rate limit of 300 requests every minute. So, in theory, you can make 432,000 API requests per day, which amounts to $4320 per day at a minimum. This brings me to the point that most customers are bottlenecked by budget and not the API itself.
That said, should budget not be a problem for you, you can take the following steps to maximize throughput on Proxycurl API.
- Make API requests in parallel. It would help if you increased the concurrency until you are maxing out the rate limit.
For each thread that is making the API request:
- Handle 503 errors by retrying immediately up to 3 times.
- Handle 500 errors by retrying up to 2 times later. (Could be an intermittent database issue).
- Handle 429 with exponential backoff.
