Updates to Rate Limits and Caching
I pushed a huge NinjaPear update yesterday, and it has four parts to it. I will talk about two items in this update for the first part that will immediately affect your use of NinjaPear: caching and rate limits.
Rate Limits
Let's start with the bad news first. I have implemented rate limits and also reduced them.
Rate limits are now live in NinjaPear. While rate limits are documented in our API documentation, they were never fully implemented, mostly because we did not have many users. Three months in, there have been a handful of instances where the NinjaPear service was taken down by a sudden deluge of concurrent API requests. In fact, it literally just happened.
I have also reduced rate limits from 300 requests/minute to 50 requests/minute for API endpoints that you pay for with credits. For the most part, almost no one will be affected other than a handful of you.
What has been nerfed severely are the free API endpoints, such as the Company Logo API or the Disposable Email Checker API. These free tools will now have a rate limit of 2 requests/minute for free/trial tiers.
To sustainably provide free APIs, the rate limit for free APIs now depends on your subscription plan:
- Free, trial, or PAYG plan: 2 requests/min
- $49/mo plan: 20 requests/min
- $299/mo plan: 50 requests/min
- $899/mo plan: 100 requests/min
- $1899/mo plan: 300 requests/min
The free API rate limit applies to Company Logo, Disposable Email Checker, View Credit Balance, and RSS feed consumption.
Sorry about this. These free API requests were consuming compute resources that weren't doing anything for us—no marketing, no attribution, and no user growth—while causing a couple of downtime issues. So, I made a conscious decision to stop supporting users who do not pay us. That said, if you're on a subscription—as many of you are—feel free to enjoy our free services, which will continue to cost you zero credits!
Caching
On to the good news! There is now a user-controllable caching system which will:
- Reduce the latency of cached responses
- Reduce your credit usage
- Allow you to manage profile freshness
For the following API endpoints, there is now a use_cache parameter:
| Endpoint | Default | Freshness window |
|---|---|---|
| GET /api/v1/customer/listing | if-present | 29 days |
| GET /api/v1/competitor/listing | if-present | 29 days |
| GET /api/v1/company/details | if-recent | 29 days |
| GET /api/v1/company/employee-count | if-recent | 29 days |
| GET /api/v1/company/updates | if-recent | 1 day |
| GET /api/v1/company/funding | if-recent | 29 days |
| GET /api/v1/employee/profile | if-recent | 29 days |
| GET /api/v1/employee/search | if-recent | 29 days |
The use_cache parameter controls the freshness of the data you receive, with the side effect of reducing your credit costs. This is how it works and the values that the use_cache parameter accepts:
use_cache=if-recent— Returns a response no older than 29 days (the freshness window might differ, but most API endpoints use 29 days). If you paid for the initial fresh response, calling the same API endpoint with the same parameters any time within those 29 days will return the cached value at no additional credit cost.- This is perfect for cases where you want "fresh-ish" data and might need to call it repeatedly from the cache with as little latency as possible without incurring additional costs.
use_cache=if-present— Returns a cached response if it exists in the cache. You only pay the first time you fetch it. Thereafter, you do not have to pay for it as long as the response remains in the cache (which is ideally forever).- This is perfect for cases when you do not care about freshness or want to minimize cost while getting results as fast as possible.
use_cache=never— Never use the cache! This always performs a live enrichment, and you will be charged every time.
With this update, the average user is expected to spend less on NinjaPear.
Cache control for the win.
The use_cache lore
I used to run Proxycurl, the largest LinkedIn scraper. We implemented the use_cache parameter because there was a subset of popular profiles that were constantly being requested. The Pareto principle applies. We did not want to keep hitting LinkedIn with the same scraping request, so we implemented the use_cache parameter to limit hits to one profile scrape per month, at most. We never allowed users to have live enrichment at will because that would expose our scraping methodology or resources.
Of course, internally, we had a secret parameter that forced a live LinkedIn scrape, which we used for debugging our data extraction pipeline since LinkedIn changes their website all the time!
Now that we do not scrape LinkedIn, we offer live enrichment as an option at all times with no additional cost! Now you know :)
Part 2 Updates Tomorrow
I will not flood you with too many changes in one update; I will share Part 2 tomorrow. Oh, and there is also a good reason why I implemented the use_cache parameter: latency. We needed good data-retrieval latency for a hot feature that we will be announcing soon.
I will keep you posted!
As usual, if you have any questions, shoot me an email at [email protected]! I reply to all emails personally!