Handling Linkedin Public Profiles that exist but cannot be scraped

So you are trying to scrape Linkedin profiles in bulk. The chances are that you are crawling the public profiles with a proxy (in a very inefficient manner, as discussed in my other post on building your own Linkedin Profile Scraper). If so, then you will come across a weird situation for which a public profile for a user cannot be found, but the profile exists if you log into Linkedin.

This is because such Linkedin profiles do not have a public profile because their profile visibility is set to Private.

You can verify that this is profile is indeed not scrapeable by searching for that profile on Google. If even Google does not have you in their search index, that means that profile does not have a public profile page, and hence it is un-scrapeable.

How does our Linkedin Person Profile endpoint deal with such profiles?

This week, we made an update to improve our Linkedin Person Profile API endpoint with a precise 404 error status code when a profile lacks a public profile. It was previously returning a general 500 error status code.

You can read up more about status code handling with Proxycurl here.