Handling Professional Social Network Public Profiles that exist but cannot be scraped

So you are trying to scrape Professional Social Network profiles in bulk. The chances are that you are crawling the public profiles with a proxy (in a very inefficient manner, as discussed in my other post on building your own Professional Social Network Profile Scraper). If so, then you will come across a weird situation for which a public profile for a user cannot be found, but the profile exists if you log into Professional Social Network.

This is because such Professional Social Network profiles do not have a public profile because their profile visibility is set to Private.

You can verify that this is profile is indeed not scrapeable by searching for that profile on Google. If even Google does not have you in their search index, that means that profile does not have a public profile page, and hence it is un-scrapeable.

How does our Professional Social Network Person Profile endpoint deal with such profiles?

This week, we made an update to improve our Professional Social Network Person Profile API endpoint with a precise 404 error status code when a profile lacks a public profile. It was previously returning a general 500 error status code.

You can read up more about status code handling with Proxycurl here.