Why is it hard to crawl Professional Social Network in scale in 2019? (Part 1)

This is a two-part series on crawling Professional Social Network in scale. In this first part, we study why Professional Social Network is a hard target to crawl. In the follow-up part two, I will dive deep into a technical tutorial on how you can crawl Professional Social Network in scale with demo code.

Everybody wants a piece of Professional Social Network, especially so since they have their data under a tight noose. Companies such as "hiQ Labs" have been sued for circumventing, but alas, the courts ruled that it is perfectly legal for companies to crawl their sites.

Before we move onto a technical guide on how you can crawl Professional Social Network, let's understand why it is hard to crawl Professional Social Network in scale:

1. You need to be logged into Professional Social Network to gain access to content

To view any profiles on Professional Social Network, you have to be logged in.

2. Professional Social Network requires Javascript to render content on the page

Professional Social Network pages load quickly, but the important data are fetched via API calls with Javascript after the page is loaded.

3. Professional Social Network blocks your IP when you crawl too much or too fast

IP address are rate limited if it is used too much.

4. Professional Social Network blocks your account when you crawl too much or too fast

Professional Social Network accounts are also rate limited if it is used too much.

Use Proxycurl to crawl Professional Social Network profiles in scale

To crawl Professional Social Network in scale, you need 3 conditions to be satisfied:

1. You will need many (residential) IP addresses
2. You will need many Professional Social Network accounts (logged in)
3. An advanced web crawler that behaves like a browser that allows you render Javascript

Proxycurl's private network of browser-based crawler nodes satisfies these requirements wholly because our proxy network employs tens of thousands of residential users around the world, using the latest bleeding edge browser to provide their computers to assist in our crawling efforts.

In part 2, I will provide a technical tutorial on how you can do this.