Scraping Linkedin Profile Using Python & Selenium

note:
This article is no longer working, please visit https://gerrysabar.medium.com/updated-scraping-linkedin-profile-using-python-tested-april-2021-10804236c22b for the updated method to scrap linkedin profile.

In scraping job, when the layout is changing so do the code for scraping the data from the website. In this case I’d like to share how to scrap Linkedin profiles from my case to get many candidates for head hunter. Searching one by one & listing them is a very tedious process, therefore Python & Selenium come in handy to help us listing so many profiles.

The workflow for scraping the job is as follow:

Alright, now let’s create a directory for our working space:

$ mkdir linkedin

go to the newly created directory then create python virtual environment:

$ python3 -m venv venv

activate the virtual environment:

source venv/bin/activate

we need to install selenium & parsel in our Python virtual environment:

$ pip install selenium
$ pip install parsel

you also need to install chromedriver, it’ll be used to mimic as a real user browsing using Chrome web browser.

Installing Chromedriver for Ubuntu:

1. Install prerequisites:

$ sudo apt-get update
$ sudo apt-get install -y unzip xvfb libxi6 libgconf-2-4
$ sudo apt-get install default-jdk

2. Install Google Chrome

$ sudo curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add
sudo echo "deb [arch=amd64]  http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list
$ sudo apt-get -y update
$ sudo apt-get -y install google-chrome-stable

3. Install Chromedriver

$ wget https://chromedriver.storage.googleapis.com/2.41/chromedriver_linux64.zip
$ unzip chromedriver_linux64.zip

4. Let’s move chromedriver to another location to make it more organizable

$ sudo mv chromedriver /usr/bin/chromedriver
$ sudo chmod +x /usr/bin/chromedriver

Note: if you’re using Windows or Mac system you can googling how to install chromedriver and take a note where your cromedriver is located at (in this article it’s located in /user/bin/chromedriver) because we’ll need the path later in our Python app.

Now, we’re going to the interesting part, let’s create a python file named linkedin.py with code as follow (don’t forget to do small adjustment):

Right after you execute it, now can get about 10 linkedin profiles along with their detail in a csv file.

Conclusion

This approach isn’t not only for Linkedin case, in fact you can use it various cases such as finding the cheapest ticket price from various website for example. You can also create a bot to posting to certain website or any other case you want to solve. In this tutorial I made the case as simple as possible, in real world case it’s often you have to deal with more detailed Linkedin profiles for example or not only accessing first page from Google result. Hopefully, this article is useful for you as a stepping stone for further journey in scraping world. Cheers!