All Collections
Case Tutorial
Social Media
Scrape the followers/following list of a public Twitter account
Scrape the followers/following list of a public Twitter account
Updated over a week ago

You are browsing a tutorial guide for the latest Octoparse version. If you are running an older version of Octoparse, we strongly recommend you upgrade because it is faster, easier, and more robust! Download and upgrade here if you haven't already done so!

With a reported, 211 million daily active users, Twitter has proven its worth in social media marketing. Users on Twitter post an average of 6000 tweets every second, making it over 500 million tweets posted daily. All of this chatter and noise is a treasure chest full of valuable information for marketers, brands, researchers, and analysts. Marketers and brands often scrape Twitter data from specific accounts (influencers and competitors) to analyze engagement and plan effective strategies.

Due to popular demand, this tutorial is the first in a series of tutorials that the Octoparse team has prepared for users with a need for Twitter data.

In this post, we are going to teach you how to scrape the followers/following list from any public account.

We will scrape the followers/following list for Nintendo of America. Check out the two sample URLs below:

Note: Although the workflows are extremely similar, you still need to create two separate tasks to scrape the two lists, using different XPaths.

The main steps are shown in the menu on the right.


1. Create a Go to Web Page - to open the target web page

Every workflow in Octoparse starts by telling Octoparse a web page to start with.

  • Enter the link of the followers/following list page into the search bar at the top of the home screen and click Start.


2. Log into Twitter in Browse mode - to save cookies for authentication

Twitter forbids direct access to followers/following lists unless you've logged in first.

  • Toggle on Browse mode and log into Twitter as you do in a normal browser

mceclip4.png
  • Click the Go to Web Page action to open its settings panel (located at the bottom right)

  • Go to the Options tab and tick Use cookies

  • Click Use cookie from the current page

  • Click Apply to save the settings

mceclip5.png
  • Turn off Browse Mode

mceclip3.png

We have now successfully saved the login information in the task workflow so that our Twitter account can be logged in when we run the task.


3. Create Extract Data - to scrape the basic info for the public account from the page heading

  • Click on the display name (e.g., Nintendo of America) and select Text on the Tips panel

  • Repeat the action to get the username

  • Click Add custom fields and select Page URL from Page-level data to get the profile URL

mceclip7.png

Tip: Twitter might change the XPath of the heading area. You might need to rewrite the XPath if the data preview section does not contain the correct information.

For the display name, the XPath is //h2[@dir="ltr" and @aria-level="2"]/span

For the username, the XPath is //h2[@dir="ltr" and @aria-level="2"]/following-sibling::div/span


4. Auto-detect the web page - to create a workflow

Twitter's infinite scroll pattern is designed to load content dynamically. Octoparse's auto-detection function can easily identify this kind of page and help you quickly create a workflow.

  • Click Auto-detect web page on the Tips and wait for the detection to complete

  • Check the data fields on the Data Preview and delete unwanted fields

  • Click Create workflow

You will see a workflow created like this:

mceclip0.png
  • Rename the fields by double-clicking them if needed


5. Add a new field to the Extract Data

The auto-detection does not scrape the bio information.

  • Click on Bio Information and select Text on the Tips panel


6. Modify the scroll settings - to scroll the page to load more followers/following

  • Click on the Scroll Page to open the settings

  • Set scroll pattern to for one screen and Repeats 600 times (or more)

  • Set wait time to 3s to fully load the list content (Important!)

  • Tick Capture data as page scrolls dynamically to minimize data loss

  • Click Apply to save the settings

scroll.jpg

7. Run the task - to get your desired data

  • Click Save on the upper right to save your task

  • Click Run next to it and wait for a Run Task window to pop up

  • Select Run on your device to run the task on your local device

  • Wait for the task to complete

Here is the sample output from a local run.

mceclip18.png

Tip: Local runs are great for task troubleshooting and quick runs. If you are dealing with more complicated tasks, it is recommended that you select Run in the Cloud to run the task in Octoparse's cloud-based platform for higher speed. Try out this premium feature by signing up for the 14-day free trial here. You can also schedule your task to run hourly, daily, or weekly and get data delivered to you regularly.

Did this answer your question?