A dropdown menu is a list of options that only appears when the user clicks on a piece of text or a button. It is often used by websites to offer a range of choices.
At times you may want to scrape the selected dropdown option as an individual data field, for example, when the drop-down options entail category information. In this article, we'll take a quick look at the different ways you can capture data on a drop-down list.
Tip: If you need guidance on how to switch between options in a drop-down menu and extract page data accordingly, check out this tutorial on how to click through options in a drop-down menu.
In Octoparse, we can either scrape the option values directly like any other data field on the webpage or scrape them from the loop item.
1. Scrape the option value directly like any other data field on the webpage
For this example, we are going to use a drown-drop list from carbase to show you how this is done.
Click on the dropdown menu box
Select Loop through the options in the dropdown menu to set up a Switch Dropdown loop
Click on the dropdown menu box again
Select Text
Untick Extract data in the loop and click Apply
You can iterate through the options in the loop list and click Extract data to see if you get the selected dropdown option in the data preview section.
2. Scrape the option value using a loop item
If in some cases the option value doesn't change even when they are selected, we'll need to scrape the option value using a loop item.
Click the dropdown menu box to set up a Switch Dropdown loop
Add an Extract Data action inside the loop
Tick Extract data in the loop and click Apply
Click on the dropdown menu and select the SELECT tag and then the first OPTION
Click Text
You can also switch between the different options from the loop to see if the data changes accordingly in the preview section.