What is "Partial Scroll"?
In most cases, you need to scroll the entire page using the default scrollbar on the right side. For these pages, the default scrolling method works well.
However, there are pages, like reviews or posts pages, that are designed differently.
Take the Google Maps reviews page as an example: https://www.google.com/maps/place/The+Schoolhouse/@51.4374112,-1.0878661,9.7z/data=!4m10!1m2!2m1!1srestaurants+in+london!3m6!1s0x4876058fd98fc091:0xbf1c07755166b551!8m2!3d51.4604646!4d-0.1757991!9m1!1b1
The reviews are displayed on the left part of the web page, and this part has a scroll bar. When you drag this bar down, the review part will scroll down and load more reviews. However, you will notice that other parts of the page will not be scrolled.
To get the review part scrolled, you will need to set up a Partial Scroll step in Octoparse.
How to set up Partial Scroll in Octoparse?
There are two ways to set it up:
1. Add a scroll action for Go to Web Page/ Click Item
(scroll to finish loading first and then extract data)
Take Go to Web Page as an example.
Click Go to Web Page
Go to the Options tab
Check Scroll down the page after it is loaded
Select Partial from the Scroll Area
Click the button to the right of the XPath input box to select the scroll area. This will auto-generate the XPath and tell this task where to scroll
Click Appy
2. Set up scroll for Loop Item (scroll and extract at the same time)
Add a Loop Item step to the workflow
Click on Loop Item and switch Loop Mode to Scroll Page
Select Partial in the scroll area
Click the button to the right of the XPath input box to select the scroll area.
Click Appy
What if the auto-generated XPath is not working?
When this happens, you can try to write a new XPath that locates the selected scroll area.
Here are the steps you can follow:
Right-click the scroll bar to inspect the source code of this web page
Find the code that matches the scroll area
Copy the code and write the corresponding XPath
In this case, the XPath should be
//div[@class="m6QErb DxyBCb kA9KIf dS8AEf XiKgde "]
Update the XPath in Octoparse
Note: If the XPath you write is not working, try creating a few more variations to match the scroll area and ensure that the selected area is scrollable.
Check out this helpful tutorial to learn more details about XPath: What is XPath and how to use it in Octoparse