When you're scraping a website that applies scroll-down/show more buttons to load more content, you may encounter this scenario:
Even if you've set up a scroll-way with Octoparse or set up to click on show more buttons, you may still find no new items added to the loop item list, or when the task runs, it still collects only a limited number of data rows.
This is because Octoparse generates the loop item in Fixed List. You may follow this tutorial to check and solve the problem. Take the picture below as an example:
You may notice that the current Loop mode is under the Fixed List mode.
Unlike the Variable List, Fixed List will only include items according to the fixed list of XPath you enter into the text box. Therefore, when dealing with dynamic items, or when the number of items varies on different pages, you need to switch the loop mode to the Variable list because it can automatically detect and match all the items corresponding to a certain XPath.
Tip: There are 6 loop modes in Octoparse. If you want to know more about these loop modes, please go to the following article: Loop Item
To solve this problem, you also need to modify the XPath. Check out this tutorial Customize element Xpath to learn more about XPath.
After changing the Loop Mode to Variable List, we can then input the correct Xapth into the Matching XPath box to locate all the target items on the page. Remember to click Apply to save the settings.
