You are browsing a tutorial guide for Octoparse latest version. If you are running an older version of Octoparse, we strongly recommend you upgrade because it is faster, easier, and more robust! Download and upgrade here if you haven't already done so!
Duplicates in Octoparse mean the data lines that are the same in all the fields. You can delete the duplicates when exporting the data if there are only a few duplicates.
But getting only a small number of valid data lines with many duplicates can be frustrating. In this FAQ tutorial, we will help you resolve the duplicates.
Error 1: When scraping multiple pages, Octoparse jumps back to previously scraped pages or keeps scraping the last page.
The auto-generated pagination XPath may not always work well. If you find Octoparse duplicates on several pages repeatedly, you will need to make some adjustments.
Solution: Modify the XPath of the pagination to make sure it locates the next page button precisely.
Click on Pagination
Enter the new XPath and click Apply to save
Tip: Check how to write an XPath here at What is XPath and how to use it in Octoparse.
Error 2: When scraping multiple pages, the AJAX timeout for pagination is too short to load the next page, and Octoparse keeps scraping the current page data.
For pages loaded with AJAX, once the timeout is not set long enough, the next page cannot be loaded, and Octoparse would scrape the current page again, producing some duplicates.
Solution: Extend the AJAX timeout to make it long enough for the page to load.
Click on Click to Paginate
Select a longer AJAX timeout
Error 3: When scraping a list of items, Octoparse only scrapes the first row of data repeatedly, or one data field is the same in all lines.
When looping through a list of items to get data, Octoparse may keep scraping from one item. Or other fields are gathered correctly from each item, but one or two fields are fixed.
This is because the Extract Data action is not associated with the Loop Item action. Two options need to be selected to associate the Extract Data and Loop Item.
Extract data in the loop in the Extract Data settings
Relative XPath in the setting of the data field
The Extract Data and Loop Item are associated with the two options selected, and Octoparse will scrape the data from each item in the loop.
Solution 1: Re-create the fields
After checking the Extract data in the loop option, click Loop Item in the workflow, and then click Extract Data
The first item will be highlighted, and we can choose elements from the highlighted area to extract the text
Solution 2: Modify the XPath of the fields directly
Click on Extract Data
Click on More and select Customize XPath
Tick Relative Xpath and enter the correct XPath
Method 1: Click Extract data then Customize Xpath
Method 2: Click Extract data, switch to the vertical view, and double-click each field to customize Xpath, which is more convenient if you need to modify several Xpaths.
