Sometimes you will encounter the problem that some websites cannot be opened in the Octoparse but work well in the browser. This may be related to the browser User Agent (UA), which is a string to tell the target website what kind of device you are accessing the page with. Octoparse offers some default UAs for you to select from, but sometimes they don't work. In this tutorial, we will show you how to add your browser's UA to Octoparse to open the target website.
You need to do 2 steps to add a custom UA you can find them both on the right and below.
1. Get a UA
Method One:
Take Chrome (Google) browser as an example. Enter "chrome://version/" in the address bar, and the following information will appear. You can copy the information after [User-Agent] in the yellow box and paste it into the Octoparse.
However, some browsers do not support the about:version method, such as Firefox (Firefox) and Edge browsers. Then the second method below can be used, and the second method is also applicable to the Chrome browser.
Method Two:
Open the website and press F12 on the keyboard
Press F12 on the keyboard to open the developer tool console or you can also right-click on the webpage and choose Inspect to access the console.
Firefox
Chrome
Refresh the web page
Click the Network option to open the following interface.
Firefox
Chrome
The Network tab shows various requests sent by the browser.
Click on one of the requests and then scroll the right part to find the User-Agent in the Headers
Headers means the message header. You can find User-Agent information in this part of the content. If any message headers do not contain UA, change the request and continue searching.
Firefox
Chrome
Copy the UA from the user-agent line
Firefox
Chrome
2. Put the UA in Octoparse
Open the task, click the setting button to enter the settings page
Click Run Settings>Add Custom User Agent
Name the User Agent you want
Paste the UA under the User Agent string and click Add to save