Batch URL input allows you to import a large number of URLs into Octoparse efficiently. This feature supports importing URLs from local files (text or spreadsheet), extracting them from another task, or generating them automatically based on a predefined pattern.
How to Batch Input URLs
Click +New in the sidebar and select Custom Task.
You will see the URL import panel, where you can choose one of 3 methods to add URLs (up to 1 million per task).
1. Import URLs from a File
You can upload URLs from CSV, TXT, or Excel (.xlsx & .xls) files
Steps:
Select Import from file.
Click Select and choose your file.
If using an Excel file, select the sheet and column containing the URLs.
Click Save to complete the import.
📝 Notes:
Only the first 100 URLs are displayed for preview.
For CSV files, ensure there is only one column with URLs - additional columns will cause errors.
2. Import URLs from Another Task
This feature connects two tasks, allowing you to use URLs scraped from a parent task in a child task without manual exports.
Steps:
Select Import from task.
Choose the Task Group and the task containing your target URLs.
Specify the field where the URLs are stored.
Click Save to complete the import.
📝 Notes:
The child task automatically updates when the parent task fetches new URLs.
Only URLs from Cloud runs in the parent task can be imported.
Supports importing more than 1 million URLs.
You can schedule the child task based on the parent task’s status.
3. Batch Generate URLs Using a Predefined Pattern
This method lets you generate multiple URLs by modifying parameters in a base URL.
Steps:
Select Batch generate.
Enter a base URL.
Highlight a parameter in the URL and click Add parameter.
Choose a Parameter Type and define its pattern.
Click Save URL to generate the list.
Parameter Types
3.1 Numbers
Set a starting number, increment/decrement value, and repeat count or end value.
Example: Generate page URLs from 1 to 100:
Start: 1, Increment: +1, Repeat: 100
3.2 Letters
Define a range (e.g., A to Z).
3.3 Time
Set a date format and range (useful for hotel booking or time-based data).
3.4 Custom List
Enter a custom list (e.g., keywords, product IDs).
💡 Tip:
You can combine multiple parameters for complex URL patterns.
Example:
Base URL:
www.example.com/[category]/[page]category = {A, B},page = {1, 2}Generated URLs:
www.example.com/A/1www.example.com/B/1www.example.com/A/2www.example.com/B/2