All Collections
Octoparse 101
Lesson 3: Refine your data
Lesson 3: Refine your data
Updated over a week ago

In this lesson, we will walk you through some practical tips on how you can refine your dataset after having the data extracted if they do not look exactly like how you want it to be.


Rename/ move/ duplicate/ delete a field

As soon as you have the data extracted and shown in Data Preview, you can now look through the dataset and start sorting your data. A few typical things you can do to refine your dataset include renaming the fields, reordering the columns, duplicating data fields, and deleting the fields that are not required for your project.

To rename a field, double-click the field name, then type in the new name directly. Note that you should only use numbers, letters, and "_" for field names.

6666666666666666666666.gif

To move a field, place your cursor at the front of the field and when the hand sign shows up, drag and drop the field to the right spot.

1.gif

To delete a field, click on the .../ More icon and select Delete -

2.png

Clean data

Octoparse provides many different ways for you to clean your data. For example, you can replace a text string, trim extra spaces, add a prefix/suffix, replace a string with RegEx, reformat date/time, and more. You can clean any single data field in one or more ways until the data meets your requirements. Some of these may require you to deal with Regular Expression with which you can use the Octoparse RegEx tool for assistance.

In Data Preview, click the show more icon for the data field you'd like to refine and select Clean data -

2.png

Select Add Step, and then select what you would like to do with the data. You can keep working with the data by adding more steps until the data meets your requirements.

5987.png
  • Replace: replace the specific string(s) in the extracted data with the new string(s) that you want.

  • Replace with Regular Expression: use a specific regular expression to replace the matched string(s) in the extracted data with the string(s) that you want.

  • Match with Regular Expression: use a specific regular expression to pick up the matched string(s) from the extracted data.

  • Trim spaces: remove the unwanted space(s) from the start or/and the end of the data extracted.

  • Add a prefix: add a string/string to the front of the data extracted.

  • Add a suffix: add a string/string to the end of the data extracted.

  • Reformat extracted date/time: shift the extracted date/time into one of the 14 built-in formats, or into your own customized format.

  • Timestamp conversion: A timestamp is a string of coded message which is used to identify a recorded date and time. You can use timestamp conversion to convert a string to the correct time format.

  • HTML: convert some specific HTML tags into plain text automatically. For example, transcode "&gt" into ">" and "&nbsp" into a space.

Note: To learn more about reformatting data and the Regular Expression tool with Octoparse? Check here!


Capture HTML code

When auto-detection is used to capture any data from a web page, Octoparse automatically extracts the text and the URL of the elements. However, you can manually customize the data field and tell Octoparse to extract any HTML code instead.

In Data Preview, click the show more icon and select "Customize field", then choose how you'd like to capture the selected data.

9.png

Extract page-level data and date & time

Octoparse offers a number of pre-defined data fields that you can use to capture page-level data, current date & time, or any fixed value conveniently.

  • Current date & time: the date and time of when the data is extracted from the web page

  • Page-level data: page URL, page title, meta keyword, meta description, and HTML source code

  • Fixed value: any fixed value you define

Click on the + sign at the upper top right corner of Data Preview. Select any pre-defined data fields that you would like to add to the dataset.

14.png

Did this answer your question?