How can I download a file using Pipes?
This document is an introduction to some of the basics of using dexi.io’s Pipes feature. It will guide you, step-by-step, through the process of creating a Pipe and the necessary data structure assets and iterate through a list of URLs to download the files represented by those URLs.
A Data Type defines the type of data to be stored and is the atom of Pipes projects. If a suitable data type doesn’t exist for your project, create one now.
- On the Projects page, click the green New button in the top-right corner of the page.
- Click New Data Type.
- If the data type should be treated as a key for results references and sorting, tick the Key? checkbox.
- Provide an ID, or name, for the Data Type.
- Choose a Type. In this case, select File.
- If the Data Type should include a default value, provide it in the Default Value field.
- Click Save.
A Data Set stores data for ready use in a Pipes project. If a suitable Data Set doesn’t exist for your project, create one now.
- On the Projects page, click the green New button in the top-right corner of the page.
- Click New Data Set.
- Under the Settings tab, enter a name for the Data Set.
- Choose a Data Type from the drop-down menu. If you just created a Data Type for this project, select it here.
- Click Save.
With a suitable Data Type and Data Set available to your project, you can now create a new Pipe.
- On the Projects page, click the green New button in the top-right corner of the page.
- Click New Robot.
- Select Pipes.
- Enter a name for the robot.
- Click Create New Robot.
- From the node menu on the right side of the page, drag the For Each Row in Data Set node (under the dexi.io category) to the diagram.
- In the Name field, enter a descriptive name for the node.
- From the Data Set drop-down menu, select the Data Set to use with the robot. If you just created a new Data Set for this project, select it here.
- From the Max Concurrency drop-down, select an appropriate number of simultaneous queries the Data Set should handle.
- Click Update.
- Hover over the right side of the For Each Row node to display the row flag.
- Click the row flag and choose As Fields from the Transform menu.
- In the Name field, enter a descriptive name for the new node.
- Click Update.
- Hover over the right side of the As Fields node to display the flag.
- Click the flag and choose HTTP Request from the HTTP menu.
- In the Name field, enter a descriptive name for the new node.
- If you wish, in the URL field, enter a default fall-back URL.
- Click Update.
- At the bottom-right corner of the page, click the Save button, then click the Close button.
Next, create a run for your Pipe to prepare for your first execution.