How can I download a file using Pipes?

How can I download a file using Pipes?

This document is an introduction to some of the basics of using dexi.io’s Pipes feature. It will guide you, step-by-step, through the process of creating a Pipe and the necessary data structure assets and iterate through a list of URLs to download the files represented by those URLs.

A Data Type defines the type of data to be stored and is the atom of Pipes projects. If a suitable data type doesn’t exist for your project, create one now.

1. On the Projects page, click the green New button in the top-right corner of the page.

2. Click New Data Type.

3. If the data type should be treated as a key for results references and sorting, tick the Key? checkbox.

4. Provide an ID, or name, for the Data Type.

5. Choose a Type. In this case, select File.

6. If the Data Type should include a default value, provide it in the Default Value field.

7. Click Save.

A Data Set stores data for ready use in a Pipes project. If a suitable Data Set doesn’t exist for your project, create one now.

1. On the Projects page, click the green New button in the top-right corner of the page.

2. Click New Data Set.

3. Under the Settings tab, enter a name for the Data Set.

4. Choose a Data Type from the drop-down menu. If you just created a Data Type for this project, select it here.

5. Click Save.

With a suitable Data Type and Data Set available to your project, you can now create a new Pipe.

1. On the Projects page, click the green New button in the top-right corner of the page.

2. Click New Robot.

3. Select Pipes.

4. Enter a name for the robot.

5. Click Create New Robot.

6. From the node menu on the right side of the page, drag the For Each Row in Data Set node (under the dexi.io category) to the diagram.

7. In the Name field, enter a descriptive name for the node.

8. From the Data Set drop-down menu, select the Data Set to use with the robot. If you just created a new Data Set for this project, select it here.

9. From the Max Concurrency drop-down, select an appropriate number of simultaneous queries the Data Set should handle.

10. Click Update.

11. Hover over the right side of the For Each Row node to display the row flag.

12. Click the row flag and choose As Fields from the Transform menu.

13. In the Name field, enter a descriptive name for the new node.

14. Click Update.

15. Hover over the right side of the As Fields node to display the flag.

16. Click the flag and choose HTTP Request from the HTTP menu.

17. In the Name field, enter a descriptive name for the new node.

18. If you wish, in the URL field, enter a default fall-back URL.

19. Click Update.

20. At the bottom-right corner of the page, click the Save button, then click the Close button.

Next, create a run for your Pipe to prepare for your first execution.