The FTP and SFTP connectors allow reading data from files and writing to files. The connector supports following modes:
- Raw mode: lines in the file are handled as strings (one string per line)
- CSV mode (comma separated values): lines in the file are delimited with e.g. a comma or semi colon, and are converted to an array (list) when reading from the file
Use "." for the Path to list the files in the root folder:
Example of listing the files in a folder under root:
Reading data from files
Example of reading data from a CSV file:
The data from the file will be a list (array), where each item in the list represents one line.
For Raw mode files, each line will be a string. Example:
"first line of file",
"second line of file"
For CSV files, the structure of each line will depend on the setting "Use first row as headers":
- If "Use first row as headers" is set to Yes: each line will be an object with the values from the CSV line, and the header line is used for keys. Example:
"first row header 1": "second row value 1",
"first row header 2": "second row value 2"
"first row header 1": "third row value 1",
"first row header 2": "third row value 2"
- If "Use first row as headers" is set to No: each line will be a list (array) with the values from the CSV line. Example:
"first row value 1",
"first row value 2"
"second row value 1",
"second row value 2"
Writing data to a file
Create a new file using the block "Create and open file". Next, use the block "Add line to file" or "Add lines to file", to add data to the newly opened file:
When adding one line to the file, the data should be a string for Raw mode, and it should be an object or list for CSV mode:
When adding multiple lines at once to a file, the data should be a list (array) where each item represents one line. The item should be a string for Raw mode and it should be an object or list for CSV mode:
Here's how the variable MyList was created for the above example:
Appending data to a file
Adding data to an existing file is currently not supported. Instead, create a Blend that performs the following steps:
- Open the existing file
- Create a new file
- Read all the data from the existing (old) file and add it to the new file
- Add the new data to the new file
- Delete the existing (old) file
Handling large amounts of data
Here's an example Blend that processes all the data from a CSV file at once, and writes it to a new file:
This will work for large amounts of data since the blocks "Read data from file", "Transform list" and "Write to file" all use an iterator. This means that the full list will not be read into memory at once. Instead the input list (from the CSV file) will be iterated using an iterator, and the output list will not be created until the data is required.
In the above example Blend, this means that the transformed list will be created as the data is written to the destination CSV file, using an iterator.
Here's another example that will also use an iterator, meaning it will work for (very) large files. A file is read from SFTP and created on Google Cloud Storage:
Note that the block "Write line to file" for Google Cloud Storage will use an internal buffer and write the data in larger chunks, so the above Blend will work efficiently for large files.