IMPORTANT: This documentation has been discontinued. Read the updated Google Storage documentation on our new documentation portal.
Google Storage allows a connection with the Google Storage service to be established and enables the following operations with files: List, Download, Upload and Delete.
Take a look at the configuration parameters of the component:
Account: for the component to make the service authentication, it’s necessary to have a PRIVATE KEY type account. To know more about credentials, click here.
Operation: operation to be executed, which can be list, download, upload or delete.
Project ID: ID of the project where the operation with the file will be made.
Bucket Name: this resource represents a bucket in the Google Cloud Storage - there's only one namespace shared by all the buckets.
Page Size: size of the page, which means, the amount of items to be returned when de List operation is used. If the value isn’t specified, all the items will be returned. If there’re more items than the amount determined in this parameter, it’s possible to request a second page (check Page Token), which returns the rest of the items.
Page Token: token used to request the next page when the List operation is used. In this next page the amount of items defined in the Page Size parameter is returned. next cursor token for the request.
File Name: name of the local file to go through download or upload (the delete function doesn't apply).
Remote File Name: name of the remote file or relative path (eg.: tmp/file.txt) for the remote file. This parameter supports Double Braces and is displayed in the Download, Upload and Delete operations.
Remote Directory: base remote directory, which can be relative (eg.: pub/tmp) or absolute (eg.: /root/pub), where the selected operation will be made. This parameter supports Double Braces and is displayed in the List, Download, Upload and Delete operations.
Generate Download Link: if the option is enabled, a public link for the file download is generated. This parameter is applicable in the Upload operation only.
Link Expiration: time for the link expiration (in milliseconds). This parameter is valid only if the Generate Download Link option is enabled.
Fail On Error: if the option is enabled, the execution of the pipeline with error will be interrupted; otherwise, the pipeline execution proceeds, but the result will show a false value for the “success” property.
IMPORTANT: some of the parameters above support Double Braces. To understand how this language works, read our article by clicking here.
Messages flow
Input
The component waits for a message in the following structure:
{
"fileName": "file",
"remoteFileName": "remoteFileName,
"remoteDirectory": "remoteDirectory
}
Local File Name overrides the default local file and Remote Path substitutes the default remote path.
Output
No specific input message is expected, but the filling of the mandatory parameters that vary according to the selected operation. See below the mandatory parameters for each operation:
List: Account, Project ID, Bucket Name
Download: Account, Project ID, Bucket Name, File Name e Remote File Name
Upload: Account, Project ID, Bucket Name, File Name e Remote File Name
Delete: Account, Project ID, Bucket Name e Remote File Name
For the Update operation, there must be a file in the pipeline local directory.
Output
When executing the component with the List operation, the following JSON structure will be generated:
{
"content": [
{
"name": "temp//tmp/processed/file-3.txt",
"contentType": "application/octet-stream",
"contentEncoding": null,
"createTime": 1581689153448,
"bucket": "test-bucket",
"size": 10
},
{ ... }
],
"pageToken": "XXXXXXXXXXXXL3RtcC9wcm9jZXNzZWQvZmlsZS0zLnR4dA==",
"fileName": null,
"remoteFileName": null,
"remoteDirectory": "",
"success": true
}
content: an array of objects for each file found
name: name of the remote file
contentType: MIME type of the file
contentEncoding: encoding of the file, if it exists
createTime: timestamp of the file creation date
bucket: bucket where the file is found
size: size of the file in bytes
pageToken: token to recover the next page
fileName: name of the local file
remoteFileName: name of the remote file
remoteDirectory: name of the base remote folder
success: "true" if the last operation has been successful
error: is displayed if there’s been an error and Fail on Error is "false"
When executing the component with the Download, Upload and Delete operations, the following JSON structure is generated:
{
"fileName": "file.txt",
"remoteFileName": "tmp/iso-8859-1-test.txt",
"remoteDirectory": "",
"success": true
}
fileName: name of the local file
remoteFileName: name of the remote file
remoteDirectory: name of the base remote folder
success: "true" if the last operation has been successful
IMPORTANT: the files manipulation inside a pipeline occurs in a protected way. All the files can be accessed with a temporary directory only, where each pipeline key gives access to its own files set.
Google Storage in Action
LIST of files
Input
Parameters
Account: google-storage-test
Operation: List
Project ID: digibee-test
Bucket Name: digibee-test-digibee-test-bucket
Remote Directory: DGB-413
Page Size: 2
Output
{
"content": [
{
"name": "DGB-413/",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552394033410,
"bucket": "digibee-test-digibee-test-bucket",
"size": 11
},
{
"name": "DGB-413/iso8859-2.txt",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552395963553,
"bucket": "digibee-test-digibee-test-bucket",
"size": 55
}
],
"pageToken": "ChVER0ItNDEzL2lzbzg4NTktMi50eHQ=",
"fileName": null,
"remoteFileName": null,
"remoteDirectory": "DGB-413",
"success": true
}
As it can be seen, the result above returns the pageToken property with a value of reference for the next page. This property will be returned when the Page Size parameter is configured (in the example it’s defined with the value 2) and also when there’re more files to be listed.
LIST of multiple files using pagination
Input
Parameters
Account: google-storage-test
Operation: List
Project ID: digibee-test
Bucket Name: digibee-test-digibee-test-bucket
Remote Directory: DGB-413
Page Size: 2
Page Token: ChVER0ItNDEzL2lzbzg4NTktMi50eHQ=
Output
{
"content": [
{
"name": "DGB-413/utf-16-test.txt",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552394973030,
"bucket": "digibee-test-digibee-test-bucket",
"size": 70
},
{
"name": "DGB-413/utf-test.txt",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552394644597,
"bucket": "digibee-test-digibee-test-bucket",
"size": 55
}
],
"fileName": null,
"remoteFileName": null,
"remoteDirectory": "DGB-413",
"success": true
}
In the result above, the pageToken property hasn’t been returned. It shows there’re no more items to be listed.
DOWNLOAD of a file
Input
Parameters
Account: google-storage-test
Operation: Download
Project ID: digibee-test
Bucket Name: digibee-test-digibee-test-bucket
File Name: iso8859-2.txt
Remote File Name: iso8859-2.txt
Remote Directory: DGB-413
Output
{
"fileName": "iso8859-2.txt",
"remoteFileName": "iso8859-2.txt",
"remoteDirectory": "DGB-413",
"success": true
}
The file download will be made in the pipeline local directory.
UPLOAD of a file
Input
Local file: file.txt
Parameters
Account: google-storage-test
Operation: Upload
Project ID: digibee-test
Bucket Name: digibee-test-digibee-test-bucket
File Name: file.txt
Remote File Name: file.txt
Remote Directory: DGB-413
Output
{
"fileName": "file.txt",
"remoteFileName": "file.txt",
"remoteDirectory": "DGB-413",
"success": true
}
UPLOAD of a file generating link for download
Input
Local file: file.txt
Parameters
Account: google-storage-test
Operation: Upload
Project ID: digibee-test
Bucket Name: digibee-test-digibee-test-bucket
File Name: file.txt
Remote File Name: file.txt
Remote Directory: DGB-413
Generate A Download Link: true
Link Expiration (in ms): 600000
Output
{
"fileName": "file.txt",
"remoteFileName": "file.txt",
"remoteDirectory": "DGB-413",
"success": true,
"urlGenerated": "<URL TO DOWNLOAD THE FILE>"
}
With the configuration of the input parameters above, the file will be available for download for 10 minutes (600000ms) through the link generated in the urlGenerated output property.
DELETE of a file
Input
Parameters
Account: google-storage-test
Operation: Delete
Project ID: digibee-test
Bucket Name: digibee-test-digibee-test-bucket
Remote File Name: file.txt
Remote Directory: DGB-413
Output
{
"fileName": null,
"remoteFileName": "file.txt",
"remoteDirectory": "DGB-413",
"success": true
}