Google Storage allows a connection with the Google Storage service to be established and enables the following operations with files: List, Download, Upload and Delete.

Take a look at the configuration parameters of the component:

  • Account: for the component to make the service authentication, it’s necessary to have a PRIVATE KEY type account. To know more about credentials, click here.

  • Operation: operation to be executed, which can be list, download, upload or delete.

  • Project ID: ID of the project where the operation with the file will be made.

  • Bucket Name: this resource represents a bucket in the Google Cloud Storage - there's only one namespace shared by all the buckets.

  • Page Size: size of the page, which means, the amount of items to be returned when de List operation is used. If the value isn’t specified, all the items will be returned. If there’re more items than the amount determined in this parameter, it’s possible to request a second page (check Page Token), which returns the rest of the items.

  • Page Token: token used to request the next page when the List operation is used. In this next page the amount of items defined in the Page Size parameter is returned. next cursor token for the request.

  • File Name: name of the local file to go through download or upload (the delete function doesn't apply).

  • Remote File Name: name of the remote file or relative path (eg.: tmp/file.txt) for the remote file. This parameter supports Double Braces and is displayed in the Download, Upload and Delete operations.

  • Remote Directory: base remote directory, which can be relative (eg.: pub/tmp) or absolute (eg.: /root/pub), where the selected operation will be made. This parameter supports Double Braces and is displayed in the List, Download, Upload and Delete operations.

  • Generate Download Link: if the option is enabled, a public link for the file download is generated. This parameter is applicable in the Upload operation only.

  • Link Expiration: time for the link expiration (in milliseconds). This parameter is valid only if the Generate Download Link option is enabled.

  • Fail On Error: if the option is enabled, the execution of the pipeline with error will be interrupted; otherwise, the pipeline execution proceeds, but the result will show a false value for the “success” property.

IMPORTANT: some of the parameters above support Double Braces. To understand how this language works, read our article by clicking here.

Messages flow

Input

The component waits for a message in the following structure:

{
"fileName": "file",
"remoteFileName": "remoteFileName,
"remoteDirectory": "remoteDirectory
}

Local File Name overrides the default local file and Remote Path substitutes the default remote path.


Output

No specific input message is expected, but the filling of the mandatory parameters that vary according to the selected operation. See below the mandatory parameters for each operation:

  • List: Account, Project ID, Bucket Name

  • Download: Account, Project ID, Bucket Name, File Name e Remote File Name

  • Upload: Account, Project ID, Bucket Name, File Name e Remote File Name

  • Delete: Account, Project ID, Bucket Name e Remote File Name

For the Update operation, there must be a file in the pipeline local directory.

Output

When executing the component with the List operation, the following JSON structure will be generated:

{
"content": [
{
"name": "temp//tmp/processed/file-3.txt",
"contentType": "application/octet-stream",
"contentEncoding": null,
"createTime": 1581689153448,
"bucket": "test-bucket",
"size": 10
},
{ ... }
],
"pageToken": "XXXXXXXXXXXXL3RtcC9wcm9jZXNzZWQvZmlsZS0zLnR4dA==",
"fileName": null,
"remoteFileName": null,
"remoteDirectory": "",
"success": true
}

  • content: an array of objects for each file found

  • name: name of the remote file

  • contentType: MIME type of the file

  • contentEncoding: encoding of the file, if it exists

  • createTime: timestamp of the file creation date

  • bucket: bucket where the file is found

  • size: size of the file in bytes

  • pageToken: token to recover the next page

  • fileName: name of the local file

  • remoteFileName: name of the remote file

  • remoteDirectory: name of the base remote folder

  • success: "true" if the last operation has been successful

  • error: is displayed if there’s been an error and Fail on Error is "false"

When executing the component with the Download, Upload and Delete operations, the following JSON structure is generated:

{
"fileName": "file.txt",
"remoteFileName": "tmp/iso-8859-1-test.txt",
"remoteDirectory": "",
"success": true
}

  • fileName: name of the local file

  • remoteFileName: name of the remote file

  • remoteDirectory: name of the base remote folder

  • success: "true" if the last operation has been successful

IMPORTANT: the files manipulation inside a pipeline occurs in a protected way. All the files can be accessed with a temporary directory only, where each pipeline key gives access to its own files set.

Google Storage in Action

LIST of files

  • Input

Parameters

Account: google-storage-test

Operation: List

Project ID: digibee-test

Bucket Name: digibee-test-digibee-test-bucket

Remote Directory: DGB-413

Page Size: 2

  • Output

{
"content": [
{
"name": "DGB-413/",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552394033410,
"bucket": "digibee-test-digibee-test-bucket",
"size": 11
},
{
"name": "DGB-413/iso8859-2.txt",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552395963553,
"bucket": "digibee-test-digibee-test-bucket",
"size": 55
}
],
"pageToken": "ChVER0ItNDEzL2lzbzg4NTktMi50eHQ=",
"fileName": null,
"remoteFileName": null,
"remoteDirectory": "DGB-413",
"success": true
}

As it can be seen, the result above returns the pageToken property with a value of reference for the next page. This property will be returned when the Page Size parameter is configured (in the example it’s defined with the value 2) and also when there’re more files to be listed.

LIST of multiple files using pagination

  • Input

Parameters

Account: google-storage-test

Operation: List

Project ID: digibee-test

Bucket Name: digibee-test-digibee-test-bucket

Remote Directory: DGB-413

Page Size: 2

Page Token: ChVER0ItNDEzL2lzbzg4NTktMi50eHQ=

  • Output

{
"content": [
{
"name": "DGB-413/utf-16-test.txt",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552394973030,
"bucket": "digibee-test-digibee-test-bucket",
"size": 70
},
{
"name": "DGB-413/utf-test.txt",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552394644597,
"bucket": "digibee-test-digibee-test-bucket",
"size": 55
}
],
"fileName": null,
"remoteFileName": null,
"remoteDirectory": "DGB-413",
"success": true
}

In the result above, the pageToken property hasn’t been returned. It shows there’re no more items to be listed.

DOWNLOAD of a file

  • Input

Parameters

Account: google-storage-test

Operation: Download

Project ID: digibee-test

Bucket Name: digibee-test-digibee-test-bucket

File Name: iso8859-2.txt

Remote File Name: iso8859-2.txt

Remote Directory: DGB-413

  • Output

{
"fileName": "iso8859-2.txt",
"remoteFileName": "iso8859-2.txt",
"remoteDirectory": "DGB-413",
"success": true
}

The file download will be made in the pipeline local directory.

UPLOAD of a file

  • Input

Local file: file.txt

Parameters

Account: google-storage-test

Operation: Upload

Project ID: digibee-test

Bucket Name: digibee-test-digibee-test-bucket

File Name: file.txt

Remote File Name: file.txt

Remote Directory: DGB-413

  • Output

{
"fileName": "file.txt",
"remoteFileName": "file.txt",
"remoteDirectory": "DGB-413",
"success": true
}

UPLOAD of a file generating link for download

  • Input

Local file: file.txt

Parameters

Account: google-storage-test

Operation: Upload

Project ID: digibee-test

Bucket Name: digibee-test-digibee-test-bucket

File Name: file.txt

Remote File Name: file.txt

Remote Directory: DGB-413

Generate A Download Link: true

Link Expiration (in ms): 600000

  • Output

{
"fileName": "file.txt",
"remoteFileName": "file.txt",
"remoteDirectory": "DGB-413",
"success": true,
"urlGenerated": "<URL TO DOWNLOAD THE FILE>"
}

With the configuration of the input parameters above, the file will be available for download for 10 minutes (600000ms) through the link generated in the urlGenerated output property.

DELETE of a file

  • Input

Parameters

Account: google-storage-test

Operation: Delete

Project ID: digibee-test

Bucket Name: digibee-test-digibee-test-bucket

Remote File Name: file.txt

Remote Directory: DGB-413

  • Output

{
"fileName": null,
"remoteFileName": "file.txt",
"remoteDirectory": "DGB-413",
"success": true
}

Did this answer your question?