IMPORTANT: This documentation has been discontinued. Read the updated Google Storage: Usage Scenarios documentation on our new documentation portal.
Scenario 1: LIST files
Let's say you have 1 or more files in Google Storage and that you want to invoke Google Storage in list mode. That way, you'll have access to the list of existing files and their respective folders.
See an example of how to make it possible:
Create a pipeline and add Google Storage.
Open the configurations of the component and choose an ACCOUNT (keep in mind this field is mandatory). If you don't have a configured account to access Google Storage, first you'll have to generate a primary key through the Google service. Click here to do that. Afterwards, configure an Account in the Digibee Platform.
Configure the other mandatory fields of the component (Project ID and Bucket Name) and the optional as well (Remote Directory, Page Size and Page Token).
Click on CONFIRM.
Connect the trigger to Google Storage.
Run a test in the pipeline (CTRL + ENTER).
You'll see a list of the available files in Google Storage according to your determined specifications:
{
"content": [
{
"name": "DGB-413/",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552394033410,
"bucket": "digibee-test-digibee-test-bucket",
"size": 11
},
{
"name": "DGB-413/iso-8859-1-test.txt",
"contentType": "application/octet-stream",
"contentEncoding": null,
"createTime": 1579552641265,
"bucket": "digibee-test-digibee-test-bucket",
"size": 65
}
],
"pageToken": "ChtER0ItNDEzL2lzby04ODU5LTEtdGVzdC50eHQ=",
"fileName": null,
"remoteFileName": null,
"remoteDirectory": "DGB-413",
"success": true
}
As you can see in the example above, the chosen option was PageSize 2, which means only 2 files will be listed. If there's no value specification in this field, then all the files will be listed. However, we recommend you to specify the value, because if you don't it can jeopardize the performance and data recovery time. Take a look at the next scenario and learn how to iterate multiple files without affecting the safety and performance.
Scenario 2: LIST multiple files using pagination
Let's say you have a great amount of files in Google Storage and that you want to invoke Google Storage in list mode. That way, you'll be able to list the files using pagination.
To do that, you must:
Create a pipeline and add Google Storage, choosing the "list" option in the Step Name field.
Open the configurations of the component and choose an ACCOUNT (keep in mind this field is mandatory). If you don't have a configured account to access Google Storage, first you'll have to generate a primary key through the Google service. Click here to do that. Afterwards, configure an Account in the Digibee Platform.
Configure the other mandatory fields of the component (Project ID and Bucket Name) and the optional as well (Remote Directory, Page Size and Page Token).
Click on CONFIRM.
Add one more Google Storage, defining Step Name as "List-2".
Configure the mandatory fields of the component one more time (Project ID, Bucket Name, Page Size and Page Token) and the optional as well (Remote Directory).
IMPORTANT: define the Page Token with Double Braces:
{{ message.pageToken }}
Double Braces give access to the previous component output. Given that, the Page Token generated by Google Storage with the "List" configuration allows the next files listing page to be fetched to Google Storage with the "List-2" configuration.
7. Connect the trigger and the components:
8. Run a test in the pipeline (CTRL + ENTER).
9. You'll see a list of the files available in Google Storage according to your determined specifications:
{
"content": [
{
"name": "DGB-413/iso8859-2.txt",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552395963553,
"bucket": "digibee-test-digibee-test-bucket",
"size": 55
},
{
"name": "DGB-413/utf-16-test.txt",
"contentType": "text/plain",
"contentEncoding": null,
"createTime": 1552394973030,
"bucket": "digibee-test-digibee-test-bucket",
"size": 70
}
],
"pageToken": "ChdER0ItNDEzL3V0Zi0xNi10ZXN0LnR4dA==",
"fileName": null,
"remoteFileName": null,
"remoteDirectory": "DGB-413",
"success": true
}
As you can see in the example above, the output shows only the files of the last page generated by Google Storage with the "List-2" configuration. However, it becomes clear how you can use the Page Token to access consecutive pages.
Scenario 3: DOWNLOAD file
Let's say you have a file in Google Storage and that you want to invoke Google Storage in download mode. That way, you'll have access to a specific file to be used by the pipeline.
Check how to do that:
Create a pipeline and add Google Storage.
Open the configurations of the component and choose an ACCOUNT (keep in mind this field is mandatory). If you don't have a configured account to access Google Storage, first you'll have to generate a primary key through the Google service. Click here to do that. Afterwards, configure an Account in the Digibee Platform.
Configure the other mandatory fields of the component (Project ID, Bucket Name and Remote File Name) and the optional as well (File Name and Remote Directory).
Click on CONFIRM.
Connect the trigger to Google Storage.
Run a test in the pipeline (CTRL + ENTER).
You'll see a confirmation of the file existence and availability in the pipeline, with the specified File Name:
{
"fileName": "iso-8859.txt",
"remoteFileName": "iso-8859-1-test.txt",
"remoteDirectory": "DGB-413",
"success": true
}
Scenario 4: UPLOAD file
Let's say you have a file in the pipeline and that you want to invoke Google Storage in upload mode. That way, the specified file becomes available in Google Storage.
Know how to do that:
Create a pipeline and add Google Storage.
Open the configurations of the component and choose an ACCOUNT (keep in mind this field is mandatory). If you don't have a configured account to access Google Storage, first you'll have to generate a primary key through the Google service. Click here to do that. Afterwards, configure an Account in the Digibee Platform.
Configure the other mandatory fields of the component (Project ID, Bucket Name and File Name) and the optional as well (Remote File Name and Remote Directory).
Click on CONFIRM.
Connect the trigger to Google Storage.
Run a test in the pipeline (CTRL + ENTER).
You'll see a confirmation of the file creation in Google Storage, with specified Remote File Name and Remote Directory:
{
"fileName": "iso-8859.txt",
"remoteFileName": "iso-8859-upload.txt",
"remoteDirectory": "DGB-413",
"success": true
}
Scenario 5: DELETE file
Let's say you have a file in Google Storage and that you want to invoke Google Storage in delete mode. That way, the file is removed from Google Storage.
To do that, just follow these steps:
Create a pipeline and add Google Storage.
Open the configurations of the component and choose an ACCOUNT (keep in mind this field is mandatory). If you don't have a configured account to access Google Storage, first you'll have to generate a primary key through the Google service. Click here to do that. Afterwards, configure an Account in the Digibee Platform.
Configure the other mandatory fields of the component (Project ID, Bucket Name and Remote File Name) and the optional as well (Remote Directory).
Click on CONFIRM.
Connect the trigger to Google Storage.
Run a test in the pipeline (CTRL + ENTER).
You'll see a confirmation of the file removal from Google Storage:
{
"fileName": null,
"remoteFileName": "iso-8859-upload.txt",
"remoteDirectory": "DGB-413",
"success": true
}