Stream DB V1

Know the component and how to use it.

Erick Rubiales avatar
Written by Erick Rubiales
Updated over a week ago

IMPORTANT: This documentation has been discontinued. Read the updated Stream DB V1 documentation on our new documentation portal.

Stream DB V1 makes operations through the connection with a database, streaming data to a subpipeline that processes them.

Take a look at the configuration parameters of the component:

  • Account: account to be used by the component.

  • Database URL: connection string to the database.

  • SQL Statement: SQL statement to be executed.

  • Column Name: name of the columns to be read.

  • Coalesce: when activated, this option controls if the data objects are coalesced into a string or if an error is generated.

  • Parallel Execution Of Each Iteration: occurs in parallel with the loop execution.

  • Fail On Error: when activated, this parameter suspends the pipeline execution only if there’s a severe occurrence in the iteration structure, disabling its complete conclusion. The “Fail On Error” parameter activation doesn’t have any connection with the errors occurred in the components used for the construction of the subpipelines (onProcess and onException).

  • Custom Connection Properties: specific connection properties defined by the user.

  • Keep Connections: if activated, the option will keep the connections to the database for 30 minutes maximum; otheriwse, it will be kept for 5 minutes only.

  • Advanced: advanced configurations.

  • Connection Test Query: SQL statement to be used before each connection is established - this is an optional parameter and must be applied to databases that don't have reliable information about the connection status.

Messages flow

Input

The component waits for a message in the following format:

{
"parameters": {
"name": "value"
...
}
}

Message "onProcess" structure

{
"column1":"data1", "column2":"data2", ...
}

Output with error

{
"code": error_code,
"error": error_message,
"processedId": the_id_column_value
}

Output

{
"total": 0,
"success": 0,
"failed": 0
}
  • total: total number of processed lines

  • success: total number of successfully processed lines

  • failed: total number of lines whose process failed

IMPORTANT: when the lines are correctly processed, their respective subpipelines return { "success": true } for each of them.

This component makes batch processing. To better understand the concept, click here.

Did this answer your question?