Stream File Reader reads a local file in a JSON structure, that currently supports CSV only, and triggers subpipelines to process each message. It should be used for heavy files.

Take a look at the configuration parameters of the component:

  • File Name: name of the local file.

  • Charset: name of the characters code for the default file reading (default UTF-8).

  • Element Identifier: attribute to be sent in case of errors.

  • Paralell Execution Of Each Iteration: occurs in parallel with the loop execution.

  • Ignore Invalid Charset: if the option is activated, the invalid charset configured in the component will be ignored along with the received file.

  • Fail On Error: if the option is enabled, the execution of the pipeline with error will be interrupted; otherwise, the pipeline execution proceeds, but the result will show a false value for the “success” property.

  • Advanced: definition of advanced parameters.

  • Skip: number of lines to be skipped before the file reading.

  • Limit: maximum number of lines to be read.

Messages flow

Input

The component waits for a message in the following format:

{
"filename": "fileName"
}

Local File Name overrides the default local file.

Output

{
"total": 0,
"success": 0,
"failed": 0
}

  • total: total number of processed lines

  • success: total number of lines successfully processed

  • failed: total number of line whose processing failed

IMPORTANT: to know if a line has been correctly processed, there must be the return { "success": true } for every processed line.

The component throws an exception if the File Name doesn't exist or can't be read.

The files manipulation inside a pipeline occurs in a protected way. All the files can be accessed with a temporary directory only, where each pipeline key gives access to its own files set.

This component makes batch processing. To better understand the concept, click here.

Did this answer your question?