Processor

processor

topic

The Processor helps to create a pipeline of functions. The processor reads inputs from the input topic, sends inputs to the first function of the pipeline, grab the output of the first function and send them as input of the second function, and continues these steps to the last function in the pipeline, then write the outputs of the last function into the output topic. If, during the running of each function, an error occurs, the input message of the function will be written into the error topic.

Parameters

Parameter Name Type Required Description

version

String

Yes

The version of the processor application that is used. To check the available processor application versions, you can check the Docker Hub

sourceTopic

String

Yes

The input topic of the processor pipeline.

entityTopic

String

Yes

The output topic of the processor pipeline.

cleanUpMode

String

No

Determines the cleanup mode of the entityTopic. Valid values are: COMPACT, DELETE.

Notes:

  • The default value for cleanUpMode is COMPACT

  • For more information, please check these links: Log Compaction (related to the COMPACT config) , Log Retention (related to the DELETE config)

cleanUpTimeHours

String

No

Determine the cleanup time in hour of the entityTopic.

Note: The default value for cleanUpTimeHours is 336

errorTopic

String

Yes

The error topic of the processor pipeline. If, during the running of each function, an error occurs, the input message of the function will be written inside the errorTopic topic.

errRetentionHours

String

No

Determine the cleanup time in hour of the errorTopic.

  • The default value for errRetentionHours is 336

  • The cleanup mode for the errorTopic is DELETE, and it is not configurable.

pipeline

List

Yes

Determine the list of functions inside the pipeline.

For access to the list of functions and documentation, please check the Processor Functions

Example

multipleFns Configured1

In this example, the processor pipeline parameters are: