Programming and Operating Model

FnFlow is a cloud-native programming and operating model for composable stream processing functions.
With FnFlow, developers can create and orchestrate pipelines of processing functions especially in the use case of entity matching and linking.

The smallest building block of FnFlow is a function with Json input and Json output. Functions are implemented as Java Function. In case of an error during the function execution, the unchanged input will be send to an error channel. Functions can be composed together by connecting the output of one function to the input of another function. The output of a function can be spread to multiple functions input. Special functions like ChangeEventEmit can route its output to a different topic.
The Match function is forming a search query from its input and is executing this query on Opensearch. The output is a list of matching entities which can be further processed.

Composed functions are forming a pipeline. Pipelines are executed in a processor application. Processor applications are deployed and managed in Kubernetes.

Each output topic is organized as Event Store holding a log of change events. The projector application can be used to write these change events to an Opensearch index.