Data handling in Flows and Bricks
A flow is processing data, Bricks store states while processing and persistent data structures are created and modified. This chapter describes the different types of data used in flows.
To reduce the risk of technical dept an keep the system maintainable in the future, dependencies should be avoided wherever possible.
Flow Packages FPs
For the titan project hierarchical data structure are used.
Port data modeling
Port data corresponds to FP data that can be processed by a brick. The Port description should be general enough to accept a wide range of FPs, but specific enough to avoid processing errors due to invalid data or data structures.
Brick Status
The Brick Status is data kept for a Brick while a flow instance is running. It is not shared with other flows or with other instances of the same flow. The combination of the status of all Bricks in the flow makes the Flow Status.
Persistent Data
Sometimes it is required to utilize persistent data to create a system status. Parameter are special kind of persistent data. Parameter sets are described for each brick and stored in different levels of visibility.
The question which additional persistent data is required and how it is modeled is not cleared yet.
Titan is an integration platform and not an enterprise storage system. Therefore it is not required to build complex data structures to store and access data. The best approach is to find a database that matches the requirements and integrate it with titan.
Different Levels of Parameter Data
Hierarchical data structures can be defined to describe parameters to control Brick processing. Each atomic Brick comes with it's own set of parameters. Atomic Brick parameters are the baseline of processing parameters, but it would be difficult to manage a system of flows only by it's atomic Brick config.
While defining a flow Brick parameters can be raised to the flow level to define a subset that controls flow processing in general. The same abstraction is possible if a composite Brick is created. The Flow level becomes the Brick level for the composite. If the composite is used in a Flow the parameters can again be raised to the flow level etc. The reduction of parameter visibility on different levels reduces the complexity while managing the system.
A system of many flows can still become complex enough to justify another level of abstraction. Flow parameters can be raised to the system level to make a set of system parameters.
Combining parameters
Bricks may use the same value for a parameter at flow level and/or system level. To simplify this situation a function is required to combine parameters to use the same value. The dependency created by doing so has to be obvious and transparent to the user.
Dynamic Parameter Sets
Parameters are not always static. Sometimes parameter sets have to be applied depending on the processed data. This special layer is currently considered, but not yet decided. It may not be part of the titan prototype.