titan Flow Engine

The titan Flow Engine provides the backend services for running data flows on a node.

The core flow engine is composed of two services:

  • Control Peer: One peer per node that takes care of managing Brick Runners on that node.

  • Brick Runner: Virtual Machine for executing the Bricks that the Control Peer has started on it's node. The Brick Runner is generic and can process
    all types of Bricks. Each runner will only execute one type of Brick at a time.

Building

titan Flow Engine requires Python 3.7 to be installed.

Depending on whether you want to contribute to the titan Flow Engine source
code you want to choose one of the following options to install:

Installing via PyPi

The flow engine is available on the Python Package Index (PyPi). It is available for Linux and Windows x64 systems. To install it without having to build it yourself open a command line and run:

pip install titanfe

Building the titan Flow Engine (Advanced Install)

Required packages for building or developing the project can be installed via the requirements_dev.txt in the project's root folder:

pip install -r requirements_dev.txt

To build and install the Flow Engine open a command line and run:

python setup.py build
pip install .

For those who also want to work on the documentation and build it locally, please use the requirements_doc.txt file to install the necessary python packages.

pip install -r requirements_doc.txt

Running the flow engine

The flow engine is being started by starting the Control Peer. The Control
Peer takes care of starting Brick Runners as needed.

Parameters for starting the Control Peer on the command line are:

Name Purpose
brick_folder Folder to store installed bricks in (default: ~/titanfe/bricks)
config_file Path to ControlPeer configuration file

The ControlPeers configuration file contains the following parameters:

Name Type Description
Kafka string Network address of Kafka instance for metrics and logging
\<hostname:port> (default: localhost:9092)
GridManager string Network address of the GridManager
\<hostname:port> (default: localhost:8080)
FlowManager string Network address of the FlowManager
\<hostname:port> (default: localhost:9002)

To try out the flow engine first download and start the services GridManager, Repository service, User service, Package service, Log Engine, and FlowManager. These services are needed for the flow engine to receive flows to process.

Adjust the ControlPeers configuration file (titanfe/apps/control_peer/config.yaml) or create a new one. Then, open a command line and run the demo flow from
within that folder:

python -m titanfe.apps.control_peer -brick_folder ~/titanfe/bricks -config_file path_to_config_file/config.yaml

Or run it from the root directory of the flow engine by providing the path to
the example:

python -m titanfe.apps.control_peer  -brick_folder ~/titanfe/bricks -config_file titanfe/apps/control_peer/config.yaml