The AQIS Engine - Airflow Part

The AQIS includes submodules supporting efficient data access and processing (including inference), an AQIS Engine to automate Extreme Data transfer and processing workflows, and a catalogue of workflow examples.

The user can choose whether to automate workflows with an elaborate workflow engine (Apache Airflow / AQIS Engine - Airflow) or with a more lightweight approach (AQIS Engine - Dask, based on Python and Dask - which includes workflow/data-orchestration capabilities). The approaches can also be combined.

To set up Airflow where your AQIS components are deployed, please follow the instructions on the official website https://airflow.apache.org.

Airflow Workflow / Directed Acyclic Graph (DAG) examples

We showcase the usage of Airflow with the other AQIS submodules via examples provided in the Workflow Catalogue.

Interfacing EXA4MIND Platform Components with Airflow

EXA4MIND relies upon provisions for enhanced interfacing between Airflow and other platform components. Most importantly,

• we are providing guidance on existing Airflow Providers and additional iRODS and Milvus Airflow Providers to facilitate access to data system backends from Airflow (see Data System Adaptors), and
• the LEXIS 2 Platform (cf. EXA4MIND Compute Module) is providing an LEXIS Airflow Provider to address its compute and data orchestration interfaces for distributed computing.