EXA4MIND's Advanced Query and Indexing System (AQIS)

The AQIS is the core EXA4MIND module enabling Extreme Data workflows. The graphics below gives an overview:

AQIS part of EXA4MIND Architecture

Third-party logos remain owned and licensed as original.
Airflow and Dask (with logos and licenses) are available under github.com/apache/airflow (BSD-3-clause license) and github.com/dask/dask (Apache 2.0 license), respectively.

The AQIS facilitates

  • with its AQIS engine (Airflow/Dask) submodule: the set-up of extreme-data transfer and processing workflows using Apache Airflow, Dask and Python, or a combination of these;
  • with its Workflow Catalogue submodule: an understanding of data-driven workflows and re-usage of parts with its groups;
  • with its Inference submodule: inference on supercomputers deplyoing the EXA4MIND inference service, and inference for natural-language to query-language translation in particular;
  • with its Data System Adaptors submodule groups: an easy and well-managed access to data backends (instantiated with our instantiation recipes or otherwise), where seamless connection management and authentication can be guaranteed via Infisical used as a Database Catalogue;
  • with its Cache Supprt submodule: the use of caching techniques - in particular to accelerate access to data from AQIS workflows running in a HPC environment.

Please have a look at the submodule (group) documentation pages for further information. The AQIS is rich in APIs, and in particular the AQIS Engine - Dask allows to program and expose general custom REST APIs (for e.g. triggering workflows in a deployed AQIS instance).