The EXA4MIND / Extreme Data Database Platform

The EXA4MIND project consortium welcomes you to the documentation of the EXA4MIND / Extreme Data Database Platform.

The platform shall enable you to:

• leverage the strongest High-Performance Computing, Cloud Computing and standalone systems for Extreme Data Mining and Analytics,
• use our “Advanced Query and Indexing System” to enhance your data analytics and AI performance and capbabilities, using high-performance data backends, and
• integrate data systems with European and FAIR data management and sharing approaches.

In short, the platform offers you the tools to enable your extreme data analytics use case. The models and submodules of the archtitecture can be flexibly deployed, so that you can concentrate on your actual needs.

Platform Architecture

_{^{Third-party logos remain owned and licensed as original.
Airflow and Dask (with logos and licenses) are available under github.com/apache/airflow (BSD-3-clause license) and github.com/dask/dask (Apache 2.0 license), respectively.}}

The platform (diagram above) offers modules to

• instantiate databases and object stores (Data System Instantiation Recipes)
• build efficient data-processing pipleines querying across various data backends (Advanced Query and Indexing System - AQIS)
• handle (pre-)processing, validation and analytics/AI tasks on your data (Toolboxes)
• deploy your data-analytics machinery across European supercomputing centres (Compute Module - LEXIS 2 Platform)
• make your data available in European data ecosystems (Dataset Connectivity and FAIR Support)

The Data Systems, AQIS, and Dataset Connectivity & FAIR Support modules together are dubbed Extreme Data Database (EDD).

Development Status and Licensing

The EXA4MIND platform is currently under significant development. Please consider this, including security aspects, when using our modules. We are happy to receive feedback from you.

The standard licensing policy of EXA4MIND uses the Apache 2.0 License (with LLVM Exceptions). This documentation is issued under CC-BY4.0 license (cf. repository) unless noted otherwise.

Further Reading

Repositories referenced from the documentation contain README.md with further instructions and LICENSE files with further information.