Database Catalogue

Accessing multiple data backends from one Extreme Data workflow, scripted using the AQIS engine, requires a complicated management of connections and credentials. To make this easy, the AQIS Engine - in particular in the Dask part

From AQIS Engine - Airflow worklows, the infisical APIs can be addressed in the same manner, taking simple inspiration from the AQIS Engine - Dask classes.

Below, we give a short walkthrough on how to install and configure Infisical for AQIS purposes.

Installing Infisical for a Self-Hosting Setup with Docker

This quick setup helps in setting up an instance of Infisical using the official docker image. The steps are inspired by the official documentation. For further details and different setup options, follow the official guide.

Prerequisites

  • A host with Docker and Docker Compose installed. Verify with docker --version.
  • Running Postgres and Redis (containers provided in the compose file, can be replaced with self hosted instances of both).

Setup

1. Download the required files

Docker compose file

curl -o docker-compose.prod.yml \
  https://raw.githubusercontent.com/Infisical/infisical/main/docker-compose.prod.yml

Example env file

curl -o .env https://raw.githubusercontent.com/Infisical/infisical/main/.env.example

Place both files in the same directory.

2. Configure required environment variables

Edit .env and replace example/test values with more secure values. At minimum set:

  • ENCRYPTION_KEY — random 16-byte hex (e.g. openssl rand -hex 16).
  • AUTH_SECRET — random 32-byte base64 (e.g. openssl rand -base64 32).
  • DB_CONNECTION_URI — Postgres connection string (db user must have required privileges).
  • REDIS_URL — Redis connection string.
  • SITE_URL, HOST, PORT — adjust for your deployment.

Other non-essential variables can be found in the documentation.

4. Start Infisical

From the directory containing docker-compose.prod.yml and .env:

docker compose -f docker-compose.prod.yml up -d

The default example serves the UI on port configured in the .env file.

Recommendations

  • The official Docker Compose template is intended for single-node / POC usage. For production environments, follow the official Infisical guides.

Usage with AQIS

Infisical serves as AQIS Database Catalogue module together with the correct configuration. I.e., it holds the list of database backends to be addressed together with the necessary connection details and secrets.

Configuring Database-Backend Details and Tokens in Infisical

1. First Steps in Setting up Infisical

Open the running instance in your browser (e.g. http://<HOST>:<PORT>). The follwoing can be done with the Infisical UI:

  1. Create an organization.
  2. Create a project (workspace) and at least one environment.
  3. Add one secret to the project so it contains data your application can read.

2. Creating a Service Token

Once a project is created, in the Infisical UI: Access Control → Service Tokens → Create token. Configure:

  • Name
  • Scope (which project environment(s) and path patterns the token may access).
  • Permissions (read, write as needed).
  • Expiration (recommended to limit lifetime).

On creation the token string will be shown once, store it securely — this will used to authenticate the API secrets retrieval request.

[!Note] Usage: when calling the API, supply the access token portion as a Bearer token (e.g. Authorization: Bearer <token> according to your Infisical instance's token format).

Keep tokens secret and rotate them in regular intervals. Consider shorter-lived tokens, least privilege scopes, and regular rotation for improved security

Configuring the AQIS Engine - Dask for Usage of Infisical

The correct configuration of the AQIS Engine - Dask for leveraging the Database Catalogue module is described in the Engine documentation and involves: