This repository contains the AI on Demand (AIoD) REST API. It is built with FastAPI that interacts with a database (MySQL) and OpenML's REST API. Both the database and the REST API are run from docker in separate containers.
The AIoD REST API will allow any kind of service to interact with the AIoD portal to discover, retrieve, and share AI resources. It forms the connection between user-facing components, such as the AIoD website or Python Client API, and the backend. The metadata for datasets, models and other resources can be accessed, added, updated and deleted through this API.
All metadata is stored in the AIoD metadata database. For every instance of the API (there will be multiple running instances, for reliability and performance), an instance of this database will run on the same machine (on the same node). The type of database is not yet determined, for now we use a simple MySQL database.
The metadata is stored in AIoD format. When the user requests an item, such as a dataset, it can be returned in AIoD format, or converted to any supported format, as requested by the user. For datasets, we will for instance support schema.org and DCAT-AP.
Requesting a dataset will therefore be simply:
To fill the database, a synchronization process must be running continuously for every platform (e.g. HuggingFace or OpenML). This synchronization service of a platform will be deployed at a single node. The synchronization service queries its platform for updates, converts the metadata to the AIoD format and updates the database.
Note that this synchronization process between the platform and the database, is different from the synchronization between database instances. The latter is under discussion in the AIoD Synchronization Meetings.
The models are found in src/database/model
. The AIoD Metadata team is responsible for
determining the fields of the metadata, whereafter the classes are implemented in this metadata
catalogue. To check the existing fields, the easiest way it to start this application (see
"Using Docker Compose") and check the automatically generated swagger documentation
(http://localhost:8000/docs).
We use inheritance to make sure that generic fields, such as name and description, are present and consistent over all resources. A partial overview of the metadata model can be found in the following figure:
- Linux/MacOS/Windows (should all work)
- Docker
- Docker Compose version 2.21.0 or higher
For development:
Python3.11
withpython3.11-dev
(sudo apt install python3.11-dev
on Debian)- Additional 'mysqlclient' dependencies. Please have a look at their installation instructions.
For production environments elasticsearch recommends -Xss4G and -Xmx8G for the JVM settings.
This parameters can be defined in the .env file.
See the elasticsearch guide.
For Keycloak, the --http-enabled=true
and --hostname-strict-https=false
should be omitted
from the docker-compose file.
This repository contains two systems; the database and the REST API. As a database we use a containerized MySQL server (through Docker), the REST API can be run locally or containerized. Information on how to install Docker is found in their documentation.
docker compose --profile examples up -d
starts the MYSQL Server, the REST API, Keycloak for Identity and access management and Nginx for reverse proxying.
Once started, you should be able to visit the REST API server at: http://localhost and Keycloak at http://localhost/aiod-auth
To authenticate to the REST API swagger interface the predefined user is: user, and password: password
To authenticate as admin to Keycloak the predefined user is: admin and password: password
To use a different DNS hostname replace localhost with it in .env and src/config.toml
This configuration is intended for development, DO NOT use it in production.
To turn if off again, use
docker compose --profile examples down
To connect to the database use ./scripts/database-connect.sql
.
mysql> SHOW DATABASES;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.03 sec)
Now, you can visit the server from your browser at localhost:8000/docs
.
You can specify different connectors using
docker compose --profile examples --profile huggingface-datasets --profile openml --profile zenodo-datasets up -d
docker compose --profile examples --profile huggingface-datasets --profile openml --profile zenodo-datasets down
Make sure you use the same profile for up
and down
, otherwise some containers might keep
running.
If you want to run the server locally, you need Python 3.11. We advise creating a virtual environment first and install the dependencies there:
python3.11 -m venv venv
source venv/bin/activate
python -m pip install .
For development, you will need to install the optional dependencies as well:
source venv/bin/activate
python -m pip install ".[dev]"
Moreover, you are encouraged to install the pre-commit hooks, so that black, mypy and the unittests run before every commit:
pre-commit install
You can run
pre-commit run --all-files
To run pre-commit manually.
After installing the dependencies you can start the server. You have 3 options:
- Run from your machine:
cd src
python main.py --reload
The --reload
argument will automatically restart the app if changes are made to the source files.
2. Run using docker. For instance using scripts/run_apiserver.sh
3. Run using DevContainer (see next subsection)
Currently, the code is by default running using the local Keycloak. To make
this work, you need to set an environment variable. You can do this by setting the
KEYCLOAK_CLIENT_SECRET
in src/.env
.
# src/.env
KEYCLOAK_CLIENT_SECRET=[SECRET]
Alternatively, you can connect to a different keycloak instance by modifying src/.env
. EGI
Checkin can for instance be used on a deployed instance - not on local host. Marco Rorro is the
go-to person to request the usage of the EGI Checkin.
The reason that EGI Checkin doesn't work on localhost, is that the redirection url of EGI Checkin is strict - as it should be. On our development keycloak, any redirection url is accepted, so that it works on local host or wherever you deploy. This should never be the case for a production instance.
See authentication README for more information.
By default, the app will connect to the database and populate it with a few items if there is no data present. You can change this behavior through parameters of the script:
-
rebuild-db: "no", "only-if-empty", "always". Default is "only-if-empty".
- no: connect to the database but don't make any modifications on startup.
- only-if-empty: if the database does not exist, create it. Then, if the tables do not exist, create them.
Then, if the tables are empty, populate according to
populate
. - always: drop the configured database and rebuild its structure from scratch.
Effectively a
DROP DATABASE
followed by aCREATE DATABASE
and the creation of the tables. The database is then repopulated according topopulate
. Important: data in the database is not restored. All data will be lost. Do not use this option if you are not sure if it is what you need.
-
populate-datasets: one or multiple of "example", "huggingface", "zenodo" or "openml". Default is nothing. Specifies what data to add the database, only used if
rebuild-db
is "only-if-empty" or "always".- nothing: don't add any data.
- example: registers two datasets and two publications.
- openml: registers datasets of OpenML, this may take a while, depending on the limit (~30 minutes).
-
populate-publications: similar to populate-datasets. Only "example" is currently implemented.
-
limit: limit the number of initial resources with which the database is populated. This limit is per resource and per platform.
Following the installation instructions above, the server may be reached at 127.0.0.1:8000
.
REST API documentation is automatically built and can be viewed at 127.0.0.1:8000/docs
.
If you want to automatically restart the server when a change is made to a file in the project, use the --reload
parameter.
It is important to realize that this also re-initializes the connection to the database, and possibly will do any
start-up work (e.g., populating the database).
The Python classes that define the database tables are found in src/database/model/. The structure is based on the metadata schema.
See src/README.md.
We provide several scripts to facilitate the scheduling of backups and the manual restoration of files. For details on these scripts and others, please see scripts/README.md.
Breaking changes of a resource include deleting a field, changing the name of an existing field, or changing the datatype of a field. Adding new fields is not a breaking change.
On a breaking change for a resource (e.g. for Dataset), a new router with a new version should be created. The existing router should be deprecated, and rewritten so that it can handle the new metadata of the database. This deprecation of a router will be visible in the Swagger documentation. Calls to a deprecated router will still work, but a response header "Deprecated" will be added with the deprecation date. The deprecated router will then be deleted on the next release.
On non-breaking changes of a resource, a new version is not needed for the corresponding router.
Example:
- Start www.aiod.eu/api/datasets/v0
- Release 1: www.aiod.eu/api/datasets/v0 (no breaking changes)
- Release 2:
- www.aiod.eu/api/datasets/v0 (deprecated)
- www.aiod.eu/api/datasets/v1
- Release 3: www.aiod.eu/api/datasets/v1
The database should always be up-to-date with the latest version of the metadata. As database migration tool, Alembic is the default choice for SQLAlchemy. The setup of this db migration for AIOD remains a TODO for now.
As changelog we use the Github tags. For each release, a release branch should be created with a bumped version in the pyproject.toml, and merged with the master. The tag should contain a message detailing all the breaking and non-breaking changes. This message should adhere to the guiding principles as described in https://keepachangelog.com/.
- Show all tags: https://github.com/aiondemand/AIOD-rest-api/tags
- Show a specific tag: https://github.com/aiondemand/AIOD-rest-api/releases/tag/0.3.20220501
This information can also be extracted using the Github REST API.
To create a new release,
- Make sure all requested functionality is merged with the
develop
branch. - From develop:
git checkout -b release/[VERSION]
. Example of version:1.1.20231129
- Update the version in
pyproject.toml
. - Test all (most of) the functionality. Checkout the project in a new directory and remove all your local images, and make sure it works out-of-the box.
- Go to https://github.com/aiondemand/AIOD-rest-api/releases and draft a new release from the release branch. Look at all closed PRs and create a changelog
- Create a PR from release branch to master
- After that's merged, create a PR from master to develop
- Deploy on the server(s):
- Check which services currently work (before the update). It's a sanity check for if a service doesn't work later.
- Update the code on the server by checking out the release
- Merge configurations as necessary
- Make sure the latest database migrations are applied: see "Schema Migrations"
- Notify everyone (e.g., in the API channel in Slack).