Merge pull request #71 from satzbeleg/feature/docker-redux

Simplifies Docker-Compose-based (development) setup

Dockerfile

@@ -0,0 +1,15 @@
+FROM postgres:13
+
+COPY dbauth/setup/000-install.sql          /docker-entrypoint-initdb.d/0000-install.sql
+COPY dbauth/setup/011-auth-email.sql       /docker-entrypoint-initdb.d/0011-auth-email.sql
+COPY dbauth/setup/012-auth-verify.sql      /docker-entrypoint-initdb.d/0012-auth-verify.sql
+COPY dbauth/setup/013-auth-linkedoauth.sql /docker-entrypoint-initdb.d/0013-auth-linkedoauth.sql
+
+COPY dbappl/setup/000-install.sql                        /docker-entrypoint-initdb.d/1000-install.sql
+COPY dbappl/setup/020-evidence-user_settings.sql         /docker-entrypoint-initdb.d/1020-evidence-user_settings.sql
+COPY dbappl/setup/021-evidence-example_items.sql         /docker-entrypoint-initdb.d/1021-evidence-example_items.sql
+COPY dbappl/setup/023-evidence-evaluated.sql             /docker-entrypoint-initdb.d/1023-evidence-evaluated.sql
+COPY dbappl/setup/024-evidence-interactivity-deleted.sql /docker-entrypoint-initdb.d/1024-evidence-interactivity-deleted.sql
+COPY dbappl/setup/030-zdlstore-sentence_buffer.sql       /docker-entrypoint-initdb.d/1030-zdlstore-sentence_buffer.sql
+COPY dbappl/setup/031-zdlstore-feature_vectors.sql       /docker-entrypoint-initdb.d/1031-zdlstore-feature_vectors.sql
+

README.md

@@ -12,85 +12,25 @@ The database containers are only reachable within the Docker network.
 Please follow the instruction of the [deployment repository](https://github.com/satzbeleg/evidence-deploy).
 
 
-## Configurated Containers
-The repository contains the docker configuration of two database systems and a UI-based database administration tool.
+## Configuration
+The repository contains configurations of two databases.
 
-- `dbauth.yml`: 
+- `dbauth/`: 
     - A Postgres database for the authentication process (e.g. email for account recovery, passwords, Google ID for OAuth).
     - `dbauth` does **not** store *application data* such as user settings! 
     - External applications only receive the `user_id`, which are random UUID4 strings, and thus represent *pseudonyms* according to Art. 4 No. 5 DSGVO (i.e. the `user_id` can be stored permanently in external applications because these are pseudonymized by design).
-- `dbappl.yml`: 
-    - A Citus/Postgres database to store application data.
+- `dbappl/`: 
+    - A Postgres database to store application data.
     - The application database does **not** store permanently *direkte personenbezogenen Daten* (DSGVO). Otherwise further functionalities have to be implemented (e.g. deletion requests of direct personal data).
     - The application database can store **temporarily** *direkte personenbezogenen Daten* (DSGVO) for sole purpose of a non-commercial research project, e.g. user surveys with the informed consent that provided data can be published under CC-* license.
     - The application stores `user_id` permanently as *Pseudonym* (Art. 4 Nr. 5 DSGVO).
 
-
-## Local Development
-1. [Build Docker Containers](#build-docker-containers)
-2. [Import Demo Data](#import-demo-data)
-3. [Backup and Recovery](#backup-and-recovery)
-
-### Build Docker Containers
-The `* .yml` files contain the container configurations of the services mentioned above.
-The file `defaults.env.sh` contains exemplary environment variables that have to be adjusted depending on the deployment scenario.
-
-```bash
-# delete persistent storage locations
-rm -rf tmp
-
-# load environment variables
-set -a
-source defaults.env.sh
-# source specific.env.sh
-
-# start containers
-# - WARNING: Don't use the `docker compose` because it cannot process `ipv4_address`!
-docker-compose -p evidence \
-    -f network.yml -f dbauth.yml -f dbappl.yml up --build
-
-# add workers to citus db
-docker-compose -p evidence -f network.yml -f dbappl.yml scale worker=3
-```
-
-
-### Import Demo Data
-In `dbauth.yml` and` dbappl.yml` SQL tables, functions and triggers are installed if they do not exist. Existing data (persistent data) will not be overwritten or accidentally deleted.
-
-A few toy data can be imported into the empty database for demonstration purposes:
-
-```sh
-cat dbauth/demo/test-user-for-app-demo.sql | docker exec -i evidence-dbauth psql --username=postgres
-cat dbappl/demo/toy-data-for-app-demo.sql | docker exec -i evidence-dbappl_master psql --username=postgres
-```
-
-
-### Backup and Recovery
-The *backup* should be carried out in the database container, i.e. `pg_dump` is executed in the container and the data is forwarded to the host.
-The reason is that the program `pg_dump` on the host might not have to have the same major version as the Postgres database in the container.
-
-```sh
-suffix=$(date +"%Y-%m-%dT%H-%M")
-container_name=evidence-database_master
-docker exec -i ${container_name} \
-    pg_dump postgres --username=postgres \
-    | gzip -9 > "postgres-${suffix}.sql.gz"
-```
-
-For *recovery*, the archive is forwarded from the host to the database container,
-and used as input in the container of `psql`.
-
-```sh
-gunzip -c "postgres-${suffix}.sql.gz" | docker exec -i ${container_name} \
-    psql --username=postgres -d postgres 
-```
-
-
 ## Appendix
 
 ### Support
 Please [open an issue](https://github.com/satzbeleg/evidence-database/issues/new) for support.
 
 ### Contributing
-Please contribute using [Github Flow](https://guides.github.com/introduction/flow/). Create a branch, add commits, and [open a pull request](https://github.com/satzbeleg/evidence-database/compare/). You are asked to sign the CLA on your first pull request.
+Please contribute using [Github Flow](https://guides.github.com/introduction/flow/). Create a branch, add commits, and [open a pull request](https://github.com/satzbeleg/evidence-database/compare/).
+You are asked to sign the CLA on your first pull request.
 

dbappl/export/export.sh

@@ -1,17 +1,17 @@
 #!/usr/bin/env bash
 
 # BWS Evaluations
-psql --host=127.0.0.1 --port=55015 --username=postgres \
-    -d postgres -t -A -F"," -c \
+docker-compose exec db psql -U evidence \
+    -d evidence -t -A -F"," -c \
     "SELECT ROW_TO_JSON(tb) FROM (SELECT * FROM evidence.evaluated_bestworst) tb;" \
     | gzip -9 > "evaluated_bestworst-$(date +%Y-%m-%d).json.gz"
 
 # Raw Sentences
-# All sents: SELECT ROW_TO_JSON(tb) FROM (SELECT * FROM evidence.sentences_cache) tb;
-psql --host=127.0.0.1 --port=55015 --username=postgres \
-    -d postgres -t -A -F"," -c \
+# All sents: SELECT ROW_TO_JSON(tb) FROM (SELECT * FROM zdlstore.sentences_cache) tb;
+docker-compose exec db psql -U evidence \
+    -d evidence -t -A -F"," -c \
     "SELECT ROW_TO_JSON(tb) FROM (
-        SELECT * FROM evidence.sentences_cache
+        SELECT * FROM zdlstore.sentences_cache
         WHERE sentence_id IN (
             SELECT (state_sentid_map->>'0')::uuid as "sent_id" FROM evidence.evaluated_bestworst 
             UNION SELECT (state_sentid_map->>'1')::uuid FROM evidence.evaluated_bestworst
@@ -21,8 +21,8 @@ psql --host=127.0.0.1 --port=55015 --username=postgres \
     | gzip -9 > "sentences_cache-$(date +%Y-%m-%d).json.gz"
 
 # Old Scores
-psql --host=127.0.0.1 --port=55015 --username=postgres \
-    -d postgres -t -A -F"," -c \
+docker-compose exec db psql -U evidence \
+    -d evidence -t -A -F"," -c \
     "SELECT ROW_TO_JSON(tb) FROM (
         SELECT sentence_id, lemma, context, score FROM evidence.example_items 
         WHERE score IS NOT NULL

dbappl/setup/000-install.sql

@@ -6,12 +6,12 @@
 -- 
 -- Load the extensions for the database
 -- 
-CREATE EXTENSION IF NOT EXISTS citus;
+-- CREATE EXTENSION IF NOT EXISTS citus;
 -- CREATE EXTENSION IF NOT EXISTS citext;
 CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
--- CREATE EXTENSION btree_gist;
-CREATE EXTENSION pg_trgm;
-CREATE EXTENSION pgcrypto;
+-- CREATE EXTENSION IF NOT EXISTS btree_gist;
+CREATE EXTENSION IF NOT EXISTS pg_trgm;
+CREATE EXTENSION IF NOT EXISTS pgcrypto;
 
 -- 
 -- Create schemes

dbauth/setup/000-install.sql

@@ -9,8 +9,8 @@
 -- CREATE EXTENSION IF NOT EXISTS citext;
 CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
 -- CREATE EXTENSION btree_gist;
-CREATE EXTENSION pg_trgm;
-CREATE EXTENSION pgcrypto;
+CREATE EXTENSION IF NOT EXISTS pg_trgm;
+CREATE EXTENSION IF NOT EXISTS pgcrypto;
 
 -- 
 -- Create schemes