BREAKING feat(dbapi): switch to postgres, fixes #430

Migration instructions:
https://github.com/desec-io/desec-stack/pull/432#issuecomment-680785746

README.md

@@ -6,7 +6,7 @@ This is a docker-compose application providing the basic stack for deSEC name se
 - `nslord`: Eventually authoritative DNS server (PowerDNS). DNSSEC keying material is generated here.
 - `nsmaster`: Stealth authoritative DNS server (PowerDNS). Receives fully signed AXFR zone transfers from `nslord`. No access to keys.
 - `api`: RESTful API to create deSEC users and domains, see [documentation](https://desec.readthedocs.io/).
-- `dbapi`, `dblord`, `dbmaster`: MariaDB database services for `api`, `nslord`, and `nsmaster`, respectively.
+- `dbapi`, `dblord`, `dbmaster`: Postgres database for `api`, MariaDB databases for `nslord` and `nsmaster`, respectively.
 - `www`: nginx instance serving static web site content and proxying to `api`
 - `celery`: A shadow instance of the `api` code for performing asynchronous tasks (email delivery).
 - `rabbitmq`: `celery`'s queue
@@ -52,7 +52,7 @@ Although most configuration is contained in this repository, some external depen
       - `DESECSTACK_API_EMAIL_PORT`: port for sending email
       - `DESECSTACK_API_SECRETKEY`: Django secret
       - `DESECSTACK_API_PSL_RESOLVER`: Resolver IP address to use for PSL lookups. If empty, the system's default resolver is used.
-      - `DESECSTACK_DBAPI_PASSWORD_desec`: mysql password for desecapi
+      - `DESECSTACK_DBAPI_PASSWORD_desec`: database password for desecapi
       - `DESECSTACK_MINIMUM_TTL_DEFAULT`: minimum TTL users can set for RRsets. The setting is per domain, and the default defined here is used on domain creation.
     - nslord-related
       - `DESECSTACK_DBLORD_PASSWORD_pdns`: mysql password for pdns on nslord
@@ -82,8 +82,8 @@ Production:
 
 Storage
 -------
-All important data is stored in the databases managed by the `db*` containers. They use Docker volumes which, by default, reside in `/var/lib/docker/volumes/desecstack_{dbapi,dblord,dbmaster}_mysql`.
-This is the location you will want to back up. (Be sure to follow standard MySQL backup practices, i.e. make sure things are consistent.)
+All important data is stored in the databases managed by the `db*` containers. They use Docker volumes which, by default, reside in `/var/lib/docker/volumes/desec-stack_{dbapi_postgres,dblord_mysql,dbmaster_mysql}`.
+This is the location you will want to back up. (Be sure to follow standard MySQL/Postgres backup practices, i.e. make sure things are consistent.)
 
 API Versions and Roadmap
 ------------------------
@@ -135,10 +135,10 @@ While there are certainly many ways to get started hacking desec-stack, here is
     For desec-stack, [docker](https://docs.docker.com/install/linux/docker-ce/ubuntu/) and [docker-compose](https://docs.docker.com/compose/install/) are required.
     Further tools that are required to start hacking are git and curl.
     Recommended, but not strictly required for desec-stack development is to use certbot along with Let's Encrypt and PyCharm.
-    jq, httpie, libmariadbclient-dev, python3-dev (>= 3.8) and python3-venv (>= 3.8) are useful if you want to follow this guide.
+    jq, httpie, libmariadbclient-dev, libpq-dev, python3-dev (>= 3.8) and python3-venv (>= 3.8) are useful if you want to follow this guide.
     The webapp requires nodejs. To install everything you need for this guide except docker and docker-compose, use
 
-       sudo apt install certbot curl git httpie jq libmariadbclient-dev nodejs npm python3-dev python3-venv libmemcached-dev
+       sudo apt install certbot curl git httpie jq libmariadbclient-dev libpq-dev nodejs npm python3-dev python3-venv libmemcached-dev
 
 1. **Get the code.** Clone this repository to your favorite location.
 

api/Dockerfile

@@ -1,6 +1,6 @@
 FROM python:3.8-alpine
 
-RUN apk add --no-cache bash dcron sqlite
+RUN apk add --no-cache bash dcron postgresql-client sqlite
 
 RUN mkdir /usr/src/app
 WORKDIR /usr/src/app
@@ -9,7 +9,7 @@ ENV PIP_DISABLE_PIP_VERSION_CHECK=1
 ENV PIP_NO_CACHE_DIR=1
 
 COPY requirements.txt /usr/src/app/
-RUN apk add --no-cache gcc freetype-dev libffi-dev musl-dev libmemcached-dev mariadb-connector-c-dev jpeg-dev zlib-dev \
+RUN apk add --no-cache gcc freetype-dev libffi-dev musl-dev libmemcached-dev postgresql-dev jpeg-dev zlib-dev \
     && pip install -r requirements.txt \
     && apk --no-cache del gcc
 RUN pip freeze

api/api/settings.py

@@ -62,21 +62,12 @@ WSGI_APPLICATION = 'desecapi.wsgi.application'
 
 DATABASES = {
     'default': {
-        'ENGINE': 'django_prometheus.db.backends.mysql',
+        'ENGINE': 'django_prometheus.db.backends.postgresql',
         'NAME': 'desec',
         'USER': 'desec',
         'PASSWORD': os.environ['DESECSTACK_DBAPI_PASSWORD_desec'],
         'HOST': 'dbapi',
-        'OPTIONS': {
-            'charset': 'utf8mb4',
-            'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
-        },
-        'TEST': {
-            'CHARSET': 'utf8mb4',
-            'COLLATION': 'utf8mb4_bin',
-        },
     },
-
 }
 
 CACHES = {

api/desecapi/serializers.py

@@ -492,14 +492,9 @@ class RRsetListSerializer(serializers.ListSerializer):
 
         # time of check (does it exist?) and time of action (create vs update) are different,
         # so for parallel requests, we can get integrity errors due to duplicate keys.
-        # This will be considered a 429-error, even though re-sending the request will be successful.
+        # We knew how to handle this with MySQL, but after switching for Postgres, we don't.
+        # Re-raise it so we get an email based on which we can learn and improve error handling.
         except OperationalError as e:
-            try:
-                if e.args[0] == 1213:
-                    # 1213 is mysql for deadlock, other OperationalErrors are treated elsewhere or not treated at all
-                    raise ConcurrencyException from e
-            except (AttributeError, KeyError):
-                pass
             raise e
         except (IntegrityError, models.RRset.DoesNotExist) as e:
             raise ConcurrencyException from e

api/requirements.txt

@@ -9,7 +9,7 @@ django-celery-email~=3.0.0
 django-prometheus~=2.0.0
 dnspython~=1.16.0
 httpretty~=0.9.0
-mysqlclient~=1.4.0
+psycopg2~=2.8.5
 prometheus-client~=0.8.0  # added to control django-prometheus' dependency version
 psl-dns~=1.0
 pylibmc~=1.6.1

api/wait

@@ -2,7 +2,7 @@
 set -e
 
 # wait for api database to come up
-host=dbapi; port=3306; n=120; i=0; while ! (echo > /dev/tcp/$host/$port) 2> /dev/null; do [[ $i -eq $n ]] && >&2 echo "$host:$port not up after $n seconds, exiting" && exit 1; echo "waiting for $host:$port to come up"; sleep 1; i=$((i+1)); done
+host=dbapi; port=5432; n=120; i=0; while ! (echo > /dev/tcp/$host/$port) 2> /dev/null; do [[ $i -eq $n ]] && >&2 echo "$host:$port not up after $n seconds, exiting" && exit 1; echo "waiting for $host:$port to come up"; sleep 1; i=$((i+1)); done
 
 # wait for pdns api to come up
 host=nslord; port=8081; n=120; i=0; while ! (echo > /dev/tcp/$host/$port) 2> /dev/null; do [[ $i -eq $n ]] && >&2 echo "$host:$port not up after $n seconds, exiting" && exit 1; echo "waiting for $host:$port to come up"; sleep 1; i=$((i+1)); done

dbapi/Dockerfile

@@ -1,13 +1,14 @@
-FROM mariadb:10.3
+FROM postgres:12-alpine
 
-# Use random throw-away root password. Our init scripts switch authentication to socket logins only
-ENV MYSQL_RANDOM_ROOT_PASSWORD=yes
+RUN apk add --no-cache pwgen
 
-# install tools used in init script
-RUN set -ex && apt-get update && apt-get -y install gettext-base && apt-get clean && rm -rf /var/lib/apt/lists/*
+ADD docker-entrypoint-initdb.d /docker-entrypoint-initdb.d
 
-COPY initdb.d/* /docker-entrypoint-initdb.d/
-RUN chown -R mysql:mysql /docker-entrypoint-initdb.d/
+USER postgres
 
 # mountable storage
-VOLUME /var/lib/mysql
+VOLUME /var/lib/postgresql/data
+
+COPY entrypoint-wrapper.sh /usr/local/bin/
+ENTRYPOINT ["entrypoint-wrapper.sh"]
+CMD ["postgres"]

dbapi/docker-entrypoint-initdb.d/init-user-db.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+set -e
+
+# Get the postgres user or set it to a default value
+if [ -n $POSTGRES_USER ]; then pg_user=$POSTGRES_USER; else pg_user="postgres"; fi
+# Get the postgres db or set it to a default value
+if [ -n $POSTGRES_DB ]; then pg_db=$POSTGRES_DB; else pg_db=$POSTGRES_USER; fi
+
+if [ -n "$POSTGRES_NON_ROOT_USER" ]; then
+psql -v ON_ERROR_STOP=1 --username "$pg_user" --dbname "$pg_db" <<-EOSQL
+    CREATE USER $POSTGRES_NON_ROOT_USER with encrypted password '$POSTGRES_NON_ROOT_USER_PASSWORD';
+    GRANT CREATE, CONNECT ON DATABASE $pg_db TO $POSTGRES_NON_ROOT_USER;
+    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, UPDATE, INSERT, DELETE, REFERENCES ON TABLES TO $POSTGRES_NON_ROOT_USER;
+EOSQL
+fi

dbapi/entrypoint-wrapper.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -Eeo pipefail
+
+# This password is set for the postgres user when initializing the database. It is not needed and thus not printed.
+export POSTGRES_PASSWORD=$(pwgen -1 -s 32)
+/usr/local/bin/docker-entrypoint.sh "$@"

dbapi/initdb.d/00-init.sh

@@ -1,6 +0,0 @@
-# https://stackoverflow.com/questions/59895/can-a-bash-script-tell-which-directory-it-is-stored-in
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-
-for file in $DIR/*.sql.var; do
-	envsubst < $file > $DIR/`basename $file .var`
-done

dbapi/initdb.d/00-init.sql

@@ -1,2 +0,0 @@
--- This file is required to exist and will be overriden by 00-init.sh.
--- If it is created only by 00-init.sh, the entrypoint script will miss it.

dbapi/initdb.d/00-init.sql.var

@@ -1,7 +0,0 @@
--- deSEC user and domain database
-CREATE DATABASE desec CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
-CREATE USER 'desec'@'${DESECSTACK_IPV4_REAR_PREFIX16}.5.%' IDENTIFIED BY '${DESECSTACK_DBAPI_PASSWORD_desec}';
-GRANT SELECT, INSERT, UPDATE, DELETE, REFERENCES, INDEX, CREATE, ALTER, DROP ON desec.* TO 'desec'@'${DESECSTACK_IPV4_REAR_PREFIX16}.5.%';
-
--- privileges for deSEC test database
-GRANT SELECT, INSERT, UPDATE, DELETE, REFERENCES, INDEX, CREATE, ALTER, DROP ON test_desec.* TO 'desec'@'${DESECSTACK_IPV4_REAR_PREFIX16}.5.%';

dbapi/initdb.d/99-finish.sql

@@ -1,4 +0,0 @@
--- Narrow down root logins
-DROP USER 'root'@'%';
-INSTALL PLUGIN unix_socket SONAME 'auth_socket';
-ALTER USER 'root'@'localhost' IDENTIFIED VIA unix_socket;

dbapi/pg_hba.conf

@@ -0,0 +1,16 @@
+# TYPE  DATABASE        USER            ADDRESS                 METHOD
+
+# "local" is for Unix domain socket connections only
+local   all             all                                     trust
+# IPv4 local connections:
+#host    all             all             127.0.0.1/32            scram-sha-256
+# IPv6 local connections:
+#host    all             all             ::1/128                 scram-sha-256
+# Allow replication connections from localhost, by a user with the
+# replication privilege.
+#local   replication     all                                     trust
+#host    replication     all             127.0.0.1/32            scram-sha-256
+#host    replication     all             ::1/128                 scram-sha-256
+
+host desec desec all scram-sha-256
+host all all all reject

docker-compose.yml

@@ -56,14 +56,21 @@ services:
     build: dbapi
     image: desec/dedyn-dbapi:latest
     init: true
-    user: mysql:mysql
+    user: postgres:postgres
+    shm_size: 256M
     volumes:
-    - dbapi_mysql:/var/lib/mysql
+    - dbapi_postgres:/var/lib/postgresql/data
+    - ./dbapi/pg_hba.conf:/usr/local/src/pg_hba.conf:ro
     environment:
     - DESECSTACK_IPV4_REAR_PREFIX16
-    - DESECSTACK_DBAPI_PASSWORD_desec
+    - POSTGRES_DB=desec
+    - POSTGRES_HOST_AUTH_METHOD=reject
+    - POSTGRES_INITDB_ARGS=--auth-host=scram-sha-256
+    - POSTGRES_NON_ROOT_USER=desec
+    - POSTGRES_NON_ROOT_USER_PASSWORD=${DESECSTACK_DBAPI_PASSWORD_desec}
     networks:
     - rearapi_dbapi
+    command: ["postgres", "-c", "hba_file=/usr/local/src/pg_hba.conf"]
     logging:
       driver: "syslog"
       options:
@@ -352,7 +359,7 @@ services:
     restart: unless-stopped
 
 volumes:
-  dbapi_mysql:
+  dbapi_postgres:
   dblord_mysql:
   dbmaster_mysql:
   openvpn-server_logs: