fix(api): ensure that Prometheus metrics comprises all uwsgi processes

Previously, when Prometheus scraped /metrics, this would be directed
to a single worker and return only that worker's metrics, causing
vast fluctuations in metrics counts.

Details: https://github.com/korfuri/django-prometheus/blob/master/documentation/exports.md

There are two ways to solve this: Expose metrics on a separate port
per worker (preferred), and using temporary file storage (tmpfs) for
the processes to write their numbers, from which prometheus_client
can sum them up. I could not get the former solution to work due to
https://github.com/korfuri/django-prometheus/issues/215, so I went
for the latter.

The tmpfs contains 4 files per process whose nominal size is 1 MB,
so we need at least 256 MB.

api/api/settings.py

@@ -214,6 +214,18 @@ CAPTCHA_VALIDITY_PERIOD = timedelta(hours=24)
 WATCHDOG_SLAVES = os.environ.get('DESECSTACK_WATCHDOG_SLAVES', '').split()
 WATCHDOG_WINDOW_SEC = 600
 
+# Prometheus (see https://github.com/korfuri/django-prometheus/blob/master/documentation/exports.md)
+#  TODO Switch to PROMETHEUS_METRICS_EXPORT_PORT_RANGE instead of this workaround, which currently necessary to due
+#  https://github.com/korfuri/django-prometheus/issues/215
+try:
+    import uwsgi
+except ImportError:
+    pass  # not running in uwsgi, e.g. management command
+else:
+    import prometheus_client
+    prometheus_client.values.ValueClass = prometheus_client.values.MultiProcessValue(
+        _pidFunc=uwsgi.worker_id)
+
 if DEBUG and not EMAIL_HOST:
     EMAIL_BACKEND = 'django.core.mail.backends.dummy.EmailBackend'
 

api/uwsgi.ini

@@ -11,3 +11,4 @@ gid = nogroup
 #master = true
 http = [::]:8080
 http-to = 127.0.0.1:3031
+env = prometheus_multiproc_dir=/var/local/django_metrics

docker-compose.yml

@@ -116,6 +116,8 @@ services:
     - nsmaster
     - celery
     - memcached
+    tmpfs:
+    - /var/local/django_metrics:size=500m
     environment:
     - DESECSTACK_DOMAIN
     - DESECSTACK_NS