Define services

infra/copycat-db/README.md

@@ -1,8 +1,8 @@
 # Copycat DB
 
-Copycat DB is a [service](../service.md) to take a backup of our database. It
-uses the Scaleway CLI to take backups of the database, and uploads them to an
-offsite bucket.
+Copycat DB is a [service](../services/README.md) to take a backup of our
+database. It uses the Scaleway CLI to take backups of the database, and uploads
+them to an offsite bucket.
 
 This bucket has an object lock configured, so backups cannot be deleted before
 expiry. Conversely, the service also deletes backups older than some threshold
@@ -14,7 +14,6 @@ In production the service runs as a cron job, scheduled using a systemd timer.
 > meant as a second layer of replication. For more details, see our [Reliability
 > and Replication Specification](https://ente.io/reliability).
 
-
 ## Quick help
 
 View service status (it gets invoked as a timer automatically, doesn't need to

infra/services/README.md

@@ -0,0 +1,103 @@
+# Services
+
+"Services" are various Docker images that we run on our instances and manage
+using systemd.
+
+All our services (including museum itself) follow the same
+pattern:
+
+* They're meant to run on vanilla Ubuntu instances. The only expectation they
+  have is for Docker to be installed.
+
+* They log to fixed, known, locations - `/root/var/log/foo.log` - so that these
+  logs can get ingested by Promtail.
+
+* Each service should consist of a Docker image (or a Docker compose file), and a
+  systemd unit file.
+
+* To start / stop / cron the service, we use the corresponding systemd command.
+
+* Each time the service runs it should pull the latest Docker image, so there is no
+separate installation/upgrade step needed. We can just restart the service, and it'll
+use the latest code.
+
+* Any credentials and/or configuration should be read by mounting the
+  appropriate file from `/root/service-name` into the running Docker container.
+
+## Systemd cheatsheet
+
+```sh
+sudo systemctl status my-service
+sudo systemctl start my-service
+sudo systemctl stop my-service
+sudo systemctl restart my-service
+```
+
+## Adding a service
+
+Create a systemd unit file (For examples, see the various `*.service` files in
+this repository).
+
+If we want the service to start on boot, add an `[Install]` section to its
+service file (_note_: there is one more step later):
+
+```
+[Install]
+WantedBy=multi-user.target
+```
+
+Copy the service file to the instance where we want to run the service.
+Services might also have some additional configuration or env files, also copy
+those to the instance.
+
+```sh
+scp services/example.service example.env <instance>:
+```
+
+SSH into the instance.
+
+```sh
+ssh <instance>
+```
+
+Move the service `/etc/systemd/service`, and any config files to their expected
+place. env and other config files that contain credentials are kept in `/root`.
+
+```sh
+sudo mv example.service /etc/systemd/system
+sudo mv example.env /root
+```
+
+If you want to start the service on boot (as spoken of in the `[Install]`
+section above), then enable it (this only needs to be done once):
+
+```sh
+sudo systemctl enable service
+```
+
+Restarts systemd so that it gets to know of the service.
+
+```sh
+sudo systemctl daemon-reload
+```
+
+Now you can manage the service using standard systemd commands.
+
+```sh
+sudo systemctl start example
+```
+
+To view stdout/err, use:
+
+```sh
+sudo journalctl --follow --unit example
+```
+
+## Logging
+
+Services should log to files in `/var/logs` within the container. This should be
+mounted to `/root/var/logs` on the instance (using the `-v` flag in the service
+file which launches the Docker container or the Docker compose cluster).
+Finally, ensure there is an entry for this log file in the
+`promtail/promtail.yaml` on that instance. The logs will then get scraped by
+Promtail and sent over to Grafana.