diff --git a/infra/copycat-db/README.md b/infra/copycat-db/README.md index 5bb556a36..4310c73d8 100644 --- a/infra/copycat-db/README.md +++ b/infra/copycat-db/README.md @@ -1,8 +1,8 @@ # Copycat DB -Copycat DB is a [service](../service.md) to take a backup of our database. It -uses the Scaleway CLI to take backups of the database, and uploads them to an -offsite bucket. +Copycat DB is a [service](../services/README.md) to take a backup of our +database. It uses the Scaleway CLI to take backups of the database, and uploads +them to an offsite bucket. This bucket has an object lock configured, so backups cannot be deleted before expiry. Conversely, the service also deletes backups older than some threshold @@ -14,7 +14,6 @@ In production the service runs as a cron job, scheduled using a systemd timer. > meant as a second layer of replication. For more details, see our [Reliability > and Replication Specification](https://ente.io/reliability). - ## Quick help View service status (it gets invoked as a timer automatically, doesn't need to diff --git a/infra/services/README.md b/infra/services/README.md new file mode 100644 index 000000000..81a82c7b7 --- /dev/null +++ b/infra/services/README.md @@ -0,0 +1,103 @@ +# Services + +"Services" are various Docker images that we run on our instances and manage +using systemd. + +All our services (including museum itself) follow the same +pattern: + +* They're meant to run on vanilla Ubuntu instances. The only expectation they + have is for Docker to be installed. + +* They log to fixed, known, locations - `/root/var/log/foo.log` - so that these + logs can get ingested by Promtail. + +* Each service should consist of a Docker image (or a Docker compose file), and a + systemd unit file. + +* To start / stop / cron the service, we use the corresponding systemd command. + +* Each time the service runs it should pull the latest Docker image, so there is no +separate installation/upgrade step needed. We can just restart the service, and it'll +use the latest code. + +* Any credentials and/or configuration should be read by mounting the + appropriate file from `/root/service-name` into the running Docker container. + +## Systemd cheatsheet + +```sh +sudo systemctl status my-service +sudo systemctl start my-service +sudo systemctl stop my-service +sudo systemctl restart my-service +``` + +## Adding a service + +Create a systemd unit file (For examples, see the various `*.service` files in +this repository). + +If we want the service to start on boot, add an `[Install]` section to its +service file (_note_: there is one more step later): + +``` +[Install] +WantedBy=multi-user.target +``` + +Copy the service file to the instance where we want to run the service. +Services might also have some additional configuration or env files, also copy +those to the instance. + +```sh +scp services/example.service example.env : +``` + +SSH into the instance. + +```sh +ssh +``` + +Move the service `/etc/systemd/service`, and any config files to their expected +place. env and other config files that contain credentials are kept in `/root`. + +```sh +sudo mv example.service /etc/systemd/system +sudo mv example.env /root +``` + +If you want to start the service on boot (as spoken of in the `[Install]` +section above), then enable it (this only needs to be done once): + +```sh +sudo systemctl enable service +``` + +Restarts systemd so that it gets to know of the service. + +```sh +sudo systemctl daemon-reload +``` + +Now you can manage the service using standard systemd commands. + +```sh +sudo systemctl start example +``` + +To view stdout/err, use: + +```sh +sudo journalctl --follow --unit example +``` + +## Logging + +Services should log to files in `/var/logs` within the container. This should be +mounted to `/root/var/logs` on the instance (using the `-v` flag in the service +file which launches the Docker container or the Docker compose cluster). +Finally, ensure there is an entry for this log file in the +`promtail/promtail.yaml` on that instance. The logs will then get scraped by +Promtail and sent over to Grafana.