add some basic how-to style documents

README.md

@@ -81,7 +81,7 @@ A full explanation of all configuration methods can be found [here](./docs/full-
 
 Please make sure to [initialize the data provider](#data-provider-initialization) before running the daemon!
 
-To start the SFTP server with default settings, simply run:
+To start SFTPGo with the default settings, simply run:
 
 ```bash
 sftpgo serve
@@ -113,6 +113,10 @@ sftpgo initprovider --help
 
 After the initialization, the database structure will be automatically checked and updated, if required, at startup.
 
+## Tutorials
+
+Some step-to-step tutorials can be found inside the source tree [howto](./docs/howto "How-to") directory.
+
 ## Authentication options
 
 ### External Authentication

docs/howto/README.md

@@ -0,0 +1,6 @@
+# Tutorials
+
+Here we collect step-to-step tutorials. SFTPGo users are encouraged to contribute!
+
+- [SFTPGo with PostgreSQL data provider and S3 backend](./postgresql-s3.md)
+- [Expose Web Admin and REST API over HTTPS and password protected](./rest-api-https-auth.md)

docs/howto/postgresql-s3.md

@@ -0,0 +1,266 @@
+# SFTPGo with PostgreSQL data provider and S3 backend
+
+This tutorial shows the installation of SFTPGo on Ubuntu 20.04 (Focal Fossa) with PostgreSQL data provider and S3 backend. SFTPGo will run as an unprivileged (non-root) user. We assume that you want to serve a single S3 bucket and you want to assign different "virtual folders" of this bucket to different SFTPGo virtual users.
+
+## Preliminary Note
+
+Before proceeding further you need to have a basic minimal installation of Ubuntu 20.04.
+
+Create the `sftpgo` user with the following command.
+
+```shell
+sudo adduser sftpgo
+```
+
+Type the user password and other info.
+
+Add the `sftpgo` user to the `sudo` group so it will be able to use `sudo`:
+
+```shell
+sudo usermod -a -G sudo sftpgo
+```
+
+Now login using this user. Confirm that you are logged in as `sftpgo` user with the following command.
+
+```shell
+whoami
+```
+
+the output should be `sftpgo`.
+
+## Install PostgreSQL
+
+Before installing any packages on the Ubuntu system, update and upgrade all packages using the `apt` commands below.
+
+```shell
+sudo apt update
+sudo apt upgrade
+```
+
+Install PostgreSQL with this `apt` command.
+
+```shell
+sudo apt -y install postgresql
+```
+
+Once installation is completed, start the PostgreSQL service and add it to the system boot.
+
+```shell
+sudo systemctl start postgresql
+sudo systemctl enable postgresql
+```
+
+Next, check the PostgreSQL service using the following command.
+
+```shell
+systemctl status postgresql
+```
+
+## Create a new PostgreSQL user
+
+PostgreSQL uses roles for user authentication and authorization, it just like Unix-Style permissions. By default, PostgreSQL creates a new user called `postgres` for basic authentication.
+
+In this step, we will create a new PostgreSQL user for SFTPGo.
+
+Login to the PostgreSQL shell using the command below.
+
+```shell
+sudo -i -u postgres psql
+```
+
+Next, create a new role `sftpgo` with the password `sftpgo_pg_pwd` using the following query.
+
+```sql
+create user "sftpgo" with encrypted password 'sftpgo_pg_pwd';
+```
+
+Next, create a new database `sftpgo.db` for the SFTPGo service using the following queries.
+
+```sql
+create database "sftpgo.db";
+grant all privileges on database "sftpgo.db" to "sftpgo";
+```
+
+Exit from the PostgreSQL shell typing `\q`.
+
+## Configure AWS credentials
+
+We assume that you want to serve a single S3 bucket and you want to assign different "virtual folders" of this bucket to different SFTPGo virtual users. In this case is very convenient to configure a credential file so SFTPGo will automatically use it and you don't need to specify the same AWS credentials for each user.
+
+You can manually create the `~/.aws/credentials` file and write your AWS credentials like this.
+
+```shell
+[default]
+aws_access_key_id=AKIAIOSFODNN7EXAMPLE
+aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
+```
+
+Alternately you can install `AWS CLI` and manage the credential using this tool.
+
+```shell
+sudo apt install awscli
+```
+
+and now set your credentials, region, and output format with the following command.
+
+```shell
+aws configure
+```
+
+Confirm that you can list your bucket contents with the following command.
+
+```shell
+aws s3 ls s3://mybucket
+```
+
+## Install SFTPGo
+
+Download a binary SFTPGo [release](https://github.com/drakkan/sftpgo/releases) or a build artifact for the [latest commit](https://github.com/drakkan/sftpgo/actions).
+
+In this tutorial we assume you downloaded a build artifact named `sftpgo-ubuntu-latest-go1.15.zip` inside the current directory.
+
+Install `unzip`, if not already installed, and extract the archive with the following commands.
+
+```shell
+sudo apt install unzip
+mkdir sftpgo_installdir
+unzip sftpgo-ubuntu-latest-go1.15.zip -d sftpgo_installdir
+```
+
+Now change the current directory to `sftpgo_installdir` and install SFTPGo.
+
+```shell
+cd sftpgo_installdir
+
+# create the required directories
+sudo mkdir -p /etc/sftpgo/hostkeys \
+  /var/lib/sftpgo/credentials \
+  /usr/share/sftpgo
+
+# install the sftpgo executable
+sudo install -Dm755 sftpgo /usr/bin/sftpgo
+# install the default configuration file, edit it if required
+sudo install -Dm644 sftpgo.json /etc/sftpgo/
+# override some configuration keys using environment variables
+sudo sh -c 'echo "SFTPGO_HTTPD__TEMPLATES_PATH=/usr/share/sftpgo/templates" > /etc/sftpgo/sftpgo.env'
+sudo sh -c 'echo "SFTPGO_HTTPD__STATIC_FILES_PATH=/usr/share/sftpgo/static" >> /etc/sftpgo/sftpgo.env'
+sudo sh -c 'echo "SFTPGO_HTTPD__BACKUPS_PATH=/var/lib/sftpgo/backups" >> /etc/sftpgo/sftpgo.env'
+sudo sh -c 'echo "SFTPGO_DATA_PROVIDER__CREDENTIALS_PATH=/var/lib/sftpgo/credentials" >> /etc/sftpgo/sftpgo.env'
+sudo sh -c 'echo "SFTPGO_SFTPD__HOST_KEYS=/etc/sftpgo/hostkeys/id_rsa,/etc/sftpgo/hostkeys/id_ecdsa" >> /etc/sftpgo/sftpgo.env'
+# install static files and templates for the web UI
+sudo cp -r static templates /usr/share/sftpgo/
+# create bash completion script and man pages
+sudo sh -c '/usr/bin/sftpgo gen completion bash > /etc/bash_completion.d/sftpgo-completion.bash'
+sudo /usr/bin/sftpgo gen man -d /usr/share/man/man1
+# enable bash completion
+source /etc/bash_completion.d/sftpgo-completion.bash
+# set proper permissions to run SFTPGo as non-root user
+sudo chown -R sftpgo:sftpgo /etc/sftpgo/hostkeys /var/lib/sftpgo
+```
+
+## Configure SFTPGo
+
+Now open the SFTPGo configuration.
+
+```shell
+sudo vi /etc/sftpgo/sftpgo.json
+```
+
+Search for the `data_provider` section and change it as follow.
+
+```json
+  "data_provider": {
+    "driver": "postgresql",
+    "name": "sftpgo.db",
+    "host": "127.0.0.1",
+    "port": 5432,
+    "username": "sftpgo",
+    "password": "sftpgo_pg_pwd",
+    ...
+    "users_base_dir": "/tmp",
+}
+```
+
+This way we set the PostgreSQL connection parameters and a default base directory for new users.
+Since we use S3 and not the local filesystem as backend we set `/tmp` as default base directory so when we add a new user the home directory will be automatically defined as the path obtained joining `/tmp` and the username.
+
+If you want to connect to PostgreSQL over a Unix Domain socket you have to set the value `/var/run/postgresql` for the `host` configuration key instead of `127.0.0.1`.
+
+You can further customize your configuration adding custom actions and other hooks. A full explanation of all configuration parameters can be found [here](../full-configuration.md).
+
+Next, initialize the data provider with the following command.
+
+```shell
+$ sftpgo initprovider -c /etc/sftpgo
+2020-09-12T21:07:50.000 DBG Initializing provider: "postgresql" config file: "/etc/sftpgo/sftpgo.json"
+2020-09-12T21:07:50.000 DBG Data provider successfully initialized
+```
+
+## Install SFTPGo systemd service
+
+Create the systemd service file `/etc/systemd/system/sftpgo.service` with the following content:
+
+```shell
+[Unit]
+Description=SFTPGo Server
+After=network.target postgresql.service
+
+[Service]
+User=sftpgo
+Group=sftpgo
+Type=simple
+WorkingDirectory=/etc/sftpgo
+Environment=SFTPGO_CONFIG_DIR=/etc/sftpgo/
+Environment=SFTPGO_LOG_FILE_PATH=
+EnvironmentFile=-/etc/sftpgo/sftpgo.env
+ExecStart=/usr/bin/sftpgo serve
+ExecReload=/bin/kill -s HUP $MAINPID
+KillMode=mixed
+PrivateTmp=true
+Restart=always
+RestartSec=10s
+
+[Install]
+WantedBy=multi-user.target
+```
+
+This way SFTPGo will run using the dedicated `sftpgo` user and the service will start after PostgreSQL.
+
+Next, start the SFTPGo service and add it to the system boot.
+
+```shell
+sudo systemctl start sftpgo
+sudo systemctl enable sftpgo
+```
+
+Next, check the SFTPGo service using the following command.
+
+```shell
+systemctl status sftpgo
+```
+
+## Add virtual users
+
+The easiest way to add virtual users is to use the built-in Web interface.
+
+You can expose the Web Admin interface over the network replacing `"bind_address": "127.0.0.1"` in the `httpd` configuration section with `"bind_address": ""` and apply the change restarting the SFTPGo service with the following command.
+
+```shell
+systemctl restart sftpgo
+```
+
+So now open the Web Admin URL.
+
+[http://127.0.0.1:8080/web](http://127.0.0.1:8080/web)
+
+Click `Add` and fill the user details, the minimum required parameters are:
+
+- `Username`
+- `Password` or `Public keys`
+- `Permissions`
+- `Home Dir` can be empty since we defined a default base dir
+- Select `Amazon S3 (Compatible)` as storage and then set `Bucket`, `Region` and optionally a `Key Prefix` if you want to restrict the user to a specific bucket virtual folder. The specified folder does not need to be pre-create. You can leave `Access Key` and `Access Secret` empty since we defined global credentials for the `sftpgo` user and we use this system user to run the SFTPGo service.
+
+You are done! Now you can connect to you SFTPGo instance using any compatible `sftp` client on port `2022`.
+
+You can mix S3 users with local users but please be aware that we are running the service as the unprivileged `sftpgo` system user so if you set storage as `local` for an SFTPGo virtual user then the home directory for this user need to be owned by the `sftpgo` system user.

docs/howto/rest-api-https-auth.md

@@ -0,0 +1,122 @@
+# Expose Web Admin and REST API over HTTPS and password protected
+
+This tutorial shows how to expose the SFTPGo web interface and REST API over HTTPS and password protect them.
+
+## Preliminary Note
+
+Before proceeding further you need to have a SFTPGo instance already configured and running.
+
+We assume:
+
+- you are running SFTPGo as service using the dedicated `sftpgo` system user
+- the SFTPGo configuration directory is `/etc/sftpgo`
+- you are running SFTPGo on Ubuntu 20.04, however this instructions can be easily adapted for other Linux variants.
+
+## Authentication Setup
+
+First install the `htpasswd` tool. We use this tool to create the users for the Web Admin/REST API.
+
+```shell
+sudo apt install apache2-utils
+```
+
+Create a user for web based authentication.
+
+```shell
+sudo htpasswd -B -c /etc/sftpgo/httpauth sftpgoweb
+```
+
+If you want to create additional users omit the `-c` option.
+
+```shell
+sudo htpasswd -B /etc/sftpgo/httpauth anotheruser
+```
+
+Next open the SFTPGo configuration.
+
+```shell
+sudo vi /etc/sftpgo/sftpgo.json
+```
+
+Search for the `httpd` section and change it as follow.
+
+```json
+  "httpd": {
+    "bind_port": 8080,
+    "bind_address": "",
+    "templates_path": "templates",
+    "static_files_path": "static",
+    "backups_path": "backups",
+    "auth_user_file": "/etc/sftpgo/httpauth",
+    "certificate_file": "",
+    "certificate_key_file": ""
+  }
+```
+
+Setting an empty `bind_address` means that the service will listen on all available network interfaces and so it will be exposed over the network.
+
+Now restart the SFTPGo service to apply the changes.
+
+```shell
+systemctl restart sftpgo
+```
+
+You are done! Now login to the Web Admin interface using the username and password created above.
+
+## Creation of a Self-Signed Certificate
+
+For demostration purpose we use a self-signed certificate here. These certificates are easy to make and do not cost money. However, they do not provide all of the security properties that certificates signed by a Public Certificate Authority (CA) aim to provide, you are encouraged to use a certificate signed by a Public CA.
+
+When creating a new SSL certificate, one needs to specify the duration validity of the same by changing the value 365 (as appearing in the message below) to the preferred number of days. It is important to mention here that the certificate so created stands to auto-expire upon completion of one year.
+
+```shell
+sudo mkdir /etc/sftpgo/ssl
+sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/sftpgo/ssl/sftpgo.key -out /etc/sftpgo/ssl/sftpgo.crt
+```
+
+The above command is rather versatile, and lets you create both the self-signed SSL certificate and the server key to safeguard it, in addition to placing both of these into the `etc/sftpgo/ssl` directory. Answer to the questions to create the certificate and the key for HTTPS.
+
+Assign the proper permissions to the generated certificates.
+
+```shell
+sudo chown -R sftpgo:sftpgo /etc/sftpgo/ssl
+```
+
+## HTTPS Setup
+
+Open the SFTPGo configuration.
+
+```shell
+sudo vi /etc/sftpgo/sftpgo.json
+```
+
+Search for the `httpd` section and change it as follow.
+
+```json
+  "httpd": {
+    "bind_port": 8080,
+    "bind_address": "",
+    "templates_path": "templates",
+    "static_files_path": "static",
+    "backups_path": "backups",
+    "auth_user_file": "/etc/sftpgo/httpauth",
+    "certificate_file": "/etc/sftpgo/ssl/sftpgo.crt",
+    "certificate_key_file": "/etc/sftpgo/ssl/sftpgo.key"
+  }
+```
+
+Now restart the SFTPGo service to apply the changes.
+
+```shell
+systemctl restart sftpgo
+```
+
+You are done! Now SFTPGo web admin and REST API are exposed over HTTPS and password protected.
+
+You can easily replace the self-signed certificate used here with a properly signed certificate.
+
+The certificate could frequently change if you use something like [let's encrypt](https://letsencrypt.org/). SFTPGo allows hot-certificate reloading using the following command.
+
+```shell
+sudo systemctl reload sftpgo
+```

docs/service.md

@@ -12,7 +12,7 @@ sudo mkdir -p /etc/sftpgo \
   /var/lib/sftpgo \
   /usr/share/sftpgo
 
-# install sftpgo executable
+# install the sftpgo executable
 sudo install -Dm755 sftpgo /usr/bin/sftpgo
 # install the default configuration file, edit it if required
 sudo install -Dm644 sftpgo.json /etc/sftpgo/

httpd/api_quota.go

@@ -121,7 +121,7 @@ func startQuotaScan(w http.ResponseWriter, r *http.Request) {
 	}
 	if common.QuotaScans.AddUserQuotaScan(user.Username) {
 		go doQuotaScan(user) //nolint:errcheck
-		sendAPIResponse(w, r, err, "Scan started", http.StatusCreated)
+		sendAPIResponse(w, r, err, "Scan started", http.StatusAccepted)
 	} else {
 		sendAPIResponse(w, r, err, "Another scan is already in progress", http.StatusConflict)
 	}
@@ -146,7 +146,7 @@ func startVFolderQuotaScan(w http.ResponseWriter, r *http.Request) {
 	}
 	if common.QuotaScans.AddVFolderQuotaScan(folder.MappedPath) {
 		go doFolderQuotaScan(folder) //nolint:errcheck
-		sendAPIResponse(w, r, err, "Scan started", http.StatusCreated)
+		sendAPIResponse(w, r, err, "Scan started", http.StatusAccepted)
 	} else {
 		sendAPIResponse(w, r, err, "Another scan is already in progress", http.StatusConflict)
 	}

httpd/httpd_test.go

@@ -1119,7 +1119,7 @@ func TestGetQuotaScans(t *testing.T) {
 func TestStartQuotaScan(t *testing.T) {
 	user, _, err := httpd.AddUser(getTestUser(), http.StatusOK)
 	assert.NoError(t, err)
-	_, err = httpd.StartQuotaScan(user, http.StatusCreated)
+	_, err = httpd.StartQuotaScan(user, http.StatusAccepted)
 	assert.NoError(t, err)
 	_, err = httpd.RemoveUser(user, http.StatusOK)
 	assert.NoError(t, err)
@@ -1128,7 +1128,7 @@ func TestStartQuotaScan(t *testing.T) {
 	}
 	_, _, err = httpd.AddFolder(folder, http.StatusOK)
 	assert.NoError(t, err)
-	_, err = httpd.StartFolderQuotaScan(folder, http.StatusCreated)
+	_, err = httpd.StartFolderQuotaScan(folder, http.StatusAccepted)
 	assert.NoError(t, err)
 	for {
 		quotaScan, _, err := httpd.GetFoldersQuotaScans(http.StatusOK)
@@ -1958,7 +1958,7 @@ func TestStartQuotaScanMock(t *testing.T) {
 	userAsJSON = getUserAsJSON(t, user)
 	req, _ = http.NewRequest(http.MethodPost, quotaScanPath, bytes.NewBuffer(userAsJSON))
 	rr = executeRequest(req)
-	checkResponseCode(t, http.StatusCreated, rr.Code)
+	checkResponseCode(t, http.StatusAccepted, rr.Code)
 
 	for {
 		var scans []common.ActiveQuotaScan
@@ -1981,7 +1981,7 @@ func TestStartQuotaScanMock(t *testing.T) {
 	}
 	req, _ = http.NewRequest(http.MethodPost, quotaScanPath, bytes.NewBuffer(userAsJSON))
 	rr = executeRequest(req)
-	checkResponseCode(t, http.StatusCreated, rr.Code)
+	checkResponseCode(t, http.StatusAccepted, rr.Code)
 
 	for {
 		var scans []common.ActiveQuotaScan
@@ -2093,7 +2093,7 @@ func TestStartFolderQuotaScanMock(t *testing.T) {
 	}
 	req, _ = http.NewRequest(http.MethodPost, quotaScanVFolderPath, bytes.NewBuffer(folderAsJSON))
 	rr = executeRequest(req)
-	checkResponseCode(t, http.StatusCreated, rr.Code)
+	checkResponseCode(t, http.StatusAccepted, rr.Code)
 	var scans []common.ActiveVirtualFolderQuotaScan
 	for {
 		req, _ = http.NewRequest(http.MethodGet, quotaScanVFolderPath, nil)

httpd/schema/openapi.yaml

@@ -144,7 +144,7 @@ paths:
             schema:
               $ref : '#/components/schemas/User'
       responses:
-        201:
+        202:
           description: successful operation
           content:
             application/json:
@@ -302,7 +302,7 @@ paths:
             schema:
               $ref : '#/components/schemas/BaseVirtualFolder'
       responses:
-        201:
+        202:
           description: successful operation
           content:
             application/json:

sftpd/sftpd_test.go

@@ -2279,7 +2279,7 @@ func TestQuotaScan(t *testing.T) {
 	// create user with the same home dir, so there is at least an untracked file
 	user, _, err = httpd.AddUser(getTestUser(usePubKey), http.StatusOK)
 	assert.NoError(t, err)
-	_, err = httpd.StartQuotaScan(user, http.StatusCreated)
+	_, err = httpd.StartQuotaScan(user, http.StatusAccepted)
 	assert.NoError(t, err)
 	assert.Eventually(t, func() bool {
 		scans, _, err := httpd.GetQuotaScans(http.StatusOK)
@@ -4454,7 +4454,7 @@ func TestVirtualFolderQuotaScan(t *testing.T) {
 		MappedPath: mappedPath,
 	}, http.StatusOK)
 	assert.NoError(t, err)
-	_, err = httpd.StartFolderQuotaScan(folder, http.StatusCreated)
+	_, err = httpd.StartFolderQuotaScan(folder, http.StatusAccepted)
 	assert.NoError(t, err)
 	assert.Eventually(t, func() bool {
 		scans, _, err := httpd.GetFoldersQuotaScans(http.StatusOK)