add some basic how-to style documents

This commit is contained in:
Nicola Murino 2020-09-13 19:43:56 +02:00
parent 61003c8079
commit 3c1300721c
9 changed files with 411 additions and 13 deletions

View file

@ -81,7 +81,7 @@ A full explanation of all configuration methods can be found [here](./docs/full-
Please make sure to [initialize the data provider](#data-provider-initialization) before running the daemon! Please make sure to [initialize the data provider](#data-provider-initialization) before running the daemon!
To start the SFTP server with default settings, simply run: To start SFTPGo with the default settings, simply run:
```bash ```bash
sftpgo serve sftpgo serve
@ -113,6 +113,10 @@ sftpgo initprovider --help
After the initialization, the database structure will be automatically checked and updated, if required, at startup. After the initialization, the database structure will be automatically checked and updated, if required, at startup.
## Tutorials
Some step-to-step tutorials can be found inside the source tree [howto](./docs/howto "How-to") directory.
## Authentication options ## Authentication options
### External Authentication ### External Authentication

6
docs/howto/README.md Normal file
View file

@ -0,0 +1,6 @@
# Tutorials
Here we collect step-to-step tutorials. SFTPGo users are encouraged to contribute!
- [SFTPGo with PostgreSQL data provider and S3 backend](./postgresql-s3.md)
- [Expose Web Admin and REST API over HTTPS and password protected](./rest-api-https-auth.md)

266
docs/howto/postgresql-s3.md Normal file
View file

@ -0,0 +1,266 @@
# SFTPGo with PostgreSQL data provider and S3 backend
This tutorial shows the installation of SFTPGo on Ubuntu 20.04 (Focal Fossa) with PostgreSQL data provider and S3 backend. SFTPGo will run as an unprivileged (non-root) user. We assume that you want to serve a single S3 bucket and you want to assign different "virtual folders" of this bucket to different SFTPGo virtual users.
## Preliminary Note
Before proceeding further you need to have a basic minimal installation of Ubuntu 20.04.
Create the `sftpgo` user with the following command.
```shell
sudo adduser sftpgo
```
Type the user password and other info.
Add the `sftpgo` user to the `sudo` group so it will be able to use `sudo`:
```shell
sudo usermod -a -G sudo sftpgo
```
Now login using this user. Confirm that you are logged in as `sftpgo` user with the following command.
```shell
whoami
```
the output should be `sftpgo`.
## Install PostgreSQL
Before installing any packages on the Ubuntu system, update and upgrade all packages using the `apt` commands below.
```shell
sudo apt update
sudo apt upgrade
```
Install PostgreSQL with this `apt` command.
```shell
sudo apt -y install postgresql
```
Once installation is completed, start the PostgreSQL service and add it to the system boot.
```shell
sudo systemctl start postgresql
sudo systemctl enable postgresql
```
Next, check the PostgreSQL service using the following command.
```shell
systemctl status postgresql
```
## Create a new PostgreSQL user
PostgreSQL uses roles for user authentication and authorization, it just like Unix-Style permissions. By default, PostgreSQL creates a new user called `postgres` for basic authentication.
In this step, we will create a new PostgreSQL user for SFTPGo.
Login to the PostgreSQL shell using the command below.
```shell
sudo -i -u postgres psql
```
Next, create a new role `sftpgo` with the password `sftpgo_pg_pwd` using the following query.
```sql
create user "sftpgo" with encrypted password 'sftpgo_pg_pwd';
```
Next, create a new database `sftpgo.db` for the SFTPGo service using the following queries.
```sql
create database "sftpgo.db";
grant all privileges on database "sftpgo.db" to "sftpgo";
```
Exit from the PostgreSQL shell typing `\q`.
## Configure AWS credentials
We assume that you want to serve a single S3 bucket and you want to assign different "virtual folders" of this bucket to different SFTPGo virtual users. In this case is very convenient to configure a credential file so SFTPGo will automatically use it and you don't need to specify the same AWS credentials for each user.
You can manually create the `~/.aws/credentials` file and write your AWS credentials like this.
```shell
[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
Alternately you can install `AWS CLI` and manage the credential using this tool.
```shell
sudo apt install awscli
```
and now set your credentials, region, and output format with the following command.
```shell
aws configure
```
Confirm that you can list your bucket contents with the following command.
```shell
aws s3 ls s3://mybucket
```
## Install SFTPGo
Download a binary SFTPGo [release](https://github.com/drakkan/sftpgo/releases) or a build artifact for the [latest commit](https://github.com/drakkan/sftpgo/actions).
In this tutorial we assume you downloaded a build artifact named `sftpgo-ubuntu-latest-go1.15.zip` inside the current directory.
Install `unzip`, if not already installed, and extract the archive with the following commands.
```shell
sudo apt install unzip
mkdir sftpgo_installdir
unzip sftpgo-ubuntu-latest-go1.15.zip -d sftpgo_installdir
```
Now change the current directory to `sftpgo_installdir` and install SFTPGo.
```shell
cd sftpgo_installdir
# create the required directories
sudo mkdir -p /etc/sftpgo/hostkeys \
/var/lib/sftpgo/credentials \
/usr/share/sftpgo
# install the sftpgo executable
sudo install -Dm755 sftpgo /usr/bin/sftpgo
# install the default configuration file, edit it if required
sudo install -Dm644 sftpgo.json /etc/sftpgo/
# override some configuration keys using environment variables
sudo sh -c 'echo "SFTPGO_HTTPD__TEMPLATES_PATH=/usr/share/sftpgo/templates" > /etc/sftpgo/sftpgo.env'
sudo sh -c 'echo "SFTPGO_HTTPD__STATIC_FILES_PATH=/usr/share/sftpgo/static" >> /etc/sftpgo/sftpgo.env'
sudo sh -c 'echo "SFTPGO_HTTPD__BACKUPS_PATH=/var/lib/sftpgo/backups" >> /etc/sftpgo/sftpgo.env'
sudo sh -c 'echo "SFTPGO_DATA_PROVIDER__CREDENTIALS_PATH=/var/lib/sftpgo/credentials" >> /etc/sftpgo/sftpgo.env'
sudo sh -c 'echo "SFTPGO_SFTPD__HOST_KEYS=/etc/sftpgo/hostkeys/id_rsa,/etc/sftpgo/hostkeys/id_ecdsa" >> /etc/sftpgo/sftpgo.env'
# install static files and templates for the web UI
sudo cp -r static templates /usr/share/sftpgo/
# create bash completion script and man pages
sudo sh -c '/usr/bin/sftpgo gen completion bash > /etc/bash_completion.d/sftpgo-completion.bash'
sudo /usr/bin/sftpgo gen man -d /usr/share/man/man1
# enable bash completion
source /etc/bash_completion.d/sftpgo-completion.bash
# set proper permissions to run SFTPGo as non-root user
sudo chown -R sftpgo:sftpgo /etc/sftpgo/hostkeys /var/lib/sftpgo
```
## Configure SFTPGo
Now open the SFTPGo configuration.
```shell
sudo vi /etc/sftpgo/sftpgo.json
```
Search for the `data_provider` section and change it as follow.
```json
"data_provider": {
"driver": "postgresql",
"name": "sftpgo.db",
"host": "127.0.0.1",
"port": 5432,
"username": "sftpgo",
"password": "sftpgo_pg_pwd",
...
"users_base_dir": "/tmp",
}
```
This way we set the PostgreSQL connection parameters and a default base directory for new users.
Since we use S3 and not the local filesystem as backend we set `/tmp` as default base directory so when we add a new user the home directory will be automatically defined as the path obtained joining `/tmp` and the username.
If you want to connect to PostgreSQL over a Unix Domain socket you have to set the value `/var/run/postgresql` for the `host` configuration key instead of `127.0.0.1`.
You can further customize your configuration adding custom actions and other hooks. A full explanation of all configuration parameters can be found [here](../full-configuration.md).
Next, initialize the data provider with the following command.
```shell
$ sftpgo initprovider -c /etc/sftpgo
2020-09-12T21:07:50.000 DBG Initializing provider: "postgresql" config file: "/etc/sftpgo/sftpgo.json"
2020-09-12T21:07:50.000 DBG Data provider successfully initialized
```
## Install SFTPGo systemd service
Create the systemd service file `/etc/systemd/system/sftpgo.service` with the following content:
```shell
[Unit]
Description=SFTPGo Server
After=network.target postgresql.service
[Service]
User=sftpgo
Group=sftpgo
Type=simple
WorkingDirectory=/etc/sftpgo
Environment=SFTPGO_CONFIG_DIR=/etc/sftpgo/
Environment=SFTPGO_LOG_FILE_PATH=
EnvironmentFile=-/etc/sftpgo/sftpgo.env
ExecStart=/usr/bin/sftpgo serve
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
PrivateTmp=true
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
```
This way SFTPGo will run using the dedicated `sftpgo` user and the service will start after PostgreSQL.
Next, start the SFTPGo service and add it to the system boot.
```shell
sudo systemctl start sftpgo
sudo systemctl enable sftpgo
```
Next, check the SFTPGo service using the following command.
```shell
systemctl status sftpgo
```
## Add virtual users
The easiest way to add virtual users is to use the built-in Web interface.
You can expose the Web Admin interface over the network replacing `"bind_address": "127.0.0.1"` in the `httpd` configuration section with `"bind_address": ""` and apply the change restarting the SFTPGo service with the following command.
```shell
systemctl restart sftpgo
```
So now open the Web Admin URL.
[http://127.0.0.1:8080/web](http://127.0.0.1:8080/web)
Click `Add` and fill the user details, the minimum required parameters are:
- `Username`
- `Password` or `Public keys`
- `Permissions`
- `Home Dir` can be empty since we defined a default base dir
- Select `Amazon S3 (Compatible)` as storage and then set `Bucket`, `Region` and optionally a `Key Prefix` if you want to restrict the user to a specific bucket virtual folder. The specified folder does not need to be pre-create. You can leave `Access Key` and `Access Secret` empty since we defined global credentials for the `sftpgo` user and we use this system user to run the SFTPGo service.
You are done! Now you can connect to you SFTPGo instance using any compatible `sftp` client on port `2022`.
You can mix S3 users with local users but please be aware that we are running the service as the unprivileged `sftpgo` system user so if you set storage as `local` for an SFTPGo virtual user then the home directory for this user need to be owned by the `sftpgo` system user.

View file

@ -0,0 +1,122 @@
# Expose Web Admin and REST API over HTTPS and password protected
This tutorial shows how to expose the SFTPGo web interface and REST API over HTTPS and password protect them.
## Preliminary Note
Before proceeding further you need to have a SFTPGo instance already configured and running.
We assume:
- you are running SFTPGo as service using the dedicated `sftpgo` system user
- the SFTPGo configuration directory is `/etc/sftpgo`
- you are running SFTPGo on Ubuntu 20.04, however this instructions can be easily adapted for other Linux variants.
## Authentication Setup
First install the `htpasswd` tool. We use this tool to create the users for the Web Admin/REST API.
```shell
sudo apt install apache2-utils
```
Create a user for web based authentication.
```shell
sudo htpasswd -B -c /etc/sftpgo/httpauth sftpgoweb
```
If you want to create additional users omit the `-c` option.
```shell
sudo htpasswd -B /etc/sftpgo/httpauth anotheruser
```
Next open the SFTPGo configuration.
```shell
sudo vi /etc/sftpgo/sftpgo.json
```
Search for the `httpd` section and change it as follow.
```json
"httpd": {
"bind_port": 8080,
"bind_address": "",
"templates_path": "templates",
"static_files_path": "static",
"backups_path": "backups",
"auth_user_file": "/etc/sftpgo/httpauth",
"certificate_file": "",
"certificate_key_file": ""
}
```
Setting an empty `bind_address` means that the service will listen on all available network interfaces and so it will be exposed over the network.
Now restart the SFTPGo service to apply the changes.
```shell
systemctl restart sftpgo
```
You are done! Now login to the Web Admin interface using the username and password created above.
## Creation of a Self-Signed Certificate
For demostration purpose we use a self-signed certificate here. These certificates are easy to make and do not cost money. However, they do not provide all of the security properties that certificates signed by a Public Certificate Authority (CA) aim to provide, you are encouraged to use a certificate signed by a Public CA.
When creating a new SSL certificate, one needs to specify the duration validity of the same by changing the value 365 (as appearing in the message below) to the preferred number of days. It is important to mention here that the certificate so created stands to auto-expire upon completion of one year.
```shell
sudo mkdir /etc/sftpgo/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/sftpgo/ssl/sftpgo.key -out /etc/sftpgo/ssl/sftpgo.crt
```
The above command is rather versatile, and lets you create both the self-signed SSL certificate and the server key to safeguard it, in addition to placing both of these into the `etc/sftpgo/ssl` directory. Answer to the questions to create the certificate and the key for HTTPS.
Assign the proper permissions to the generated certificates.
```shell
sudo chown -R sftpgo:sftpgo /etc/sftpgo/ssl
```
## HTTPS Setup
Open the SFTPGo configuration.
```shell
sudo vi /etc/sftpgo/sftpgo.json
```
Search for the `httpd` section and change it as follow.
```json
"httpd": {
"bind_port": 8080,
"bind_address": "",
"templates_path": "templates",
"static_files_path": "static",
"backups_path": "backups",
"auth_user_file": "/etc/sftpgo/httpauth",
"certificate_file": "/etc/sftpgo/ssl/sftpgo.crt",
"certificate_key_file": "/etc/sftpgo/ssl/sftpgo.key"
}
```
Now restart the SFTPGo service to apply the changes.
```shell
systemctl restart sftpgo
```
You are done! Now SFTPGo web admin and REST API are exposed over HTTPS and password protected.
You can easily replace the self-signed certificate used here with a properly signed certificate.
The certificate could frequently change if you use something like [let's encrypt](https://letsencrypt.org/). SFTPGo allows hot-certificate reloading using the following command.
```shell
sudo systemctl reload sftpgo
```

View file

@ -12,7 +12,7 @@ sudo mkdir -p /etc/sftpgo \
/var/lib/sftpgo \ /var/lib/sftpgo \
/usr/share/sftpgo /usr/share/sftpgo
# install sftpgo executable # install the sftpgo executable
sudo install -Dm755 sftpgo /usr/bin/sftpgo sudo install -Dm755 sftpgo /usr/bin/sftpgo
# install the default configuration file, edit it if required # install the default configuration file, edit it if required
sudo install -Dm644 sftpgo.json /etc/sftpgo/ sudo install -Dm644 sftpgo.json /etc/sftpgo/

View file

@ -121,7 +121,7 @@ func startQuotaScan(w http.ResponseWriter, r *http.Request) {
} }
if common.QuotaScans.AddUserQuotaScan(user.Username) { if common.QuotaScans.AddUserQuotaScan(user.Username) {
go doQuotaScan(user) //nolint:errcheck go doQuotaScan(user) //nolint:errcheck
sendAPIResponse(w, r, err, "Scan started", http.StatusCreated) sendAPIResponse(w, r, err, "Scan started", http.StatusAccepted)
} else { } else {
sendAPIResponse(w, r, err, "Another scan is already in progress", http.StatusConflict) sendAPIResponse(w, r, err, "Another scan is already in progress", http.StatusConflict)
} }
@ -146,7 +146,7 @@ func startVFolderQuotaScan(w http.ResponseWriter, r *http.Request) {
} }
if common.QuotaScans.AddVFolderQuotaScan(folder.MappedPath) { if common.QuotaScans.AddVFolderQuotaScan(folder.MappedPath) {
go doFolderQuotaScan(folder) //nolint:errcheck go doFolderQuotaScan(folder) //nolint:errcheck
sendAPIResponse(w, r, err, "Scan started", http.StatusCreated) sendAPIResponse(w, r, err, "Scan started", http.StatusAccepted)
} else { } else {
sendAPIResponse(w, r, err, "Another scan is already in progress", http.StatusConflict) sendAPIResponse(w, r, err, "Another scan is already in progress", http.StatusConflict)
} }

View file

@ -1119,7 +1119,7 @@ func TestGetQuotaScans(t *testing.T) {
func TestStartQuotaScan(t *testing.T) { func TestStartQuotaScan(t *testing.T) {
user, _, err := httpd.AddUser(getTestUser(), http.StatusOK) user, _, err := httpd.AddUser(getTestUser(), http.StatusOK)
assert.NoError(t, err) assert.NoError(t, err)
_, err = httpd.StartQuotaScan(user, http.StatusCreated) _, err = httpd.StartQuotaScan(user, http.StatusAccepted)
assert.NoError(t, err) assert.NoError(t, err)
_, err = httpd.RemoveUser(user, http.StatusOK) _, err = httpd.RemoveUser(user, http.StatusOK)
assert.NoError(t, err) assert.NoError(t, err)
@ -1128,7 +1128,7 @@ func TestStartQuotaScan(t *testing.T) {
} }
_, _, err = httpd.AddFolder(folder, http.StatusOK) _, _, err = httpd.AddFolder(folder, http.StatusOK)
assert.NoError(t, err) assert.NoError(t, err)
_, err = httpd.StartFolderQuotaScan(folder, http.StatusCreated) _, err = httpd.StartFolderQuotaScan(folder, http.StatusAccepted)
assert.NoError(t, err) assert.NoError(t, err)
for { for {
quotaScan, _, err := httpd.GetFoldersQuotaScans(http.StatusOK) quotaScan, _, err := httpd.GetFoldersQuotaScans(http.StatusOK)
@ -1958,7 +1958,7 @@ func TestStartQuotaScanMock(t *testing.T) {
userAsJSON = getUserAsJSON(t, user) userAsJSON = getUserAsJSON(t, user)
req, _ = http.NewRequest(http.MethodPost, quotaScanPath, bytes.NewBuffer(userAsJSON)) req, _ = http.NewRequest(http.MethodPost, quotaScanPath, bytes.NewBuffer(userAsJSON))
rr = executeRequest(req) rr = executeRequest(req)
checkResponseCode(t, http.StatusCreated, rr.Code) checkResponseCode(t, http.StatusAccepted, rr.Code)
for { for {
var scans []common.ActiveQuotaScan var scans []common.ActiveQuotaScan
@ -1981,7 +1981,7 @@ func TestStartQuotaScanMock(t *testing.T) {
} }
req, _ = http.NewRequest(http.MethodPost, quotaScanPath, bytes.NewBuffer(userAsJSON)) req, _ = http.NewRequest(http.MethodPost, quotaScanPath, bytes.NewBuffer(userAsJSON))
rr = executeRequest(req) rr = executeRequest(req)
checkResponseCode(t, http.StatusCreated, rr.Code) checkResponseCode(t, http.StatusAccepted, rr.Code)
for { for {
var scans []common.ActiveQuotaScan var scans []common.ActiveQuotaScan
@ -2093,7 +2093,7 @@ func TestStartFolderQuotaScanMock(t *testing.T) {
} }
req, _ = http.NewRequest(http.MethodPost, quotaScanVFolderPath, bytes.NewBuffer(folderAsJSON)) req, _ = http.NewRequest(http.MethodPost, quotaScanVFolderPath, bytes.NewBuffer(folderAsJSON))
rr = executeRequest(req) rr = executeRequest(req)
checkResponseCode(t, http.StatusCreated, rr.Code) checkResponseCode(t, http.StatusAccepted, rr.Code)
var scans []common.ActiveVirtualFolderQuotaScan var scans []common.ActiveVirtualFolderQuotaScan
for { for {
req, _ = http.NewRequest(http.MethodGet, quotaScanVFolderPath, nil) req, _ = http.NewRequest(http.MethodGet, quotaScanVFolderPath, nil)

View file

@ -144,7 +144,7 @@ paths:
schema: schema:
$ref : '#/components/schemas/User' $ref : '#/components/schemas/User'
responses: responses:
201: 202:
description: successful operation description: successful operation
content: content:
application/json: application/json:
@ -302,7 +302,7 @@ paths:
schema: schema:
$ref : '#/components/schemas/BaseVirtualFolder' $ref : '#/components/schemas/BaseVirtualFolder'
responses: responses:
201: 202:
description: successful operation description: successful operation
content: content:
application/json: application/json:

View file

@ -2279,7 +2279,7 @@ func TestQuotaScan(t *testing.T) {
// create user with the same home dir, so there is at least an untracked file // create user with the same home dir, so there is at least an untracked file
user, _, err = httpd.AddUser(getTestUser(usePubKey), http.StatusOK) user, _, err = httpd.AddUser(getTestUser(usePubKey), http.StatusOK)
assert.NoError(t, err) assert.NoError(t, err)
_, err = httpd.StartQuotaScan(user, http.StatusCreated) _, err = httpd.StartQuotaScan(user, http.StatusAccepted)
assert.NoError(t, err) assert.NoError(t, err)
assert.Eventually(t, func() bool { assert.Eventually(t, func() bool {
scans, _, err := httpd.GetQuotaScans(http.StatusOK) scans, _, err := httpd.GetQuotaScans(http.StatusOK)
@ -4454,7 +4454,7 @@ func TestVirtualFolderQuotaScan(t *testing.T) {
MappedPath: mappedPath, MappedPath: mappedPath,
}, http.StatusOK) }, http.StatusOK)
assert.NoError(t, err) assert.NoError(t, err)
_, err = httpd.StartFolderQuotaScan(folder, http.StatusCreated) _, err = httpd.StartFolderQuotaScan(folder, http.StatusAccepted)
assert.NoError(t, err) assert.NoError(t, err)
assert.Eventually(t, func() bool { assert.Eventually(t, func() bool {
scans, _, err := httpd.GetFoldersQuotaScans(http.StatusOK) scans, _, err := httpd.GetFoldersQuotaScans(http.StatusOK)