add rsync support ...

and better document quota management issues for system commands.

rsync and git are not enabled in the default config so don't install
them in sample Dockerfiles, simply add a comment to facilitate their
installation if needed

Fixes #44

README.md

@@ -154,11 +154,12 @@ The `sftpgo` configuration file contains the following sections:
     - `macs`, list of strings. available MAC (message authentication code) algorithms in preference order. Leave empty to use default values. The supported values can be found here: [`crypto/ssh`](https://github.com/golang/crypto/blob/master/ssh/common.go#L84 "Supported MACs")
     - `login_banner_file`, path to the login banner file. The contents of the specified file, if any, are sent to the remote user before authentication is allowed. It can be a path relative to the config dir or an absolute one. Leave empty to send no login banner
     - `setstat_mode`, integer. 0 means "normal mode": requests for changing permissions, owner/group and access/modification times are executed. 1 means "ignore mode": requests for changing permissions, owner/group and access/modification times are silently ignored.
-    - `enabled_ssh_commands`, list of enabled SSH commands. These SSH commands are enabled by default: `md5sum`, `sha1sum`, `cd`, `pwd`. `*` enables all supported commands. We support the following SSH commands:
-      - `scp`, SCP is an experimental feature, we have our own SCP implementation since we can't rely on scp system command to proper handle permissions, quota and user's home dir restrictions. The SCP protocol is quite simple but there is no official docs about it, so we need more testing and feedbacks before enabling it by default. We may not handle some borderline cases or have sneaky bugs. Please do accurate tests yourself before enabling SCP and let us known if something does not work as expected for your use cases. SCP between two remote hosts is supported using the `-3` scp option.
+    - `enabled_ssh_commands`, list of enabled SSH commands. These SSH commands are enabled by default: `md5sum`, `sha1sum`, `cd`, `pwd`. `*` enables all supported commands. Some commands are implemented directly inside SFTPGo, while for other commands we use system commands that need to be installed and in your system's `PATH`. For system commands we have no direct control on file creation/deletion and so quota check is suboptimal: if quota is enabled, the number of files is checked at the command begin and not while new files are created. The allowed size is calculated as the difference between the max quota and the used one and it is checked against the bytes transferred via SSH. The command is aborted if it uploads more bytes than the remaining allowed size calculated at the command start. Anyway we see the bytes that the remote command send to the local command via SSH, these bytes contain both protocol commands and files and so the size of the files is different from the size trasferred via SSH: for example a command can send compressed files or a protocol command (few bytes) could delete a big file. To mitigate this issue quotas are recalculated at the command end with a full home directory scan, this could be heavy for big directories. If you need system commands and quotas you could consider to disable quota restrictions and periodically update quota usage yourself using the REST API. We support the following SSH commands:
+      - `scp`, SCP is an experimental feature, we have our own SCP implementation since we can't rely on "scp" system command to proper handle quotas and user's home dir restrictions. The SCP protocol is quite simple but there is no official docs about it, so we need more testing and feedbacks before enabling it by default. We may not handle some borderline cases or have sneaky bugs. Please do accurate tests yourself before enabling SCP and let us known if something does not work as expected for your use cases. SCP between two remote hosts is supported using the `-3` scp option.
       - `md5sum`, `sha1sum`, `sha256sum`, `sha384sum`, `sha512sum`. Useful to check message digests for uploaded files. These commands are implemented inside SFTPGo so they work even if the matching system commands are not available, for example on Windows.
       - `cd`, `pwd`. Some SFTP clients does not support the SFTP SSH_FXP_REALPATH packet type and so they use `cd` and `pwd` SSH commands to get the initial directory. Currently `cd` do nothing and `pwd` always returns the `/` path.
-      - `git-receive-pack`, `git-upload-pack`, `git-upload-archive`. These commands enable `git` support, they need to be installed and in your system's `PATH`. Since we execute system commands we have no direct control on file creation/deletion and so quota check is suboptimal: if quota is enabled, the number of files is checked at the command begin and not while new files are created. The allowed size is calculated as the difference between the max quota and the used one. The command is aborted if it uploads more bytes than the remaining allowed size calculated at the command start. Quotas are recalculated at the command end with a full home directory scan, this could be heavy for big directories.
+      - `git-receive-pack`, `git-upload-pack`, `git-upload-archive`. These commands enable support for Git repo over SSH, they need to be installed and in your system's `PATH`.
+      - `rsync`. The `rsync` command need to be installed and in your system's `PATH`. We cannot avoid that rsync create symlinks so if the user has the permission to create symlinks we add the option `--safe-links` to the received rsync command if it is not already set. This should prevent to create symlinks that point outside the home dir. If the user cannot create symlinks we add the option `--munge-links`, if it is not already set. This should make symlinks unusable (but manually recoverable)
 - **"data_provider"**, the configuration for the data provider
     - `driver`, string. Supported drivers are `sqlite`, `mysql`, `postgresql`, `bolt`, `memory`
     - `name`, string. Database name. For driver `sqlite` this can be the database name relative to the config dir or the absolute path to the SQLite database.

docker/sftpgo/alpine/Dockerfile

@@ -9,10 +9,12 @@ RUN go build -i -ldflags "-s -w -X github.com/drakkan/sftpgo/utils.commit=`git d
 
 FROM alpine:latest
 
-# git is optional it allows to serve Git repositories over SSH
-RUN  apk add --no-cache ca-certificates su-exec git \
+RUN apk add --no-cache ca-certificates su-exec \
   && mkdir -p /data /etc/sftpgo /srv/sftpgo/config /srv/sftpgo/web
 
+# git and rsync are optional, uncomment the next line to add support for them if needed
+#RUN apk add --no-cache git rsync
+
 COPY --from=builder /go/bin/sftpgo /bin/
 COPY --from=builder /go/src/github.com/drakkan/sftpgo/sftpgo.json /etc/sftpgo/sftpgo.json
 COPY --from=builder /go/src/github.com/drakkan/sftpgo/templates /srv/sftpgo/web/templates

docker/sftpgo/debian/Dockerfile

@@ -10,8 +10,8 @@ RUN go build -i -ldflags "-s -w -X github.com/drakkan/sftpgo/utils.commit=`git d
 # now define the run environment
 FROM debian:latest
 
-# git is optional it allows to serve Git repositories over SSH
-RUN apt-get update && apt-get install -y git
+# git and rsync are optional, uncomment the next line to add support for them if needed
+#RUN apt-get update && apt-get install -y git rsync
 
 ARG BASE_DIR=/app
 ARG DATA_REL_DIR=data

sftpd/internal_test.go

@@ -431,6 +431,46 @@ func TestSSHCommandQuotaScan(t *testing.T) {
 	}
 }
 
+func TestRsyncOptions(t *testing.T) {
+	conn := Connection{
+		User: dataprovider.User{
+			Permissions: []string{dataprovider.PermAny},
+			HomeDir:     os.TempDir(),
+		},
+	}
+	sshCmd := sshCommand{
+		command:    "rsync",
+		connection: conn,
+		args:       []string{"--server", "-vlogDtprze.iLsfxC", ".", "/"},
+	}
+	cmd, err := sshCmd.getSystemCommand()
+	if err != nil {
+		t.Errorf("unexpected error: %v", err)
+	}
+	if !utils.IsStringInSlice("--safe-links", cmd.cmd.Args) {
+		t.Errorf("--safe-links must be added if the user has the create symlinks permission")
+	}
+	conn = Connection{
+		User: dataprovider.User{
+			Permissions: []string{dataprovider.PermDownload, dataprovider.PermUpload, dataprovider.PermCreateDirs,
+				dataprovider.PermListItems, dataprovider.PermOverwrite, dataprovider.PermDelete, dataprovider.PermRename},
+			HomeDir: os.TempDir(),
+		},
+	}
+	sshCmd = sshCommand{
+		command:    "rsync",
+		connection: conn,
+		args:       []string{"--server", "-vlogDtprze.iLsfxC", ".", "/"},
+	}
+	cmd, err = sshCmd.getSystemCommand()
+	if err != nil {
+		t.Errorf("unexpected error: %v", err)
+	}
+	if !utils.IsStringInSlice("--munge-links", cmd.cmd.Args) {
+		t.Errorf("--munge-links must be added if the user has the create symlinks permission")
+	}
+}
+
 func TestSystemCommandErrors(t *testing.T) {
 	buf := make([]byte, 65535)
 	stdErrBuf := make([]byte, 65535)

sftpd/sftpd.go

@@ -63,10 +63,10 @@ var (
 	uploadMode           int
 	setstatMode          int
 	supportedSSHCommands = []string{"scp", "md5sum", "sha1sum", "sha256sum", "sha384sum", "sha512sum", "cd", "pwd",
-		"git-receive-pack", "git-upload-pack", "git-upload-archive"}
+		"git-receive-pack", "git-upload-pack", "git-upload-archive", "rsync"}
 	defaultSSHCommands = []string{"md5sum", "sha1sum", "cd", "pwd"}
 	sshHashCommands    = []string{"md5sum", "sha1sum", "sha256sum", "sha384sum", "sha512sum"}
-	gitCommands        = []string{"git-receive-pack", "git-upload-pack", "git-upload-archive"}
+	systemCommands     = []string{"git-receive-pack", "git-upload-pack", "git-upload-archive", "rsync"}
 )
 
 type connectionTransfer struct {

sftpd/ssh_cmd.go

@@ -84,7 +84,7 @@ func (c *sshCommand) handle() error {
 	updateConnectionActivity(c.connection.ID)
 	if utils.IsStringInSlice(c.command, sshHashCommands) {
 		return c.handleHashCommands()
-	} else if utils.IsStringInSlice(c.command, gitCommands) {
+	} else if utils.IsStringInSlice(c.command, systemCommands) {
 		command, err := c.getSystemCommand()
 		if err != nil {
 			return c.sendErrorResponse(err)
@@ -201,8 +201,8 @@ func (c *sshCommand) executeSystemCommand(command systemCommand) error {
 		addTransfer(&transfer)
 		defer removeTransfer(&transfer)
 		w, e := transfer.copyFromReaderToWriter(stdin, c.connection.channel, remainingQuotaSize)
-		c.connection.Log(logger.LevelDebug, logSenderSSH, "command: %#v, copy to sdtin ended, written: %v, remaining quota: %v, err: %v",
-			c.connection.command, w, remainingQuotaSize, e)
+		c.connection.Log(logger.LevelDebug, logSenderSSH, "command: %#v, copy from remote command to sdtin ended, written: %v, "+
+			"initial remaining quota: %v, err: %v", c.connection.command, w, remainingQuotaSize, e)
 		if e != nil {
 			once.Do(closeCmdOnError)
 		}
@@ -228,7 +228,7 @@ func (c *sshCommand) executeSystemCommand(command systemCommand) error {
 		addTransfer(&transfer)
 		defer removeTransfer(&transfer)
 		w, e := transfer.copyFromReaderToWriter(c.connection.channel, stdout, 0)
-		c.connection.Log(logger.LevelDebug, logSenderSSH, "command: %#v, copy from sdtout ended, written: %v err: %v",
+		c.connection.Log(logger.LevelDebug, logSenderSSH, "command: %#v, copy from sdtout to remote command ended, written: %v err: %v",
 			c.connection.command, w, e)
 		if e != nil {
 			once.Do(closeCmdOnError)
@@ -256,7 +256,7 @@ func (c *sshCommand) executeSystemCommand(command systemCommand) error {
 		addTransfer(&transfer)
 		defer removeTransfer(&transfer)
 		w, e := transfer.copyFromReaderToWriter(c.connection.channel.Stderr(), stderr, 0)
-		c.connection.Log(logger.LevelDebug, logSenderSSH, "command: %#v, copy from sdterr ended, written: %v err: %v",
+		c.connection.Log(logger.LevelDebug, logSenderSSH, "command: %#v, copy from sdterr to remote command ended, written: %v err: %v",
 			c.connection.command, w, e)
 		if e != nil || w > 0 {
 			once.Do(closeCmdOnError)
@@ -277,6 +277,23 @@ func (c *sshCommand) getSystemCommand() (systemCommand, error) {
 	}
 	args := make([]string, len(c.args))
 	copy(args, c.args)
+	if c.command == "rsync" {
+		// we cannot avoid that rsync create symlinks so if the user has the permission
+		// to create symlinks we add the option --safe-links to the received rsync command if
+		// it is not already set. This should prevent to create symlinks that point outside
+		// the home dir.
+		// If the user cannot create symlinks we add the option --munge-links, if it is not
+		// already set. This should make symlinks unusable (but manually recoverable)
+		if c.connection.User.HasPerm(dataprovider.PermCreateSymlinks) {
+			if !utils.IsStringInSlice("--safe-links", args) {
+				args = append([]string{"--safe-links"}, args...)
+			}
+		} else {
+			if !utils.IsStringInSlice("--munge-links", args) {
+				args = append([]string{"--munge-links"}, args...)
+			}
+		}
+	}
 	var path string
 	if len(c.args) > 0 {
 		var err error