Commit graph

121 commits

Author SHA1 Message Date
Nicola Murino
c64c080159 fix test cases on Windows
We have to rework TestRelativePaths and TestResolvePaths if we want to run
them for Cloud Storage on Windows too: we use filesystem path while Cloud
Storage providers expect Unix paths.
On Windows is important to check the local filesystem so skip Cloud Storage
providers test cases for now
2020-02-02 22:40:10 +01:00
Nicola Murino
bcaf283c35 memory provider: load users from a dump file
The `memory` provider can load users from a dump obtained using the
`dumpdata` REST API. This dump file can be configured using the
dataprovider `name` configuration key. It will be loaded at startup
and can be reloaded on demand using a `SIGHUP` on Unix based systems
and a `paramchange` request to the running service on Windows.

Fixes #66
2020-02-02 22:20:39 +01:00
Nicola Murino
31a433cda2 update deps and simplify some code 2020-01-31 23:26:56 +01:00
Nicola Murino
3491717c26 add support for serving Google Cloud Storage over SFTP/SCP
Each user can be mapped with a Google Cloud Storage bucket or a bucket
virtual folder
2020-01-31 19:04:00 +01:00
Nicola Murino
6884ce3f3e sftp actions: add a parameter to distinguish local and remote files 2020-01-23 23:23:28 +01:00
Nicola Murino
5f4efc9148 S3: add metrics 2020-01-23 23:17:00 +01:00
Nicola Murino
d481294519 S3: fix quota update after an upload error
S3 uploads are atomic, if the upload fails we have no partial file so we
have to update the user quota only if the upload succeed
2020-01-23 10:19:56 +01:00
Nicola Murino
9ff303b8c0 add support for keyboard interactive authentication
Fixes #64
2020-01-21 10:54:05 +01:00
Nicola Murino
4463421028 S3: add support for serving virtual folders
inside the same bucket each user can be assigned to a virtual folder.
This is similar to a chroot directory for local filesystem
2020-01-19 23:23:09 +01:00
Nicola Murino
d75f56b914 vfs: store root dir
so we don't need to pass it over and over
2020-01-19 13:58:55 +01:00
Nicola Murino
a4834f4a83 add basic S3-Compatible Object Storage support
we have now an interface for filesystem backeds, this make easy to add
new filesystem backends
2020-01-19 07:41:05 +01:00
Nicola Murino
27dbcf0066 fix test cases on Windows 2020-01-12 08:25:08 +01:00
Nicola Murino
ec194d73d2 update pkg/sftp to a stable version ...
... now that it contains all the needed patches.
Remove an hack for setstat with empty attrs, it is now handled in pkg/sftp.
Update other dependencies too.
2020-01-11 14:29:30 +01:00
Nicola Murino
1d9bb54073 transfers: improve errors detection
We can now properly report write errors if for example no space left on
device.

For downloads we check the downloaded size with the expected one
2020-01-10 19:20:22 +01:00
Nicola Murino
eec60d6309 custom actions: add env vars
action parameters can now be readed from env vars too.
Added a timeout for the command execution
2020-01-09 12:00:37 +01:00
Nicola Murino
8e604f888a improve docs and test cases 2020-01-07 09:39:20 +01:00
Nicola Murino
531091906d add support for authentication using external programs
Fixes #62
2020-01-06 21:42:41 +01:00
Nicola Murino
e046b35b97 check permissions against sftp path
instead of building filesystem paths and then checking permissions against
path relative to the home dir that is the initial sftp path
2020-01-05 11:41:25 +01:00
Nicola Murino
1b1c740b29 Add support for allowed/denied IP/Mask
Login can be restricted to specific ranges of IP address or to a specific IP
address.

Please apply the appropriate SQL upgrade script to add the filter field to your
database.

The filter database field will allow to add other filters without requiring a
new database migration
2019-12-30 18:37:50 +01:00
Nicola Murino
ad5436e3f6 ssh commands: improve command ended detection
Sometime we can have this error:

read |0: file already closed

reading from the command standard error, this means that the command is
already finished so we don't need to do nothing.
This happen randomically while running the test cases on travis.
2019-12-30 00:10:03 +01:00
Nicola Murino
20606a0043 sftpd test: add a debug log
The git push test sometime fails when running on travis.
The issue cannot be replicated locally so print the logs to try to
understand what is happening
2019-12-29 23:27:32 +01:00
Nicola Murino
6aff8c2f5e add support for checking passwords in md5crypt ($1$) format
this is an old and unsafe schema but it is still useful to import users
from legacy systems
2019-12-29 07:43:59 +01:00
Nicola Murino
ae094d3479 add backup/restore REST API 2019-12-27 23:12:44 +01:00
Nicola Murino
f49c280a7f fix test cases on Windows
SysProcAttr.Credential is not available on Windows we need to move the
WrapCmd test in a separate file to be able to build test cases on Windows,
skipping the test is not enough
2019-12-26 08:29:38 +01:00
Nicola Murino
ae812e55af sftpd: explicitly disallow some commands on root directory
It was possible to remove an empty root dir or create a symlink to it.
We now return a Permission Denied error if we detect an attempt to remove,
renaming or symlinking the root directory
2019-12-25 23:37:37 +01:00
Nicola Murino
489101668c add per directory permissions
we can now have permissions such as these ones

{"/":["*"],"/somedir":["list","download"]}

The old permissions are automatically converted to the new structure,
no database migration is needed
2019-12-25 18:20:19 +01:00
Nicola Murino
39fc9b73e9 sftp setstat: guard against empty attrs
It seems that there are some clients that sends Setstat requests with
no attrs:

https://github.com/pkg/sftp/issues/325

I haven't never seen this myself, anyway we now return ErrSSHFxBadMessage
and log the client version in such cases
2019-12-04 08:31:47 +01:00
Nicola Murino
80a5138115 add rsync support ...
and better document quota management issues for system commands.

rsync and git are not enabled in the default config so don't install
them in sample Dockerfiles, simply add a comment to facilitate their
installation if needed

Fixes #44
2019-11-29 15:24:56 +01:00
Nicola Murino
bc844105b2 ssh system commands: wait on write from command out to network
we only need to wait for the write from the local command to
the ssh channel. There is no need to wait for the write from ssh
channel to the local command stdin
2019-11-27 22:33:17 +01:00
Nicola Murino
0a025aabfd add support for Git over SSH
We use the system commands "git-receive-pack", "git-upload-pack" and
"git-upload-archive". they need to be installed and in your system's
PATH. Since we execute system commands we have no direct control on
file creation/deletion and so quota check is suboptimal: if quota is
enabled, the number of files is checked at the command begin and not
while new files are created.
The allowed size is calculated as the difference between the max quota
and the used one. The command is aborted if it uploads more bytes than
the remaining allowed size calculated at the command start. Quotas are
recalculated at the command end with a full home directory scan, this
could be heavy for big directories.
2019-11-26 22:26:42 +01:00
Nicola Murino
b3729e4666 log ssh commands in "command logs" category 2019-11-19 11:38:39 +01:00
Nicola Murino
9c4dbbc3f8 sftpd: add support for some SSH commands
md5sum, sha1sum are used by rclone.
cd, pwd improve the support for RemoteFiles mobile app.

These commands are all implemented inside SFTPGo so they work even
if the matching system commands are not available, for example on Windows
2019-11-18 23:30:37 +01:00
Nicola Murino
ca6cb34d98 sftpd: add support for chtimes
This improve rclone compatibility
2019-11-16 10:23:41 +01:00
Nicola Murino
fc442d7862 sftpd: document chmod/chown on Windows
chmod is partially supported and chown is not supported on Windows.

Skip unsupported test cases on Windows
2019-11-15 17:09:00 +01:00
Nicola Murino
3ac5af47f2 minor fixes 2019-11-15 12:26:52 +01:00
Nicola Murino
bb37a1c1ce sftpd: add support for chmod/chown
added matching permissions too and a new setting "setstat_mode".
Setting setstat_mode to 1 you can keep the previous behaviour that
silently ignore setstat requests
2019-11-15 12:15:07 +01:00
Nicola Murino
5be1d1be69 sftpd: send exit-status message on close
this fix restic compatibility
2019-11-14 16:49:42 +01:00
Nicola Murino
08e85f6be9 sftpd: return sftp.ErrSSHFxNoSuchFile if the client ask the file for a missing path
some clients expected this error and not the generic one if the path is missing
2019-11-14 14:18:43 +01:00
Nicola Murino
acdf351047 dataprovider: add custom command and/or HTTP notifications on users add, update and delete
This way custom logic can be implemented for example to create a UNIX user
as asked in #58
2019-11-14 11:06:03 +01:00
Nicola Murino
c2ff50c917 dataprovider: add support for user status and expiration
an user can now be disabled or expired.

If you are using an SQL database as dataprovider please remember to
execute the sql update script inside "sql" folder.

Fixes #57
2019-11-13 11:36:21 +01:00
Nicola Murino
363b9ccc7f sftpd: explicitly configure supported SFTP extensions
update pkg/sftp to a git revision that includes the needed patch

https://github.com/pkg/sftp/pull/315
2019-11-12 07:37:47 +01:00
Nicola Murino
4ff34b3e53 logger: add specific logs for failed attempts to initialize a connection
This should allow for better integration in tools like fail2ban.

Hopefully fix #59
2019-11-11 15:20:00 +01:00
Nicola Murino
77db2bd3d1 fix posix rename
update pkg/sftp to a git revision that includes:

https://github.com/pkg/sftp/pull/316

add a test case here too and update other deps
2019-11-08 08:43:27 +01:00
Nicola Murino
8cd0aec417 add memory data provider and use it for portable mode 2019-10-25 18:37:12 +02:00
Nicola Murino
a4cddf4f7f add portable mode
Portable mode is a convenient way to share a single directory on demand
2019-10-24 18:50:35 +02:00
Nicola Murino
44d403cf9c sftpd_test: use path.Join for SFTP/SCP path
filepath.Join could use an OS dependent separator
2019-10-16 12:57:06 +02:00
Nicola Murino
8682ae4a54 sftpd: make file/dir removal and creation more standard
- remove a non empty directory. Before: the directory contents were
removed recursively. Now: removing a non empty directory fails.

- make a directory in a non existent path: Before: any necessary parents
were created. Now: it fails.

- remove a file. Before: files, directories and symlinks were removed.
Now: only files and symlink are removed, removing a directory using "Remove"
instead of "Rmdir" fails.

Upload a file in a non existent directory. Before: any necessary parents
were created. Now: it fails.

Now SFTPGo behaves as OpenSSH.
2019-10-16 07:48:22 +02:00
Nicola Murino
f98a29a1e0 sftpd: port to non deprecated pkg/sftp error types
see https://github.com/pkg/sftp/pull/304
2019-10-14 22:44:57 +02:00
Nicola Murino
62224debd2 simplify closing a connection
get the connection from the map instead of cycling all the open connections
2019-10-12 15:12:43 +02:00
Nicola Murino
871e2ccbbf sftpd: refactor connection closing
we have not known bugs with the previous implementation anyway this one
is cleaner: the underlying network connection is directly related with
SFTP/SCP connections.
This should better protect us against buggy clients and edge cases
2019-10-10 09:04:17 +02:00