Commit graph

250 commits

Author SHA1 Message Date
Nicola Murino
6aff8c2f5e add support for checking passwords in md5crypt ($1$) format
this is an old and unsafe schema but it is still useful to import users
from legacy systems
2019-12-29 07:43:59 +01:00
Nicola Murino
ae094d3479 add backup/restore REST API 2019-12-27 23:12:44 +01:00
Nicola Murino
f49c280a7f fix test cases on Windows
SysProcAttr.Credential is not available on Windows we need to move the
WrapCmd test in a separate file to be able to build test cases on Windows,
skipping the test is not enough
2019-12-26 08:29:38 +01:00
Nicola Murino
ae812e55af sftpd: explicitly disallow some commands on root directory
It was possible to remove an empty root dir or create a symlink to it.
We now return a Permission Denied error if we detect an attempt to remove,
renaming or symlinking the root directory
2019-12-25 23:37:37 +01:00
Nicola Murino
489101668c add per directory permissions
we can now have permissions such as these ones

{"/":["*"],"/somedir":["list","download"]}

The old permissions are automatically converted to the new structure,
no database migration is needed
2019-12-25 18:20:19 +01:00
Nicola Murino
39fc9b73e9 sftp setstat: guard against empty attrs
It seems that there are some clients that sends Setstat requests with
no attrs:

https://github.com/pkg/sftp/issues/325

I haven't never seen this myself, anyway we now return ErrSSHFxBadMessage
and log the client version in such cases
2019-12-04 08:31:47 +01:00
Nicola Murino
80a5138115 add rsync support ...
and better document quota management issues for system commands.

rsync and git are not enabled in the default config so don't install
them in sample Dockerfiles, simply add a comment to facilitate their
installation if needed

Fixes #44
2019-11-29 15:24:56 +01:00
Nicola Murino
bc844105b2 ssh system commands: wait on write from command out to network
we only need to wait for the write from the local command to
the ssh channel. There is no need to wait for the write from ssh
channel to the local command stdin
2019-11-27 22:33:17 +01:00
Nicola Murino
0a025aabfd add support for Git over SSH
We use the system commands "git-receive-pack", "git-upload-pack" and
"git-upload-archive". they need to be installed and in your system's
PATH. Since we execute system commands we have no direct control on
file creation/deletion and so quota check is suboptimal: if quota is
enabled, the number of files is checked at the command begin and not
while new files are created.
The allowed size is calculated as the difference between the max quota
and the used one. The command is aborted if it uploads more bytes than
the remaining allowed size calculated at the command start. Quotas are
recalculated at the command end with a full home directory scan, this
could be heavy for big directories.
2019-11-26 22:26:42 +01:00
Nicola Murino
b3729e4666 log ssh commands in "command logs" category 2019-11-19 11:38:39 +01:00
Nicola Murino
9c4dbbc3f8 sftpd: add support for some SSH commands
md5sum, sha1sum are used by rclone.
cd, pwd improve the support for RemoteFiles mobile app.

These commands are all implemented inside SFTPGo so they work even
if the matching system commands are not available, for example on Windows
2019-11-18 23:30:37 +01:00
Nicola Murino
ca6cb34d98 sftpd: add support for chtimes
This improve rclone compatibility
2019-11-16 10:23:41 +01:00
Nicola Murino
fc442d7862 sftpd: document chmod/chown on Windows
chmod is partially supported and chown is not supported on Windows.

Skip unsupported test cases on Windows
2019-11-15 17:09:00 +01:00
Nicola Murino
3ac5af47f2 minor fixes 2019-11-15 12:26:52 +01:00
Nicola Murino
bb37a1c1ce sftpd: add support for chmod/chown
added matching permissions too and a new setting "setstat_mode".
Setting setstat_mode to 1 you can keep the previous behaviour that
silently ignore setstat requests
2019-11-15 12:15:07 +01:00
Nicola Murino
5be1d1be69 sftpd: send exit-status message on close
this fix restic compatibility
2019-11-14 16:49:42 +01:00
Nicola Murino
08e85f6be9 sftpd: return sftp.ErrSSHFxNoSuchFile if the client ask the file for a missing path
some clients expected this error and not the generic one if the path is missing
2019-11-14 14:18:43 +01:00
Nicola Murino
acdf351047 dataprovider: add custom command and/or HTTP notifications on users add, update and delete
This way custom logic can be implemented for example to create a UNIX user
as asked in #58
2019-11-14 11:06:03 +01:00
Nicola Murino
c2ff50c917 dataprovider: add support for user status and expiration
an user can now be disabled or expired.

If you are using an SQL database as dataprovider please remember to
execute the sql update script inside "sql" folder.

Fixes #57
2019-11-13 11:36:21 +01:00
Nicola Murino
363b9ccc7f sftpd: explicitly configure supported SFTP extensions
update pkg/sftp to a git revision that includes the needed patch

https://github.com/pkg/sftp/pull/315
2019-11-12 07:37:47 +01:00
Nicola Murino
4ff34b3e53 logger: add specific logs for failed attempts to initialize a connection
This should allow for better integration in tools like fail2ban.

Hopefully fix #59
2019-11-11 15:20:00 +01:00
Nicola Murino
77db2bd3d1 fix posix rename
update pkg/sftp to a git revision that includes:

https://github.com/pkg/sftp/pull/316

add a test case here too and update other deps
2019-11-08 08:43:27 +01:00
Nicola Murino
8cd0aec417 add memory data provider and use it for portable mode 2019-10-25 18:37:12 +02:00
Nicola Murino
a4cddf4f7f add portable mode
Portable mode is a convenient way to share a single directory on demand
2019-10-24 18:50:35 +02:00
Nicola Murino
44d403cf9c sftpd_test: use path.Join for SFTP/SCP path
filepath.Join could use an OS dependent separator
2019-10-16 12:57:06 +02:00
Nicola Murino
8682ae4a54 sftpd: make file/dir removal and creation more standard
- remove a non empty directory. Before: the directory contents were
removed recursively. Now: removing a non empty directory fails.

- make a directory in a non existent path: Before: any necessary parents
were created. Now: it fails.

- remove a file. Before: files, directories and symlinks were removed.
Now: only files and symlink are removed, removing a directory using "Remove"
instead of "Rmdir" fails.

Upload a file in a non existent directory. Before: any necessary parents
were created. Now: it fails.

Now SFTPGo behaves as OpenSSH.
2019-10-16 07:48:22 +02:00
Nicola Murino
f98a29a1e0 sftpd: port to non deprecated pkg/sftp error types
see https://github.com/pkg/sftp/pull/304
2019-10-14 22:44:57 +02:00
Nicola Murino
62224debd2 simplify closing a connection
get the connection from the map instead of cycling all the open connections
2019-10-12 15:12:43 +02:00
Nicola Murino
871e2ccbbf sftpd: refactor connection closing
we have not known bugs with the previous implementation anyway this one
is cleaner: the underlying network connection is directly related with
SFTP/SCP connections.
This should better protect us against buggy clients and edge cases
2019-10-10 09:04:17 +02:00
Nicola Murino
4b5ce3913e ssh handshake: add a deadline for handshake to complete
we use a 2 minutes timeout as OpenSSH
2019-10-09 19:07:35 +02:00
Nicola Murino
1d917561fe sftpd: add support for upload resume
we support resume only if the client sets the correct offset while resuming
the upload.
Based on the specs the offset is optional for resume, but all the tested
clients sets a right offset.
If an invalid offset is given we interrupt the transfer with the error
"Invalid write offset ..."

See https://github.com/pkg/sftp/issues/295

This commit add a new upload mode: "atomic with resume support", this acts
as atomic but if there is an upload error the temporary file is renamed
to the requested path and not deleted, this way a client can reconnect
and resume the upload
2019-10-09 17:33:30 +02:00
Nicola Murino
afd312f26a add a basic web interface
The builtin web interface allows to manage users and connections
2019-10-07 18:19:01 +02:00
Nicola Murino
df96ea7e9f add a new permission for overwriting existing files
The upload permission is required to allow file overwrite
2019-09-17 08:53:45 +02:00
Nicola Murino
bba78763e1 add a test case and document sha512crypt passwords support 2019-09-15 12:23:27 +02:00
Nicola Murino
7eb5b01169 add Prometheus support
some basic counters and gauges are now exposed
2019-09-13 18:45:36 +02:00
Nicola Murino
fd59f35108 update pkg/sftp to the latest master
Our pull request to handle transfer errors is now merged, so updating
pkg/sftp should fix #36
2019-09-13 08:30:22 +02:00
Nicola Murino
3d13fe15c3 connections: close the ssh channel before the network connection
This way if pkg/sftp is stuck in Serve() method should be unlocked.
2019-09-11 16:29:56 +02:00
Nicola Murino
9794ca7ee0 connection: add a close method
and use it instead of netConn directly
2019-09-11 12:46:21 +02:00
Nicola Murino
784f75f45b use net.Conn instead of ssh.Conn to close connections 2019-09-11 09:41:46 +02:00
Nicola Murino
c1effdf701 atomic upload mode: remove temporary file on error
If a transfer error is detected, then the partial temporary file will
be removed and not renamed to requested path
2019-09-10 18:47:21 +02:00
Nicola Murino
e79f7010b4 docker alpine: use the latest stable docker tag for both run and build env
so we don't need to update our Dockerfile each time a new alpine or golang
version is released
2019-09-09 12:58:07 +02:00
Nicola Murino
f4507aeec2 improve transfer error log and TestSCPErrors 2019-09-09 08:57:11 +02:00
Nicola Murino
e09ea00d8b improve transfer error log 2019-09-07 23:27:37 +02:00
Nicola Murino
bc5779e26f actions: don't execute actions on errors
detect upload/download errors and don't execute actions if a transfer
error happen.

To detect SFTP errors this patch is needed:

https://github.com/pkg/sftp/pull/307
2019-09-07 23:10:20 +02:00
Nicola Murino
2a7e56ed29 docker: minor fixes 2019-09-07 18:21:03 +02:00
Nicola Murino
4f1c2c094f improve logging
this partially revert #45
2019-09-06 15:19:01 +02:00
Jo Vandeginste
abbb7f272b Refactor the logging system
* created a "Log" function for type "Connection"
* created a "log" function for type "Provider"
* replace logger calls to Log/log where possible

I also renamed PGSSQL to PGSQL, as this seemed to be a typo

Signed-off-by: Jo Vandeginste <Jo.Vandeginste@kuleuven.be>
2019-09-06 13:43:21 +02:00
Nicola Murino
a26343a694 improve more logs 2019-09-05 23:42:00 +02:00
Jo Vandeginste
25260297aa Show info about public key during login
This will show the key fingerprint and the associated comment, or
"password" when password was used, during login.

Eg.:

```
message":"User id: 1, logged in with: \"public_key:SHA256:FV3+wlAKGzYy7+J02786fh8N8c06+jga/mdiSOSPT7g:jo@desktop\",
```

or

```
message":"User id: 1, logged in with: \"password\",
...`

Signed-off-by: Jo Vandeginste <Jo.Vandeginste@kuleuven.be>
2019-09-05 22:58:14 +02:00
Jo Vandeginste
bb589c6fc8 Show user info, remove some extraneous connection ids
Signed-off-by: Jo Vandeginste <Jo.Vandeginste@kuleuven.be>
2019-09-05 22:58:14 +02:00
Jo Vandeginste
0737c672f5 Add connectionID to as many entries as possible
Signed-off-by: Jo Vandeginste <Jo.Vandeginste@kuleuven.be>
2019-09-05 22:58:14 +02:00
Jo Vandeginste
795ffc4c5f Add quotes to certain strings
Signed-off-by: Jo Vandeginste <Jo.Vandeginste@kuleuven.be>
2019-09-05 16:14:31 +02:00
Nicola Murino
49a40f7a0b sftpd: add configuration options for allowed ciphers, MACs and KEX algorithms
add support for login banner too

Fixes #32
2019-09-03 12:08:09 +02:00
Nicola Murino
dc5eeb54fd scp: fix quota update after file overwrite
added a test case too
2019-09-02 23:12:41 +02:00
Nicola Murino
25fb639465 actions: properly release resources once the external command exits 2019-08-30 03:58:54 +02:00
Nicola Murino
d4969cde03 scp: add test case and document remote to remote transfers 2019-08-25 13:51:54 +02:00
Nicola Murino
1c5aac0dc4 simplify scp upload code and add some test cases 2019-08-24 22:44:01 +02:00
Nicola Murino
e50c521c33 add SCP support
SCP is an experimental feature, we have our own SCP implementation
since we can't rely on scp system command to proper handle permissions,
quota and user's home dir restrictions. The SCP protocol is quite simple
but there is no official docs about it, so we need more testing and
feedbacks before enabling it by default.
We may not handle some borderline cases or have sneaky bugs.

This commit contains some breaking changes to the REST API.
SFTPGo API should be stable now and I hope no more breaking changes
before the first stable release.
2019-08-24 14:41:15 +02:00
Nicola Murino
8dc3501a1f add test cases for password stored using bcrypt and pbkdf2 2019-08-17 16:54:02 +02:00
Nicola Murino
cb87fe811a dataprovider move db handle to provider struct
This is needed to support non SQL providers
2019-08-11 14:53:37 +02:00
Nicola Murino
51aacae3c5 version: only git commit and build date must be modifiable
Improved some test cases too
2019-08-08 21:42:07 +02:00
Nicola Murino
90607d4f86 sftpd stats: add file path for active upload/download 2019-08-08 19:33:16 +02:00
Nicola Murino
4f4489d3f1 add version info 2019-08-08 10:01:33 +02:00
Nicola Murino
2aca4479a5 rename public_key in public_keys
remove compatibility layer to convert public keys newline delimited
in json list
2019-08-07 23:41:10 +02:00
Nicola Murino
8f421b7d0f switch to viper for configuration and use cobra for cli 2019-08-07 22:46:52 +02:00
Nicola Murino
4dc52ee3e9 api_utils: return response body too
useful for debug and to build external API, for example a CLI interface
2019-08-04 21:37:31 +02:00
Nicola Murino
4d3c1fbad6 add more test cases 2019-08-04 12:35:33 +02:00
Nicola Murino
61b5a1d9a3 simplify FileOpenFlags to os.Flags conversion 2019-08-04 11:02:38 +02:00
Nicola Murino
80b9c40489 add support for atomic upload
Atomic uploads are now configurable. The default upload mode remains
non atomic
2019-08-04 09:37:58 +02:00
Nicola Murino
b8dafecdd9 transfer: initialize lastActivity
This avoid a random value for transfer of empty file where no reads or writes happen
2019-08-02 15:49:51 +02:00
Nicola Murino
8c40684989 sftpd: simplify initialization code
now gocyclo is happy again
2019-08-02 11:17:23 +02:00
Nicola Murino
8d4964c16d convert public key from newline delimited string to a real array
Added a compatibility layer that will convert newline delimited keys to array
when the user is fetched from the database.
This code will be removed in future versions please update your public keys,
you only need to resave the users using the REST API.
2019-08-01 22:42:46 +02:00
Nicola Murino
788e068e13 Improve documentation 2019-08-01 15:39:30 +02:00
Jo Vandeginste
bd7b6a785e Support multiple private (host) keys
With this patch, one can configure one or more private (host) keys in
the configuration file.

I made it a nested struct, so we can add more options later. Eg. host
certificates might be a useful addition if I can figure out how this is
done in golang's crypto/ssh...

Signed-off-by: Jo Vandeginste <Jo.Vandeginste@kuleuven.be>
2019-08-01 14:57:15 +02:00
Nicola Murino
1566e43cd7 set track_quota to 2 as default 2019-08-01 09:24:35 +02:00
Nicola Murino
a7a3d533e7 Add documentation and tests for multiple public keys support 2019-07-31 17:06:12 +02:00
Nicola Murino
34543fd861 simplify test case for quota checking after file replace 2019-07-31 14:21:56 +02:00
Nicola Murino
89986b9305 add new test cases 2019-07-31 14:11:44 +02:00
Nicola Murino
fab21dcf51 Print initialization errors to standard output
This way debugging configuration and initialization issues is simpler
2019-07-31 08:14:31 +02:00
Nicola Murino
8058178ea0 Improve documentation 2019-07-30 20:51:29 +02:00
Philip Hofstetter
88a288ccfe make the server banner configurable 2019-07-29 08:07:59 +02:00
Nicola Murino
0bed768909 allow quota tracking only for users with quota restrictions
Setting "track_quota" to 2 in sftpgo.conf will avoid a database update
after each file upload/deletion if the user has no quota restrictions
2019-07-28 22:04:50 +02:00
Nicola Murino
88fedd577d sftpd handler: remove eccessive logging and prepare the code to support upload resume
Upload resume is currently disabled for this upstream bug:

https://github.com/pkg/sftp/issues/295
2019-07-28 13:40:12 +02:00
Nicola Murino
c547c5c9ee quota tracking: fix concurrent updates
added a test case to check quota size for upload that replace an existing file
2019-07-28 13:24:46 +02:00
Nicola Murino
e177366be9 windows: fix test cases 2019-07-27 21:19:30 +02:00
Nicola Murino
4a46b84dd5 Improve http action notification
Send the notification using a goroutine and set the request timeout to 15 seconds
2019-07-27 19:29:33 +02:00
Nicola Murino
25f7043fb3 sftpd transfer: close the file before executing any action ...
... that could potentially move or delete it
2019-07-27 11:17:32 +02:00
Nicola Murino
48451a9924 add support for custom actions
Configurable custom commands and/or HTTP notifications on SFTP upload, download, delete or rename
2019-07-27 09:38:09 +02:00
Nicola Murino
aaa4c22911 sftpd: test case for SetStat
SetStat is silently ignored. Ownership and permissions are configured globally
or per account
2019-07-26 17:18:51 +02:00
Nicola Murino
5d3a93baae sftpd: improve test cases 2019-07-26 16:15:42 +02:00
Nicola Murino
9f61415832 add more test cases 2019-07-26 15:08:08 +02:00
Nicola Murino
ff8fb80e3c fix check for used quota
remove unused data provider from sftpd configuration struct
2019-07-26 13:07:08 +02:00
Nicola Murino
7360b0f73c test cases: wait 100 milliseconds before recheck if the test server is up
this avoid a lot of useless warnings in travis logs
2019-07-26 11:41:58 +02:00
Nicola Murino
417b173c78 Improve test cases and logging 2019-07-26 11:35:12 +02:00
Vincent Castellano
69e941483a
Fix typo in ActiveQuotaScan json annotations 2019-07-25 21:55:26 -07:00
Nicola Murino
5c861a7c46 sftpd config: MaxAuthTries is now configurable 2019-07-21 12:02:24 +02:00
Nicola Murino
14f919df47 simplify some code 2019-07-21 00:19:17 +02:00
Nicola Murino
fea9ae1de2 fix another misspell 2019-07-20 21:20:16 +02:00
Nicola Murino
22b8dc7f1d fix misspells and ineffectual assignments 2019-07-20 21:17:53 +02:00
Nicola Murino
31cd4d7139 first version 2019-07-20 12:26:52 +02:00