Commit graph

232 commits

Author SHA1 Message Date
Nicola Murino
0a9c4914aa pre-login program: allow to create a new user too
clarify the difference between dynamic user creation/update and external
authentication
2020-03-27 23:26:22 +01:00
Nicola Murino
f284008fb5 enable scp in default configuration
remove the deprecated enable_scp setting
2020-03-26 23:38:24 +01:00
Nicola Murino
81c8e8d898 add profiler support
profiling is now available via the HTTP base URL /debug/pprof/

examples, use this URL to start and download a 30 seconds CPU profile:

/debug/pprof/profile?seconds=30

use this URL to profile used memory:

/debug/pprof/heap?gc=1

use this URL to profile allocated memory:

/debug/pprof/allocs?gc=1

Full docs here:

https://golang.org/pkg/net/http/pprof/
2020-03-15 15:16:35 +01:00
Nicola Murino
ed1c7cac17 update deps
we now use git master for pkg/sftp: it includes the performance patches
from my copy branch.
2020-03-02 10:13:49 +01:00
Nicola Murino
833b702b90 proxy protocol: add list of allowed IP addresses and IP ranges
"proxy_allowed" setting allows to specify the allowed IP address and IP
ranges that can send the proxy header. This setting combined with
"proxy_protocol" allows to ignore the header or to reject connections
that send the proxy header from a non listed IP
2020-03-01 23:12:28 +01:00
Nicola Murino
b885d453a2 filters: we can now set allowed and denied files extensions 2020-03-01 22:10:29 +01:00
Nicola Murino
7163fde724 proxy protocol: added an option to make the proxy header required
now we can configure SFTPGo to accept or reject requests without the proxy
header when the proxy protocol is enabled
2020-02-29 00:02:06 +01:00
Nicola Murino
830e3d1f64 Support for HAProxy PROXY protocol
you can proxy and/or load balance the SFTP/SCP service without losing
the information about the client's address.
2020-02-27 09:21:30 +01:00
Mengsk
637463a068 Rename before_login_program to pre_login_program
and some documentation update
2020-02-25 16:34:54 +01:00
Nicola Murino
eb1b869b73 virtual folders fixes
scp now properly handles virtual folders.

rsync is disabled for users with virtual folders: we execute a system
command and it is not aware about virtual folders.

git is not allowed if the repo path is inside a virtual folder
2020-02-24 18:54:35 +01:00
Nicola Murino
703ccc8d91 add support for dynamic users modifications
A custom program can be executed before the users login to modify the
configurations for the user trying to login.
You can, for example, allow login based on time range.

Fixes #77
2020-02-23 18:50:59 +01:00
Nicola Murino
45b9366dd0 add support for virtual folders
directories outside the user home directory can be exposed as virtual folders
2020-02-23 11:30:26 +01:00
Nicola Murino
bc11cdd8d5 add support for per user authentication methods
You can, for example, deny one or more authentication methods to one or
more users.
2020-02-19 22:39:30 +01:00
Nicola Murino
8eff2df39c subdir perms: allow empty perms
empty perms will allow nothing on the specified subdir.

Non empty permissions for the "/" dir are still required.

Fixes #70
2020-02-10 19:28:35 +01:00
Nicola Murino
8b039e0447 httpd: add support for basic auth and HTTPS 2020-02-04 00:08:00 +01:00
Nicola Murino
c64c080159 fix test cases on Windows
We have to rework TestRelativePaths and TestResolvePaths if we want to run
them for Cloud Storage on Windows too: we use filesystem path while Cloud
Storage providers expect Unix paths.
On Windows is important to check the local filesystem so skip Cloud Storage
providers test cases for now
2020-02-02 22:40:10 +01:00
Nicola Murino
bcaf283c35 memory provider: load users from a dump file
The `memory` provider can load users from a dump obtained using the
`dumpdata` REST API. This dump file can be configured using the
dataprovider `name` configuration key. It will be loaded at startup
and can be reloaded on demand using a `SIGHUP` on Unix based systems
and a `paramchange` request to the running service on Windows.

Fixes #66
2020-02-02 22:20:39 +01:00
Nicola Murino
31a433cda2 update deps and simplify some code 2020-01-31 23:26:56 +01:00
Nicola Murino
3491717c26 add support for serving Google Cloud Storage over SFTP/SCP
Each user can be mapped with a Google Cloud Storage bucket or a bucket
virtual folder
2020-01-31 19:04:00 +01:00
Nicola Murino
d481294519 S3: fix quota update after an upload error
S3 uploads are atomic, if the upload fails we have no partial file so we
have to update the user quota only if the upload succeed
2020-01-23 10:19:56 +01:00
Nicola Murino
9ff303b8c0 add support for keyboard interactive authentication
Fixes #64
2020-01-21 10:54:05 +01:00
Nicola Murino
4463421028 S3: add support for serving virtual folders
inside the same bucket each user can be assigned to a virtual folder.
This is similar to a chroot directory for local filesystem
2020-01-19 23:23:09 +01:00
Nicola Murino
d75f56b914 vfs: store root dir
so we don't need to pass it over and over
2020-01-19 13:58:55 +01:00
Nicola Murino
a4834f4a83 add basic S3-Compatible Object Storage support
we have now an interface for filesystem backeds, this make easy to add
new filesystem backends
2020-01-19 07:41:05 +01:00
Nicola Murino
8e604f888a improve docs and test cases 2020-01-07 09:39:20 +01:00
Nicola Murino
531091906d add support for authentication using external programs
Fixes #62
2020-01-06 21:42:41 +01:00
Nicola Murino
e046b35b97 check permissions against sftp path
instead of building filesystem paths and then checking permissions against
path relative to the home dir that is the initial sftp path
2020-01-05 11:41:25 +01:00
Nicola Murino
1b1c740b29 Add support for allowed/denied IP/Mask
Login can be restricted to specific ranges of IP address or to a specific IP
address.

Please apply the appropriate SQL upgrade script to add the filter field to your
database.

The filter database field will allow to add other filters without requiring a
new database migration
2019-12-30 18:37:50 +01:00
Nicola Murino
20606a0043 sftpd test: add a debug log
The git push test sometime fails when running on travis.
The issue cannot be replicated locally so print the logs to try to
understand what is happening
2019-12-29 23:27:32 +01:00
Nicola Murino
6aff8c2f5e add support for checking passwords in md5crypt ($1$) format
this is an old and unsafe schema but it is still useful to import users
from legacy systems
2019-12-29 07:43:59 +01:00
Nicola Murino
ae094d3479 add backup/restore REST API 2019-12-27 23:12:44 +01:00
Nicola Murino
ae812e55af sftpd: explicitly disallow some commands on root directory
It was possible to remove an empty root dir or create a symlink to it.
We now return a Permission Denied error if we detect an attempt to remove,
renaming or symlinking the root directory
2019-12-25 23:37:37 +01:00
Nicola Murino
489101668c add per directory permissions
we can now have permissions such as these ones

{"/":["*"],"/somedir":["list","download"]}

The old permissions are automatically converted to the new structure,
no database migration is needed
2019-12-25 18:20:19 +01:00
Nicola Murino
0a025aabfd add support for Git over SSH
We use the system commands "git-receive-pack", "git-upload-pack" and
"git-upload-archive". they need to be installed and in your system's
PATH. Since we execute system commands we have no direct control on
file creation/deletion and so quota check is suboptimal: if quota is
enabled, the number of files is checked at the command begin and not
while new files are created.
The allowed size is calculated as the difference between the max quota
and the used one. The command is aborted if it uploads more bytes than
the remaining allowed size calculated at the command start. Quotas are
recalculated at the command end with a full home directory scan, this
could be heavy for big directories.
2019-11-26 22:26:42 +01:00
Nicola Murino
9c4dbbc3f8 sftpd: add support for some SSH commands
md5sum, sha1sum are used by rclone.
cd, pwd improve the support for RemoteFiles mobile app.

These commands are all implemented inside SFTPGo so they work even
if the matching system commands are not available, for example on Windows
2019-11-18 23:30:37 +01:00
Nicola Murino
ca6cb34d98 sftpd: add support for chtimes
This improve rclone compatibility
2019-11-16 10:23:41 +01:00
Nicola Murino
fc442d7862 sftpd: document chmod/chown on Windows
chmod is partially supported and chown is not supported on Windows.

Skip unsupported test cases on Windows
2019-11-15 17:09:00 +01:00
Nicola Murino
3ac5af47f2 minor fixes 2019-11-15 12:26:52 +01:00
Nicola Murino
bb37a1c1ce sftpd: add support for chmod/chown
added matching permissions too and a new setting "setstat_mode".
Setting setstat_mode to 1 you can keep the previous behaviour that
silently ignore setstat requests
2019-11-15 12:15:07 +01:00
Nicola Murino
08e85f6be9 sftpd: return sftp.ErrSSHFxNoSuchFile if the client ask the file for a missing path
some clients expected this error and not the generic one if the path is missing
2019-11-14 14:18:43 +01:00
Nicola Murino
c2ff50c917 dataprovider: add support for user status and expiration
an user can now be disabled or expired.

If you are using an SQL database as dataprovider please remember to
execute the sql update script inside "sql" folder.

Fixes #57
2019-11-13 11:36:21 +01:00
Nicola Murino
77db2bd3d1 fix posix rename
update pkg/sftp to a git revision that includes:

https://github.com/pkg/sftp/pull/316

add a test case here too and update other deps
2019-11-08 08:43:27 +01:00
Nicola Murino
8cd0aec417 add memory data provider and use it for portable mode 2019-10-25 18:37:12 +02:00
Nicola Murino
44d403cf9c sftpd_test: use path.Join for SFTP/SCP path
filepath.Join could use an OS dependent separator
2019-10-16 12:57:06 +02:00
Nicola Murino
8682ae4a54 sftpd: make file/dir removal and creation more standard
- remove a non empty directory. Before: the directory contents were
removed recursively. Now: removing a non empty directory fails.

- make a directory in a non existent path: Before: any necessary parents
were created. Now: it fails.

- remove a file. Before: files, directories and symlinks were removed.
Now: only files and symlink are removed, removing a directory using "Remove"
instead of "Rmdir" fails.

Upload a file in a non existent directory. Before: any necessary parents
were created. Now: it fails.

Now SFTPGo behaves as OpenSSH.
2019-10-16 07:48:22 +02:00
Nicola Murino
1d917561fe sftpd: add support for upload resume
we support resume only if the client sets the correct offset while resuming
the upload.
Based on the specs the offset is optional for resume, but all the tested
clients sets a right offset.
If an invalid offset is given we interrupt the transfer with the error
"Invalid write offset ..."

See https://github.com/pkg/sftp/issues/295

This commit add a new upload mode: "atomic with resume support", this acts
as atomic but if there is an upload error the temporary file is renamed
to the requested path and not deleted, this way a client can reconnect
and resume the upload
2019-10-09 17:33:30 +02:00
Nicola Murino
afd312f26a add a basic web interface
The builtin web interface allows to manage users and connections
2019-10-07 18:19:01 +02:00
Nicola Murino
df96ea7e9f add a new permission for overwriting existing files
The upload permission is required to allow file overwrite
2019-09-17 08:53:45 +02:00
Nicola Murino
bba78763e1 add a test case and document sha512crypt passwords support 2019-09-15 12:23:27 +02:00
Nicola Murino
fd59f35108 update pkg/sftp to the latest master
Our pull request to handle transfer errors is now merged, so updating
pkg/sftp should fix #36
2019-09-13 08:30:22 +02:00
Nicola Murino
c1effdf701 atomic upload mode: remove temporary file on error
If a transfer error is detected, then the partial temporary file will
be removed and not renamed to requested path
2019-09-10 18:47:21 +02:00
Nicola Murino
f4507aeec2 improve transfer error log and TestSCPErrors 2019-09-09 08:57:11 +02:00
Nicola Murino
bc5779e26f actions: don't execute actions on errors
detect upload/download errors and don't execute actions if a transfer
error happen.

To detect SFTP errors this patch is needed:

https://github.com/pkg/sftp/pull/307
2019-09-07 23:10:20 +02:00
Jo Vandeginste
0737c672f5 Add connectionID to as many entries as possible
Signed-off-by: Jo Vandeginste <Jo.Vandeginste@kuleuven.be>
2019-09-05 22:58:14 +02:00
Nicola Murino
49a40f7a0b sftpd: add configuration options for allowed ciphers, MACs and KEX algorithms
add support for login banner too

Fixes #32
2019-09-03 12:08:09 +02:00
Nicola Murino
dc5eeb54fd scp: fix quota update after file overwrite
added a test case too
2019-09-02 23:12:41 +02:00
Nicola Murino
d4969cde03 scp: add test case and document remote to remote transfers 2019-08-25 13:51:54 +02:00
Nicola Murino
1c5aac0dc4 simplify scp upload code and add some test cases 2019-08-24 22:44:01 +02:00
Nicola Murino
e50c521c33 add SCP support
SCP is an experimental feature, we have our own SCP implementation
since we can't rely on scp system command to proper handle permissions,
quota and user's home dir restrictions. The SCP protocol is quite simple
but there is no official docs about it, so we need more testing and
feedbacks before enabling it by default.
We may not handle some borderline cases or have sneaky bugs.

This commit contains some breaking changes to the REST API.
SFTPGo API should be stable now and I hope no more breaking changes
before the first stable release.
2019-08-24 14:41:15 +02:00
Nicola Murino
8dc3501a1f add test cases for password stored using bcrypt and pbkdf2 2019-08-17 16:54:02 +02:00
Nicola Murino
51aacae3c5 version: only git commit and build date must be modifiable
Improved some test cases too
2019-08-08 21:42:07 +02:00
Nicola Murino
4f4489d3f1 add version info 2019-08-08 10:01:33 +02:00
Nicola Murino
2aca4479a5 rename public_key in public_keys
remove compatibility layer to convert public keys newline delimited
in json list
2019-08-07 23:41:10 +02:00
Nicola Murino
8f421b7d0f switch to viper for configuration and use cobra for cli 2019-08-07 22:46:52 +02:00
Nicola Murino
4dc52ee3e9 api_utils: return response body too
useful for debug and to build external API, for example a CLI interface
2019-08-04 21:37:31 +02:00
Nicola Murino
80b9c40489 add support for atomic upload
Atomic uploads are now configurable. The default upload mode remains
non atomic
2019-08-04 09:37:58 +02:00
Nicola Murino
8d4964c16d convert public key from newline delimited string to a real array
Added a compatibility layer that will convert newline delimited keys to array
when the user is fetched from the database.
This code will be removed in future versions please update your public keys,
you only need to resave the users using the REST API.
2019-08-01 22:42:46 +02:00
Nicola Murino
1566e43cd7 set track_quota to 2 as default 2019-08-01 09:24:35 +02:00
Nicola Murino
a7a3d533e7 Add documentation and tests for multiple public keys support 2019-07-31 17:06:12 +02:00
Nicola Murino
34543fd861 simplify test case for quota checking after file replace 2019-07-31 14:21:56 +02:00
Nicola Murino
89986b9305 add new test cases 2019-07-31 14:11:44 +02:00
Nicola Murino
c547c5c9ee quota tracking: fix concurrent updates
added a test case to check quota size for upload that replace an existing file
2019-07-28 13:24:46 +02:00
Nicola Murino
e177366be9 windows: fix test cases 2019-07-27 21:19:30 +02:00
Nicola Murino
48451a9924 add support for custom actions
Configurable custom commands and/or HTTP notifications on SFTP upload, download, delete or rename
2019-07-27 09:38:09 +02:00
Nicola Murino
aaa4c22911 sftpd: test case for SetStat
SetStat is silently ignored. Ownership and permissions are configured globally
or per account
2019-07-26 17:18:51 +02:00
Nicola Murino
5d3a93baae sftpd: improve test cases 2019-07-26 16:15:42 +02:00
Nicola Murino
9f61415832 add more test cases 2019-07-26 15:08:08 +02:00
Nicola Murino
7360b0f73c test cases: wait 100 milliseconds before recheck if the test server is up
this avoid a lot of useless warnings in travis logs
2019-07-26 11:41:58 +02:00
Nicola Murino
417b173c78 Improve test cases and logging 2019-07-26 11:35:12 +02:00
Nicola Murino
14f919df47 simplify some code 2019-07-21 00:19:17 +02:00
Nicola Murino
fea9ae1de2 fix another misspell 2019-07-20 21:20:16 +02:00
Nicola Murino
31cd4d7139 first version 2019-07-20 12:26:52 +02:00