Commit graph

260 commits

Author SHA1 Message Date
Nicola Murino
62b20cd884 scp: check for write errors
exits as soon as there is a write error instead of get the same error when
the transfer is closed
2020-02-19 11:26:40 +01:00
Nicola Murino
ae8ed75ae5 gcs: add support for automatic credentials
We can now also support implicit credentials using the Application
Default Credentials strategy
2020-02-19 09:41:15 +01:00
Nicola Murino
c8cc81cf4a sftpd: autogenerate ecdsa key
With default configuration we now generate RSA and ECDSA server keys.
2020-02-16 18:17:39 +01:00
Nicola Murino
79c8b6cbc2 keyboard interactive auth: allows to automatically check the user password
This simplify the common pattern where the user password and a one time
token is requested: now the external program can delegate password check
to SFTPGo and verify the token itself
2020-02-16 11:43:52 +01:00
Nicola Murino
58253968fc s3: improve credentials validation
access secret can now be empty, so check if not empty before encrypting
the secret
2020-02-16 10:14:44 +01:00
Enes Çakır
dbd75209df s3: add auth options to README 2020-02-16 09:06:25 +01:00
Enes Çakır
da01848855 s3: enable shared config state 2020-02-16 09:06:25 +01:00
Nicola Murino
0b7be1175d parse ssh commands with shlex
instead of use our bugged home made method.

Fixes #72
2020-02-14 16:17:32 +01:00
Enes Çakır
3479a7e438 docker: remove serve command from entrypoint in alpine 2020-02-14 15:13:39 +01:00
Enes Çakır
4f5c67e7df portable: set data-provider name to empty 2020-02-14 15:13:10 +01:00
Nicola Murino
b99495ebbb sftpd download: remove check for download size
some clients, for example rclone can request only part of a file, we have
no way to detect this so we haven't return an error if the downloaded size
does not match the file size
2020-02-14 11:09:16 +01:00
Nicola Murino
0061978db8 docker: better clarify when ca-certificates package is needed 2020-02-14 08:29:31 +01:00
Mazinger
e011f793ec
CA certificate for Debian Docker image (#71)
* ca-certificate package included to ensure ssl connections

* comment instruction
2020-02-14 06:34:16 +01:00
Nicola Murino
5b47292366 sftpd: improve error logs
Now logs for cloud filesystems are more readable.

Also use standard output as default for debian Dockerfile
2020-02-13 08:26:45 +01:00
Nicola Murino
8eff2df39c subdir perms: allow empty perms
empty perms will allow nothing on the specified subdir.

Non empty permissions for the "/" dir are still required.

Fixes #70
2020-02-10 19:28:35 +01:00
Nicola Murino
7bfe0ddf80 ssh commands: fix parsing commands with space
For now we support "\" escaping style
2020-02-08 23:33:06 +01:00
Nicola Murino
d6fa853a37 add support for integrated database schema migrations
added the "initprovider" command to initialize the database structure.
If we change the database schema the required changes will be checked
at startup and automatically applyed.
2020-02-08 14:44:25 +01:00
Nicola Murino
553cceab42 dataprovider actions: add more users fiels as env vars 2020-02-05 22:17:03 +01:00
Nicola Murino
5bfaae9202 httpd: allow to reload the https certificate without restarting the service
HTTPS certificate can be reloaded on demand sending a SIGHUP signal on
Unix based systems and a "paramchange" request to the running service on
Windows
2020-02-04 23:21:33 +01:00
Nicola Murino
9359669cd4 s3: set upload concurrency to 2
we upload a file while receiving it via SFTP not a file stored on a local
disk. We use concurrent uploads only to be able to send files of arbitrary
size, so concurrency is not really useful here. Setting the concurrency to
2 we have a max difference of 10 MB between the writer (sftp client) and
the reader (aws sdk), with the default concurrency value this difference
is 25MB.
2020-02-04 23:14:55 +01:00
Nicola Murino
8b039e0447 httpd: add support for basic auth and HTTPS 2020-02-04 00:08:00 +01:00
Nicola Murino
c64c080159 fix test cases on Windows
We have to rework TestRelativePaths and TestResolvePaths if we want to run
them for Cloud Storage on Windows too: we use filesystem path while Cloud
Storage providers expect Unix paths.
On Windows is important to check the local filesystem so skip Cloud Storage
providers test cases for now
2020-02-02 22:40:10 +01:00
Nicola Murino
bcaf283c35 memory provider: load users from a dump file
The `memory` provider can load users from a dump obtained using the
`dumpdata` REST API. This dump file can be configured using the
dataprovider `name` configuration key. It will be loaded at startup
and can be reloaded on demand using a `SIGHUP` on Unix based systems
and a `paramchange` request to the running service on Windows.

Fixes #66
2020-02-02 22:20:39 +01:00
Nicola Murino
31a433cda2 update deps and simplify some code 2020-01-31 23:26:56 +01:00
Nicola Murino
e647f3626e loaddata: add an option that allows to not modify existing users 2020-01-31 19:10:45 +01:00
Nicola Murino
3491717c26 add support for serving Google Cloud Storage over SFTP/SCP
Each user can be mapped with a Google Cloud Storage bucket or a bucket
virtual folder
2020-01-31 19:04:00 +01:00
thomsh
45a13f5f4e Update README add section for dataprovider init 2020-01-31 08:56:15 +01:00
Nicola Murino
6884ce3f3e sftp actions: add a parameter to distinguish local and remote files 2020-01-23 23:23:28 +01:00
Nicola Murino
5f4efc9148 S3: add metrics 2020-01-23 23:17:00 +01:00
Nicola Murino
d481294519 S3: fix quota update after an upload error
S3 uploads are atomic, if the upload fails we have no partial file so we
have to update the user quota only if the upload succeed
2020-01-23 10:19:56 +01:00
Nicola Murino
7ebbbe5c29 S3: update pipeat to the latest commit in my fork
Here are the main improvements:

- unliked files works on windows too
- the uploads are now synced on the lower speed between the SFTP client write
and the upload speed to S3

This commit increase the external auth timeout to 60 seconds too
2020-01-22 19:42:23 +01:00
Nicola Murino
9ff303b8c0 add support for keyboard interactive authentication
Fixes #64
2020-01-21 10:54:05 +01:00
Nicola Murino
4463421028 S3: add support for serving virtual folders
inside the same bucket each user can be assigned to a virtual folder.
This is similar to a chroot directory for local filesystem
2020-01-19 23:23:09 +01:00
Nicola Murino
d75f56b914 vfs: store root dir
so we don't need to pass it over and over
2020-01-19 13:58:55 +01:00
Nicola Murino
a4834f4a83 add basic S3-Compatible Object Storage support
we have now an interface for filesystem backeds, this make easy to add
new filesystem backends
2020-01-19 07:41:05 +01:00
Nicola Murino
0b42dbc3c3 back to development 2020-01-12 19:16:18 +01:00
Nicola Murino
2013ba497c macOS: add backups dir to the sample launchd service 2020-01-12 19:02:09 +01:00
Nicola Murino
2be37217cf set version to 0.9.5 2020-01-12 14:56:07 +01:00
Nicola Murino
55a03c2e2b REST API CLI: improve color/no-color arguments 2020-01-12 14:54:41 +01:00
Nicola Murino
27dbcf0066 fix test cases on Windows 2020-01-12 08:25:08 +01:00
Nicola Murino
ec194d73d2 update pkg/sftp to a stable version ...
... now that it contains all the needed patches.
Remove an hack for setstat with empty attrs, it is now handled in pkg/sftp.
Update other dependencies too.
2020-01-11 14:29:30 +01:00
Nicola Murino
1d9bb54073 transfers: improve errors detection
We can now properly report write errors if for example no space left on
device.

For downloads we check the downloaded size with the expected one
2020-01-10 19:20:22 +01:00
Nicola Murino
5cf4a47b48 web interface: fix setting expiration date
editor autoformatting changed the meaning of the code
2020-01-09 12:14:33 +01:00
Nicola Murino
eec60d6309 custom actions: add env vars
action parameters can now be readed from env vars too.
Added a timeout for the command execution
2020-01-09 12:00:37 +01:00
Paweł K
37c602a477 Expand environment variables for external auth program
Use os.Environ() as a base instead of empty variable. Currently the environment of executed external auth program only contains SFTPGO_AUTHD* variables and therefore the program lacks additional context when started.
2020-01-09 08:33:40 +01:00
Nicola Murino
8e604f888a improve docs and test cases 2020-01-07 09:39:20 +01:00
Nicola Murino
531091906d add support for authentication using external programs
Fixes #62
2020-01-06 21:42:41 +01:00
Nicola Murino
e046b35b97 check permissions against sftp path
instead of building filesystem paths and then checking permissions against
path relative to the home dir that is the initial sftp path
2020-01-05 11:41:25 +01:00
Nicola Murino
eb2ddc4798 small README improvements 2020-01-01 23:44:33 +01:00
Nicola Murino
aee9312cea better document how to reuse existing users 2019-12-31 14:22:42 +01:00