Commit graph

295 commits

Author SHA1 Message Date
Nicola Murino
ea74aca165 doc: some other minor improvements 2020-03-05 00:01:40 +01:00
Nicola Murino
9b119765fc docs: minor improvements 2020-03-04 23:51:16 +01:00
Jo Vandeginste
df02496145 Refactor docs 2020-03-04 23:10:58 +01:00
Germs2004
f9fc5792fd fixed minor typos and edits 2020-03-04 08:01:02 +01:00
Nicola Murino
016abda6d7 improve docs 2020-03-03 23:25:23 +01:00
Nicola Murino
ed11e1128a docs: clarify the initprovider command is required for SQLite too 2020-03-02 10:34:31 +01:00
Nicola Murino
833b702b90 proxy protocol: add list of allowed IP addresses and IP ranges
"proxy_allowed" setting allows to specify the allowed IP address and IP
ranges that can send the proxy header. This setting combined with
"proxy_protocol" allows to ignore the header or to reject connections
that send the proxy header from a non listed IP
2020-03-01 23:12:28 +01:00
Nicola Murino
b885d453a2 filters: we can now set allowed and denied files extensions 2020-03-01 22:10:29 +01:00
Nicola Murino
7163fde724 proxy protocol: added an option to make the proxy header required
now we can configure SFTPGo to accept or reject requests without the proxy
header when the proxy protocol is enabled
2020-02-29 00:02:06 +01:00
Nicola Murino
830e3d1f64 Support for HAProxy PROXY protocol
you can proxy and/or load balance the SFTP/SCP service without losing
the information about the client's address.
2020-02-27 09:21:30 +01:00
Mengsk
637463a068 Rename before_login_program to pre_login_program
and some documentation update
2020-02-25 16:34:54 +01:00
Nicola Murino
e69536f540 fixed some typos and improved a log 2020-02-25 12:46:52 +01:00
Mengsk
c516780289 Documentation update 1 2020-02-25 12:41:28 +01:00
Nicola Murino
eb1b869b73 virtual folders fixes
scp now properly handles virtual folders.

rsync is disabled for users with virtual folders: we execute a system
command and it is not aware about virtual folders.

git is not allowed if the repo path is inside a virtual folder
2020-02-24 18:54:35 +01:00
Nicola Murino
703ccc8d91 add support for dynamic users modifications
A custom program can be executed before the users login to modify the
configurations for the user trying to login.
You can, for example, allow login based on time range.

Fixes #77
2020-02-23 18:50:59 +01:00
Nicola Murino
45b9366dd0 add support for virtual folders
directories outside the user home directory can be exposed as virtual folders
2020-02-23 11:30:26 +01:00
Nicola Murino
bc11cdd8d5 add support for per user authentication methods
You can, for example, deny one or more authentication methods to one or
more users.
2020-02-19 22:39:30 +01:00
Nicola Murino
ae8ed75ae5 gcs: add support for automatic credentials
We can now also support implicit credentials using the Application
Default Credentials strategy
2020-02-19 09:41:15 +01:00
Nicola Murino
c8cc81cf4a sftpd: autogenerate ecdsa key
With default configuration we now generate RSA and ECDSA server keys.
2020-02-16 18:17:39 +01:00
Nicola Murino
79c8b6cbc2 keyboard interactive auth: allows to automatically check the user password
This simplify the common pattern where the user password and a one time
token is requested: now the external program can delegate password check
to SFTPGo and verify the token itself
2020-02-16 11:43:52 +01:00
Nicola Murino
58253968fc s3: improve credentials validation
access secret can now be empty, so check if not empty before encrypting
the secret
2020-02-16 10:14:44 +01:00
Enes Çakır
dbd75209df s3: add auth options to README 2020-02-16 09:06:25 +01:00
Nicola Murino
0b7be1175d parse ssh commands with shlex
instead of use our bugged home made method.

Fixes #72
2020-02-14 16:17:32 +01:00
Nicola Murino
5b47292366 sftpd: improve error logs
Now logs for cloud filesystems are more readable.

Also use standard output as default for debian Dockerfile
2020-02-13 08:26:45 +01:00
Nicola Murino
7bfe0ddf80 ssh commands: fix parsing commands with space
For now we support "\" escaping style
2020-02-08 23:33:06 +01:00
Nicola Murino
d6fa853a37 add support for integrated database schema migrations
added the "initprovider" command to initialize the database structure.
If we change the database schema the required changes will be checked
at startup and automatically applyed.
2020-02-08 14:44:25 +01:00
Nicola Murino
553cceab42 dataprovider actions: add more users fiels as env vars 2020-02-05 22:17:03 +01:00
Nicola Murino
5bfaae9202 httpd: allow to reload the https certificate without restarting the service
HTTPS certificate can be reloaded on demand sending a SIGHUP signal on
Unix based systems and a "paramchange" request to the running service on
Windows
2020-02-04 23:21:33 +01:00
Nicola Murino
8b039e0447 httpd: add support for basic auth and HTTPS 2020-02-04 00:08:00 +01:00
Nicola Murino
bcaf283c35 memory provider: load users from a dump file
The `memory` provider can load users from a dump obtained using the
`dumpdata` REST API. This dump file can be configured using the
dataprovider `name` configuration key. It will be loaded at startup
and can be reloaded on demand using a `SIGHUP` on Unix based systems
and a `paramchange` request to the running service on Windows.

Fixes #66
2020-02-02 22:20:39 +01:00
Nicola Murino
3491717c26 add support for serving Google Cloud Storage over SFTP/SCP
Each user can be mapped with a Google Cloud Storage bucket or a bucket
virtual folder
2020-01-31 19:04:00 +01:00
thomsh
45a13f5f4e Update README add section for dataprovider init 2020-01-31 08:56:15 +01:00
Nicola Murino
6884ce3f3e sftp actions: add a parameter to distinguish local and remote files 2020-01-23 23:23:28 +01:00
Nicola Murino
5f4efc9148 S3: add metrics 2020-01-23 23:17:00 +01:00
Nicola Murino
7ebbbe5c29 S3: update pipeat to the latest commit in my fork
Here are the main improvements:

- unliked files works on windows too
- the uploads are now synced on the lower speed between the SFTP client write
and the upload speed to S3

This commit increase the external auth timeout to 60 seconds too
2020-01-22 19:42:23 +01:00
Nicola Murino
9ff303b8c0 add support for keyboard interactive authentication
Fixes #64
2020-01-21 10:54:05 +01:00
Nicola Murino
4463421028 S3: add support for serving virtual folders
inside the same bucket each user can be assigned to a virtual folder.
This is similar to a chroot directory for local filesystem
2020-01-19 23:23:09 +01:00
Nicola Murino
d75f56b914 vfs: store root dir
so we don't need to pass it over and over
2020-01-19 13:58:55 +01:00
Nicola Murino
a4834f4a83 add basic S3-Compatible Object Storage support
we have now an interface for filesystem backeds, this make easy to add
new filesystem backends
2020-01-19 07:41:05 +01:00
Nicola Murino
eec60d6309 custom actions: add env vars
action parameters can now be readed from env vars too.
Added a timeout for the command execution
2020-01-09 12:00:37 +01:00
Nicola Murino
8e604f888a improve docs and test cases 2020-01-07 09:39:20 +01:00
Nicola Murino
531091906d add support for authentication using external programs
Fixes #62
2020-01-06 21:42:41 +01:00
Nicola Murino
eb2ddc4798 small README improvements 2020-01-01 23:44:33 +01:00
Nicola Murino
aee9312cea better document how to reuse existing users 2019-12-31 14:22:42 +01:00
Nicola Murino
6a99a5cb9f Improve README 2019-12-31 11:11:07 +01:00
RIVIERE Fabien
8e0ca88421 Add systemD/journalD sftpgo Fail2ban configuration 2019-12-31 10:08:51 +01:00
Nicola Murino
1b1c740b29 Add support for allowed/denied IP/Mask
Login can be restricted to specific ranges of IP address or to a specific IP
address.

Please apply the appropriate SQL upgrade script to add the filter field to your
database.

The filter database field will allow to add other filters without requiring a
new database migration
2019-12-30 18:37:50 +01:00
Nicola Murino
ad5436e3f6 ssh commands: improve command ended detection
Sometime we can have this error:

read |0: file already closed

reading from the command standard error, this means that the command is
already finished so we don't need to do nothing.
This happen randomically while running the test cases on travis.
2019-12-30 00:10:03 +01:00
Nicola Murino
6aff8c2f5e add support for checking passwords in md5crypt ($1$) format
this is an old and unsafe schema but it is still useful to import users
from legacy systems
2019-12-29 07:43:59 +01:00
Nicola Murino
ae094d3479 add backup/restore REST API 2019-12-27 23:12:44 +01:00
Nicola Murino
489101668c add per directory permissions
we can now have permissions such as these ones

{"/":["*"],"/somedir":["list","download"]}

The old permissions are automatically converted to the new structure,
no database migration is needed
2019-12-25 18:20:19 +01:00
Nicola Murino
f8fd5c067c docker alpine: fix example
execute 20191112.sql too
2019-12-06 21:57:29 +01:00
Nicola Murino
80a5138115 add rsync support ...
and better document quota management issues for system commands.

rsync and git are not enabled in the default config so don't install
them in sample Dockerfiles, simply add a comment to facilitate their
installation if needed

Fixes #44
2019-11-29 15:24:56 +01:00
Nicola Murino
0a025aabfd add support for Git over SSH
We use the system commands "git-receive-pack", "git-upload-pack" and
"git-upload-archive". they need to be installed and in your system's
PATH. Since we execute system commands we have no direct control on
file creation/deletion and so quota check is suboptimal: if quota is
enabled, the number of files is checked at the command begin and not
while new files are created.
The allowed size is calculated as the difference between the max quota
and the used one. The command is aborted if it uploads more bytes than
the remaining allowed size calculated at the command start. Quotas are
recalculated at the command end with a full home directory scan, this
could be heavy for big directories.
2019-11-26 22:26:42 +01:00
Nicola Murino
7a8b1645ef set version to 0.9.4 2019-11-22 21:27:56 +01:00
Nicola Murino
b3729e4666 log ssh commands in "command logs" category 2019-11-19 11:38:39 +01:00
Nicola Murino
9c4dbbc3f8 sftpd: add support for some SSH commands
md5sum, sha1sum are used by rclone.
cd, pwd improve the support for RemoteFiles mobile app.

These commands are all implemented inside SFTPGo so they work even
if the matching system commands are not available, for example on Windows
2019-11-18 23:30:37 +01:00
Nicola Murino
ca6cb34d98 sftpd: add support for chtimes
This improve rclone compatibility
2019-11-16 10:23:41 +01:00
Nicola Murino
fc442d7862 sftpd: document chmod/chown on Windows
chmod is partially supported and chown is not supported on Windows.

Skip unsupported test cases on Windows
2019-11-15 17:09:00 +01:00
Nicola Murino
bb37a1c1ce sftpd: add support for chmod/chown
added matching permissions too and a new setting "setstat_mode".
Setting setstat_mode to 1 you can keep the previous behaviour that
silently ignore setstat requests
2019-11-15 12:15:07 +01:00
Nicola Murino
f3de83707f improve README 2019-11-14 17:43:14 +01:00
Nicola Murino
08e85f6be9 sftpd: return sftp.ErrSSHFxNoSuchFile if the client ask the file for a missing path
some clients expected this error and not the generic one if the path is missing
2019-11-14 14:18:43 +01:00
Nicola Murino
acdf351047 dataprovider: add custom command and/or HTTP notifications on users add, update and delete
This way custom logic can be implemented for example to create a UNIX user
as asked in #58
2019-11-14 11:06:03 +01:00
Nicola Murino
c2ff50c917 dataprovider: add support for user status and expiration
an user can now be disabled or expired.

If you are using an SQL database as dataprovider please remember to
execute the sql update script inside "sql" folder.

Fixes #57
2019-11-13 11:36:21 +01:00
Nicola Murino
74367a65cc failed connection logs: rename host to client_ip 2019-11-11 19:53:27 +01:00
Nicola Murino
2221d3307a Improve README 2019-11-11 18:28:18 +01:00
Nicola Murino
4ff34b3e53 logger: add specific logs for failed attempts to initialize a connection
This should allow for better integration in tools like fail2ban.

Hopefully fix #59
2019-11-11 15:20:00 +01:00
Nicola Murino
77db2bd3d1 fix posix rename
update pkg/sftp to a git revision that includes:

https://github.com/pkg/sftp/pull/316

add a test case here too and update other deps
2019-11-08 08:43:27 +01:00
Nicola Murino
758f2ee834 improve README 2019-10-26 20:42:46 +02:00
Nicola Murino
c5a6ca5650 portable mode: advertise service via multicast dns
Fixes #51
2019-10-26 18:25:53 +02:00
Nicola Murino
b409523d5c document memory data provider 2019-10-25 19:01:01 +02:00
Nicola Murino
a4cddf4f7f add portable mode
Portable mode is a convenient way to share a single directory on demand
2019-10-24 18:50:35 +02:00
Roman Isko
d970e757eb Use relative to the current branch links in readme 2019-10-22 10:00:09 +02:00
Nicola Murino
083d9f76c6 add commit hash and build date to the version logged when starting the daemon 2019-10-21 21:18:45 +02:00
Nicola Murino
9cf4653425 improve docs for available Arch Linux packages 2019-10-20 09:23:04 +02:00
Nicola Murino
1d917561fe sftpd: add support for upload resume
we support resume only if the client sets the correct offset while resuming
the upload.
Based on the specs the offset is optional for resume, but all the tested
clients sets a right offset.
If an invalid offset is given we interrupt the transfer with the error
"Invalid write offset ..."

See https://github.com/pkg/sftp/issues/295

This commit add a new upload mode: "atomic with resume support", this acts
as atomic but if there is an upload error the temporary file is renamed
to the requested path and not deleted, this way a client can reconnect
and resume the upload
2019-10-09 17:33:30 +02:00
Nicola Murino
60d4a3e1b5 improve docs 2019-10-08 08:20:26 +02:00
Nicola Murino
3e0558c0e9 add web interface support to windows setup ...
... and other small improvements
2019-10-07 22:37:28 +02:00
Nicola Murino
afd312f26a add a basic web interface
The builtin web interface allows to manage users and connections
2019-10-07 18:19:01 +02:00
Nicola Murino
00dd5db226 add support for users' default base dir 2019-09-28 22:48:52 +02:00
Nicola Murino
0d8506c93d set version to 0.9.2 2019-09-18 22:19:34 +02:00
Nicola Murino
df96ea7e9f add a new permission for overwriting existing files
The upload permission is required to allow file overwrite
2019-09-17 08:53:45 +02:00
Nicola Murino
580fae7a8f minor improvements 2019-09-16 18:11:35 +02:00
Nicola Murino
4a1baaee69 windows service: improve doc 2019-09-16 09:22:27 +02:00
Nicola Murino
f3f38f5f09 add Windows Service support 2019-09-16 08:52:58 +02:00
Nicola Murino
bba78763e1 add a test case and document sha512crypt passwords support 2019-09-15 12:23:27 +02:00
Nicola Murino
018b10808d improve SQL provider availability check adding a timeout 2019-09-14 16:18:31 +02:00
Jo Vandeginste
29aadbf3e3 log redacted passwords
Closes #48
2019-09-13 21:57:52 +02:00
Nicola Murino
7eb5b01169 add Prometheus support
some basic counters and gauges are now exposed
2019-09-13 18:45:36 +02:00
Nicola Murino
e7eb3476b7 dataprovider: remove transaction for quota update
The update is atomic so no transaction is needed.
Addionally a transaction will ask for a new connection to the pool
and this can deadlock if the pool has a max connection limit too low.

Also make configurable the pool size instead of hard code to the cpu number.

Fixes #47
2019-09-13 08:14:07 +02:00
Nicola Murino
6b33d3d9f1 add version to the default server banner 2019-09-12 09:21:58 +02:00
Nicola Murino
784f75f45b use net.Conn instead of ssh.Conn to close connections 2019-09-11 09:41:46 +02:00
Nicola Murino
c1effdf701 atomic upload mode: remove temporary file on error
If a transfer error is detected, then the partial temporary file will
be removed and not renamed to requested path
2019-09-10 18:47:21 +02:00
Nicola Murino
7010f513e3 macOS: add launchd sample plist 2019-09-10 08:03:53 +02:00
Nicola Murino
e8db7d8539 improve configuration docs
viper will not use env vars if no configuration file is found

https://github.com/spf13/viper/issues/584

As workaround we could manually bind/set a default for each configuration
option using viper.SetDefault("key") and then generate a default config
using viper.Get("key").
This manual solution is error prone and it will become increasingly
difficult to maintain since the configuration options will grow, so
we avoid it for now.

Let's see if viper will solve this issue

Fixes #35
2019-09-04 09:09:17 +02:00
Nicola Murino
3441b75a58 allow empty log file, use the standard output in this case
Fixes #34
2019-09-03 23:13:33 +02:00
Nicola Murino
49a40f7a0b sftpd: add configuration options for allowed ciphers, MACs and KEX algorithms
add support for login banner too

Fixes #32
2019-09-03 12:08:09 +02:00
Nicola Murino
dc5eeb54fd scp: fix quota update after file overwrite
added a test case too
2019-09-02 23:12:41 +02:00
Nicola Murino
4a1b67454e set version to 0.9.1 2019-09-02 09:36:02 +02:00
Nicola Murino
838426b3a4 improve docs 2019-08-29 13:52:22 +02:00
Nicola Murino
0e6113eef6 remove uneeded deps using go mod tidy 2019-08-27 15:59:00 +02:00
Nicola Murino
d4969cde03 scp: add test case and document remote to remote transfers 2019-08-25 13:51:54 +02:00
Nicola Murino
e50c521c33 add SCP support
SCP is an experimental feature, we have our own SCP implementation
since we can't rely on scp system command to proper handle permissions,
quota and user's home dir restrictions. The SCP protocol is quite simple
but there is no official docs about it, so we need more testing and
feedbacks before enabling it by default.
We may not handle some borderline cases or have sneaky bugs.

This commit contains some breaking changes to the REST API.
SFTPGo API should be stable now and I hope no more breaking changes
before the first stable release.
2019-08-24 14:41:15 +02:00
Nicola Murino
2c05791624 document how to store passwords using bcrypt or pbkdf2 using REST API 2019-08-18 12:52:29 +02:00
Nicola Murino
133f2e8601 add support for checking pbkdf2 passwords 2019-08-17 15:20:49 +02:00
Nicola Murino
9d342cb125 dataprovider: remove clear text password checking
passwords must be stored using argon2id or bcrypt
2019-08-17 08:44:43 +02:00
Nicola Murino
0bcc97aef6 add Awesome Go badge
SFTPGo is an awesome project now!
2019-08-12 23:38:43 +02:00
Nicola Murino
96a39a36bb dataprovider: add support for bbolt key/value store
This way there is an alternative for embedded/small systems if CGO
is disabled at build time and so SQLite support cannot be compiled
2019-08-12 18:31:31 +02:00
Nicola Murino
cb87fe811a dataprovider move db handle to provider struct
This is needed to support non SQL providers
2019-08-11 14:53:37 +02:00
Nicola Murino
51aacae3c5 version: only git commit and build date must be modifiable
Improved some test cases too
2019-08-08 21:42:07 +02:00
Nicola Murino
90607d4f86 sftpd stats: add file path for active upload/download 2019-08-08 19:33:16 +02:00
Nicola Murino
4d3c8fb6c4 add get version example 2019-08-08 10:13:32 +02:00
Nicola Murino
4f4489d3f1 add version info 2019-08-08 10:01:33 +02:00
Nicola Murino
5ad222fc53 python cli: add a flag to ignore verifying the SSL certificate 2019-08-07 23:06:26 +02:00
Nicola Murino
8f421b7d0f switch to viper for configuration and use cobra for cli 2019-08-07 22:46:52 +02:00
Nicola Murino
80b9c40489 add support for atomic upload
Atomic uploads are now configurable. The default upload mode remains
non atomic
2019-08-04 09:37:58 +02:00
Nicola Murino
b8dafecdd9 transfer: initialize lastActivity
This avoid a random value for transfer of empty file where no reads or writes happen
2019-08-02 15:49:51 +02:00
Nicola Murino
ba3f9d891a allow to set default arguments values from env vars 2019-08-02 09:47:14 +02:00
Nicola Murino
73c61cda31 arguments: document --arg form
go flag library supports one or two minus signs, they are equivalent.
Other Go command line parsers such as kingpin only support the -- variant
2019-08-02 00:26:20 +02:00
Nicola Murino
266216e372 Add sample cli client for REST API
It isn't written in Go to show interoperability with other programming languages
2019-08-02 00:00:48 +02:00
Nicola Murino
8d4964c16d convert public key from newline delimited string to a real array
Added a compatibility layer that will convert newline delimited keys to array
when the user is fetched from the database.
This code will be removed in future versions please update your public keys,
you only need to resave the users using the REST API.
2019-08-01 22:42:46 +02:00
Nicola Murino
788e068e13 Improve documentation 2019-08-01 15:39:30 +02:00
Jo Vandeginste
bd7b6a785e Support multiple private (host) keys
With this patch, one can configure one or more private (host) keys in
the configuration file.

I made it a nested struct, so we can add more options later. Eg. host
certificates might be a useful addition if I can figure out how this is
done in golang's crypto/ssh...

Signed-off-by: Jo Vandeginste <Jo.Vandeginste@kuleuven.be>
2019-08-01 14:57:15 +02:00
Nicola Murino
1566e43cd7 set track_quota to 2 as default 2019-08-01 09:24:35 +02:00
Nicola Murino
a7a3d533e7 Add documentation and tests for multiple public keys support 2019-07-31 17:06:12 +02:00
Jo Vandeginste
99e89f59c9 Add missing newline, add code highlighting hint 2019-07-31 14:49:21 +02:00
Nicola Murino
2a8ab620f3 improve logger
log file size, rotation policy and compression are now configurable
2019-07-31 14:06:55 +02:00
Nicola Murino
8058178ea0 Improve documentation 2019-07-30 20:51:29 +02:00
Nicola Murino
eb79c4effa add license badge 2019-07-29 09:26:36 +02:00
Nicola Murino
7dd03ee24e document bcrypt support 2019-07-29 09:20:33 +02:00
Nicola Murino
2456d5c3bf document the new banner configuration
use SFTPGo as default banner and ensure that the banner is not empty
2019-07-29 08:53:22 +02:00
Nicola Murino
0bed768909 allow quota tracking only for users with quota restrictions
Setting "track_quota" to 2 in sftpgo.conf will avoid a database update
after each file upload/deletion if the user has no quota restrictions
2019-07-28 22:04:50 +02:00
Nicola Murino
9987821003 rename last_quota_scan to last_quota_update
Existing databases must be updated using the script 20190728.sql
2019-07-28 19:29:32 +02:00
Nicola Murino
b728a06984 travis: add osx 2019-07-27 22:45:11 +02:00
Nicola Murino
c4fbca9ea2 HTTP server is now optional
Setting bind_port to 0 the HTTP server will not be started
2019-07-27 14:11:14 +02:00
Nicola Murino
6a6e31cceb clarify notifications on folder deletion 2019-07-27 09:52:57 +02:00
Nicola Murino
48451a9924 add support for custom actions
Configurable custom commands and/or HTTP notifications on SFTP upload, download, delete or rename
2019-07-27 09:38:09 +02:00
jungle-boogie
70ae68a7c4 with -> will
and other small changes
2019-07-27 05:19:32 +02:00
Tim Sedlmeyer
eb18e30a65 Fix some spelling errors 2019-07-26 16:20:40 +02:00
Nicola Murino
1348862e52 add code coverage badge 2019-07-26 11:56:01 +02:00
Nicola Murino
5c861a7c46 sftpd config: MaxAuthTries is now configurable 2019-07-21 12:02:24 +02:00
Nicola Murino
6053a0617f README: add Build Status and Go Report Card links 2019-07-21 11:45:11 +02:00
Nicola Murino
1019d74243 postgres 9.x is supported too 2019-07-20 18:11:28 +02:00
Nicola Murino
6378b58529 fix Arch Linux package link 2019-07-20 12:52:21 +02:00
Nicola Murino
31cd4d7139 first version 2019-07-20 12:26:52 +02:00