beenull/sftpgo-mirror
1
0
Fork 0
mirror of https://github.com/drakkan/sftpgo.git synced 2024-11-22 23:50:32 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6
1152 commits 8 branches 49 tags 49 MiB
Commit graph

155 commits

Author SHA1 Message Date
Nicola Murino
c63b923ec3
cryptfs: add support for atomic uploads 2021-05-31 21:45:29 +02:00
Nicola Murino
423d8306be
webclient: allow to download multiple files as zip 2021-05-30 23:07:46 +02:00
Nicola Murino
fc7066a25c
cross device rename: remove the source if copy suceeded 2021-05-27 22:23:14 +02:00
Nicola Murino
e1bf46c6a5
local fs rename: if it fails with a cross device error try a copy 
I don't want to add a new setting for this, at least until we get the
first complain for a slow rename :)

Fixes #440
 2021-05-27 20:14:12 +02:00
Nicola Murino
3b46e6a6fb
add support for a global temp path 
Fixes #436
 2021-05-27 15:38:27 +02:00
Nicola Murino
25a44030f9
actions: add pre-download and pre-upload 
Downloads and uploads can be denied based on hook response
 2021-05-26 07:48:37 +02:00
Nicola Murino
0cb5c49cf3
map path resolution errors to Permission errors 
this way the affected paths will be ignored in WebDAV

Fixes #432
 2021-05-21 13:04:22 +02:00
Nicola Murino
23d9ebfc91
add a basic front-end web interface for end-users 
Fixes #339 #321 #398
 2021-05-06 21:35:43 +02:00
Nicola Murino
5be4b6bd44
localfs: fix subdir check if the user has the root dir as home 2021-04-25 14:36:29 +02:00
Nicola Murino
74b51f0ad3
update nfpm 2021-04-23 22:53:13 +02:00
Nicola Murino
74e0223eb9
remove sha256-simd usage 
sha256-simd is now deprecated

https://github.com/minio/sha256-simd/issues/58

This could slow down sha256 computation on some CPU
 2021-04-05 18:23:40 +02:00
Nicola Murino
f895059660
web: add responsive table style to connections too 
Fixed a small issue for sftpfs too
 2021-04-05 11:28:28 +02:00
Nicola Murino
fdf3f23df5
allow to disable some hooks on a per-user basis 
This way you can, for example, mix external and internal users
 2021-04-04 22:32:25 +02:00
Nicola Murino
d92861a8e8
sftpfs: disable buffering for downloads if concurrent reads are disabled 2021-04-04 09:53:29 +02:00
Nicola Murino
1ee843757d
fix OpenAPI schema 2021-04-03 17:09:08 +02:00
Nicola Murino
ea26d7786c
sftpfs: add buffering support 
this way we improve performance over high latency networks
 2021-04-03 16:00:55 +02:00
Nicola Murino
2f56375121
improve SFTP loop detection 2021-04-01 18:53:48 +02:00
Nicola Murino
183bedd6ed
webui: add responsive extension 2021-03-28 11:02:11 +02:00
Nicola Murino
2a89a8f664
webui: minor improvements 2021-03-27 22:23:01 +01:00
Nicola Murino
9ad750da54
WebDAV: try to preserve the lock fs as much as possible 2021-03-27 19:10:27 +01:00
Nicola Murino
5e375f56dd
kms: add a lock, secrets could be modified concurrently for cached users 
also reduce the size of the JSON payload omitting empty secrets
 2021-03-22 19:03:25 +01:00
Nicola Murino
54c0c1b80d
Windows: manually check if we can bind on the configured port/ports 
Windows allows the coexistence of three types of sockets on the same
transport-layer service port, for example, 127.0.0.1:8080, [::1]:8080
and [::ffff:0.0.0.0]:8080

Go don't properly handles this, so we use a ugly hack

Fixes #350
 2021-03-21 22:21:04 +01:00
Nicola Murino
d6dc3a507e
extend virtual folders support to all storage backends 
Fixes #241
 2021-03-21 19:15:47 +01:00
Nicola Murino
0286da2356
try to auto create virtual folders if missing 2021-03-10 22:30:56 +01:00
Nicola Murino
055506e518
sftpfs: add an option to disable concurrent reads 2021-03-06 15:41:40 +01:00
Nicola Murino
88122ba2f8
update jwtauth to v5 2021-03-05 18:50:45 +01:00
Nicola Murino
ca3e15578e
Use new methods in the io and os packages instead of ioutil ones 
ioutil is deprecated in Go 1.16 and SFTPGo is an application, not
a library, we have no reason to keep compatibility with old Go
versions.

Go 1.16 fix some cifs related issues too.
 2021-02-25 21:53:04 +01:00
Nicola Murino
2146b83343
data providers: add filesystem to folder ... 
... and some descriptive fields.
The filesystem support for virtual folders will be implemented in
future commits
 2021-02-24 19:40:29 +01:00
Nicola Murino
3e1b07324d
GCS: remove compat code 2021-02-22 22:06:23 +01:00
Nicola Murino
78a837e8f1
remove other compat code 2021-02-22 09:13:26 +01:00
Nicola Murino
be9230e85b
micro optimizations spotted using the go-critic linter 2021-02-16 19:11:36 +01:00
Nicola Murino
46176a54b4
minor doc fixes 2021-02-14 22:08:08 +01:00
Nicola Murino
1ac66d27b6
Use IEC units for byte counting everywhere 2021-02-12 22:16:35 +01:00
Nicola Murino
51f110bc7b
sftpd: add statvfs@openssh.com support 2021-02-11 19:45:52 +01:00
Nicola Murino
70bb3c34ce
sftpfs: improve endpoint validation 
Validation will fail if the endpoint is not specified as host:port
 2021-02-03 11:29:04 +01:00
Nicola Murino
78bf808322
virtual folders: change dataprovider structure 
This way we no longer depend on the local file system path and so we can
add support for cloud backends in future updates
 2021-02-01 19:04:15 +01:00
Nicola Murino
ea096db8e4
sftpfs: set the correct file mode 2021-01-23 10:32:15 +01:00
Nicola Murino
0caeb68680
sftpfs: fix stat info 2021-01-23 09:42:49 +01:00
Giorgio Pellero
d42fcc3786
s3: don't paginate to find zero-byte-keyed dirs (#277) 
Fixes #275
 2021-01-14 12:01:25 +01:00
Nicola Murino
5d4f758c47
GCS: don't paginate to find compat "dirs" 2021-01-12 19:22:12 +01:00
Nicola Murino
2a95d031ea
FTP: add support for AVBL command 2020-12-25 11:14:08 +01:00
Jochen Munz
5b1d8666b3
S3fs: Handle non-ascii filename in rename operations (#257) 
SFTP is based on UTF-8 filenames, so non-ASCII filenames get transported with utf-8 escaped character sequences.
At least for the S3fs provider, if such a file is stored in a nested path it cannot be used as the source for a rename operations.

This adds the necessary escaping of the path fragments.

The patch is not required for MinIO but it doesn't hurt
 2020-12-24 11:13:42 +01:00
Nicola Murino
7ab7941ddd
sftpfs: fix race condition 2020-12-23 17:15:55 +01:00
Nicola Murino
f876c728ad
add support for the latest ftpserverlib and azblob versions 2020-12-17 13:40:36 +01:00
Nicola Murino
a6985075b9
add sftpfs storage backend 
Fixes #224
 2020-12-12 10:31:09 +01:00
Nicola Murino
034d89876d
webdav: fix proppatch handling 
also respect login delay for cached webdav users and check the home dir as
soon as the user authenticates

Fixes #239
 2020-12-06 08:19:41 +01:00
Nicola Murino
4a88ea5c03
add Data At Rest Encryption support 2020-12-05 13:48:13 +01:00
Nicola Murino
634b723b5d
add KMS support 
Fixes #226
 2020-11-30 21:46:34 +01:00
Nicola Murino
99cd1ccfe5
S3: fix empty directory detection 
when listing empty directory MinIO returns no contents while S3 returns
1 object with the key equal to the prefix. Make detection work in both
cases

Fixes #227
 2020-11-23 15:36:42 +01:00
Nicola Murino
dccc583b5d
add a dedicated struct to store encrypted credentials 
also gcs credentials are now encrypted, both on disk and inside the
provider.

Data provider is automatically migrated and load data will accept
old format too but you should upgrade to the new format to avoid future
issues
 2020-11-22 21:53:04 +01:00
Nicola Murino
ca0ff0d630
add a File interface so we can avoid to use os.File directly 2020-11-17 19:36:39 +01:00
Nicola Murino
1a6863f4b1
GCS uploads: check Close() error 
some code simplification too
 2020-11-13 18:40:18 +01:00
Nicola Murino
5720d40fee
add setstat_mode 2 
in this mode chmod/chtimes/chown can be silently ignored only for cloud
based file systems

Fixes #223
 2020-11-12 10:39:46 +01:00
Nicola Murino
c6ab6f94e7
azblob: container level SAS cannot access container properties 
so return the root directory without checking if the bucket exists
 2020-11-05 15:03:35 +01:00
Nicola Murino
0a14297b48
webdav: performance improvements and bug fixes 
we need my custom golang/x/net/webdav fork for now

https://github.com/drakkan/net/tree/sftpgo
 2020-11-04 19:11:40 +01:00
Nicola Murino
6ad4cc317c
cloud backends: stat and other performance improvements 2020-11-02 19:16:12 +01:00
Nicola Murino
fcfdd633f6
Azure Blob: update SDK and add access tier support 2020-10-30 22:17:17 +01:00
Nicola Murino
e54828a7b8
add metrics for Azure Blob storage 2020-10-26 19:01:17 +01:00
Nicola Murino
f2acde789d
portable mode: add Azure Blob support 2020-10-25 21:42:43 +01:00
Nicola Murino
9b49f63a97
azure: implement multipart uploads using low level API 
The high level wrapper seems to hang if there are network issues
 2020-10-25 17:41:04 +01:00
Nicola Murino
14bcc6f2fc
s3, azblob: check upper limit for part size 2020-10-25 12:10:11 +01:00
Nicola Murino
5ff8f75917
add Azure Blob support 2020-10-25 08:18:48 +01:00
Sean Hildebrand
db7e81e9d0
add prefer_database_credentials configuration parameter 
When true, users' Google Cloud Storage credentials will be written to
the data provider instead of disk.
Pre-existing credentials on disk will be used as a fallback

Fixes #201
 2020-10-22 10:42:40 +02:00
Nicola Murino
5eabaf98e0
gcs: remove a superfluous debug log 2020-09-29 09:17:08 +02:00
Nicola Murino
d1f0e9ae9f
CGS: implement MimeTyper interface 2020-09-28 22:12:46 +02:00
Thomas Blommaert
cd56039ab7
GCS mime-type detection (#179) 
Fixes #178
 2020-09-28 21:52:18 +02:00
Nicola Murino
55515fee95
update deps, GCS can no finally use attribute selection 
See https://github.com/googleapis/google-cloud-go/pull/2661
 2020-09-28 12:51:19 +02:00
Nicola Murino
bf708cb8bc osfs: improve isSubDir check 2020-09-21 19:32:33 +02:00
Nicola Murino
6c1a7449fe ssh commands: return better error messages 
This improve the fix for #171 and return better error message for
SSH commands other than SCP too
 2020-09-19 10:14:30 +02:00
Nicola Murino
209badf10c scp: return better error messages 
Fixes #171
 2020-09-18 19:13:09 +02:00
Nicola Murino
02e35ee002 sftpd: add Readlink support 2020-08-22 14:52:17 +02:00
Nicola Murino
f41ce6619f sftpd: add SSH_FXP_FSETSTAT support 
This change will fix file editing from sshfs, we need this patch

https://github.com/pkg/sftp/pull/373

for pkg/sftp to support this feature
 2020-08-20 13:54:36 +02:00
Nicola Murino
a9e21c282a add WebDAV support 
Fixes #147
 2020-08-11 23:56:10 +02:00
Nicola Murino
fa41bfd06a Cloud backends: add support for FTP REST command 
So partial downloads are now supported as for local fs
 2020-08-03 18:03:09 +02:00
Nicola Murino
2a9f7db1e2 Cloud FS: don't propagate the error if removing a folder returns not found 
for Cloud FS the folders are virtual and they, generally, disappear when the
last file is removed.

This fix doesn't work for FTP protocol for now.

Fixes #149
 2020-07-31 19:24:57 +02:00
Nicola Murino
93ce96d011 add support for the venerable FTP protocol 
Fixes #46
 2020-07-29 21:56:56 +02:00
Nicola Murino
4e41a5583d refactoring: add common package 
The common package defines the interfaces that a protocol must implement
and contain code that can be shared among supported protocols.

This way should be easier to support new protocols
 2020-07-24 23:39:38 +02:00
Nicola Murino
cf541d62ea recursive permissions check before renaming/copying directories 2020-06-26 23:38:29 +02:00
Nicola Murino
23a80b01b6 add build tag to disable metrics 2020-06-19 17:08:51 +02:00
Nicola Murino
e86089a9f3 quota: improve size check 
get the remaining allowed size when an upload starts and check it against the
uploaded bytes

Fixes #128
 2020-06-18 22:38:03 +02:00
Nicola Murino
3ceba7a147 sftpgo-copy: add quota limits check 2020-06-16 22:49:18 +02:00
Nicola Murino
7807fa7cc2 use os.ModePerm for files and directory creation 2020-06-08 19:40:17 +02:00
Nicola Murino
8306b6bde6 refactor virtual folders 
The same virtual folder can now be shared among users and different
folder quota limits for each user are supported.

Fixes #120
 2020-06-07 23:30:18 +02:00
Nicola Murino
ad53429cf1 add support for build tag to allow to disable some features 
The following build tags are available:

- "nogcs", disable Google Cloud Storage backend
- "nos3", disable S3 Compabible Object Storage backends
- "nobolt", disable Bolt data provider
- "nomysql", disable MySQL data provider
- "nopgsql", disable PostgreSQL data provider
- "nosqlite", disable SQLite data provider
- "noportable", disable portable mode
 2020-05-23 11:58:05 +02:00
Nicola Murino
cfa710037c cloud backends: fix SFTP error message for some write failures 
Fixes #119

Signed-off-by: Nicola Murino <nicola.murino@gmail.com>
 2020-05-19 19:17:43 +02:00
Nicola Murino
f02e24437a add more linters 
test cases migration to testify is now complete.
Linters are enabled for test cases too
 2020-05-06 19:36:34 +02:00
Nicola Murino
f369fdf6f2 httpclient: add a configuration parameter to skip TLS certificate validation 
In this mode, TLS is susceptible to man-in-the-middle attacks.
This should be used only for testing.
 2020-05-03 11:37:50 +02:00
Nicola Murino
b006c5f914 NewOsFs: return an interface and not a pointer 2020-05-02 15:01:56 +02:00
Nicola Murino
3f75d46a16 sftpd: add support for excluding virtual folders from user quota limit 
Fixes #110
 2020-05-01 15:27:53 +02:00
Nicola Murino
d70959c34c fix some lint issues 2020-04-30 14:23:55 +02:00
Nicola Murino
075bbe2aef added test case that checks quota for files inside virtual folders 2020-03-29 11:10:03 +02:00
Nicola Murino
1770da545d s3: upload concurrency is now configurable 
Please note that if the upload bandwidth between the SFTP client and
SFTPGo is greater than the upload bandwidth between SFTPGo and S3 then
the SFTP client have to wait for the upload of the last parts to S3
after it ends the file upload to SFTPGo, and it may time out.
Keep this in mind if you customize parts size and upload concurrency
 2020-03-13 19:13:58 +01:00
Nicola Murino
de3e69f846 s3: add documentation and test cases for upload part size 2020-03-13 17:28:55 +01:00
Michael Bonfils
cdf1233065 s3: export PartSize parameter 
By default AWS SDK use part_size of 5 MB. For big files,
it is not ideal case. For Hadoop, it is not uncommon to
use 512 MB.
 2020-03-13 17:28:04 +01:00
Nicola Murino
45b9366dd0 add support for virtual folders 
directories outside the user home directory can be exposed as virtual folders
 2020-02-23 11:30:26 +01:00
Nicola Murino
ae8ed75ae5 gcs: add support for automatic credentials 
We can now also support implicit credentials using the Application
Default Credentials strategy
 2020-02-19 09:41:15 +01:00
Nicola Murino
58253968fc s3: improve credentials validation 
access secret can now be empty, so check if not empty before encrypting
the secret
 2020-02-16 10:14:44 +01:00
Enes Çakır
da01848855 s3: enable shared config state 2020-02-16 09:06:25 +01:00
Nicola Murino
9359669cd4 s3: set upload concurrency to 2 
we upload a file while receiving it via SFTP not a file stored on a local
disk. We use concurrent uploads only to be able to send files of arbitrary
size, so concurrency is not really useful here. Setting the concurrency to
2 we have a max difference of 10 MB between the writer (sftp client) and
the reader (aws sdk), with the default concurrency value this difference
is 25MB.
 2020-02-04 23:14:55 +01:00
Nicola Murino
8b039e0447 httpd: add support for basic auth and HTTPS 2020-02-04 00:08:00 +01:00