Nicola Murino
de3e69f846
s3: add documentation and test cases for upload part size
2020-03-13 17:28:55 +01:00
Michael Bonfils
cdf1233065
s3: export PartSize parameter
...
By default AWS SDK use part_size of 5 MB. For big files,
it is not ideal case. For Hadoop, it is not uncommon to
use 512 MB.
2020-03-13 17:28:04 +01:00
Nicola Murino
45b9366dd0
add support for virtual folders
...
directories outside the user home directory can be exposed as virtual folders
2020-02-23 11:30:26 +01:00
Nicola Murino
ae8ed75ae5
gcs: add support for automatic credentials
...
We can now also support implicit credentials using the Application
Default Credentials strategy
2020-02-19 09:41:15 +01:00
Nicola Murino
58253968fc
s3: improve credentials validation
...
access secret can now be empty, so check if not empty before encrypting
the secret
2020-02-16 10:14:44 +01:00
Enes Çakır
da01848855
s3: enable shared config state
2020-02-16 09:06:25 +01:00
Nicola Murino
9359669cd4
s3: set upload concurrency to 2
...
we upload a file while receiving it via SFTP not a file stored on a local
disk. We use concurrent uploads only to be able to send files of arbitrary
size, so concurrency is not really useful here. Setting the concurrency to
2 we have a max difference of 10 MB between the writer (sftp client) and
the reader (aws sdk), with the default concurrency value this difference
is 25MB.
2020-02-04 23:14:55 +01:00
Nicola Murino
8b039e0447
httpd: add support for basic auth and HTTPS
2020-02-04 00:08:00 +01:00
Nicola Murino
3491717c26
add support for serving Google Cloud Storage over SFTP/SCP
...
Each user can be mapped with a Google Cloud Storage bucket or a bucket
virtual folder
2020-01-31 19:04:00 +01:00
Nicola Murino
5f4efc9148
S3: add metrics
2020-01-23 23:17:00 +01:00
Nicola Murino
4463421028
S3: add support for serving virtual folders
...
inside the same bucket each user can be assigned to a virtual folder.
This is similar to a chroot directory for local filesystem
2020-01-19 23:23:09 +01:00
Nicola Murino
d75f56b914
vfs: store root dir
...
so we don't need to pass it over and over
2020-01-19 13:58:55 +01:00
Nicola Murino
a4834f4a83
add basic S3-Compatible Object Storage support
...
we have now an interface for filesystem backeds, this make easy to add
new filesystem backends
2020-01-19 07:41:05 +01:00
