s3: set upload concurrency to 2
we upload a file while receiving it via SFTP not a file stored on a local disk. We use concurrent uploads only to be able to send files of arbitrary size, so concurrency is not really useful here. Setting the concurrency to 2 we have a max difference of 10 MB between the writer (sftp client) and the reader (aws sdk), with the default concurrency value this difference is 25MB.
This commit is contained in:
parent
8b039e0447
commit
9359669cd4
1 changed files with 2 additions and 0 deletions
|
@ -188,6 +188,8 @@ func (fs S3Fs) Create(name string, flag int) (*os.File, *pipeat.PipeWriterAt, fu
|
|||
Key: aws.String(key),
|
||||
Body: r,
|
||||
StorageClass: utils.NilIfEmpty(fs.config.StorageClass),
|
||||
}, func(u *s3manager.Uploader) {
|
||||
u.Concurrency = 2
|
||||
})
|
||||
r.CloseWithError(err)
|
||||
fsLog(fs, logger.LevelDebug, "upload completed, path: %#v, response: %v, readed bytes: %v, err: %v",
|
||||
|
|
Loading…
Add table
Reference in a new issue