beenull/sftpgo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add support for serving Google Cloud Storage over SFTP/SCP

Browse source 
Each user can be mapped with a Google Cloud Storage bucket or a bucket
virtual folder
This commit is contained in:
Nicola Murino 2020-01-31 19:04:00 +01:00
parent 45a13f5f4e
commit 3491717c26
33 changed files with 1632 additions and 165 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
README.md
View file

@ -22,7 +22,7 @@ Full featured and highly configurable SFTP server
- Atomic uploads are configurable.
- Support for Git repositories over SSH.
- SCP and rsync are supported.
- Support for serving S3 Compatible Object Storage over SFTP.
- Support for serving local filesystem, S3 Compatible Object Storage and Google Cloud Storage over SFTP/SCP.
- Prometheus metrics are exposed.
- REST API for users management, backup, restore and real time reports of the active connections with possibility of forcibly closing a connection.
- Web based interface to easily manage users and connections.
@ -178,6 +178,7 @@ The `sftpgo` configuration file contains the following sections:
- `http_notification_url`, a valid URL. Leave empty to disable.
- `external_auth_program`, string. Absolute path to an external program to use for users authentication. See the "External Authentication" paragraph for more details.
- `external_auth_scope`, integer. 0 means all supported authetication scopes (passwords, public keys and keyboard interactive). 1 means passwords only. 2 means public keys only. 4 means key keyboard interactive only. The flags can be combined, for example 6 means public keys and keyboard interactive
- `credentials_path`, string. It defines the directory for storing user provided credential files such as Google Cloud Storage credentials. This can be an absolute path or a path relative to the config dir
- **"httpd"**, the configuration for the HTTP server used to serve REST API
- `bind_port`, integer. The port used for serving HTTP requests. Set to 0 to disable HTTP server. Default: 8080
- `bind_address`, string. Leave blank to listen on all available network interfaces. Default: "127.0.0.1"
@ -232,7 +233,8 @@ Here is a full example showing the default config in JSON format:
"http_notification_url": ""
},
"external_auth_program": "",
"external_auth_scope": 0
"external_auth_scope": 0,
"credentials_path": "credentials"
},
"httpd": {
"bind_port": 8080,
@ -270,10 +272,12 @@ Please note that to override configuration options with environment variables a
### Data provider initialization
Before starting `sftpgo serve` a data provider must be configured.
Before starting `sftpgo serve` please ensure that the configured dataprovider is properly initialized.
SQL based data providers (SQLite, MySQL, PostgreSQL) requires the creation of a database containing the required tables. Memory and bolt data providers does not require an initialization.
SQL scripts to create the required database structure can be found inside the source tree [sql](./sql "sql") directory. The SQL scripts filename is, by convention, the date as `YYYYMMDD` and the suffix `.sql`. You need to apply all the SQL scripts for your database ordered by name, for example `20190828.sql` must be applied before `20191112.sql` and so on.
Example for `sqlite`: `find sql/sqlite/ -type f -iname '*.sql' -print |sort -n|xargs cat |sqlite3 sftpgo.db`
Example for `SQLite`: `find sql/sqlite/ -type f -iname '*.sql' -print | sort -n |xargs cat | sqlite3 sftpgo.db`
### Starting SFTGo in server mode
@ -482,13 +486,13 @@ The HTTP request has a 15 seconds timeout.
## S3 Compabible Object Storage backends
Each user can be mapped with an S3-Compatible bucket or a bucket virtual folder, this way the mapped bucket/virtual folder is exposed over SFTP/SCP.
Each user can be mapped to whole bucket or to a bucket virtual folder, this way the mapped bucket/virtual folder is exposed over SFTP/SCP.
Specifying a different `key_prefix` you can assign different virtual folders of the same bucket to different users. This is similar to a chroot directory for local filesystem. The virtual folder identified by `key_prefix` does not need to be pre-created.
Specifying a different `key_prefix` you can assign different virtual folders of the same bucket to different users. This is similar to a chroot directory for local filesystem. Each SFTP/SCP user can only access to the assigned virtual folder and to its contents The virtual folder identified by `key_prefix` does not need to be pre-created.
SFTPGo uses multipart uploads and parallel downloads for storing and retrieving files from S3.
SFTPGo tries to automatically create the mapped bucket if it does not exists but it's a better idea to pre-create the bucket and to assign to it the wanted options such as automatic encryption and authorizations.
The configured bucket must exist.
Some SFTP commands doesn't work over S3:
@ -504,6 +508,21 @@ Other notes:
- For server side encryption you have to configure the mapped bucket to automatically encrypt objects.
- A local home directory is still required to store temporary files.
## Google Cloud Storage backend
Each user can be mapped with a Google Cloud Storage bucket or a bucket virtual folder, this way the mapped bucket/virtual folder is exposed over SFTP/SCP. This backend is very similar to the S3 backend and it has the same limitations.
## Other Storage backends
Adding new storage backends it's quite easy:
- implement the [Fs interface](./vfs/vfs.go#L18 "interface for filesystem backends").
- update the user method `GetFilesystem` to return the new backend
- update the web interface and the REST API CLI
- add the flags for the new storage backed to the `portable` mode
Anyway some backends require a pay per use account (or they offer free account for a limited time period only), to be able to add support for such backends or to review pull requests please provide a test account. The test account must be available over the time to be able to maintain the backend and do basic tests before each new release.
## Portable mode
SFTPGo allows to share a single directory on demand using the `portable` subcommand:
@ -523,7 +542,11 @@ Flags:
-C, --advertise-credentials If the SFTP service is advertised via multicast DNS this flag allows to put username/password inside the advertised TXT record
-S, --advertise-service Advertise SFTP service using multicast DNS (default true)
-d, --directory string Path to the directory to serve. This can be an absolute path or a path relative to the current directory (default ".")
-f, --fs-provider int 0 means local filesystem, 1 S3 compatible
-f, --fs-provider int 0 means local filesystem, 1 Amazon S3 compatible, 2 Google Cloud Storage
--gcs-bucket string
--gcs-credentials-file string Google Cloud Storage JSON credentials file
--gcs-key-prefix string Allows to restrict access to the virtual folder identified by this prefix and its contents
--gcs-storage-class string
-h, --help help for portable
-l, --log-file-path string Leave empty to disable logging
-p, --password string Leave empty to use an auto generated value
@ -592,6 +615,10 @@ For each account the following properties can be configured:
- `s3_endpoint`, specifies s3 endpoint (server) different from AWS
- `s3_storage_class`
- `s3_key_prefix`, allows to restrict access to the virtual folder identified by this prefix and its contents
- `gcs_bucket`, required for GCS filesystem
- `gcs_credentials`, Google Cloud Storage JSON credentials base64 encoded
- `gcs_storage_class`
- `gcs_key_prefix`, allows to restrict access to the virtual folder identified by this prefix and its contents
These properties are stored inside the data provider.

40
cmd/portable.go
View file

@ -1,6 +1,10 @@
package cmd
import (
"encoding/base64"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"github.com/drakkan/sftpgo/dataprovider"
@ -29,6 +33,10 @@ var (
portableS3Endpoint string
portableS3StorageClass string
portableS3KeyPrefix string
portableGCSBucket string
portableGCSCredentialsFile string
portableGCSStorageClass string
portableGCSKeyPrefix string
portableCmd = &cobra.Command{
Use: "portable",
Short: "Serve a single directory",
@ -44,6 +52,24 @@ Please take a look at the usage below to customize the serving parameters`,
}
permissions := make(map[string][]string)
permissions["/"] = portablePermissions
portableGCSCredentials := ""
if portableFsProvider == 2 {
fi, err := os.Stat(portableGCSCredentialsFile)
if err != nil {
fmt.Printf("Invalid GCS credentials file: %v\n", err)
return
}
if fi.Size() > 1048576 {
fmt.Printf("Invalid GCS credentials file: %#v is too big %v/1048576 bytes\n", portableGCSCredentialsFile,
fi.Size())
return
}
creds, err := ioutil.ReadFile(portableGCSCredentialsFile)
if err != nil {
fmt.Printf("Unable to read credentials file: %v\n", err)
}
portableGCSCredentials = base64.StdEncoding.EncodeToString(creds)
}
service := service.Service{
ConfigDir: defaultConfigDir,
ConfigFile: defaultConfigName,
@ -73,6 +99,12 @@ Please take a look at the usage below to customize the serving parameters`,
StorageClass: portableS3StorageClass,
KeyPrefix: portableS3KeyPrefix,
},
GCSConfig: vfs.GCSFsConfig{
Bucket: portableGCSBucket,
Credentials: portableGCSCredentials,
StorageClass: portableGCSStorageClass,
KeyPrefix: portableGCSKeyPrefix,
},
},
},
}
@ -100,7 +132,8 @@ func init() {
"Advertise SFTP service using multicast DNS")
portableCmd.Flags().BoolVarP(&portableAdvertiseCredentials, "advertise-credentials", "C", false,
"If the SFTP service is advertised via multicast DNS this flag allows to put username/password inside the advertised TXT record")
portableCmd.Flags().IntVarP(&portableFsProvider, "fs-provider", "f", 0, "0 means local filesystem, 1 S3 compatible")
portableCmd.Flags().IntVarP(&portableFsProvider, "fs-provider", "f", 0, "0 means local filesystem, 1 Amazon S3 compatible, "+
"2 Google Cloud Storage")
portableCmd.Flags().StringVar(&portableS3Bucket, "s3-bucket", "", "")
portableCmd.Flags().StringVar(&portableS3Region, "s3-region", "", "")
portableCmd.Flags().StringVar(&portableS3AccessKey, "s3-access-key", "", "")
@ -109,5 +142,10 @@ func init() {
portableCmd.Flags().StringVar(&portableS3StorageClass, "s3-storage-class", "", "")
portableCmd.Flags().StringVar(&portableS3KeyPrefix, "s3-key-prefix", "", "Allows to restrict access to the virtual folder "+
"identified by this prefix and its contents")
portableCmd.Flags().StringVar(&portableGCSBucket, "gcs-bucket", "", "")
portableCmd.Flags().StringVar(&portableGCSStorageClass, "gcs-storage-class", "", "")
portableCmd.Flags().StringVar(&portableGCSKeyPrefix, "gcs-key-prefix", "", "Allows to restrict access to the virtual folder "+
"identified by this prefix and its contents")
portableCmd.Flags().StringVar(&portableGCSCredentialsFile, "gcs-credentials-file", "", "Google Cloud Storage JSON credentials file")
rootCmd.AddCommand(portableCmd)
}

7
config/config.go
View file

@ -84,6 +84,7 @@ func init() {
},
ExternalAuthProgram: "",
ExternalAuthScope: 0,
CredentialsPath: "credentials",
},
HTTPDConfig: httpd.Conf{
BindPort: 8080,
@ -178,6 +179,12 @@ func LoadConfig(configDir, configName string) error {
logger.Warn(logSender, "", "Configuration error: %v", err)
logger.WarnToConsole("Configuration error: %v", err)
}
if len(globalConf.ProviderConf.CredentialsPath) == 0 {
err = fmt.Errorf("invalid credentials path, reset to \"credentials\"")
globalConf.ProviderConf.CredentialsPath = "credentials"
logger.Warn(logSender, "", "Configuration error: %v", err)
logger.WarnToConsole("Configuration error: %v", err)
}
logger.Debug(logSender, "", "config file used: '%v', config loaded: %+v", viper.ConfigFileUsed(), getRedactedGlobalConf())
return err
}

21
config/config_test.go
View file

@ -119,6 +119,27 @@ func TestInvalidExternalAuthScope(t *testing.T) {
os.Remove(configFilePath)
}
func TestInvalidCredentialsPath(t *testing.T) {
configDir := ".."
confName := tempConfigName + ".json"
configFilePath := filepath.Join(configDir, confName)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
providerConf.CredentialsPath = ""
c := make(map[string]dataprovider.Config)
c["data_provider"] = providerConf
jsonConf, _ := json.Marshal(c)
err := ioutil.WriteFile(configFilePath, jsonConf, 0666)
if err != nil {
t.Errorf("error saving temporary configuration")
}
err = config.LoadConfig(configDir, tempConfigName)
if err == nil {
t.Errorf("Loading configuration with credentials path must fail")
}
os.Remove(configFilePath)
}
func TestSetGetConfig(t *testing.T) {
sftpdConf := config.GetSFTPDConfig()
sftpdConf.IdleTimeout = 3

4
dataprovider/bolt.go
View file

@ -319,6 +319,10 @@ func (p BoltProvider) dumpUsers() ([]User, error) {
if err != nil {
return err
}
err = addCredentialsToUser(&user)
if err != nil {
return err
}
users = append(users, user)
}
return err

71
dataprovider/dataprovider.go
View file

@ -16,6 +16,7 @@ import (
"errors"
"fmt"
"hash"
"io/ioutil"
"net"
"net/http"
"net/url"
@ -85,6 +86,7 @@ var (
availabilityTicker *time.Ticker
availabilityTickerDone chan bool
errWrongPassword = errors.New("password does not match")
credentialsDirPath string
)
// Actions to execute on user create, update, delete.
@ -179,6 +181,10 @@ type Config struct {
// you can combine the scopes, for example 3 means password and public key, 5 password and keyboard
// interactive and so on
ExternalAuthScope int `json:"external_auth_scope" mapstructure:"external_auth_scope"`
// CredentialsPath defines the directory for storing user provided credential files such as
// Google Cloud Storage credentials. It can be a path relative to the config dir or an
// absolute path
CredentialsPath string `json:"credentials_path" mapstructure:"credentials_path"`
}
type keyboardAuthProgramResponse struct {
@ -268,6 +274,9 @@ func Initialize(cnf Config, basePath string) error {
return err
}
}
if err := validateCredentialsDir(basePath); err != nil {
return err
}
if config.Driver == SQLiteDataProviderName {
err = initializeSQLiteProvider(basePath)
@ -509,6 +518,25 @@ func validateFilters(user *User) error {
return nil
}
func saveGCSCredentials(user *User) error {
if user.FsConfig.Provider != 2 {
return nil
}
if len(user.FsConfig.GCSConfig.Credentials) == 0 {
return nil
}
decoded, err := base64.StdEncoding.DecodeString(user.FsConfig.GCSConfig.Credentials)
if err != nil {
return &ValidationError{err: fmt.Sprintf("could not validate GCS credentials: %v", err)}
}
err = ioutil.WriteFile(user.getGCSCredentialsFilePath(), decoded, 0600)
if err != nil {
return &ValidationError{err: fmt.Sprintf("could not save GCS credentials: %v", err)}
}
user.FsConfig.GCSConfig.Credentials = ""
return nil
}
func validateFilesystemConfig(user *User) error {
if user.FsConfig.Provider == 1 {
err := vfs.ValidateS3FsConfig(&user.FsConfig.S3Config)
@ -524,9 +552,16 @@ func validateFilesystemConfig(user *User) error {
user.FsConfig.S3Config.AccessSecret = accessSecret
}
return nil
} else if user.FsConfig.Provider == 2 {
err := vfs.ValidateGCSFsConfig(&user.FsConfig.GCSConfig, user.getGCSCredentialsFilePath())
if err != nil {
return &ValidationError{err: fmt.Sprintf("could not validate GCS config: %v", err)}
}
return nil
}
user.FsConfig.Provider = 0
user.FsConfig.S3Config = vfs.S3FsConfig{}
user.FsConfig.GCSConfig = vfs.GCSFsConfig{}
return nil
}
@ -563,6 +598,9 @@ func validateUser(user *User) error {
if err := validateFilters(user); err != nil {
return err
}
if err := saveGCSCredentials(user); err != nil {
return err
}
return nil
}
@ -704,10 +742,24 @@ func HideUserSensitiveData(user *User) User {
user.Password = ""
if user.FsConfig.Provider == 1 {
user.FsConfig.S3Config.AccessSecret = utils.RemoveDecryptionKey(user.FsConfig.S3Config.AccessSecret)
} else if user.FsConfig.Provider == 2 {
user.FsConfig.GCSConfig.Credentials = ""
}
return *user
}
func addCredentialsToUser(user *User) error {
if user.FsConfig.Provider != 2 {
return nil
}
cred, err := ioutil.ReadFile(user.getGCSCredentialsFilePath())
if err != nil {
return err
}
user.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString(cred)
return nil
}
func getSSLMode() string {
if config.Driver == PGSQLDataProviderName {
if config.SSLMode == 0 {
@ -748,6 +800,25 @@ func startAvailabilityTimer() {
}()
}
func validateCredentialsDir(basePath string) error {
if filepath.IsAbs(config.CredentialsPath) {
credentialsDirPath = config.CredentialsPath
} else {
credentialsDirPath = filepath.Join(basePath, config.CredentialsPath)
}
fi, err := os.Stat(credentialsDirPath)
if err == nil {
if !fi.IsDir() {
return errors.New("Credential path is not a valid directory")
}
return nil
}
if !os.IsNotExist(err) {
return err
}
return os.MkdirAll(credentialsDirPath, 0700)
}
func checkDataprovider() {
err := provider.checkAvailability()
if err != nil {

4
dataprovider/memory.go
View file

@ -224,6 +224,10 @@ func (p MemoryProvider) dumpUsers() ([]User, error) {
}
for _, username := range p.dbHandle.usernames {
user := p.dbHandle.users[username]
err = addCredentialsToUser(&user)
if err != nil {
return users, err
}
users = append(users, user)
}
return users, err

11
dataprovider/sqlcommon.go
View file

@ -232,11 +232,14 @@ func sqlCommonDumpUsers(dbHandle *sql.DB) ([]User, error) {
defer rows.Close()
for rows.Next() {
u, err := getUserFromDbRow(nil, rows)
if err == nil {
users = append(users, u)
} else {
break
if err != nil {
return users, err
}
err = addCredentialsToUser(&u)
if err != nil {
return users, err
}
users = append(users, u)
}
}

26
dataprovider/user.go
View file

@ -55,9 +55,10 @@ type UserFilters struct {
// Filesystem defines cloud storage filesystem details
type Filesystem struct {
// 0 local filesystem, 1 Amazon S3 compatible
// 0 local filesystem, 1 Amazon S3 compatible, 2 Google Cloud Storage
Provider int `json:"provider"`
S3Config vfs.S3FsConfig `json:"s3config,omitempty"`
GCSConfig vfs.GCSFsConfig `json:"gcsconfig,omitempty"`
}
// User defines an SFTP user
@ -73,7 +74,7 @@ type User struct {
ExpirationDate int64 `json:"expiration_date"`
// Password used for password authentication.
// For users created using SFTPGo REST API the password is be stored using argon2id hashing algo.
// Checking passwords stored with bcrypt, pbkdf2 and sha512crypt is supported too.
// Checking passwords stored with bcrypt, pbkdf2, md5crypt and sha512crypt is supported too.
Password string `json:"password,omitempty"`
// PublicKeys used for public key authentication. At least one between password and a public key is mandatory
PublicKeys []string `json:"public_keys,omitempty"`
@ -113,6 +114,10 @@ type User struct {
func (u *User) GetFilesystem(connectionID string) (vfs.Fs, error) {
if u.FsConfig.Provider == 1 {
return vfs.NewS3Fs(connectionID, u.GetHomeDir(), u.FsConfig.S3Config)
} else if u.FsConfig.Provider == 2 {
config := u.FsConfig.GCSConfig
config.CredentialFile = u.getGCSCredentialsFilePath()
return vfs.NewGCSFs(connectionID, u.GetHomeDir(), config)
}
return vfs.NewOsFs(connectionID, u.GetHomeDir()), nil
}
@ -321,7 +326,8 @@ func (u *User) GetBandwidthAsString() string {
}
// GetInfoString returns user's info as string.
// Number of public keys, max sessions, uid and gid are returned
// Storage provider, number of public keys, max sessions, uid,
// gid, denied and allowed IP/Mask are returned
func (u *User) GetInfoString() string {
var result string
if u.LastLogin > 0 {
@ -329,7 +335,9 @@ func (u *User) GetInfoString() string {
result += fmt.Sprintf("Last login: %v ", t.Format("2006-01-02 15:04:05")) // YYYY-MM-DD HH:MM:SS
}
if u.FsConfig.Provider == 1 {
result += fmt.Sprintf("Storage: S3")
result += fmt.Sprintf("Storage: S3 ")
} else if u.FsConfig.Provider == 2 {
result += fmt.Sprintf("Storage: GCS ")
}
if len(u.PublicKeys) > 0 {
result += fmt.Sprintf("Public keys: %v ", len(u.PublicKeys))
@ -410,6 +418,12 @@ func (u *User) getACopy() User {
StorageClass: u.FsConfig.S3Config.StorageClass,
KeyPrefix: u.FsConfig.S3Config.KeyPrefix,
},
GCSConfig: vfs.GCSFsConfig{
Bucket: u.FsConfig.GCSConfig.Bucket,
CredentialFile: u.FsConfig.GCSConfig.CredentialFile,
StorageClass: u.FsConfig.GCSConfig.StorageClass,
KeyPrefix: u.FsConfig.GCSConfig.KeyPrefix,
},
}
return User{
@ -458,3 +472,7 @@ func (u *User) getNotificationFieldsAsEnvVars(action string) []string {
fmt.Sprintf("SFTPGO_USER_UID=%v", u.UID),
fmt.Sprintf("SFTPGO_USER_GID=%v", u.GID)}
}
func (u *User) getGCSCredentialsFilePath() string {
return filepath.Join(credentialsDirPath, fmt.Sprintf("%v_gcs_credentials.json", u.Username))
}

1
docker/sftpgo/alpine/README.md
View file

@ -12,6 +12,7 @@ sudo groupadd -g 1003 sftpgrp && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20190828.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20191112.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20191230.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20200116.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sftpgo.json -o /home/sftpuser/conf/sftpgo.json
# Get and build SFTPGo image

9
go.mod
View file

@ -3,6 +3,8 @@ module github.com/drakkan/sftpgo
go 1.13
require (
cloud.google.com/go v0.52.0 // indirect
cloud.google.com/go/storage v1.5.0
github.com/alexedwards/argon2id v0.0.0-20190612080829-01a59b2b8802
github.com/aws/aws-sdk-go v1.28.3
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
@ -10,6 +12,7 @@ require (
github.com/go-chi/chi v4.0.2+incompatible
github.com/go-chi/render v1.0.1
github.com/go-sql-driver/mysql v1.5.0
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/grandcat/zeroconf v0.0.0-20190424104450-85eadb44205c
github.com/lib/pq v1.3.0
github.com/mattn/go-sqlite3 v2.0.2+incompatible
@ -23,7 +26,11 @@ require (
github.com/spf13/viper v1.6.1
go.etcd.io/bbolt v1.3.3
golang.org/x/crypto v0.0.0-20200109152110-61a87790db17
golang.org/x/sys v0.0.0-20191220142924-d4481acd189f
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a // indirect
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82
golang.org/x/tools v0.0.0-20200124200720-1b668f209185 // indirect
google.golang.org/api v0.15.0
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.0.0
)

183
go.sum
View file

@ -1,5 +1,26 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0 h1:0E3eE8MX426vUOs7aHfI7aN1BrIzzzf4ccKCSfSjGmc=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0 h1:GGslhk/BU052LPlnI1vpp3fcbUs+hQ3E+Doti/3/vF8=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0 h1:RPUcBvDeYgQFMfQu1eBMq6piD1SXmLH+vK3qjewZPus=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
@ -16,10 +37,14 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4=
github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
@ -34,6 +59,8 @@ github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZm
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/drakkan/pipeat v0.0.0-20200123131427-11c048cfc0ec h1:DXfzg1NXoesnFzdCyyi2uU3o1o0XiWTN2ZcpWDE7MCk=
github.com/drakkan/pipeat v0.0.0-20200123131427-11c048cfc0ec/go.mod h1:wNYvIpR5rIhoezOYcpxcXz4HbIEOu7A45EqlQCA+h+w=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
@ -41,6 +68,8 @@ github.com/go-chi/chi v4.0.2+incompatible h1:maB6vn6FqCxrpz4FqWdh4+lwpyZIQS7YEAU
github.com/go-chi/chi v4.0.2+incompatible/go.mod h1:eB3wogJHnLi3x/kFX2A+IbTBlXxmMeXJVKy9tTv1XzQ=
github.com/go-chi/render v1.0.1 h1:4/5tis2cKaNdnv9zFLfXzcquC9HbeZgCnxGnKrltBS8=
github.com/go-chi/render v1.0.1/go.mod h1:pq4Rr7HbnsdaeHagklXub+p6Wd16Af5l9koip1OvJns=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
@ -52,15 +81,35 @@ github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7a
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/grandcat/zeroconf v0.0.0-20190424104450-85eadb44205c h1:svzQzfVE9t7Y1CGULS5PsMWs4/H4Au/ZTJzU/0CKgqc=
@ -68,14 +117,20 @@ github.com/grandcat/zeroconf v0.0.0-20190424104450-85eadb44205c/go.mod h1:YjKB0W
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1 h1:6QPYqodiu3GuPL+7mfx+NwDdp2eTkp9IfEUpgAwUN0o=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
@ -124,6 +179,7 @@ github.com/prometheus/client_golang v1.3.0 h1:miYCvYqFXtl/J9FIy8eNpBfYthAEFg+Ys0
github.com/prometheus/client_golang v1.3.0/go.mod h1:hJaj2vgQTGQmVCsAACORcieXFeDPbaTKGT+JTgUa3og=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.1.0 h1:ElTg5tNp4DqfV7UQjDqv2+RJlNzsDtvNAWccbItceIE=
github.com/prometheus/client_model v0.1.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
@ -138,6 +194,7 @@ github.com/prometheus/procfs v0.0.8 h1:+fpWZdT24pJBiqJdAwYBjPSk+5YmQzYNPYzQsdzLk
github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
github.com/rs/zerolog v1.17.2 h1:RMRHFw2+wF7LO0QqtELQwo8hqSmqISyCJeFeAAuWcRo=
@ -178,34 +235,80 @@ github.com/zenazn/goji v0.9.0/go.mod h1:7S9M489iMyHBNxwZnk9/EHS098H4/F6TATF2mIxt
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.3 h1:MUGmc65QhB3pIlaQ5bB4LwqSj6GIonVJXpZiaKNyaKk=
go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2 h1:75k/FF0Q2YM8QYo07VPddOLBslDt1MZOdEslOHvmzAs=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200109152110-61a87790db17 h1:nVJ3guKA9qdkEQ3TUdXI9QSINo2CUPM/cySEvw2w8I0=
golang.org/x/crypto v0.0.0-20200109152110-61a87790db17/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299 h1:zQpM52jfKHG6II1ISZY1ZcpygvuSFZpLwfluuF89XOg=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a h1:7Wlg8L54In96HTWOaI4sreLJ6qfyGuvSau5el3fK41Y=
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f h1:J5lckAjkw6qYlOZNj90mLYNTEKDvWeuc1yieZ8qUzUE=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478 h1:l5EDrHhldLYb3ZRHDUhXF7Om7MvYXnkV9/iQNo1lX6g=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 h1:efeOvDhwQ29Dj3SdAV/MJf8oukgn+8D8WgaCaRMchF8=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa h1:F+8P+gmewFQYRk6JoLQLwjBCTu3mcIURZfNkVweuRKA=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 h1:pE8b58s1HRDMi8RDc79m0HISf9D4TzseP40cEA6IGfs=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@ -214,29 +317,103 @@ golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191220142924-d4481acd189f h1:68K/z8GLUxV76xGSqwTWw2gyk/jwn79LUL43rES2g8o=
golang.org/x/sys v0.0.0-20191220142924-d4481acd189f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8 h1:JA8d3MPx/IToSyXZG/RhwYEtfrKO1Fxrqe8KrkiLXKM=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82 h1:ywK/j/KkyTHcdyYSZNXGjMwgmDSfjglYZ3vStQ/gSCU=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190828213141-aed303cbaa74/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4 h1:Toz2IK7k8rbltAXwNAxKcn9OzqyNfMUhUNjz3sL0NMk=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200124200720-1b668f209185 h1:UhyNb/h6VU6sPOVb6x118tYR91HRLBUtWS01bOSFAz4=
golang.org/x/tools v0.0.0-20200124200720-1b668f209185/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.15.0 h1:yzlyyDW/J0w8yNFJIhiAJy4kq74S+1DOLdawELNxFMA=
google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb h1:ADPHZzpzM4tk4V4S5cnCrr5SwzvlrPRmqqCuJDB8UTs=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150 h1:VPpdpQkGvFicX9yo4G5oxZPi9ALBnEOZblPSa/Wa2m4=
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.26.0 h1:2dTRdpdFEEhJYQD8EMLB61nnrzSCTbG38PhqdhvOltg=
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/ini.v1 v1.51.0 h1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=
gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
@ -248,3 +425,9 @@ gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=

1
httpd/api_quota.go
View file

@ -14,6 +14,7 @@ func getQuotaScans(w http.ResponseWriter, r *http.Request) {
}
func startQuotaScan(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
var u dataprovider.User
err := render.DecodeJSON(r.Body, &u)
if err != nil {

14
httpd/api_user.go
View file

@ -73,6 +73,7 @@ func getUserByID(w http.ResponseWriter, r *http.Request) {
}
func addUser(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
var user dataprovider.User
err := render.DecodeJSON(r.Body, &user)
if err != nil {
@ -93,6 +94,7 @@ func addUser(w http.ResponseWriter, r *http.Request) {
}
func updateUser(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
userID, err := strconv.ParseInt(chi.URLParam(r, "userID"), 10, 64)
if err != nil {
err = errors.New("Invalid userID")
@ -100,10 +102,10 @@ func updateUser(w http.ResponseWriter, r *http.Request) {
return
}
user, err := dataprovider.GetUserByID(dataProvider, userID)
oldPermissions := user.Permissions
oldS3AccessSecret := ""
currentPermissions := user.Permissions
currentS3AccessSecret := ""
if user.FsConfig.Provider == 1 {
oldS3AccessSecret = user.FsConfig.S3Config.AccessSecret
currentS3AccessSecret = user.FsConfig.S3Config.AccessSecret
}
user.Permissions = make(map[string][]string)
if _, ok := err.(*dataprovider.RecordNotFoundError); ok {
@ -120,13 +122,13 @@ func updateUser(w http.ResponseWriter, r *http.Request) {
}
// we use new Permissions if passed otherwise the old ones
if len(user.Permissions) == 0 {
user.Permissions = oldPermissions
user.Permissions = currentPermissions
}
// we use the new access secret if different from the old one and not empty
if user.FsConfig.Provider == 1 {
if utils.RemoveDecryptionKey(oldS3AccessSecret) == user.FsConfig.S3Config.AccessSecret ||
if utils.RemoveDecryptionKey(currentS3AccessSecret) == user.FsConfig.S3Config.AccessSecret ||
len(user.FsConfig.S3Config.AccessSecret) == 0 {
user.FsConfig.S3Config.AccessSecret = oldS3AccessSecret
user.FsConfig.S3Config.AccessSecret = currentS3AccessSecret
}
}
if user.ID != userID {

10
httpd/api_utils.go
View file

@ -439,6 +439,16 @@ func compareUserFsConfig(expected *dataprovider.User, actual *dataprovider.User)
expected.FsConfig.S3Config.KeyPrefix+"/" != actual.FsConfig.S3Config.KeyPrefix {
return errors.New("S3 key prefix mismatch")
}
if expected.FsConfig.GCSConfig.Bucket != actual.FsConfig.GCSConfig.Bucket {
return errors.New("GCS bucket mismatch")
}
if expected.FsConfig.GCSConfig.StorageClass != actual.FsConfig.GCSConfig.StorageClass {
return errors.New("GCS storage class mismatch")
}
if expected.FsConfig.GCSConfig.KeyPrefix != actual.FsConfig.GCSConfig.KeyPrefix &&
expected.FsConfig.GCSConfig.KeyPrefix+"/" != actual.FsConfig.GCSConfig.KeyPrefix {
return errors.New("GCS key prefix mismatch")
}
return nil
}

1
httpd/httpd.go
View file

@ -33,6 +33,7 @@ const (
webConnectionsPath = "/web/connections"
webStaticFilesPath = "/static"
maxRestoreSize = 10485760 // 10 MB
maxRequestSize = 1048576 // 1MB
)
var (

325
httpd/httpd_test.go
View file

@ -3,9 +3,12 @@ package httpd_test
import (
"bytes"
"crypto/rand"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"mime/multipart"
"net"
"net/http"
"net/http/httptest"
@ -56,6 +59,7 @@ var (
defaultPerms = []string{dataprovider.PermAny}
homeBasePath string
backupsPath string
credentialsPath string
testServer *httptest.Server
providerDriverName string
)
@ -66,7 +70,10 @@ func TestMain(m *testing.M) {
logger.InitLogger(logfilePath, 5, 1, 28, false, zerolog.DebugLevel)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
credentialsPath = filepath.Join(os.TempDir(), "test_credentials")
providerConf.CredentialsPath = credentialsPath
providerDriverName = providerConf.Driver
os.RemoveAll(credentialsPath)
err := dataprovider.Initialize(providerConf, configDir)
if err != nil {
@ -102,6 +109,7 @@ func TestMain(m *testing.M) {
exitCode := m.Run()
os.Remove(logfilePath)
os.RemoveAll(backupsPath)
os.RemoveAll(credentialsPath)
os.Exit(exitCode)
}
@ -250,6 +258,8 @@ func TestAddUserInvalidFsConfig(t *testing.T) {
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
os.RemoveAll(credentialsPath)
os.MkdirAll(credentialsPath, 0700)
u.FsConfig.S3Config.Bucket = "test"
u.FsConfig.S3Config.Region = "eu-west-1"
u.FsConfig.S3Config.AccessKey = "access-key"
@ -261,6 +271,32 @@ func TestAddUserInvalidFsConfig(t *testing.T) {
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
u = getTestUser()
u.FsConfig.Provider = 2
u.FsConfig.GCSConfig.Bucket = ""
_, _, err = httpd.AddUser(u, http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
u.FsConfig.GCSConfig.Bucket = "test"
u.FsConfig.GCSConfig.StorageClass = "Standard"
u.FsConfig.GCSConfig.KeyPrefix = "/somedir/subdir/"
u.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString([]byte("test"))
_, _, err = httpd.AddUser(u, http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
u.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir/"
u.FsConfig.GCSConfig.Credentials = ""
_, _, err = httpd.AddUser(u, http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
u.FsConfig.GCSConfig.Credentials = "no base64 encoded"
_, _, err = httpd.AddUser(u, http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
}
func TestUserPublicKey(t *testing.T) {
@ -363,6 +399,56 @@ func TestUserS3Config(t *testing.T) {
}
}
func TestUserGCSConfig(t *testing.T) {
user, _, err := httpd.AddUser(getTestUser(), http.StatusOK)
if err != nil {
t.Errorf("unable to add user: %v", err)
}
os.RemoveAll(credentialsPath)
os.MkdirAll(credentialsPath, 0700)
user.FsConfig.Provider = 2
user.FsConfig.GCSConfig.Bucket = "test"
user.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString([]byte("fake credentials"))
user, _, err = httpd.UpdateUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to update user: %v", err)
}
_, err = httpd.RemoveUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to remove: %v", err)
}
user.Password = defaultPassword
user.ID = 0
// the user will be added since the credentials file is found
user, _, err = httpd.AddUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to add user: %v", err)
}
user.FsConfig.Provider = 1
user.FsConfig.S3Config.Bucket = "test1"
user.FsConfig.S3Config.Region = "us-east-1"
user.FsConfig.S3Config.AccessKey = "Server-Access-Key1"
user.FsConfig.S3Config.AccessSecret = "secret"
user.FsConfig.S3Config.Endpoint = "http://localhost:9000"
user.FsConfig.S3Config.KeyPrefix = "somedir/subdir"
user, _, err = httpd.UpdateUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to update user: %v", err)
}
user.FsConfig.Provider = 2
user.FsConfig.GCSConfig.Bucket = "test1"
user.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString([]byte("fake credentials"))
user, _, err = httpd.UpdateUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to update user: %v", err)
}
_, err = httpd.RemoveUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to remove: %v", err)
}
}
func TestUpdateUserNoCredentials(t *testing.T) {
user, _, err := httpd.AddUser(getTestUser(), http.StatusOK)
if err != nil {
@ -594,6 +680,8 @@ func TestUserBaseDir(t *testing.T) {
dataprovider.Close(dataProvider)
config.LoadConfig(configDir, "")
providerConf = config.GetProviderConf()
providerConf.CredentialsPath = credentialsPath
os.RemoveAll(credentialsPath)
err = dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
@ -646,6 +734,8 @@ func TestProviderErrors(t *testing.T) {
os.Remove(backupFilePath)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
providerConf.CredentialsPath = credentialsPath
os.RemoveAll(credentialsPath)
err = dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
@ -655,7 +745,17 @@ func TestProviderErrors(t *testing.T) {
}
func TestDumpdata(t *testing.T) {
_, _, err := httpd.Dumpdata("", http.StatusBadRequest)
dataProvider := dataprovider.GetProvider()
dataprovider.Close(dataProvider)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
err := dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
}
httpd.SetDataProvider(dataprovider.GetProvider())
sftpd.SetDataProvider(dataprovider.GetProvider())
_, _, err = httpd.Dumpdata("", http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error: %v", err)
}
@ -680,6 +780,15 @@ func TestDumpdata(t *testing.T) {
}
os.Chmod(backupsPath, 0755)
}
providerConf = config.GetProviderConf()
providerConf.CredentialsPath = credentialsPath
os.RemoveAll(credentialsPath)
err = dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
}
httpd.SetDataProvider(dataprovider.GetProvider())
sftpd.SetDataProvider(dataprovider.GetProvider())
}
func TestLoaddata(t *testing.T) {
@ -1228,16 +1337,23 @@ func TestBasicWebUsersMock(t *testing.T) {
checkResponseCode(t, http.StatusBadRequest, rr.Code)
form := make(url.Values)
form.Set("username", user.Username)
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
b, contentType, _ := getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), strings.NewReader(form.Encode()))
b, contentType, _ = getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/0", strings.NewReader(form.Encode()))
b, contentType, _ = getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/0", &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusNotFound, rr.Code)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/a", strings.NewReader(form.Encode()))
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/a", &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusBadRequest, rr.Code)
req, _ = http.NewRequest(http.MethodDelete, userPath+"/"+strconv.FormatInt(user.ID, 10), nil)
@ -1261,90 +1377,103 @@ func TestWebUserAddMock(t *testing.T) {
form.Set("expiration_date", "")
form.Set("permissions", "*")
form.Set("sub_dirs_permissions", " /subdir:list ,download ")
b, contentType, _ := getMultipartFormData(form, "", "")
// test invalid url escape
req, _ := http.NewRequest(http.MethodPost, webUserPath+"?a=%2", strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ := http.NewRequest(http.MethodPost, webUserPath+"?a=%2", &b)
req.Header.Set("Content-Type", contentType)
rr := executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("public_keys", testPubKey)
form.Set("uid", strconv.FormatInt(int64(user.UID), 10))
form.Set("gid", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid gid
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("gid", "0")
form.Set("max_sessions", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid max sessions
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("max_sessions", "0")
form.Set("quota_size", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid quota size
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("quota_size", "0")
form.Set("quota_files", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid quota files
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("quota_files", "0")
form.Set("upload_bandwidth", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid upload bandwidth
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("upload_bandwidth", strconv.FormatInt(user.UploadBandwidth, 10))
form.Set("download_bandwidth", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid download bandwidth
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("download_bandwidth", strconv.FormatInt(user.DownloadBandwidth, 10))
form.Set("status", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid status
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("status", strconv.Itoa(user.Status))
form.Set("expiration_date", "123")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid expiration date
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("expiration_date", "")
form.Set("allowed_ip", "invalid,ip")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid allowed_ip
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("allowed_ip", "")
form.Set("denied_ip", "192.168.1.2") // it should be 192.168.1.2/32
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid denied_ip
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("denied_ip", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
b, contentType, _ = getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusSeeOther, rr.Code)
// the user already exists, was created with the above request
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
b, contentType, _ = getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
@ -1356,7 +1485,7 @@ func TestWebUserAddMock(t *testing.T) {
t.Errorf("Error decoding users: %v", err)
}
if len(users) != 1 {
t.Errorf("1 user is expected")
t.Errorf("1 user is expected, actual: %v", len(users))
}
newUser := users[0]
if newUser.UID != user.UID {
@ -1413,8 +1542,9 @@ func TestWebUserUpdateMock(t *testing.T) {
form.Set("expiration_date", "2020-01-01 00:00:00")
form.Set("allowed_ip", " 192.168.1.3/32, 192.168.2.0/24 ")
form.Set("denied_ip", " 10.0.0.2/32 ")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
b, contentType, _ := getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusSeeOther, rr.Code)
req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
@ -1504,8 +1634,9 @@ func TestWebUserS3Mock(t *testing.T) {
form.Set("s3_storage_class", user.FsConfig.S3Config.StorageClass)
form.Set("s3_endpoint", user.FsConfig.S3Config.Endpoint)
form.Set("s3_key_prefix", user.FsConfig.S3Config.KeyPrefix)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
b, contentType, _ := getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusSeeOther, rr.Code)
req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
@ -1552,6 +1683,97 @@ func TestWebUserS3Mock(t *testing.T) {
checkResponseCode(t, http.StatusOK, rr.Code)
}
func TestWebUserGCSMock(t *testing.T) {
user := getTestUser()
userAsJSON := getUserAsJSON(t, user)
req, _ := http.NewRequest(http.MethodPost, userPath, bytes.NewBuffer(userAsJSON))
rr := executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
err := render.DecodeJSON(rr.Body, &user)
if err != nil {
t.Errorf("Error get user: %v", err)
}
credentialsFilePath := filepath.Join(os.TempDir(), "gcs.json")
err = createTestFile(credentialsFilePath, 0)
if err != nil {
t.Errorf("unable to create credential test file: %v", err)
}
user.FsConfig.Provider = 2
user.FsConfig.GCSConfig.Bucket = "test"
user.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir/"
user.FsConfig.GCSConfig.StorageClass = "standard"
form := make(url.Values)
form.Set("username", user.Username)
form.Set("home_dir", user.HomeDir)
form.Set("uid", "0")
form.Set("gid", strconv.FormatInt(int64(user.GID), 10))
form.Set("max_sessions", strconv.FormatInt(int64(user.MaxSessions), 10))
form.Set("quota_size", strconv.FormatInt(user.QuotaSize, 10))
form.Set("quota_files", strconv.FormatInt(int64(user.QuotaFiles), 10))
form.Set("upload_bandwidth", "0")
form.Set("download_bandwidth", "0")
form.Set("permissions", "*")
form.Set("sub_dirs_permissions", "")
form.Set("status", strconv.Itoa(user.Status))
form.Set("expiration_date", "2020-01-01 00:00:00")
form.Set("allowed_ip", "")
form.Set("denied_ip", "")
form.Set("fs_provider", "2")
form.Set("gcs_bucket", user.FsConfig.GCSConfig.Bucket)
form.Set("gcs_storage_class", user.FsConfig.GCSConfig.StorageClass)
form.Set("gcs_key_prefix", user.FsConfig.GCSConfig.KeyPrefix)
b, contentType, _ := getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
b, contentType, _ = getMultipartFormData(form, "gcs_credential_file", credentialsFilePath)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
err = createTestFile(credentialsFilePath, 4096)
if err != nil {
t.Errorf("unable to create credential test file: %v", err)
}
b, contentType, _ = getMultipartFormData(form, "gcs_credential_file", credentialsFilePath)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusSeeOther, rr.Code)
req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
var users []dataprovider.User
err = render.DecodeJSON(rr.Body, &users)
if err != nil {
t.Errorf("Error decoding users: %v", err)
}
if len(users) != 1 {
t.Errorf("1 user is expected")
}
updateUser := users[0]
if updateUser.ExpirationDate != 1577836800000 {
t.Errorf("invalid expiration date: %v", updateUser.ExpirationDate)
}
if updateUser.FsConfig.Provider != user.FsConfig.Provider {
t.Error("fs provider mismatch")
}
if updateUser.FsConfig.GCSConfig.Bucket != user.FsConfig.GCSConfig.Bucket {
t.Error("GCS bucket mismatch")
}
if updateUser.FsConfig.GCSConfig.StorageClass != user.FsConfig.GCSConfig.StorageClass {
t.Error("GCS storage class mismatch")
}
if updateUser.FsConfig.GCSConfig.KeyPrefix != user.FsConfig.GCSConfig.KeyPrefix {
t.Error("GCS key prefix mismatch")
}
req, _ = http.NewRequest(http.MethodDelete, userPath+"/"+strconv.FormatInt(user.ID, 10), nil)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
os.Remove(credentialsFilePath)
}
func TestProviderClosedMock(t *testing.T) {
if providerDriverName == dataprovider.BoltDataProviderName {
t.Skip("skipping test provider errors for bolt provider")
@ -1571,6 +1793,8 @@ func TestProviderClosedMock(t *testing.T) {
checkResponseCode(t, http.StatusInternalServerError, rr.Code)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
providerConf.CredentialsPath = credentialsPath
os.RemoveAll(credentialsPath)
err := dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
@ -1644,9 +1868,38 @@ func createTestFile(path string, size int64) error {
os.MkdirAll(baseDir, 0777)
}
content := make([]byte, size)
if size > 0 {
_, err := rand.Read(content)
if err != nil {
return err
}
}
return ioutil.WriteFile(path, content, 0666)
}
func getMultipartFormData(values url.Values, fileFieldName, filePath string) (bytes.Buffer, string, error) {
var b bytes.Buffer
w := multipart.NewWriter(&b)
for k, v := range values {
for _, s := range v {
if err := w.WriteField(k, s); err != nil {
return b, "", err
}
}
}
if len(fileFieldName) > 0 && len(filePath) > 0 {
fw, err := w.CreateFormFile(fileFieldName, filepath.Base(filePath))
if err != nil {
return b, "", err
}
f, err := os.Open(filePath)
if err != nil {
return b, "", err
}
if _, err = io.Copy(fw, f); err != nil {
return b, "", err
}
}
err := w.Close()
return b, w.FormDataContentType(), err
}

34
httpd/internal_test.go
View file

@ -6,7 +6,9 @@ import (
"html/template"
"net/http"
"net/http/httptest"
"net/url"
"os"
"strings"
"testing"
"github.com/drakkan/sftpgo/dataprovider"
@ -280,6 +282,38 @@ func TestCompareUserFsConfig(t *testing.T) {
if err == nil {
t.Errorf("S3 key prefix does not match")
}
expected.FsConfig.S3Config.KeyPrefix = ""
expected.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir"
err = compareUserFsConfig(expected, actual)
if err == nil {
t.Errorf("GCS key prefix does not match")
}
expected.FsConfig.GCSConfig.KeyPrefix = ""
expected.FsConfig.GCSConfig.Bucket = "bucket"
err = compareUserFsConfig(expected, actual)
if err == nil {
t.Errorf("GCS bucket does not match")
}
expected.FsConfig.GCSConfig.Bucket = ""
expected.FsConfig.GCSConfig.StorageClass = "Standard"
err = compareUserFsConfig(expected, actual)
if err == nil {
t.Errorf("GCS storage class does not match")
}
expected.FsConfig.GCSConfig.StorageClass = ""
}
func TestGCSWebInvalidFormFile(t *testing.T) {
form := make(url.Values)
form.Set("username", "test_username")
form.Set("fs_provider", "2")
req, _ := http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.ParseForm()
_, err := getFsConfigFromUserPostFields(req)
if err != http.ErrNotMultipart {
t.Errorf("unexpected error: %v", err)
}
}
func TestApiCallsWithBadURL(t *testing.T) {

26
httpd/schema/openapi.yaml
View file

@ -2,7 +2,7 @@ openapi: 3.0.1
info:
title: SFTPGo
description: 'SFTPGo REST API'
version: 1.5.0
version: 1.6.0
servers:
- url: /api/v1
@ -743,6 +743,26 @@ components:
- access_secret
nullable: true
description: S3 Compatible Object Storage configuration details
GCSConfig:
type: object
properties:
bucket:
type: string
minLength: 1
credentials:
type: string
format: byte
description: Google Cloud Storage JSON credentials base64 encoded. This field must be populated only when adding/updating an user. It will be always omitted, since there are sensitive data, when you search/get users. The credentials will be stored in the configured "credentials_path"
storage_class:
type: string
key_prefix:
type: string
description: key_prefix is similar to a chroot directory for a local filesystem. If specified the SFTP user will only see contents that starts with this prefix and so you can restrict access to a specific virtual folder. The prefix, if not empty, must not start with "/" and must end with "/". If empty the whole bucket contents will be available
example: folder/subfolder/
required:
- bucket
nullable: true
description: Google Cloud Storage configuration details
FilesystemConfig:
type: object
properties:
@ -751,12 +771,16 @@ components:
enum:
- 0
- 1
- 2
description: >
Providers:
* `0` - local filesystem
* `1` - S3 Compatible Object Storage
* `2` - Google Cloud Storage
s3config:
$ref: '#/components/schemas/S3Config'
gcsconfig:
$ref: '#/components/schemas/GCSConfig'
description: Storage filesystem details
User:
type: object

37
httpd/web.go
View file

@ -1,8 +1,11 @@
package httpd
import (
"encoding/base64"
"errors"
"fmt"
"html/template"
"io/ioutil"
"net/http"
"path/filepath"
"strconv"
@ -224,7 +227,7 @@ func getFiltersFromUserPostFields(r *http.Request) dataprovider.UserFilters {
return filters
}
func getFsConfigFromUserPostFields(r *http.Request) dataprovider.Filesystem {
func getFsConfigFromUserPostFields(r *http.Request) (dataprovider.Filesystem, error) {
var fs dataprovider.Filesystem
provider, err := strconv.Atoi(r.Form.Get("fs_provider"))
if err != nil {
@ -239,13 +242,33 @@ func getFsConfigFromUserPostFields(r *http.Request) dataprovider.Filesystem {
fs.S3Config.Endpoint = r.Form.Get("s3_endpoint")
fs.S3Config.StorageClass = r.Form.Get("s3_storage_class")
fs.S3Config.KeyPrefix = r.Form.Get("s3_key_prefix")
} else if fs.Provider == 2 {
fs.GCSConfig.Bucket = r.Form.Get("gcs_bucket")
fs.GCSConfig.StorageClass = r.Form.Get("gcs_storage_class")
fs.GCSConfig.KeyPrefix = r.Form.Get("gcs_key_prefix")
credentials, _, err := r.FormFile("gcs_credential_file")
if err == http.ErrMissingFile {
return fs, nil
}
return fs
if err != nil {
return fs, err
}
defer credentials.Close()
fileBytes, err := ioutil.ReadAll(credentials)
if err != nil || len(fileBytes) == 0 {
if len(fileBytes) == 0 {
err = errors.New("credentials file size must be greater than 0")
}
return fs, err
}
fs.GCSConfig.Credentials = base64.StdEncoding.EncodeToString(fileBytes)
}
return fs, nil
}
func getUserFromPostFields(r *http.Request) (dataprovider.User, error) {
var user dataprovider.User
err := r.ParseForm()
err := r.ParseMultipartForm(maxRequestSize)
if err != nil {
return user, err
}
@ -292,6 +315,10 @@ func getUserFromPostFields(r *http.Request) (dataprovider.User, error) {
}
expirationDateMillis = utils.GetTimeAsMsSinceEpoch(expirationDate)
}
fsConfig, err := getFsConfigFromUserPostFields(r)
if err != nil {
return user, err
}
user = dataprovider.User{
Username: r.Form.Get("username"),
Password: r.Form.Get("password"),
@ -308,7 +335,7 @@ func getUserFromPostFields(r *http.Request) (dataprovider.User, error) {
Status: status,
ExpirationDate: expirationDateMillis,
Filters: getFiltersFromUserPostFields(r),
FsConfig: getFsConfigFromUserPostFields(r),
FsConfig: fsConfig,
}
return user, err
}
@ -365,6 +392,7 @@ func handleWebUpdateUserGet(userID string, w http.ResponseWriter, r *http.Reques
}
func handleWebAddUserPost(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
user, err := getUserFromPostFields(r)
if err != nil {
renderAddUserPage(w, user, err.Error())
@ -379,6 +407,7 @@ func handleWebAddUserPost(w http.ResponseWriter, r *http.Request) {
}
func handleWebUpdateUserPost(userID string, w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
id, err := strconv.ParseInt(userID, 10, 64)
if err != nil {
renderBadRequestPage(w, err)

148
metrics/metrics.go
View file

@ -246,22 +246,94 @@ var (
Help: "The total number of successful S3 head bucket requests",
})
// totalS3CreateBucket is the metric that reports the total successful S3 create bucket requests
totalS3CreateBucket = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_s3_create_bucket",
Help: "The total number of successful S3 create bucket requests",
})
// totalS3HeadBucketErrors is the metric that reports the total S3 head bucket errors
totalS3HeadBucketErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_s3_head_bucket_errors",
Help: "The total number of S3 head bucket errors",
})
// totalS3CreateBucketErrors is the metric that reports the total S3 create bucket errors
totalS3CreateBucketErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_s3_create_bucket_errors",
Help: "The total number of S3 create bucket errors",
// totalGCSUploads is the metric that reports the total number of successful GCS uploads
totalGCSUploads = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_uploads_total",
Help: "The total number of successful GCS uploads",
})
// totalGCSDownloads is the metric that reports the total number of successful GCS downloads
totalGCSDownloads = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_downloads_total",
Help: "The total number of successful GCS downloads",
})
// totalGCSUploadErrors is the metric that reports the total number of GCS upload errors
totalGCSUploadErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_upload_errors_total",
Help: "The total number of GCS upload errors",
})
// totalGCSDownloadErrors is the metric that reports the total number of GCS download errors
totalGCSDownloadErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_download_errors_total",
Help: "The total number of GCS download errors",
})
// totalGCSUploadSize is the metric that reports the total GCS uploads size as bytes
totalGCSUploadSize = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_upload_size",
Help: "The total GCS upload size as bytes, partial uploads are included",
})
// totalGCSDownloadSize is the metric that reports the total GCS downloads size as bytes
totalGCSDownloadSize = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_download_size",
Help: "The total GCS download size as bytes, partial downloads are included",
})
// totalS3ListObjects is the metric that reports the total successful GCS list objects requests
totalGCSListObjects = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_list_objects",
Help: "The total number of successful GCS list objects requests",
})
// totalGCSCopyObject is the metric that reports the total successful GCS copy object requests
totalGCSCopyObject = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_copy_object",
Help: "The total number of successful GCS copy object requests",
})
// totalGCSDeleteObject is the metric that reports the total successful S3 delete object requests
totalGCSDeleteObject = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_delete_object",
Help: "The total number of successful GCS delete object requests",
})
// totalGCSListObjectsError is the metric that reports the total GCS list objects errors
totalGCSListObjectsErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_list_objects_errors",
Help: "The total number of GCS list objects errors",
})
// totalGCSCopyObjectErrors is the metric that reports the total GCS copy object errors
totalGCSCopyObjectErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_copy_object_errors",
Help: "The total number of GCS copy object errors",
})
// totalGCSDeleteObjectErrors is the metric that reports the total GCS delete object errors
totalGCSDeleteObjectErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_delete_object_errors",
Help: "The total number of GCS delete object errors",
})
// totalGCSHeadBucket is the metric that reports the total successful GCS head bucket requests
totalGCSHeadBucket = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_head_bucket",
Help: "The total number of successful GCS head bucket requests",
})
// totalGCSHeadBucketErrors is the metric that reports the total GCS head bucket errors
totalGCSHeadBucketErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_head_bucket_errors",
Help: "The total number of GCS head bucket errors",
})
)
@ -343,12 +415,60 @@ func S3HeadBucketCompleted(err error) {
}
}
// S3CreateBucketCompleted updates metrics after an S3 create bucket request terminates
func S3CreateBucketCompleted(err error) {
// GCSTransferCompleted updates metrics after a GCS upload or a download
func GCSTransferCompleted(bytes int64, transferKind int, err error) {
if transferKind == 0 {
// upload
if err == nil {
totalS3CreateBucket.Inc()
totalGCSUploads.Inc()
} else {
totalS3CreateBucketErrors.Inc()
totalGCSUploadErrors.Inc()
}
totalGCSUploadSize.Add(float64(bytes))
} else {
// download
if err == nil {
totalGCSDownloads.Inc()
} else {
totalGCSDownloadErrors.Inc()
}
totalGCSDownloadSize.Add(float64(bytes))
}
}
// GCSListObjectsCompleted updates metrics after a GCS list objects request terminates
func GCSListObjectsCompleted(err error) {
if err == nil {
totalGCSListObjects.Inc()
} else {
totalGCSListObjectsErrors.Inc()
}
}
// GCSCopyObjectCompleted updates metrics after a GCS copy object request terminates
func GCSCopyObjectCompleted(err error) {
if err == nil {
totalGCSCopyObject.Inc()
} else {
totalGCSCopyObjectErrors.Inc()
}
}
// GCSDeleteObjectCompleted updates metrics after a GCS delete object request terminates
func GCSDeleteObjectCompleted(err error) {
if err == nil {
totalGCSDeleteObject.Inc()
} else {
totalGCSDeleteObjectErrors.Inc()
}
}
// GCSHeadBucketCompleted updates metrics after a GCS head bucket request terminates
func GCSHeadBucketCompleted(err error) {
if err == nil {
totalGCSHeadBucket.Inc()
} else {
totalGCSHeadBucketErrors.Inc()
}
}

3
scripts/README.md
View file

@ -54,6 +54,7 @@ Output:
"download_bandwidth": 60,
"expiration_date": 1546297200000,
"filesystem": {
"gcsconfig": {},
"provider": 1,
"s3config": {
"access_key": "accesskey",
@ -139,6 +140,7 @@ Output:
"download_bandwidth": 80,
"expiration_date": 0,
"filesystem": {
"gcsconfig": {},
"provider": 0,
"s3config": {}
},
@ -191,6 +193,7 @@ Output:
"download_bandwidth": 80,
"expiration_date": 0,
"filesystem": {
"gcsconfig": {},
"provider": 0,
"s3config": {}
},

44
scripts/sftpgo_api_cli.py
View file

@ -1,5 +1,6 @@
#!/usr/bin/env python
import argparse
import base64
from datetime import datetime
import json
import platform
@ -74,7 +75,7 @@ class SFTPGoApiRequests:
max_sessions=0, quota_size=0, quota_files=0, permissions={}, upload_bandwidth=0, download_bandwidth=0,
status=1, expiration_date=0, allowed_ip=[], denied_ip=[], fs_provider='local', s3_bucket='',
s3_region='', s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='',
s3_key_prefix=''):
s3_key_prefix='', gcs_bucket='', gcs_key_prefix='', gcs_storage_class='', gcs_credentials_file=''):
user = {"id":user_id, "username":username, "uid":uid, "gid":gid,
"max_sessions":max_sessions, "quota_size":quota_size, "quota_files":quota_files,
"upload_bandwidth":upload_bandwidth, "download_bandwidth":download_bandwidth,
@ -92,9 +93,9 @@ class SFTPGoApiRequests:
user.update({"permissions":permissions})
if allowed_ip or denied_ip:
user.update({"filters":self.buildFilters(allowed_ip, denied_ip)})
user.update({"filesystem":self.buildFsConfig(fs_provider, s3_bucket, s3_region, s3_access_key,
s3_access_secret, s3_endpoint, s3_storage_class,
s3_key_prefix)})
user.update({"filesystem":self.buildFsConfig(fs_provider, s3_bucket, s3_region, s3_access_key, s3_access_secret,
s3_endpoint, s3_storage_class, s3_key_prefix, gcs_bucket,
gcs_key_prefix, gcs_storage_class, gcs_credentials_file)})
return user
def buildPermissions(self, root_perms, subdirs_perms):
@ -129,13 +130,19 @@ class SFTPGoApiRequests:
return filters
def buildFsConfig(self, fs_provider, s3_bucket, s3_region, s3_access_key, s3_access_secret, s3_endpoint,
s3_storage_class, s3_key_prefix):
s3_storage_class, s3_key_prefix, gcs_bucket, gcs_key_prefix, gcs_storage_class, gcs_credentials_file):
fs_config = {'provider':0}
if fs_provider == 'S3':
s3config = {'bucket':s3_bucket, 'region':s3_region, 'access_key':s3_access_key, 'access_secret':
s3_access_secret, 'endpoint':s3_endpoint, 'storage_class':s3_storage_class, 'key_prefix':
s3_key_prefix}
fs_config.update({'provider':1, 's3config':s3config})
elif fs_provider == 'GCS':
gcsconfig = {'bucket':gcs_bucket, 'key_prefix':gcs_key_prefix, 'storage_class':gcs_storage_class}
if gcs_credentials_file:
with open(gcs_credentials_file) as creds:
gcsconfig.update({'credentials':base64.b64encode(creds.read().encode('UTF-8')).decode('UTF-8')})
fs_config.update({'provider':2, 'gcsconfig':gcsconfig})
return fs_config
def getUsers(self, limit=100, offset=0, order="ASC", username=""):
@ -150,11 +157,13 @@ class SFTPGoApiRequests:
def addUser(self, username="", password="", public_keys="", home_dir="", uid=0, gid=0, max_sessions=0, quota_size=0,
quota_files=0, perms=[], upload_bandwidth=0, download_bandwidth=0, status=1, expiration_date=0,
subdirs_permissions=[], allowed_ip=[], denied_ip=[], fs_provider='local', s3_bucket='', s3_region='',
s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='', s3_key_prefix=''):
s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='', s3_key_prefix='', gcs_bucket='',
gcs_key_prefix='', gcs_storage_class='', gcs_credentials_file=''):
u = self.buildUserObject(0, username, password, public_keys, home_dir, uid, gid, max_sessions,
quota_size, quota_files, self.buildPermissions(perms, subdirs_permissions), upload_bandwidth, download_bandwidth,
status, expiration_date, allowed_ip, denied_ip, fs_provider, s3_bucket, s3_region,
s3_access_key, s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix)
status, expiration_date, allowed_ip, denied_ip, fs_provider, s3_bucket, s3_region, s3_access_key,
s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix, gcs_bucket, gcs_key_prefix, gcs_storage_class,
gcs_credentials_file)
r = requests.post(self.userPath, json=u, auth=self.auth, verify=self.verify)
self.printResponse(r)
@ -162,11 +171,12 @@ class SFTPGoApiRequests:
quota_size=0, quota_files=0, perms=[], upload_bandwidth=0, download_bandwidth=0, status=1,
expiration_date=0, subdirs_permissions=[], allowed_ip=[], denied_ip=[], fs_provider='local',
s3_bucket='', s3_region='', s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='',
s3_key_prefix=''):
s3_key_prefix='', gcs_bucket='', gcs_key_prefix='', gcs_storage_class='', gcs_credentials_file=''):
u = self.buildUserObject(user_id, username, password, public_keys, home_dir, uid, gid, max_sessions,
quota_size, quota_files, self.buildPermissions(perms, subdirs_permissions), upload_bandwidth, download_bandwidth,
status, expiration_date, allowed_ip, denied_ip, fs_provider, s3_bucket, s3_region, s3_access_key,
s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix)
s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix, gcs_bucket, gcs_key_prefix, gcs_storage_class,
gcs_credentials_file)
r = requests.put(urlparse.urljoin(self.userPath, "user/" + str(user_id)), json=u, auth=self.auth, verify=self.verify)
self.printResponse(r)
@ -420,7 +430,7 @@ def addCommonUserArguments(parser):
help='Allowed IP/Mask in CIDR notation. For example "192.168.2.0/24" or "2001:db8::/32". Default: %(default)s')
parser.add_argument('-N', '--denied-ip', type=str, nargs='+', default=[],
help='Denied IP/Mask in CIDR notation. For example "192.168.2.0/24" or "2001:db8::/32". Default: %(default)s')
parser.add_argument('--fs', type=str, default='local', choices=['local', 'S3'],
parser.add_argument('--fs', type=str, default='local', choices=['local', 'S3', 'GCS'],
help='Filesystem provider. Default: %(default)s')
parser.add_argument('--s3-bucket', type=str, default='', help='Default: %(default)s')
parser.add_argument('--s3-key-prefix', type=str, default='', help='Virtual root directory. If non empty only this ' +
@ -431,6 +441,12 @@ def addCommonUserArguments(parser):
parser.add_argument('--s3-access-secret', type=str, default='', help='Default: %(default)s')
parser.add_argument('--s3-endpoint', type=str, default='', help='Default: %(default)s')
parser.add_argument('--s3-storage-class', type=str, default='', help='Default: %(default)s')
parser.add_argument('--gcs-bucket', type=str, default='', help='Default: %(default)s')
parser.add_argument('--gcs-key-prefix', type=str, default='', help='Virtual root directory. If non empty only this ' +
'directory and its contents will be available. Cannot start with "/". For example "folder/subfolder/".' +
' Default: %(default)s')
parser.add_argument('--gcs-storage-class', type=str, default='', help='Default: %(default)s')
parser.add_argument('--gcs-credentials-file', type=str, default='', help='Default: %(default)s')
if __name__ == '__main__':
@ -534,14 +550,16 @@ if __name__ == '__main__':
args.quota_size, args.quota_files, args.permissions, args.upload_bandwidth, args.download_bandwidth,
args.status, getDatetimeAsMillisSinceEpoch(args.expiration_date), args.subdirs_permissions, args.allowed_ip,
args.denied_ip, args.fs, args.s3_bucket, args.s3_region, args.s3_access_key, args.s3_access_secret,
args.s3_endpoint, args.s3_storage_class, args.s3_key_prefix)
args.s3_endpoint, args.s3_storage_class, args.s3_key_prefix, args.gcs_bucket, args.gcs_key_prefix,
args.gcs_storage_class, args.gcs_credentials_file)
elif args.command == 'update-user':
api.updateUser(args.id, args.username, args.password, args.public_keys, args.home_dir, args.uid, args.gid,
args.max_sessions, args.quota_size, args.quota_files, args.permissions, args.upload_bandwidth,
args.download_bandwidth, args.status, getDatetimeAsMillisSinceEpoch(args.expiration_date),
args.subdirs_permissions, args.allowed_ip, args.denied_ip, args.fs, args.s3_bucket, args.s3_region,
args.s3_access_key, args.s3_access_secret, args.s3_endpoint, args.s3_storage_class,
args.s3_key_prefix)
args.s3_key_prefix, args.gcs_bucket, args.gcs_key_prefix, args.gcs_storage_class,
args.gcs_credentials_file)
elif args.command == 'delete-user':
api.deleteUser(args.id)
elif args.command == 'get-users':

2
service/service.go
View file

@ -6,6 +6,7 @@ import (
"math/rand"
"os"
"os/signal"
"path/filepath"
"strings"
"syscall"
"time"
@ -146,6 +147,7 @@ func (s *Service) StartPortableMode(sftpdPort int, enabledSSHCommands []string,
}
dataProviderConf := config.GetProviderConf()
dataProviderConf.Driver = dataprovider.MemoryDataProviderName
dataProviderConf.CredentialsPath = filepath.Join(os.TempDir(), "credentials")
config.SetProviderConf(dataProviderConf)
httpdConf := config.GetHTTPDConfig()
httpdConf.BindPort = 0

20
sftpd/sftpd_test.go
View file

@ -1053,7 +1053,7 @@ func TestLoginInvalidFs(t *testing.T) {
t.Errorf("unable to add user: %v", err)
}
// we update the database using sqlite3 CLI since we cannot add an user with an invalid config
time.Sleep(150 * time.Millisecond)
time.Sleep(200 * time.Millisecond)
updateUserQuery := fmt.Sprintf("UPDATE users SET filesystem='{\"provider\":1}' WHERE id=%v", user.ID)
cmd := exec.Command("sqlite3", dbPath, updateUserQuery)
out, err := cmd.CombinedOutput()
@ -3039,7 +3039,11 @@ func TestRelativePaths(t *testing.T) {
KeyPrefix: strings.TrimPrefix(user.GetHomeDir(), "/") + "/",
}
s3fs, _ := vfs.NewS3Fs("", user.GetHomeDir(), s3config)
filesystems = append(filesystems, s3fs)
gcsConfig := vfs.GCSFsConfig{
KeyPrefix: strings.TrimPrefix(user.GetHomeDir(), "/") + "/",
}
gcsfs, _ := vfs.NewGCSFs("", user.GetHomeDir(), gcsConfig)
filesystems = append(filesystems, s3fs, gcsfs)
for _, fs := range filesystems {
path = filepath.Join(user.HomeDir, "/")
rel = fs.GetRelativePath(path)
@ -3104,7 +3108,11 @@ func TestResolvePaths(t *testing.T) {
}
os.MkdirAll(user.GetHomeDir(), 0777)
s3fs, _ := vfs.NewS3Fs("", user.GetHomeDir(), s3config)
filesystems = append(filesystems, s3fs)
gcsConfig := vfs.GCSFsConfig{
KeyPrefix: strings.TrimPrefix(user.GetHomeDir(), "/") + "/",
}
gcsfs, _ := vfs.NewGCSFs("", user.GetHomeDir(), gcsConfig)
filesystems = append(filesystems, s3fs, gcsfs)
for _, fs := range filesystems {
path = "/"
resolved, _ = fs.ResolvePath(filepath.ToSlash(path))
@ -3509,7 +3517,8 @@ func TestSCPBasicHandling(t *testing.T) {
t.Errorf("stat for the downloaded file must succeed")
} else {
if fi.Size() != testFileSize {
t.Errorf("size of the file downloaded via SCP does not match the expected one")
t.Errorf("size of the file downloaded via SCP does not match the expected one: %v/%v",
fi.Size(), testFileSize)
}
}
os.Remove(localPath)
@ -3582,7 +3591,8 @@ func TestSCPUploadFileOverwrite(t *testing.T) {
t.Errorf("stat for the downloaded file must succeed")
} else {
if fi.Size() != testFileSize {
t.Errorf("size of the file downloaded via SCP does not match the expected one")
t.Errorf("size of the file downloaded via SCP does not match the expected one: %v/%v",
fi.Size(), testFileSize)
}
}
os.Remove(localPath)

3
sftpgo.json
View file

@ -42,7 +42,8 @@
"http_notification_url": ""
},
"external_auth_program": "",
"external_auth_scope": 0
"external_auth_scope": 0,
"credentials_path": "credentials"
},
"httpd": {
"bind_port": 8080,

68
templates/user.html
View file

@ -15,7 +15,7 @@
<div class="card-body text-form-error">{{.Error}}</div>
</div>
{{end}}
<form id="user_form" action="{{.CurrentURL}}" method="POST" autocomplete="off">
<form id="user_form" enctype="multipart/form-data" action="{{.CurrentURL}}" method="POST" autocomplete="off">
<div class="form-group row">
<label for="idUsername" class="col-sm-2 col-form-label">Username</label>
<div class="col-sm-10">
@ -194,14 +194,15 @@
<div class="form-group row">
<label for="idFilesystem" class="col-sm-2 col-form-label">Storage</label>
<div class="col-sm-10">
<select class="form-control" id="idFilesystem" name="fs_provider">
<select class="form-control" id="idFilesystem" name="fs_provider" onchange="onFilesystemChanged(this.value)">
<option value="0" {{if eq .User.FsConfig.Provider 0 }}selected{{end}}>local</option>
<option value="1" {{if eq .User.FsConfig.Provider 1 }}selected{{end}}>S3</option>
<option value="1" {{if eq .User.FsConfig.Provider 1 }}selected{{end}}>Amazon S3 (Compatible)</option>
<option value="2" {{if eq .User.FsConfig.Provider 2 }}selected{{end}}>Google Cloud Storage</option>
</select>
</div>
</div>
<div class="form-group row">
<div class="form-group row s3">
<label for="idS3Bucket" class="col-sm-2 col-form-label">S3 Bucket</label>
<div class="col-sm-3">
<input type="text" class="form-control" id="idS3Bucket" name="s3_bucket" placeholder=""
@ -215,7 +216,7 @@
</div>
</div>
<div class="form-group row">
<div class="form-group row s3">
<label for="idS3AccessKey" class="col-sm-2 col-form-label">S3 Access Key</label>
<div class="col-sm-3">
<input type="text" class="form-control" id="idS3AccessKey" name="s3_access_key" placeholder=""
@ -229,7 +230,7 @@
</div>
</div>
<div class="form-group row">
<div class="form-group row s3">
<label for="idS3StorageClass" class="col-sm-2 col-form-label">S3 Storage Class</label>
<div class="col-sm-3">
<input type="text" class="form-control" id="idS3StorageClass" name="s3_storage_class" placeholder=""
@ -243,7 +244,7 @@
</div>
</div>
<div class="form-group row">
<div class="form-group row s3">
<label for="idS3KeyPrefix" class="col-sm-2 col-form-label">S3 Key Prefix</label>
<div class="col-sm-10">
<input type="text" class="form-control" id="idS3KeyPrefix" name="s3_key_prefix" placeholder=""
@ -254,6 +255,43 @@
</div>
</div>
<div class="form-group row gcs">
<label for="idGCSBucket" class="col-sm-2 col-form-label">GCS Bucket</label>
<div class="col-sm-10">
<input type="text" class="form-control" id="idGCSBucket" name="gcs_bucket" placeholder=""
value="{{.User.FsConfig.GCSConfig.Bucket}}" maxlength="255">
</div>
</div>
<div class="form-group row gcs">
<label for="idGCSCredentialFile" class="col-sm-2 col-form-label">GCS Credential file</label>
<div class="col-sm-4">
<input type="file" class="form-control-file" id="idGCSCredentialFile" name="gcs_credential_file"
aria-describedby="GCSCredentialsHelpBlock">
<small id="GCSCredentialsHelpBlock" class="form-text text-muted">
Add or update credentials from a JSON file
</small>
</div>
<div class="col-sm-1"></div>
<label for="idGCSStorageClass" class="col-sm-2 col-form-label">GCS Storage Class</label>
<div class="col-sm-3">
<input type="text" class="form-control" id="idGCSStorageClass" name="gcs_storage_class" placeholder=""
value="{{.User.FsConfig.GCSConfig.StorageClass}}" maxlength="255">
</div>
</div>
<div class="form-group row gcs">
<label for="idGCSKeyPrefix" class="col-sm-2 col-form-label">GCS Key Prefix</label>
<div class="col-sm-10">
<input type="text" class="form-control" id="idGCSKeyPrefix" name="gcs_key_prefix" placeholder=""
value="{{.User.FsConfig.GCSConfig.KeyPrefix}}" maxlength="255" aria-describedby="GCSKeyPrefixHelpBlock">
<small id="GCSKeyPrefixHelpBlock" class="form-text text-muted">
Similar to a chroot for local filesystem. Cannot start with "/". Example: "somedir/subdir/".
</small>
</div>
</div>
<input type="hidden" name="expiration_date" id="hidden_start_datetime" value="">
<button type="submit" class="btn btn-primary float-right mt-3 mb-5 px-5 px-3">Submit</button>
</form>
@ -295,6 +333,22 @@
}
return true;
});
onFilesystemChanged('{{.User.FsConfig.Provider}}');
});
function onFilesystemChanged(val){
if (val == '1'){
$('.form-group.row.gcs').hide();
$('.form-group.row.s3').show();
} else if (val == '2'){
$('.form-group.row.gcs').show();
$('.form-group.row.s3').hide();
} else {
$('.form-group.row.gcs').hide();
$('.form-group.row.s3').hide();
}
}
</script>
{{end}}

22
vfs/s3fileinfo.go → vfs/fileinfo.go
View file

@ -5,8 +5,8 @@ import (
"time"
)
// S3FileInfo implements os.FileInfo for a file in S3.
type S3FileInfo struct {
// FileInfo implements os.FileInfo for a file in S3.
type FileInfo struct {
name string
sizeInBytes int64
modTime time.Time
@ -14,14 +14,14 @@ type S3FileInfo struct {
sys interface{}
}
// NewS3FileInfo creates file info.
func NewS3FileInfo(name string, isDirectory bool, sizeInBytes int64, modTime time.Time) S3FileInfo {
// NewFileInfo creates file info.
func NewFileInfo(name string, isDirectory bool, sizeInBytes int64, modTime time.Time) FileInfo {
mode := os.FileMode(0644)
if isDirectory {
mode = os.FileMode(0755) | os.ModeDir
}
return S3FileInfo{
return FileInfo{
name: name,
sizeInBytes: sizeInBytes,
modTime: modTime,
@ -30,31 +30,31 @@ func NewS3FileInfo(name string, isDirectory bool, sizeInBytes int64, modTime tim
}
// Name provides the base name of the file.
func (fi S3FileInfo) Name() string {
func (fi FileInfo) Name() string {
return fi.name
}
// Size provides the length in bytes for a file.
func (fi S3FileInfo) Size() int64 {
func (fi FileInfo) Size() int64 {
return fi.sizeInBytes
}
// Mode provides the file mode bits
func (fi S3FileInfo) Mode() os.FileMode {
func (fi FileInfo) Mode() os.FileMode {
return fi.mode
}
// ModTime provides the last modification time.
func (fi S3FileInfo) ModTime() time.Time {
func (fi FileInfo) ModTime() time.Time {
return fi.modTime
}
// IsDir provides the abbreviation for Mode().IsDir()
func (fi S3FileInfo) IsDir() bool {
func (fi FileInfo) IsDir() bool {
return fi.mode&os.ModeDir != 0
}
// Sys provides the underlying data source (can return nil)
func (fi S3FileInfo) Sys() interface{} {
func (fi FileInfo) Sys() interface{} {
return fi.getFileInfoSys()
}

2
vfs/s3fileinfo_unix.go → vfs/fileinfo_unix.go
View file

@ -21,7 +21,7 @@ func init() {
}
}
func (fi S3FileInfo) getFileInfoSys() interface{} {
func (fi FileInfo) getFileInfoSys() interface{} {
return &syscall.Stat_t{
Uid: uint32(defaultUID),
Gid: uint32(defaultGID)}

2
vfs/s3fileinfo_windows.go → vfs/fileinfo_windows.go
View file

@ -2,6 +2,6 @@ package vfs
import "syscall"
func (fi S3FileInfo) getFileInfoSys() interface{} {
func (fi FileInfo) getFileInfoSys() interface{} {
return syscall.Win32FileAttributeData{}
}

511
vfs/gcsfs.go Normal file
View file

@ -0,0 +1,511 @@
package vfs
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"os"
"path"
"strings"
"time"
"cloud.google.com/go/storage"
"github.com/drakkan/sftpgo/logger"
"github.com/drakkan/sftpgo/metrics"
"github.com/eikenb/pipeat"
"google.golang.org/api/googleapi"
"google.golang.org/api/iterator"
"google.golang.org/api/option"
)
var (
// we cannot use attrs selection until this bug is fixed:
//
// https://github.com/googleapis/google-cloud-go/issues/1763
//
gcsDefaultFieldsSelection = []string{"Name", "Size", "Deleted", "Updated"}
)
// GCSFsConfig defines the configuration for Google Cloud Storage based filesystem
type GCSFsConfig struct {
Bucket string `json:"bucket,omitempty"`
// KeyPrefix is similar to a chroot directory for local filesystem.
// If specified the SFTP user will only see objects that starts with
// this prefix and so you can restrict access to a specific virtual
// folder. The prefix, if not empty, must not start with "/" and must
// end with "/".
// If empty the whole bucket contents will be available
KeyPrefix string `json:"key_prefix,omitempty"`
CredentialFile string `json:"-"`
Credentials string `json:"credentials,omitempty"`
StorageClass string `json:"storage_class,omitempty"`
}
// GCSFs is a Fs implementation for Google Cloud Storage.
type GCSFs struct {
connectionID string
localTempDir string
config GCSFsConfig
svc *storage.Client
ctxTimeout time.Duration
ctxLongTimeout time.Duration
}
// NewGCSFs returns an GCSFs object that allows to interact with Google Cloud Storage
func NewGCSFs(connectionID, localTempDir string, config GCSFsConfig) (Fs, error) {
var err error
fs := GCSFs{
connectionID: connectionID,
localTempDir: localTempDir,
config: config,
ctxTimeout: 30 * time.Second,
ctxLongTimeout: 300 * time.Second,
}
if err = ValidateGCSFsConfig(&fs.config, fs.config.CredentialFile); err != nil {
return fs, err
}
ctx := context.Background()
fs.svc, err = storage.NewClient(ctx, option.WithCredentialsFile(fs.config.CredentialFile))
return fs, err
}
// Name returns the name for the Fs implementation
func (fs GCSFs) Name() string {
return fmt.Sprintf("GCSFs bucket: %#v", fs.config.Bucket)
}
// ConnectionID returns the SSH connection ID associated to this Fs implementation
func (fs GCSFs) ConnectionID() string {
return fs.connectionID
}
// Stat returns a FileInfo describing the named file
func (fs GCSFs) Stat(name string) (os.FileInfo, error) {
var result FileInfo
var err error
if len(name) == 0 || name == "." {
err := fs.checkIfBucketExists()
if err != nil {
return result, err
}
return NewFileInfo(name, true, 0, time.Time{}), nil
}
if fs.config.KeyPrefix == name+"/" {
return NewFileInfo(name, true, 0, time.Time{}), nil
}
prefix := fs.getPrefixForStat(name)
query := &storage.Query{Prefix: prefix, Delimiter: "/"}
/*err = query.SetAttrSelection(gcsDefaultFieldsSelection)
if err != nil {
return result, err
}*/
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
bkt := fs.svc.Bucket(fs.config.Bucket)
it := bkt.Objects(ctx, query)
for {
attrs, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
metrics.GCSListObjectsCompleted(err)
return result, err
}
if len(attrs.Prefix) > 0 {
if fs.isEqual(attrs.Prefix, name) {
result = NewFileInfo(name, true, 0, time.Time{})
}
} else {
if !attrs.Deleted.IsZero() {
continue
}
if fs.isEqual(attrs.Name, name) {
isDir := strings.HasSuffix(attrs.Name, "/")
result = NewFileInfo(name, isDir, attrs.Size, attrs.Updated)
}
}
}
metrics.GCSListObjectsCompleted(nil)
if len(result.Name()) == 0 {
err = errors.New("404 no such file or directory")
}
return result, err
}
// Lstat returns a FileInfo describing the named file
func (fs GCSFs) Lstat(name string) (os.FileInfo, error) {
return fs.Stat(name)
}
// Open opens the named file for reading
func (fs GCSFs) Open(name string) (*os.File, *pipeat.PipeReaderAt, func(), error) {
r, w, err := pipeat.AsyncWriterPipeInDir(fs.localTempDir)
if err != nil {
return nil, nil, nil, err
}
bkt := fs.svc.Bucket(fs.config.Bucket)
obj := bkt.Object(name)
ctx, cancelFn := context.WithCancel(context.Background())
objectReader, err := obj.NewReader(ctx)
if err != nil {
r.Close()
w.Close()
cancelFn()
return nil, nil, nil, err
}
go func() {
defer cancelFn()
defer objectReader.Close()
n, err := io.Copy(w, objectReader)
w.CloseWithError(err)
fsLog(fs, logger.LevelDebug, "download completed, path: %#v size: %v, err: %v", name, n, err)
metrics.GCSTransferCompleted(n, 1, err)
}()
return nil, r, cancelFn, nil
}
// Create creates or opens the named file for writing
func (fs GCSFs) Create(name string, flag int) (*os.File, *pipeat.PipeWriterAt, func(), error) {
r, w, err := pipeat.PipeInDir(fs.localTempDir)
if err != nil {
return nil, nil, nil, err
}
bkt := fs.svc.Bucket(fs.config.Bucket)
obj := bkt.Object(name)
ctx, cancelFn := context.WithCancel(context.Background())
objectWriter := obj.NewWriter(ctx)
if len(fs.config.StorageClass) > 0 {
objectWriter.ObjectAttrs.StorageClass = fs.config.StorageClass
}
go func() {
defer cancelFn()
defer objectWriter.Close()
n, err := io.Copy(objectWriter, r)
r.CloseWithError(err)
fsLog(fs, logger.LevelDebug, "upload completed, path: %#v, readed bytes: %v, err: %v", name, n, err)
metrics.GCSTransferCompleted(n, 0, err)
}()
return nil, w, cancelFn, nil
}
// Rename renames (moves) source to target.
// We don't support renaming non empty directories since we should
// rename all the contents too and this could take long time: think
// about directories with thousands of files, for each file we should
// execute a CopyObject call.
func (fs GCSFs) Rename(source, target string) error {
if source == target {
return nil
}
fi, err := fs.Stat(source)
if err != nil {
return err
}
if fi.IsDir() {
contents, err := fs.ReadDir(source)
if err != nil {
return err
}
if len(contents) > 0 {
return fmt.Errorf("Cannot rename non empty directory: %#v", source)
}
if !strings.HasSuffix(source, "/") {
source += "/"
}
if !strings.HasSuffix(target, "/") {
target += "/"
}
}
src := fs.svc.Bucket(fs.config.Bucket).Object(source)
dst := fs.svc.Bucket(fs.config.Bucket).Object(target)
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
copier := dst.CopierFrom(src)
if len(fs.config.StorageClass) > 0 {
copier.StorageClass = fs.config.StorageClass
}
_, err = copier.Run(ctx)
metrics.GCSCopyObjectCompleted(err)
if err != nil {
return err
}
return fs.Remove(source, fi.IsDir())
}
// Remove removes the named file or (empty) directory.
func (fs GCSFs) Remove(name string, isDir bool) error {
if isDir {
contents, err := fs.ReadDir(name)
if err != nil {
return err
}
if len(contents) > 0 {
return fmt.Errorf("Cannot remove non empty directory: %#v", name)
}
if !strings.HasSuffix(name, "/") {
name += "/"
}
}
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
err := fs.svc.Bucket(fs.config.Bucket).Object(name).Delete(ctx)
metrics.GCSDeleteObjectCompleted(err)
return err
}
// Mkdir creates a new directory with the specified name and default permissions
func (fs GCSFs) Mkdir(name string) error {
_, err := fs.Stat(name)
if !fs.IsNotExist(err) {
return err
}
if !strings.HasSuffix(name, "/") {
name += "/"
}
_, w, _, err := fs.Create(name, 0)
if err != nil {
return err
}
return w.Close()
}
// Symlink creates source as a symbolic link to target.
func (GCSFs) Symlink(source, target string) error {
return errors.New("403 symlinks are not supported")
}
// Chown changes the numeric uid and gid of the named file.
// Silently ignored.
func (GCSFs) Chown(name string, uid int, gid int) error {
return nil
}
// Chmod changes the mode of the named file to mode.
// Silently ignored.
func (GCSFs) Chmod(name string, mode os.FileMode) error {
return nil
}
// Chtimes changes the access and modification times of the named file.
// Silently ignored.
func (GCSFs) Chtimes(name string, atime, mtime time.Time) error {
return errors.New("403 chtimes is not supported")
}
// ReadDir reads the directory named by dirname and returns
// a list of directory entries.
func (fs GCSFs) ReadDir(dirname string) ([]os.FileInfo, error) {
var result []os.FileInfo
// dirname deve essere già cleaned
prefix := ""
if len(dirname) > 0 && dirname != "." {
prefix = strings.TrimPrefix(dirname, "/")
if !strings.HasSuffix(prefix, "/") {
prefix += "/"
}
}
query := &storage.Query{Prefix: prefix, Delimiter: "/"}
/*err := query.SetAttrSelection(gcsDefaultFieldsSelection)
if err != nil {
return result, err
}*/
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
bkt := fs.svc.Bucket(fs.config.Bucket)
it := bkt.Objects(ctx, query)
for {
attrs, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
metrics.GCSListObjectsCompleted(err)
return result, err
}
if len(attrs.Prefix) > 0 {
name, _ := fs.resolve(attrs.Prefix, prefix)
result = append(result, NewFileInfo(name, true, 0, time.Time{}))
} else {
name, isDir := fs.resolve(attrs.Name, prefix)
if len(name) == 0 {
continue
}
if !attrs.Deleted.IsZero() {
continue
}
result = append(result, NewFileInfo(name, isDir, attrs.Size, attrs.Updated))
}
}
metrics.GCSListObjectsCompleted(nil)
return result, nil
}
// IsUploadResumeSupported returns true if upload resume is supported.
// SFTP Resume is not supported on S3
func (GCSFs) IsUploadResumeSupported() bool {
return false
}
// IsAtomicUploadSupported returns true if atomic upload is supported.
// S3 uploads are already atomic, we don't need to upload to a temporary
// file
func (GCSFs) IsAtomicUploadSupported() bool {
return false
}
// IsNotExist returns a boolean indicating whether the error is known to
// report that a file or directory does not exist
func (GCSFs) IsNotExist(err error) bool {
if err == nil {
return false
}
if err == storage.ErrObjectNotExist || err == storage.ErrBucketNotExist {
return true
}
if e, ok := err.(*googleapi.Error); ok {
if e.Code == http.StatusNotFound {
return true
}
}
return strings.Contains(err.Error(), "404")
}
// IsPermission returns a boolean indicating whether the error is known to
// report that permission is denied.
func (GCSFs) IsPermission(err error) bool {
if err == nil {
return false
}
if e, ok := err.(*googleapi.Error); ok {
if e.Code == http.StatusForbidden || e.Code == http.StatusUnauthorized {
return true
}
}
return strings.Contains(err.Error(), "403")
}
// CheckRootPath creates the specified root directory if it does not exists
func (fs GCSFs) CheckRootPath(username string, uid int, gid int) bool {
// we need a local directory for temporary files
osFs := NewOsFs(fs.ConnectionID(), fs.localTempDir)
osFs.CheckRootPath(username, uid, gid)
return fs.checkIfBucketExists() != nil
}
// ScanRootDirContents returns the number of files contained in the bucket,
// and their size
func (fs GCSFs) ScanRootDirContents() (int, int64, error) {
numFiles := 0
size := int64(0)
query := &storage.Query{Prefix: fs.config.KeyPrefix}
err := query.SetAttrSelection(gcsDefaultFieldsSelection)
if err != nil {
return numFiles, size, err
}
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxLongTimeout))
defer cancelFn()
bkt := fs.svc.Bucket(fs.config.Bucket)
it := bkt.Objects(ctx, query)
for {
attrs, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
metrics.GCSListObjectsCompleted(err)
return numFiles, size, err
}
if !attrs.Deleted.IsZero() {
continue
}
numFiles++
size += attrs.Size
}
metrics.GCSListObjectsCompleted(nil)
return numFiles, size, err
}
// GetAtomicUploadPath returns the path to use for an atomic upload.
// S3 uploads are already atomic, we never call this method for S3
func (GCSFs) GetAtomicUploadPath(name string) string {
return ""
}
// GetRelativePath returns the path for a file relative to the user's home dir.
// This is the path as seen by SFTP users
func (fs GCSFs) GetRelativePath(name string) string {
rel := path.Clean(name)
if rel == "." {
rel = ""
}
if !path.IsAbs(rel) {
rel = "/" + rel
}
if len(fs.config.KeyPrefix) > 0 {
if !strings.HasPrefix(rel, "/"+fs.config.KeyPrefix) {
rel = "/"
}
rel = path.Clean("/" + strings.TrimPrefix(rel, "/"+fs.config.KeyPrefix))
}
return rel
}
// Join joins any number of path elements into a single path
func (GCSFs) Join(elem ...string) string {
return strings.TrimPrefix(path.Join(elem...), "/")
}
// ResolvePath returns the matching filesystem path for the specified sftp path
func (fs GCSFs) ResolvePath(sftpPath string) (string, error) {
if !path.IsAbs(sftpPath) {
sftpPath = path.Clean("/" + sftpPath)
}
return fs.Join(fs.config.KeyPrefix, strings.TrimPrefix(sftpPath, "/")), nil
}
func (fs *GCSFs) resolve(name string, prefix string) (string, bool) {
result := strings.TrimPrefix(name, prefix)
isDir := strings.HasSuffix(result, "/")
if isDir {
result = strings.TrimSuffix(result, "/")
}
return result, isDir
}
func (fs *GCSFs) isEqual(key string, sftpName string) bool {
if key == sftpName {
return true
}
if key == sftpName+"/" {
return true
}
if key+"/" == sftpName {
return true
}
return false
}
func (fs *GCSFs) checkIfBucketExists() error {
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
bkt := fs.svc.Bucket(fs.config.Bucket)
_, err := bkt.Attrs(ctx)
metrics.GCSHeadBucketCompleted(err)
return err
}
func (fs *GCSFs) getPrefixForStat(name string) string {
prefix := path.Dir(name)
if prefix == "/" || prefix == "." || len(prefix) == 0 {
prefix = ""
} else {
prefix = strings.TrimPrefix(prefix, "/")
if !strings.HasSuffix(prefix, "/") {
prefix += "/"
}
}
return prefix
}

39
vfs/s3fs.go
View file

@ -21,15 +21,15 @@ import (
"github.com/eikenb/pipeat"
)
// S3FsConfig defines the configuration for S3fs
// S3FsConfig defines the configuration for S3 based filesystem
type S3FsConfig struct {
Bucket string `json:"bucket,omitempty"`
// KeyPrefix is similar to a chroot directory for local filesystem.
// If specified the SFTP user will only see contents that starts with
// If specified the SFTP user will only see objects that starts with
// this prefix and so you can restrict access to a specific virtual
// folder. The prefix, if not empty, must not start with "/" and must
// end with "/".
//If empty the whole bucket contents will be available
// If empty the whole bucket contents will be available
KeyPrefix string `json:"key_prefix,omitempty"`
Region string `json:"region,omitempty"`
AccessKey string `json:"access_key,omitempty"`
@ -70,7 +70,6 @@ func NewS3Fs(connectionID, localTempDir string, config S3FsConfig) (Fs, error) {
Region: aws.String(fs.config.Region),
Credentials: credentials.NewStaticCredentials(fs.config.AccessKey, fs.config.AccessSecret, ""),
}
//config.WithLogLevel(aws.LogDebugWithHTTPBody)
if len(fs.config.Endpoint) > 0 {
awsConfig.Endpoint = aws.String(fs.config.Endpoint)
awsConfig.S3ForcePathStyle = aws.Bool(true)
@ -95,16 +94,16 @@ func (fs S3Fs) ConnectionID() string {
// Stat returns a FileInfo describing the named file
func (fs S3Fs) Stat(name string) (os.FileInfo, error) {
var result S3FileInfo
var result FileInfo
if name == "/" || name == "." {
err := fs.checkIfBucketExists()
if err != nil {
return result, err
}
return NewS3FileInfo(name, true, 0, time.Time{}), nil
return NewFileInfo(name, true, 0, time.Time{}), nil
}
if "/"+fs.config.KeyPrefix == name+"/" {
return NewS3FileInfo(name, true, 0, time.Time{}), nil
return NewFileInfo(name, true, 0, time.Time{}), nil
}
prefix := path.Dir(name)
if prefix == "/" || prefix == "." {
@ -124,7 +123,7 @@ func (fs S3Fs) Stat(name string) (os.FileInfo, error) {
}, func(page *s3.ListObjectsV2Output, lastPage bool) bool {
for _, p := range page.CommonPrefixes {
if fs.isEqual(p.Prefix, name) {
result = NewS3FileInfo(name, true, 0, time.Time{})
result = NewFileInfo(name, true, 0, time.Time{})
return false
}
}
@ -133,7 +132,7 @@ func (fs S3Fs) Stat(name string) (os.FileInfo, error) {
objectSize := *fileObject.Size
objectModTime := *fileObject.LastModified
isDir := strings.HasSuffix(*fileObject.Key, "/")
result = NewS3FileInfo(name, isDir, objectSize, objectModTime)
result = NewFileInfo(name, isDir, objectSize, objectModTime)
return false
}
}
@ -325,7 +324,7 @@ func (fs S3Fs) ReadDir(dirname string) ([]os.FileInfo, error) {
}, func(page *s3.ListObjectsV2Output, lastPage bool) bool {
for _, p := range page.CommonPrefixes {
name, isDir := fs.resolve(p.Prefix, prefix)
result = append(result, NewS3FileInfo(name, isDir, 0, time.Time{}))
result = append(result, NewFileInfo(name, isDir, 0, time.Time{}))
}
for _, fileObject := range page.Contents {
objectSize := *fileObject.Size
@ -334,7 +333,7 @@ func (fs S3Fs) ReadDir(dirname string) ([]os.FileInfo, error) {
if len(name) == 0 {
continue
}
result = append(result, NewS3FileInfo(name, isDir, objectSize, objectModTime))
result = append(result, NewFileInfo(name, isDir, objectSize, objectModTime))
}
return true
})
@ -394,23 +393,7 @@ func (fs S3Fs) CheckRootPath(username string, uid int, gid int) bool {
// we need a local directory for temporary files
osFs := NewOsFs(fs.ConnectionID(), fs.localTempDir)
osFs.CheckRootPath(username, uid, gid)
err := fs.checkIfBucketExists()
if err == nil {
return true
}
if !fs.IsNotExist(err) {
return false
}
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
input := &s3.CreateBucketInput{
Bucket: aws.String(fs.config.Bucket),
}
_, err = fs.svc.CreateBucketWithContext(ctx, input)
fsLog(fs, logger.LevelDebug, "bucket %#v for user %#v does not exists, try to create, error: %v",
fs.config.Bucket, username, err)
metrics.S3CreateBucketCompleted(err)
return err == nil
return fs.checkIfBucketExists() != nil
}
// ScanRootDirContents returns the number of files contained in the bucket,

29
vfs/vfs.go
View file

@ -3,6 +3,7 @@ package vfs
import (
"errors"
"fmt"
"os"
"path"
"runtime"
@ -14,7 +15,7 @@ import (
"github.com/pkg/sftp"
)
// Fs defines the interface for filesystems backends
// Fs defines the interface for filesystem backends
type Fs interface {
Name() string
ConnectionID() string
@ -94,6 +95,32 @@ func ValidateS3FsConfig(config *S3FsConfig) error {
return nil
}
// ValidateGCSFsConfig returns nil if the specified GCS config is valid, otherwise an error
func ValidateGCSFsConfig(config *GCSFsConfig, credentialsFilePath string) error {
if len(config.Bucket) == 0 {
return errors.New("bucket cannot be empty")
}
if len(config.KeyPrefix) > 0 {
if strings.HasPrefix(config.KeyPrefix, "/") {
return errors.New("key_prefix cannot start with /")
}
config.KeyPrefix = path.Clean(config.KeyPrefix)
if !strings.HasSuffix(config.KeyPrefix, "/") {
config.KeyPrefix += "/"
}
}
if len(config.Credentials) == 0 {
fi, err := os.Stat(credentialsFilePath)
if err != nil {
return fmt.Errorf("invalid credentials %v", err)
}
if fi.Size() == 0 {
return errors.New("credentials cannot be empty")
}
}
return nil
}
// SetPathPermissions calls fs.Chown.
// It does nothing for local filesystem on windows
func SetPathPermissions(fs Fs, path string, uid int, gid int) {