|
@@ -212,7 +212,7 @@ a `direct-lvm` configuration.
|
|
|
> and have images you want to keep, `push` them Docker Hub or your private
|
|
|
> Docker Trusted Registry before attempting this procedure.
|
|
|
|
|
|
-The procedure below will create a 90GB data volume and 4GB metadata volume to
|
|
|
+The procedure below will create a logical volume configured as a thin pool to
|
|
|
use as backing for the storage pool. It assumes that you have a spare block
|
|
|
device at `/dev/xvdf` with enough free space to complete the task. The device
|
|
|
identifier and volume sizes may be be different in your environment and you
|
|
@@ -221,106 +221,146 @@ assumes that the Docker daemon is in the `stopped` state.
|
|
|
|
|
|
1. Log in to the Docker host you want to configure and stop the Docker daemon.
|
|
|
|
|
|
-2. If it exists, delete your existing image store by removing the
|
|
|
-`/var/lib/docker` directory.
|
|
|
+2. Install the LVM2 package.
|
|
|
+ The LVM2 package includes the userspace toolset that provides logical volume
|
|
|
+ management facilities on linux.
|
|
|
+
|
|
|
+3. Create a physical volume replacing `/dev/xvdf` with your block device.
|
|
|
|
|
|
```bash
|
|
|
- $ sudo rm -rf /var/lib/docker
|
|
|
+ $ pvcreate /dev/xvdf
|
|
|
```
|
|
|
|
|
|
-3. Create an LVM physical volume (PV) on your spare block device using the
|
|
|
-`pvcreate` command.
|
|
|
+4. Create a 'docker' volume group.
|
|
|
|
|
|
```bash
|
|
|
- $ sudo pvcreate /dev/xvdf
|
|
|
- Physical volume `/dev/xvdf` successfully created
|
|
|
+ $ vgcreate docker /dev/xvdf
|
|
|
```
|
|
|
|
|
|
- The device identifier may be different on your system. Remember to substitute
|
|
|
- your value in the command above. If your host is running on AWS EC2, you may
|
|
|
- need to install `lvm2` and <a href="http://goo.gl/Q5pUwG"
|
|
|
- target="_blank">attach an EBS device</a> to use this procedure.
|
|
|
+5. Create a thin pool named `thinpool`.
|
|
|
+
|
|
|
+ In this example, the data logical is 95% of the 'docker' volume group size.
|
|
|
+ Leaving this free space allows for auto expanding of either the data or
|
|
|
+ metadata if space runs low as a temporary stopgap.
|
|
|
+
|
|
|
+ ```bash
|
|
|
+ $ lvcreate --wipesignatures y -n thinpool docker -l 95%VG
|
|
|
+ $ lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
|
|
|
+ ```
|
|
|
|
|
|
-4. Create a new volume group (VG) called `vg-docker` using the PV created in
|
|
|
-the previous step.
|
|
|
+6. Convert the pool to a thin pool.
|
|
|
|
|
|
```bash
|
|
|
- $ sudo vgcreate vg-docker /dev/xvdf
|
|
|
- Volume group `vg-docker` successfully created
|
|
|
+ $ lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta
|
|
|
```
|
|
|
|
|
|
-5. Create a new 90GB logical volume (LV) called `data` from space in the
|
|
|
-`vg-docker` volume group.
|
|
|
+7. Configure autoextension of thin pools via an `lvm` profile.
|
|
|
|
|
|
```bash
|
|
|
- $ sudo lvcreate -L 90G -n data vg-docker
|
|
|
- Logical volume `data` created.
|
|
|
+ $ vi /etc/lvm/profile/docker-thinpool.profile
|
|
|
+ ```
|
|
|
+
|
|
|
+8. Specify 'thin_pool_autoextend_threshold' value.
|
|
|
+
|
|
|
+ The value should be the percentage of space used before `lvm` attempts
|
|
|
+ to autoextend the available space (100 = disabled).
|
|
|
+
|
|
|
+ ```
|
|
|
+ thin_pool_autoextend_threshold = 80
|
|
|
+ ```
|
|
|
+
|
|
|
+9. Modify the `thin_pool_autoextend_percent` for when thin pool autoextension occurs.
|
|
|
+
|
|
|
+ The value's setting is the perentage of space to increase the thin pool (100 =
|
|
|
+ disabled)
|
|
|
+
|
|
|
+ ```
|
|
|
+ thin_pool_autoextend_percent = 20
|
|
|
```
|
|
|
|
|
|
- The command creates an LVM logical volume called `data` and an associated
|
|
|
- block device file at `/dev/vg-docker/data`. In a later step, you instruct the
|
|
|
- `devicemapper` storage driver to use this block device to store image and
|
|
|
- container data.
|
|
|
+10. Check your work, your `docker-thinpool.profile` file should appear similar to the following:
|
|
|
|
|
|
- If you receive a signature detection warning, make sure you are working on
|
|
|
- the correct devices before continuing. Signature warnings indicate that the
|
|
|
- device you're working on is currently in use by LVM or has been used by LVM in
|
|
|
- the past.
|
|
|
+ An example `/etc/lvm/profile/docker-thinpool.profile` file:
|
|
|
|
|
|
-6. Create a new logical volume (LV) called `metadata` from space in the
|
|
|
-`vg-docker` volume group.
|
|
|
+ ```
|
|
|
+ activation {
|
|
|
+ thin_pool_autoextend_threshold=80
|
|
|
+ thin_pool_autoextend_percent=20
|
|
|
+ }
|
|
|
+ ```
|
|
|
+
|
|
|
+11. Apply your new lvm profile
|
|
|
|
|
|
```bash
|
|
|
- $ sudo lvcreate -L 4G -n metadata vg-docker
|
|
|
- Logical volume `metadata` created.
|
|
|
+ $ lvchange --metadataprofile docker-thinpool docker/thinpool
|
|
|
```
|
|
|
|
|
|
- This creates an LVM logical volume called `metadata` and an associated
|
|
|
- block device file at `/dev/vg-docker/metadata`. In the next step you instruct
|
|
|
- the `devicemapper` storage driver to use this block device to store image and
|
|
|
- container metadata.
|
|
|
+12. Verify the `lv` is monitored.
|
|
|
|
|
|
-7. Start the Docker daemon with the `devicemapper` storage driver and the
|
|
|
-`--storage-opt` flags.
|
|
|
+ ```bash
|
|
|
+ $ lvs -o+seg_monitor
|
|
|
+ ```
|
|
|
|
|
|
- The `data` and `metadata` devices that you pass to the `--storage-opt`
|
|
|
- options were created in the previous steps.
|
|
|
+13. If the Docker daemon was previously started, clear your graph driver directory.
|
|
|
+
|
|
|
+ Clearing your graph driver removes any images, containers, and volumes in your
|
|
|
+ Docker installation.
|
|
|
|
|
|
```bash
|
|
|
- $ sudo docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata &
|
|
|
- [1] 2163
|
|
|
- [root@ip-10-0-0-75 centos]# INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
|
|
|
- INFO[0027] Option DefaultDriver: bridge
|
|
|
- INFO[0027] Option DefaultNetwork: bridge
|
|
|
- <-- output truncated -->
|
|
|
- INFO[0027] Daemon has completed initialization
|
|
|
- INFO[0027] Docker daemon commit=1b09a95 graphdriver=aufs version=1.11.0-dev
|
|
|
+ $ rm -rf /var/lib/docker/*
|
|
|
```
|
|
|
|
|
|
- It is also possible to set the `--storage-driver` and `--storage-opt` flags
|
|
|
- in the Docker config file and start the daemon normally using the `service` or
|
|
|
- `systemd` commands.
|
|
|
+14. Configure the Docker daemon with specific devicemapper options.
|
|
|
|
|
|
-8. Use the `docker info` command to verify that the daemon is using `data` and
|
|
|
-`metadata` devices you created.
|
|
|
+ There are two ways to do this. You can set options on the commmand line if you start the daemon there:
|
|
|
|
|
|
```bash
|
|
|
- $ sudo docker info
|
|
|
- INFO[0180] GET /v1.20/info
|
|
|
- Containers: 0
|
|
|
- Images: 0
|
|
|
- Storage Driver: devicemapper
|
|
|
- Pool Name: docker-202:1-1032-pool
|
|
|
- Pool Blocksize: 65.54 kB
|
|
|
- Backing Filesystem: xfs
|
|
|
- Data file: /dev/vg-docker/data
|
|
|
- Metadata file: /dev/vg-docker/metadata
|
|
|
- [...]
|
|
|
+ --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true
|
|
|
```
|
|
|
|
|
|
- The output of the command above shows the storage driver as `devicemapper`.
|
|
|
- The last two lines also confirm that the correct devices are being used for
|
|
|
- the `Data file` and the `Metadata file`.
|
|
|
+ You can also set them for startup in the `daemon.json` configuration, for example:
|
|
|
+
|
|
|
+ ```json
|
|
|
+ {
|
|
|
+ "storage-driver": "devicemapper",
|
|
|
+ "storage-opts": [
|
|
|
+ "dm.thinpooldev=/dev/mapper/docker-thinpool",
|
|
|
+ "dm.use_deferred_removal=true"
|
|
|
+ ]
|
|
|
+ }
|
|
|
+ ```
|
|
|
+
|
|
|
+15. If using systemd and modifying the daemon configuration via unit or drop-in file, reload systemd to scan for changes.
|
|
|
+
|
|
|
+ ```bash
|
|
|
+ $ systemctl daemon-reload
|
|
|
+ ```
|
|
|
+
|
|
|
+16. Start the Docker daemon.
|
|
|
+
|
|
|
+ ```bash
|
|
|
+ $ systemctl start docker
|
|
|
+ ```
|
|
|
+
|
|
|
+After you start the Docker daemon, ensure you monitor your thin pool and volume
|
|
|
+group free space. While the volume group will auto-extend, it can still fill
|
|
|
+up. To monitor logical volumes, use `lvs` without options or `lvs -a` to see tha
|
|
|
+data and metadata sizes. To monitor volume group free space, use the `vgs` command.
|
|
|
+
|
|
|
+Logs can show the auto-extension of the thin pool when it hits the threshold, to
|
|
|
+view the logs use:
|
|
|
+
|
|
|
+```bash
|
|
|
+$ journalctl -fu dm-event.service
|
|
|
+```
|
|
|
+
|
|
|
+If you run into repeated problems with thin pool, you can use the
|
|
|
+`dm.min_free_space` option to tune the Engine behavior. This value ensures that
|
|
|
+operations fail with a warning when the free space is at or near the minimum.
|
|
|
+For information, see <a
|
|
|
+href="../../../reference/commandline/daemon/#storage-driver-options"
|
|
|
+target="_blank">the storage driver options in the Engine daemon reference</a>.
|
|
|
+
|
|
|
|
|
|
### Examine devicemapper structures on the host
|
|
|
|
|
@@ -329,20 +369,20 @@ You can use the `lsblk` command to see the device files created above and the
|
|
|
|
|
|
```bash
|
|
|
$ sudo lsblk
|
|
|
-NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
|
|
-xvda 202:0 0 8G 0 disk
|
|
|
-└─xvda1 202:1 0 8G 0 part /
|
|
|
-xvdf 202:80 0 10G 0 disk
|
|
|
-├─vg--docker-data 253:0 0 90G 0 lvm
|
|
|
-│ └─docker-202:1-1032-pool 253:2 0 10G 0 dm
|
|
|
-└─vg--docker-metadata 253:1 0 4G 0 lvm
|
|
|
- └─docker-202:1-1032-pool 253:2 0 10G 0 dm
|
|
|
+NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
|
|
+xvda 202:0 0 8G 0 disk
|
|
|
+└─xvda1 202:1 0 8G 0 part /
|
|
|
+xvdf 202:80 0 10G 0 disk
|
|
|
+├─vg--docker-data 253:0 0 90G 0 lvm
|
|
|
+│ └─docker-202:1-1032-pool 253:2 0 10G 0 dm
|
|
|
+└─vg--docker-metadata 253:1 0 4G 0 lvm
|
|
|
+ └─docker-202:1-1032-pool 253:2 0 10G 0 dm
|
|
|
```
|
|
|
|
|
|
The diagram below shows the image from prior examples updated with the detail
|
|
|
from the `lsblk` command above.
|
|
|
|
|
|
-
|
|
|
+
|
|
|
|
|
|
In the diagram, the pool is named `Docker-202:1-1032-pool` and spans the `data`
|
|
|
and `metadata` devices created earlier. The `devicemapper` constructs the pool
|
|
@@ -427,12 +467,10 @@ The `Data Space` values show that the pool is 100GB total. This example extends
|
|
|
3. Verify the file size changed.
|
|
|
|
|
|
```bash
|
|
|
- $ sudo ls -al /var/lib/docker/devicemapper/devicemapper/
|
|
|
- total 1175492
|
|
|
- drwx------ 2 root root 4096 Mar 29 02:45 .
|
|
|
- drwx------ 5 root root 4096 Mar 29 02:48 ..
|
|
|
- -rw------- 1 root root 214748364800 Mar 31 11:20 data
|
|
|
- -rw------- 1 root root 2147483648 Mar 31 11:17 metadata
|
|
|
+ $ sudo ls -lh /var/lib/docker/devicemapper/devicemapper/
|
|
|
+ total 1.2G
|
|
|
+ -rw------- 1 root root 200G Apr 14 08:47 data
|
|
|
+ -rw------- 1 root root 2.0G Apr 19 13:27 metadata
|
|
|
```
|
|
|
|
|
|
4. Reload data loop device
|
|
@@ -450,13 +488,14 @@ The `Data Space` values show that the pool is 100GB total. This example extends
|
|
|
a. Get the pool name first.
|
|
|
|
|
|
```bash
|
|
|
- $ sudo dmsetup status docker-8:1-123141-pool: 0 209715200 thin-pool 91
|
|
|
+ $ sudo dmsetup status | grep pool
|
|
|
+ docker-8:1-123141-pool: 0 209715200 thin-pool 91
|
|
|
422/524288 18338/1638400 - rw discard_passdown queue_if_no_space -
|
|
|
```
|
|
|
|
|
|
The name is the string before the colon.
|
|
|
|
|
|
- b. Dump the device mapper table first.
|
|
|
+ b. Dump the device mapper table first.
|
|
|
|
|
|
```bash
|
|
|
$ sudo dmsetup table docker-8:1-123141-pool
|
|
@@ -469,6 +508,7 @@ The `Data Space` values show that the pool is 100GB total. This example extends
|
|
|
reflect the new number of 512 byte sectors in the disk. For example, as the
|
|
|
new loop size is 200GB, change the second number to 419430400.
|
|
|
|
|
|
+
|
|
|
d. Reload the thin pool with the new sector number
|
|
|
|
|
|
```bash
|
|
@@ -533,7 +573,7 @@ disk partition.
|
|
|
|
|
|
c. Calculate the real total sectors of the thin pool now. we can use `blockdev` to get the real size of data lv.
|
|
|
|
|
|
- Change the second number of the table info (i.e. the disk end sector) to
|
|
|
+ Change the second number of the table info (i.e. the number of sectors) to
|
|
|
reflect the new number of 512 byte sectors in the disk. For example, as the
|
|
|
new data `lv` size is `264132100096` bytes, change the second number to
|
|
|
`515883008`.
|
|
@@ -592,8 +632,8 @@ There are several other things that impact the performance of the
|
|
|
|
|
|
- **The mode.** The default mode for Docker running the `devicemapper` storage
|
|
|
driver is `loop-lvm`. This mode uses sparse files and suffers from poor
|
|
|
- performance. It is **not recommended for production**. The recommended mode
|
|
|
- for production environments is `direct-lvm` where the storage driver writes
|
|
|
+ performance. It is **not recommended for production**. The recommended mode for
|
|
|
+ production environments is `direct-lvm` where the storage driver writes
|
|
|
directly to raw block devices.
|
|
|
|
|
|
- **High speed storage.** For best performance you should place the `Data file`
|
|
@@ -601,10 +641,10 @@ There are several other things that impact the performance of the
|
|
|
attached storage or from a SAN or NAS array.
|
|
|
|
|
|
- **Memory usage.** `devicemapper` is not the most memory efficient Docker
|
|
|
- storage driver. Launching *n* copies of the same container loads *n* copies
|
|
|
- of its files into memory. This can have a memory impact on your Docker host.
|
|
|
- As a result, the `devicemapper` storage driver may not be the best choice for
|
|
|
- PaaS and other high density use cases.
|
|
|
+ storage driver. Launching *n* copies of the same container loads *n* copies of
|
|
|
+ its files into memory. This can have a memory impact on your Docker host. As a
|
|
|
+ result, the `devicemapper` storage driver may not be the best choice for PaaS
|
|
|
+ and other high density use cases.
|
|
|
|
|
|
One final point, data volumes provide the best and most predictable
|
|
|
performance. This is because they bypass the storage driver and do not incur
|
|
@@ -618,4 +658,4 @@ data volumes.
|
|
|
* [Select a storage driver](selectadriver.md)
|
|
|
* [AUFS storage driver in practice](aufs-driver.md)
|
|
|
* [Btrfs storage driver in practice](btrfs-driver.md)
|
|
|
-* [daemon reference](../../reference/commandline/dockerd#storage-driver-options)
|
|
|
+* [daemon reference](../../reference/commandline/daemon#storage-driver-options)
|