From 3cf8c53515ff8144a63c6d4418738b5d4f4416a7 Mon Sep 17 00:00:00 2001 From: Chun Chen Date: Tue, 5 Apr 2016 15:35:24 +0800 Subject: [PATCH 01/11] Add docs about how to extend devicemapper thin pool Signed-off-by: Chun Chen Update to device mapper Entering comments Signed-off-by: Mary Anthony (cherry picked from commit a7b2f87b0637a3711e90c0f1d0a99dd5d32bd60f) Signed-off-by: Sebastiaan van Stijn --- .../storagedriver/device-mapper-driver.md | 327 ++++++++---------- 1 file changed, 147 insertions(+), 180 deletions(-) diff --git a/docs/userguide/storagedriver/device-mapper-driver.md b/docs/userguide/storagedriver/device-mapper-driver.md index 1802d10167..90f7424b9d 100644 --- a/docs/userguide/storagedriver/device-mapper-driver.md +++ b/docs/userguide/storagedriver/device-mapper-driver.md @@ -16,12 +16,10 @@ leverages the thin provisioning and snapshotting capabilities of this framework for image and container management. This article refers to the Device Mapper storage driver as `devicemapper`, and the kernel framework as `Device Mapper`. - >**Note**: The [Commercially Supported Docker Engine (CS-Engine) running on RHEL and CentOS Linux](https://www.docker.com/compatibility-maintenance) requires that you use the `devicemapper` storage driver. - ## An alternative to AUFS Docker originally ran on Ubuntu and Debian Linux and used AUFS for its storage @@ -61,20 +59,20 @@ With `devicemapper` the high level process for creating images is as follows: 1. The `devicemapper` storage driver creates a thin pool. - The pool is created from block devices or loop mounted sparse files (more -on this later). + The pool is created from block devices or loop mounted sparse files (more + on this later). 2. Next it creates a *base device*. - A base device is a thin device with a filesystem. You can see which -filesystem is in use by running the `docker info` command and checking the -`Backing filesystem` value. + A base device is a thin device with a filesystem. You can see which + filesystem is in use by running the `docker info` command and checking the + `Backing filesystem` value. 3. Each new image (and image layer) is a snapshot of this base device. - These are thin provisioned copy-on-write snapshots. This means that they -are initially empty and only consume space from the pool when data is written -to them. + These are thin provisioned copy-on-write snapshots. This means that they + are initially empty and only consume space from the pool when data is written + to them. With `devicemapper`, container layers are snapshots of the image they are created from. Just as with images, container snapshots are thin provisioned @@ -109,9 +107,9 @@ block (`0x44f`) in an example container. 1. An application makes a read request for block `0x44f` in the container. - Because the container is a thin snapshot of an image it does not have the -data. Instead, it has a pointer (PTR) to where the data is stored in the image -snapshot lower down in the image stack. + Because the container is a thin snapshot of an image it does not have the + data. Instead, it has a pointer (PTR) to where the data is stored in the image + snapshot lower down in the image stack. 2. The storage driver follows the pointer to block `0xf33` in the snapshot relating to image layer `a005...`. @@ -121,7 +119,7 @@ snapshot to memory in the container. 4. The storage driver returns the data to the requesting application. -### Write examples +## Write examples With the `devicemapper` driver, writing new data to a container is accomplished by an *allocate-on-demand* operation. Updating existing data uses a @@ -132,7 +130,7 @@ For example, when making a small change to a large file in a container, the `devicemapper` storage driver does not copy the entire file. It only copies the blocks to be modified. Each block is 64KB. -#### Writing new data +### Writing new data To write 56KB of new data to a container: @@ -141,12 +139,12 @@ To write 56KB of new data to a container: 2. The allocate-on-demand operation allocates a single new 64KB block to the container's snapshot. - If the write operation is larger than 64KB, multiple new blocks are -allocated to the container's snapshot. + If the write operation is larger than 64KB, multiple new blocks are + allocated to the container's snapshot. 3. The data is written to the newly allocated block. -#### Overwriting existing data +### Overwriting existing data To modify existing data for the first time: @@ -163,7 +161,7 @@ The application in the container is unaware of any of these allocate-on-demand and copy-on-write operations. However, they may add latency to the application's read and write operations. -## Configuring Docker with Device Mapper +## Configure Docker with devicemapper The `devicemapper` is the default Docker storage driver on some Linux distributions. This includes RHEL and most of its forks. Currently, the @@ -182,18 +180,20 @@ deployments should not run under `loop-lvm` mode. You can detect the mode by viewing the `docker info` command: - $ sudo docker info - Containers: 0 - Images: 0 - Storage Driver: devicemapper - Pool Name: docker-202:2-25220302-pool - Pool Blocksize: 65.54 kB - Backing Filesystem: xfs - ... - Data loop file: /var/lib/docker/devicemapper/devicemapper/data - Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata - Library Version: 1.02.93-RHEL7 (2015-01-28) - ... +```bash +$ sudo docker info +Containers: 0 +Images: 0 +Storage Driver: devicemapper + Pool Name: docker-202:2-25220302-pool + Pool Blocksize: 65.54 kB + Backing Filesystem: xfs + [...] + Data loop file: /var/lib/docker/devicemapper/devicemapper/data + Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata + Library Version: 1.02.93-RHEL7 (2015-01-28) + [...] + ``` The output above shows a Docker host running with the `devicemapper` storage driver operating in `loop-lvm` mode. This is indicated by the fact that the @@ -203,175 +203,141 @@ files. ### Configure direct-lvm mode for production -The preferred configuration for production deployments is `direct lvm`. This +The preferred configuration for production deployments is `direct-lvm`. This mode uses block devices to create the thin pool. The following procedure shows you how to configure a Docker host to use the `devicemapper` storage driver in a `direct-lvm` configuration. -> **Caution:** If you have already run the Engine daemon on your Docker host +> **Caution:** If you have already run the Docker daemon on your Docker host > and have images you want to keep, `push` them Docker Hub or your private > Docker Trusted Registry before attempting this procedure. The procedure below will create a 90GB data volume and 4GB metadata volume to use as backing for the storage pool. It assumes that you have a spare block -device at `/dev/sdd` with enough free space to complete the task. The device +device at `/dev/xvdf` with enough free space to complete the task. The device identifier and volume sizes may be be different in your environment and you -should substitute your own values throughout the procedure. +should substitute your own values throughout the procedure. The procedure also +assumes that the Docker daemon is in the `stopped` state. -The procedure also assumes that the Engine daemon is in the `stopped` state. -Any existing images or data are lost by this process. +1. Log in to the Docker host you want to configure and stop the Docker daemon. -1. Log in to the Docker host you want to configure. -2. If it is running, stop the Engine daemon. -3. Install the logical volume management version 2. +2. If it exists, delete your existing image store by removing the +`/var/lib/docker` directory. - ```bash - $ yum install lvm2 - ``` -4. Create a physical volume replacing `/dev/sdd` with your block device. + ```bash + $ sudo rm -rf /var/lib/docker + ``` - ```bash - $ pvcreate /dev/sdd - ``` +3. Create an LVM physical volume (PV) on your spare block device using the +`pvcreate` command. -5. Create a 'docker' volume group. + ```bash + $ sudo pvcreate /dev/xvdf + Physical volume `/dev/xvdf` successfully created + ``` - ```bash - $ vgcreate docker /dev/sdd - ``` + The device identifier may be different on your system. Remember to substitute + your value in the command above. If your host is running on AWS EC2, you may + need to install `lvm2` and attach an EBS device to use this procedure. -6. Create a thin pool named `thinpool`. +4. Create a new volume group (VG) called `vg-docker` using the PV created in +the previous step. - In this example, the data logical is 95% of the 'docker' volume group size. - Leaving this free space allows for auto expanding of either the data or - metadata if space runs low as a temporary stopgap. + ```bash + $ sudo vgcreate vg-docker /dev/xvdf + Volume group `vg-docker` successfully created + ``` - ```bash - $ lvcreate --wipesignatures y -n thinpool docker -l 95%VG - $ lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG - ``` +5. Create a new 90GB logical volume (LV) called `data` from space in the +`vg-docker` volume group. -7. Convert the pool to a thin pool. + ```bash + $ sudo lvcreate -L 90G -n data vg-docker + Logical volume `data` created. + ``` - ```bash - $ lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta - ``` + The command creates an LVM logical volume called `data` and an associated + block device file at `/dev/vg-docker/data`. In a later step, you instruct the + `devicemapper` storage driver to use this block device to store image and + container data. -8. Configure autoextension of thin pools via an `lvm` profile. + If you receive a signature detection warning, make sure you are working on + the correct devices before continuing. Signature warnings indicate that the + device you're working on is currently in use by LVM or has been used by LVM in + the past. - ```bash - $ vi /etc/lvm/profile/docker-thinpool.profile - ``` +6. Create a new logical volume (LV) called `metadata` from space in the +`vg-docker` volume group. -9. Specify 'thin_pool_autoextend_threshold' value. + ```bash + $ sudo lvcreate -L 4G -n metadata vg-docker + Logical volume `metadata` created. + ``` - The value should be the percentage of space used before `lvm` attempts - to autoextend the available space (100 = disabled). + This creates an LVM logical volume called `metadata` and an associated + block device file at `/dev/vg-docker/metadata`. In the next step you instruct + the `devicemapper` storage driver to use this block device to store image and + container metadata. - ``` - thin_pool_autoextend_threshold = 80 - ``` +7. Start the Docker daemon with the `devicemapper` storage driver and the +`--storage-opt` flags. -10. Modify the `thin_pool_autoextend_percent` for when thin pool autoextension occurs. + The `data` and `metadata` devices that you pass to the `--storage-opt` + options were created in the previous steps. - The value's setting is the perentage of space to increase the thin pool (100 = - disabled) + ```bash + $ sudo docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata & + [1] 2163 + [root@ip-10-0-0-75 centos]# INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) + INFO[0027] Option DefaultDriver: bridge + INFO[0027] Option DefaultNetwork: bridge + <-- output truncated --> + INFO[0027] Daemon has completed initialization + INFO[0027] Docker daemon commit=1b09a95 graphdriver=aufs version=1.11.0-dev + ``` - ``` - thin_pool_autoextend_percent = 20 - ``` + It is also possible to set the `--storage-driver` and `--storage-opt` flags + in the Docker config file and start the daemon normally using the `service` or + `systemd` commands. -11. Check your work, your `docker-thinpool.profile` file should appear similar to the following: +8. Use the `docker info` command to verify that the daemon is using `data` and +`metadata` devices you created. - An example `/etc/lvm/profile/docker-thinpool.profile` file: - - ``` - activation { - thin_pool_autoextend_threshold=80 - thin_pool_autoextend_percent=20 - } - ``` - -12. Apply your new lvm profile - - ```bash - $ lvchange --metadataprofile docker-thinpool docker/thinpool - ``` - -13. Verify the `lv` is monitored. - - ```bash - $ lvs -o+seg_monitor - ``` - -14. If Engine was previously started, clear your graph driver directory. - - Clearing your graph driver removes any images and containers in your Docker - installation. - - ```bash - $ rm -rf /var/lib/docker/* - ``` - -14. Configure the Engine daemon with specific devicemapper options. - - There are two ways to do this. You can set options on the commmand line if you start the daemon there: - - ```bash - --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true - ``` - - You can also set them for startup in the `daemon.json` configuration, for example: - - ```json - { - "storage-driver": "devicemapper", - "storage-opts": [ - "dm.thinpooldev=/dev/mapper/docker-thinpool", - "dm.use_deferred_removal=true" - ] - } - ``` -15. Start the Engine daemon. - - ```bash - $ systemctl start docker - ``` - -After you start the Engine daemon, ensure you monitor your thin pool and volume -group free space. While the volume group will auto-extend, it can still fill -up. To monitor logical volumes, use `lvs` without options or `lvs -a` to see tha -data and metadata sizes. To monitor volume group free space, use the `vgs` command. - -Logs can show the auto-extension of the thin pool when it hits the threshold, to -view the logs use: - -```bash -journalctl -fu dm-event.service -``` - -If you run into repeated problems with thin pool, you can use the -`dm.min_free_space` option to tune the Engine behavior. This value ensures that -operations fail with a warning when the free space is at or near the minimum. -For information, see the storage driver options in the Engine daemon reference. + ```bash + $ sudo docker info + INFO[0180] GET /v1.20/info + Containers: 0 + Images: 0 + Storage Driver: devicemapper + Pool Name: docker-202:1-1032-pool + Pool Blocksize: 65.54 kB + Backing Filesystem: xfs + Data file: /dev/vg-docker/data + Metadata file: /dev/vg-docker/metadata + [...] + ``` + The output of the command above shows the storage driver as `devicemapper`. + The last two lines also confirm that the correct devices are being used for + the `Data file` and the `Metadata file`. ### Examine devicemapper structures on the host You can use the `lsblk` command to see the device files created above and the `pool` that the `devicemapper` storage driver creates on top of them. - $ sudo lsblk - NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT - xvda 202:0 0 8G 0 disk - └─xvda1 202:1 0 8G 0 part / - xvdf 202:80 0 10G 0 disk - ├─vg--docker-data 253:0 0 90G 0 lvm - │ └─docker-202:1-1032-pool 253:2 0 10G 0 dm - └─vg--docker-metadata 253:1 0 4G 0 lvm - └─docker-202:1-1032-pool 253:2 0 10G 0 dm +```bash +$ sudo lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +xvda 202:0 0 8G 0 disk +└─xvda1 202:1 0 8G 0 part / +xvdf 202:80 0 10G 0 disk +├─vg--docker-data 253:0 0 90G 0 lvm +│ └─docker-202:1-1032-pool 253:2 0 10G 0 dm +└─vg--docker-metadata 253:1 0 4G 0 lvm + └─docker-202:1-1032-pool 253:2 0 10G 0 dm +``` The diagram below shows the image from prior examples updated with the detail from the `lsblk` command above. @@ -379,8 +345,8 @@ from the `lsblk` command above. ![](http://farm1.staticflickr.com/703/22116692899_0471e5e160_b.jpg) In the diagram, the pool is named `Docker-202:1-1032-pool` and spans the `data` - and `metadata` devices created earlier. The `devicemapper` constructs the pool - name as follows: +and `metadata` devices created earlier. The `devicemapper` constructs the pool +name as follows: ``` Docker-MAJ:MIN-INO-pool @@ -440,18 +406,18 @@ Logging Driver: json-file [...] ``` -The `Data Space` values show that the pool is 100GiB total. This example extends the pool to 200GiB. +The `Data Space` values show that the pool is 100GB total. This example extends the pool to 200GB. 1. List the sizes of the devices. ```bash $ sudo ls -lh /var/lib/docker/devicemapper/devicemapper/ - total 1.2G - -rw------- 1 root root 100G Apr 14 08:47 data - -rw------- 1 root root 2.0G Apr 19 13:27 metadata + total 1175492 + -rw------- 1 root root 100G Mar 30 05:22 data + -rw------- 1 root root 2.0G Mar 31 11:17 metadata ``` -2. Truncate `data` file to 200GiB. +2. Truncate `data` file to the size of the `metadata` file (approximage 200GB). ```bash $ sudo truncate -s 214748364800 /var/lib/docker/devicemapper/devicemapper/data @@ -460,10 +426,12 @@ The `Data Space` values show that the pool is 100GiB total. This example extends 3. Verify the file size changed. ```bash - $ sudo ls -lh /var/lib/docker/devicemapper/devicemapper/ - total 1.2G - -rw------- 1 root root 200G Apr 14 08:47 data - -rw------- 1 root root 2.0G Apr 19 13:27 metadata + $ sudo ls -al /var/lib/docker/devicemapper/devicemapper/ + total 1175492 + drwx------ 2 root root 4096 Mar 29 02:45 . + drwx------ 5 root root 4096 Mar 29 02:48 .. + -rw------- 1 root root 214748364800 Mar 31 11:20 data + -rw------- 1 root root 2147483648 Mar 31 11:17 metadata ``` 4. Reload data loop device @@ -480,19 +448,19 @@ The `Data Space` values show that the pool is 100GiB total. This example extends a. Get the pool name first. - $ sudo dmsetup status | grep pool - docker-8:1-123141-pool: 0 209715200 thin-pool 91 422/524288 18338/1638400 - rw discard_passdown queue_if_no_space - + $ sudo dmsetup status docker-8:1-123141-pool: 0 209715200 thin-pool 91 + 422/524288 18338/1638400 - rw discard_passdown queue_if_no_space - The name is the string before the colon. - b. Dump the device mapper table first. + b. Dump the device mapper table first. $ sudo dmsetup table docker-8:1-123141-pool 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing c. Calculate the real total sectors of the thin pool now. - Change the second number of the table info (i.e. the number of sectors) to reflect the new number of 512 byte sectors in the disk. For example, as the new loop size is 200GiB, change the second number to 419430400. + Change the second number of the table info (i.e. the disk end sector) to reflect the new number of 512 byte sectors in the disk. For example, as the new loop size is 200GB, change the second number to 419430400. d. Reload the thin pool with the new sector number @@ -514,7 +482,7 @@ $ ./device_tool resize 200GB ### For a direct-lvm mode configuration In this example, you extend the capacity of a running device that uses the -`direct-lvm` configuration. This example assumes you are using the `/dev/sdh1` +`direct-lvm` configuration. This example assumes you are using the `/dev/sdh1` disk partition. 1. Extend the volume group (VG) `vg-docker`. @@ -550,7 +518,7 @@ disk partition. c. Calculate the real total sectors of the thin pool now. we can use `blockdev` to get the real size of data lv. - Change the second number of the table info (i.e. the number of sectors) to + Change the second number of the table info (i.e. the disk end sector) to reflect the new number of 512 byte sectors in the disk. For example, as the new data `lv` size is `264132100096` bytes, change the second number to `515883008`. @@ -562,7 +530,6 @@ disk partition. $ sudo dmsetup suspend docker-253:17-1835016-pool && sudo dmsetup reload docker-253:17-1835016-pool --table '0 515883008 thin-pool 252:0 252:1 128 32768 1 skip_block_zeroing' && sudo dmsetup resume docker-253:17-1835016-pool - ## Device Mapper and Docker performance It is important to understand the impact that allocate-on-demand and From ea8b5e07b63fb42854d1102a03041994ebac2b4c Mon Sep 17 00:00:00 2001 From: Liron Levin Date: Mon, 2 May 2016 22:19:23 +0300 Subject: [PATCH 02/11] Remove response modification sections from authorization design doc Signed-off-by: Liron Levin (cherry picked from commit 638096431a36435aae0e30b6c76c134a4728ed5d) Signed-off-by: Sebastiaan van Stijn --- docs/extend/plugins_authorization.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 31a6a072ab..8a1352c6d2 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -190,17 +190,10 @@ should implement the following two methods: { "Allow": "Determined whether the user is allowed or not", "Msg": "The authorization message", - "Err": "The error message if things go wrong", - "ModifiedBody": "Byte array containing a modified body of the raw HTTP body (or nil if no changes required)", - "ModifiedHeader": "Byte array containing a modified header of the HTTP response (or nil if no changes required)", - "ModifiedStatusCode": "int containing the modified version of the status code (or 0 if not change is required)" + "Err": "The error message if things go wrong" } ``` -The modified response enables the authorization plugin to manipulate the content -of the HTTP response. In case of more than one plugin, each subsequent plugin -receives a response (optionally) modified by a previous plugin. - ### Request authorization Each plugin must support two request authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message. From 8fc5559841bb163ce35162db3312273a6d3467a4 Mon Sep 17 00:00:00 2001 From: Eric Yang Date: Wed, 4 May 2016 19:13:00 +0800 Subject: [PATCH 03/11] fix typo fix typo Signed-off-by: Qizhao Yang (cherry picked from commit 176e9e2ffc192862f104b13bdf651551c1e492f1) Signed-off-by: Sebastiaan van Stijn --- docs/userguide/storagedriver/imagesandcontainers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/userguide/storagedriver/imagesandcontainers.md b/docs/userguide/storagedriver/imagesandcontainers.md index 627729595c..1ec8a098ac 100644 --- a/docs/userguide/storagedriver/imagesandcontainers.md +++ b/docs/userguide/storagedriver/imagesandcontainers.md @@ -42,7 +42,7 @@ image. Docker 1.10 introduced a new content addressable storage model. This is a completely new way to address image and layer data on disk. Previously, image -and layer data was referenced and stored using a a randomly generated UUID. In +and layer data was referenced and stored using a randomly generated UUID. In the new model this is replaced by a secure *content hash*. The new model improves security, provides a built-in way to avoid ID From 2c531b4fb7644c76874aef9b38f6fde76923d292 Mon Sep 17 00:00:00 2001 From: Doug Davis Date: Wed, 4 May 2016 05:15:57 -0700 Subject: [PATCH 04/11] Remove unnecessary double-double quotes Signed-off-by: Doug Davis (cherry picked from commit 8eb2188bd93c01533e201a37e1e674f5b53cb597) Signed-off-by: Sebastiaan van Stijn --- docs/reference/run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/run.md b/docs/reference/run.md index e0b7a2bbe3..8e43c71086 100644 --- a/docs/reference/run.md +++ b/docs/reference/run.md @@ -1331,7 +1331,7 @@ If the operator uses `--link` when starting a new client container in the default bridge network, then the client container can access the exposed port via a private networking interface. If `--link` is used when starting a container in a user-defined network as -described in [*Docker network overview*""](../userguide/networking/index.md)), +described in [*Docker network overview*](../userguide/networking/index.md)), it will provide a named alias for the container being linked to. ### ENV (environment variables) From cef1a10f5e7a1821b36e39d1260e9cd53080c514 Mon Sep 17 00:00:00 2001 From: Tomasz Kopczynski Date: Wed, 4 May 2016 22:35:57 +0200 Subject: [PATCH 05/11] Docs: fixing typos in admin/supervisor Signed-off-by: Tomasz Kopczynski (cherry picked from commit 74d382ff8de9a870d48e7fd4cc63bb9ee254f3de) Signed-off-by: Sebastiaan van Stijn --- docs/admin/using_supervisord.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/using_supervisord.md b/docs/admin/using_supervisord.md index f8a2625012..683154b09d 100644 --- a/docs/admin/using_supervisord.md +++ b/docs/admin/using_supervisord.md @@ -90,7 +90,7 @@ to start Supervisor when our container launches. EXPOSE 22 80 CMD ["/usr/bin/supervisord"] -Here We've exposed ports 22 and 80 on the container and we're running +Here we've exposed ports 22 and 80 on the container and we're running the `/usr/bin/supervisord` binary when the container launches. @@ -102,7 +102,7 @@ We can now build our new image. ## Running our Supervisor container -Once We've got a built image we can launch a container from it. +Once we've got a built image we can launch a container from it. $ docker run -p 22 -p 80 -t -i /supervisord 2013-11-25 18:53:22,312 CRIT Supervisor running as root (no user in config file) From b34165ae3725e6600cec15b6d63b2ad924a246d9 Mon Sep 17 00:00:00 2001 From: objectified Date: Thu, 5 May 2016 13:08:22 +0200 Subject: [PATCH 06/11] remove trailing comma from top command Signed-off-by: objectified (cherry picked from commit c7e738d641e83ab955be423b70b9ed364dba9637) Signed-off-by: Sebastiaan van Stijn --- docs/userguide/containers/dockervolumes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/userguide/containers/dockervolumes.md b/docs/userguide/containers/dockervolumes.md index 9c960d1fe9..93e58c2741 100644 --- a/docs/userguide/containers/dockervolumes.md +++ b/docs/userguide/containers/dockervolumes.md @@ -345,7 +345,7 @@ source. When the container is deleted, you should instruction the Engine daemon to clean up anonymous volumes. To do this, use the `--rm` option, for example: ```bash -$ docker run --rm -v /foo -v awesome:/bar busybox top, +$ docker run --rm -v /foo -v awesome:/bar busybox top ``` This command creates an anonymous `/foo` volume. When the container is removed, From 2f4b69229a0152c65a84a36dfbd4dcc453c53608 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 6 May 2016 01:52:51 +0200 Subject: [PATCH 07/11] docs: update supervisord example This updates the supervisor example documentation to use an up-to-date version of Ubuntu. Also reduced the use of "royal We", and tweaked some language. Finally, added some language hints for code-highlighting. Signed-off-by: Sebastiaan van Stijn (cherry picked from commit e38678e6601cc597b621aaf3cf630419a7963ae9) Signed-off-by: Sebastiaan van Stijn --- docs/admin/using_supervisord.md | 168 +++++++++++++++++++------------- 1 file changed, 102 insertions(+), 66 deletions(-) diff --git a/docs/admin/using_supervisord.md b/docs/admin/using_supervisord.md index 683154b09d..8e52f18197 100644 --- a/docs/admin/using_supervisord.md +++ b/docs/admin/using_supervisord.md @@ -15,104 +15,140 @@ parent = "engine_admin" > - **If you don't like sudo** then see [*Giving non-root > access*](../installation/binaries.md#giving-non-root-access) -Traditionally a Docker container runs a single process when it is -launched, for example an Apache daemon or a SSH server daemon. Often -though you want to run more than one process in a container. There are a -number of ways you can achieve this ranging from using a simple Bash -script as the value of your container's `CMD` instruction to installing -a process management tool. +Traditionally a Docker container runs a single process when it is launched, for +example an Apache daemon or a SSH server daemon. Often though you want to run +more than one process in a container. There are a number of ways you can +achieve this ranging from using a simple Bash script as the value of your +container's `CMD` instruction to installing a process management tool. -In this example we're going to make use of the process management tool, -[Supervisor](http://supervisord.org/), to manage multiple processes in -our container. Using Supervisor allows us to better control, manage, and -restart the processes we want to run. To demonstrate this we're going to -install and manage both an SSH daemon and an Apache daemon. +In this example you're going to make use of the process management tool, +[Supervisor](http://supervisord.org/), to manage multiple processes in a +container. Using Supervisor allows you to better control, manage, and restart +the processes inside the container. To demonstrate this we're going to install +and manage both an SSH daemon and an Apache daemon. ## Creating a Dockerfile -Let's start by creating a basic `Dockerfile` for our -new image. +Let's start by creating a basic `Dockerfile` for our new image. - FROM ubuntu:13.04 - MAINTAINER examples@docker.com +```Dockerfile +FROM ubuntu:16.04 +MAINTAINER examples@docker.com +``` ## Installing Supervisor -We can now install our SSH and Apache daemons as well as Supervisor in -our container. +You can now install the SSH and Apache daemons as well as Supervisor in the +container. - RUN apt-get update && apt-get install -y openssh-server apache2 supervisor - RUN mkdir -p /var/lock/apache2 /var/run/apache2 /var/run/sshd /var/log/supervisor +```Dockerfile +RUN apt-get update && apt-get install -y openssh-server apache2 supervisor +RUN mkdir -p /var/lock/apache2 /var/run/apache2 /var/run/sshd /var/log/supervisor +``` -Here we're installing the `openssh-server`, -`apache2` and `supervisor` -(which provides the Supervisor daemon) packages. We're also creating four -new directories that are needed to run our SSH daemon and Supervisor. +The first `RUN` instruction installs the `openssh-server`, `apache2` and +`supervisor` (which provides the Supervisor daemon) packages. The next `RUN` +instruction creates four new directories that are needed to run the SSH daemon +and Supervisor. ## Adding Supervisor's configuration file -Now let's add a configuration file for Supervisor. The default file is -called `supervisord.conf` and is located in -`/etc/supervisor/conf.d/`. +Now let's add a configuration file for Supervisor. The default file is called +`supervisord.conf` and is located in `/etc/supervisor/conf.d/`. - COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf +```Dockerfile +COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf +``` -Let's see what is inside our `supervisord.conf` -file. +Let's see what is inside the `supervisord.conf` file. - [supervisord] - nodaemon=true +```ini +[supervisord] +nodaemon=true - [program:sshd] - command=/usr/sbin/sshd -D +[program:sshd] +command=/usr/sbin/sshd -D - [program:apache2] - command=/bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND" +[program:apache2] +command=/bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND" +``` -The `supervisord.conf` configuration file contains -directives that configure Supervisor and the processes it manages. The -first block `[supervisord]` provides configuration -for Supervisor itself. We're using one directive, `nodaemon` -which tells Supervisor to run interactively rather than -daemonize. +The `supervisord.conf` configuration file contains directives that configure +Supervisor and the processes it manages. The first block `[supervisord]` +provides configuration for Supervisor itself. The `nodaemon` directive is used, +which tells Supervisor to run interactively rather than daemonize. -The next two blocks manage the services we wish to control. Each block -controls a separate process. The blocks contain a single directive, -`command`, which specifies what command to run to -start each process. +The next two blocks manage the services we wish to control. Each block controls +a separate process. The blocks contain a single directive, `command`, which +specifies what command to run to start each process. ## Exposing ports and running Supervisor -Now let's finish our `Dockerfile` by exposing some -required ports and specifying the `CMD` instruction -to start Supervisor when our container launches. - - EXPOSE 22 80 - CMD ["/usr/bin/supervisord"] - -Here we've exposed ports 22 and 80 on the container and we're running -the `/usr/bin/supervisord` binary when the container +Now let's finish the `Dockerfile` by exposing some required ports and +specifying the `CMD` instruction to start Supervisor when our container launches. +```Dockerfile +EXPOSE 22 80 +CMD ["/usr/bin/supervisord"] +``` + +These instructions tell Docker that ports 22 and 80 are exposed by the +container and that the `/usr/bin/supervisord` binary should be executed when +the container launches. + ## Building our image -We can now build our new image. +Your completed Dockerfile now looks like this: - $ docker build -t /supervisord . +```Dockerfile +FROM ubuntu:16.04 +MAINTAINER examples@docker.com -## Running our Supervisor container +RUN apt-get update && apt-get install -y openssh-server apache2 supervisor +RUN mkdir -p /var/lock/apache2 /var/run/apache2 /var/run/sshd /var/log/supervisor -Once we've got a built image we can launch a container from it. +COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf - $ docker run -p 22 -p 80 -t -i /supervisord - 2013-11-25 18:53:22,312 CRIT Supervisor running as root (no user in config file) - 2013-11-25 18:53:22,312 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing - 2013-11-25 18:53:22,342 INFO supervisord started with pid 1 - 2013-11-25 18:53:23,346 INFO spawned: 'sshd' with pid 6 - 2013-11-25 18:53:23,349 INFO spawned: 'apache2' with pid 7 - . . . +EXPOSE 22 80 +CMD ["/usr/bin/supervisord"] +``` -We've launched a new container interactively using the `docker run` command. +And your `supervisord.conf` file looks like this; + +```ini +[supervisord] +nodaemon=true + +[program:sshd] +command=/usr/sbin/sshd -D + +[program:apache2] +command=/bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND" +``` + + +You can now build the image using this command; + +```bash +$ docker build -t mysupervisord . +``` + +## Running your Supervisor container + +Once you have built your image you can launch a container from it. + +```bash +$ docker run -p 22 -p 80 -t -i mysupervisord +2013-11-25 18:53:22,312 CRIT Supervisor running as root (no user in config file) +2013-11-25 18:53:22,312 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing +2013-11-25 18:53:22,342 INFO supervisord started with pid 1 +2013-11-25 18:53:23,346 INFO spawned: 'sshd' with pid 6 +2013-11-25 18:53:23,349 INFO spawned: 'apache2' with pid 7 +... +``` + +You launched a new container interactively using the `docker run` command. That container has run Supervisor and launched the SSH and Apache daemons with it. We've specified the `-p` flag to expose ports 22 and 80. From here we can now identify the exposed ports and connect to one or both of the SSH and Apache From b706ee90ca937d7cfc57ce61aa7aeae5c519e7f3 Mon Sep 17 00:00:00 2001 From: Lars Kellogg-Stedman Date: Fri, 6 May 2016 15:52:42 -0400 Subject: [PATCH 08/11] docs: note requirements for systemd drop-in filenames the documentations says that you can drop "a file" into the `docker.service.d` directory, but does not note that the file must end with `.conf` in order to be recognized by systemd. This can lead to some [confusion][] if readers are not previously familiar with systemd. [confusion]: https://botbot.me/freenode/docker/2016-05-06/?msg=65605541&page=11 Signed-off-by: Lars Kellogg-Stedman (cherry picked from commit 987b03054af8bfa5147465d4fe5d5a554cd5fee3) Signed-off-by: Sebastiaan van Stijn --- docs/admin/systemd.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/admin/systemd.md b/docs/admin/systemd.md index ff1d54607f..ef44e8e9ee 100644 --- a/docs/admin/systemd.md +++ b/docs/admin/systemd.md @@ -33,15 +33,19 @@ If you want Docker to start at boot, you should also: There are a number of ways to configure the daemon flags and environment variables for your Docker daemon. -The recommended way is to use a systemd drop-in file. These are local files in -the `/etc/systemd/system/docker.service.d` directory. This could also be -`/etc/systemd/system/docker.service`, which also works for overriding the -defaults from `/lib/systemd/system/docker.service`. +The recommended way is to use a systemd drop-in file (as described in +the systemd.unit +documentation). These are local files named `.conf` in the +`/etc/systemd/system/docker.service.d` directory. This could also be +`/etc/systemd/system/docker.service`, which also works for overriding +the defaults from `/lib/systemd/system/docker.service`. -However, if you had previously used a package which had an `EnvironmentFile` -(often pointing to `/etc/sysconfig/docker`) then for backwards compatibility, -you drop a file in the `/etc/systemd/system/docker.service.d` -directory including the following: +However, if you had previously used a package which had an +`EnvironmentFile` (often pointing to `/etc/sysconfig/docker`) then for +backwards compatibility, you drop a file with a `.conf` extension into +the `/etc/systemd/system/docker.service.d` directory including the +following: [Service] EnvironmentFile=-/etc/sysconfig/docker From ef76dd07612c890b0d03891429617cc6076815f8 Mon Sep 17 00:00:00 2001 From: Yuan Sun Date: Sat, 7 May 2016 17:15:02 +0800 Subject: [PATCH 09/11] from inheritted to inherited Signed-off-by: Yuan Sun (cherry picked from commit fe1130b7bab0d0724133b4d0e2aa0ca5a5dc0222) Signed-off-by: Sebastiaan van Stijn --- docs/reference/run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/run.md b/docs/reference/run.md index 8e43c71086..942bba948f 100644 --- a/docs/reference/run.md +++ b/docs/reference/run.md @@ -594,7 +594,7 @@ associated with the container when the container is removed. This is similar to running `docker rm -v my-container`. Only volumes that are specified without a name are removed. For example, with `docker run --rm -v /foo -v awesome:/bar busybox top`, the volume for `/foo` will be removed, -but the volume for `/bar` will not. Volumes inheritted via `--volumes-from` will be removed +but the volume for `/bar` will not. Volumes inherited via `--volumes-from` will be removed with the same logic -- if the original volume was specified with a name it will **not** be removed. ## Security configuration From 7e83ae34dc8953eed4a0d0fdb9aa33b843102b5a Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Sat, 7 May 2016 08:46:15 -0700 Subject: [PATCH 10/11] Add the missing subtitle in deprecated docs for --security-opt. The colon separator(`:`) of `--security-opt` flag was deprecated in 1.11.0. However, the subtitle in deprecated docs is missing so it is placed under the same subtitle as the deprecated `-e` and `--email` flags. This fix adds the missing subtitle in deprecated docs. Signed-off-by: Yong Tang (cherry picked from commit 018c22880d594b1de06216aa05186f2ebe9a1e9c) Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index f1c1fb0a77..d4702c0f90 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -21,6 +21,11 @@ The following list of features are deprecated in Engine. The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. +### Separator (`:`) of `--security-opt` flag on `docker run` +**Deprecated In Release: v1.11** + +**Target For Removal In Release: v1.13** + The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consinstency with other similar flags, like `--storage-opt`. ### Ambiguous event fields in API From 8d485949d6d3464fed00e002ea1611f601ed8532 Mon Sep 17 00:00:00 2001 From: Tonis Tiigi Date: Mon, 9 May 2016 10:40:11 -0700 Subject: [PATCH 11/11] docs: clarify docker attach Signed-off-by: Tonis Tiigi (cherry picked from commit da1dbd2093ec040f05f4daf9fc8ca28dc8262ec8) Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/attach.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/reference/commandline/attach.md b/docs/reference/commandline/attach.md index dfe2908c3b..b28b6fc263 100644 --- a/docs/reference/commandline/attach.md +++ b/docs/reference/commandline/attach.md @@ -39,6 +39,15 @@ using `CTRL-p CTRL-q` key sequence. It is forbidden to redirect the standard input of a `docker attach` command while attaching to a tty-enabled container (i.e.: launched with `-t`). +While a client is connected to container's stdio using `docker attach`, Docker +uses a ~1MB memory buffer to maximize the throughput of the application. If +this buffer is filled, the speed of the API connection will start to have an +effect on the process output writing speed. This is similar to other +applications like SSH. Because of this, it is not recommended to run +performance critical applications that generate a lot of output in the +foreground over a slow client connection. Instead, users should use the +`docker logs` command to get access to the logs. + ## Override the detach sequence