Merge pull request #1970 from dotcloud/cleanup-hack

Cleanup and reorganize docs and tooling for contributors and maintainers

CONTRIBUTING.md

@@ -90,53 +90,6 @@ regenerated occasionally from the git commit history, so a mismatch may result
 in your changes being overwritten.
 
 
-## Decision process
-
-### How are decisions made?
-
-Short answer: with pull requests to the docker repository.
-
-Docker is an open-source project with an open design philosophy. This means that the repository is the source of truth for EVERY aspect of the project,
-including its philosophy, design, roadmap and APIs. *If it's part of the project, it's in the repo. It's in the repo, it's part of the project.*
-
-As a result, all decisions can be expressed as changes to the repository. An implementation change is a change to the source code. An API change is a change to
-the API specification. A philosophy change is a change to the philosophy manifesto. And so on.
-
-All decisions affecting docker, big and small, follow the same 3 steps:
-
-* Step 1: Open a pull request. Anyone can do this.
-
-* Step 2: Discuss the pull request. Anyone can do this.
-
-* Step 3: Accept or refuse a pull request. The relevant maintainer does this (see below "Who decides what?")
-
-
-### Who decides what?
-
-So all decisions are pull requests, and the relevant maintainer makes the decision by accepting or refusing the pull request.
-But how do we identify the relevant maintainer for a given pull request?
-
-Docker follows the timeless, highly efficient and totally unfair system known as [Benevolent dictator for life](http://en.wikipedia.org/wiki/Benevolent_Dictator_for_Life),
-with yours truly, Solomon Hykes, in the role of BDFL.
-This means that all decisions are made by default by me. Since making every decision myself would be highly unscalable, in practice decisions are spread across multiple maintainers.
-
-The relevant maintainer for a pull request is assigned in 3 steps:
-
-* Step 1: Determine the subdirectory affected by the pull request. This might be src/registry, docs/source/api, or any other part of the repo.
-
-* Step 2: Find the MAINTAINERS file which affects this directory. If the directory itself does not have a MAINTAINERS file, work your way up the the repo hierarchy until you find one.
-
-* Step 3: The first maintainer listed is the primary maintainer. The pull request is assigned to him. He may assign it to other listed maintainers, at his discretion.
-
-
-### I'm a maintainer, should I make pull requests too?
-
-Primary maintainers are not required to create pull requests when changing their own subdirectory, but secondary maintainers are.
-
-### Who assigns maintainers?
-
-Solomon.
-
 ### How can I become a maintainer?
 
 * Step 1: learn the component inside out
@@ -146,15 +99,3 @@ Solomon.
 Don't forget: being a maintainer is a time investment. Make sure you will have time to make yourself available.
 You don't have to be a maintainer to make a difference on the project!
 
-### What are a maintainer's responsibility?
-
-It is every maintainer's responsibility to:
-
-* 1) Expose a clear roadmap for improving their component.
-* 2) Deliver prompt feedback and decisions on pull requests.
-* 3) Be available to anyone with questions, bug reports, criticism etc. on their component. This includes irc, github requests and the mailing list.
-* 4) Make sure their component respects the philosophy, design and roadmap of the project.
-
-### How is this process changed?
-
-Just like everything else: by making a pull request :)

hack/CONTRIBUTORS.md

@@ -0,0 +1 @@
+../CONTRIBUTING.md

hack/MAINTAINERS.md

@@ -0,0 +1,78 @@
+# The Docker maintainer manual
+
+## Introduction
+
+Dear maintainer. Thank you for investing the time and energy to help make Docker as
+useful as possible. Maintaining a project is difficult, sometimes unrewarding work.
+Sure, you will get to contribute cool features to the project. But most of your time
+will be spent reviewing, cleaning up, documenting, andswering questions, justifying
+design decisions - while everyone has all the fun! But remember - the quality of the
+maintainers work is what distinguishes the good projects from the great.
+So please be proud of your work, even the unglamourous parts, and encourage a culture
+of appreciation and respect for *every* aspect of improving the project - not just the
+hot new features.
+
+This document is a manual for maintainers old and new. It explains what is expected of
+maintainers, how they should work, and what tools are available to them.
+
+This is a living document - if you see something out of date or missing, speak up!
+
+
+## What are a maintainer's responsibility?
+
+It is every maintainer's responsibility to:
+
+* 1) Expose a clear roadmap for improving their component.
+* 2) Deliver prompt feedback and decisions on pull requests.
+* 3) Be available to anyone with questions, bug reports, criticism etc. on their component. This includes irc, github requests and the mailing list.
+* 4) Make sure their component respects the philosophy, design and roadmap of the project.
+
+
+## How are decisions made?
+
+Short answer: with pull requests to the docker repository.
+
+Docker is an open-source project with an open design philosophy. This means that the repository is the source of truth for EVERY aspect of the project,
+including its philosophy, design, roadmap and APIs. *If it's part of the project, it's in the repo. It's in the repo, it's part of the project.*
+
+As a result, all decisions can be expressed as changes to the repository. An implementation change is a change to the source code. An API change is a change to
+the API specification. A philosophy change is a change to the philosophy manifesto. And so on.
+
+All decisions affecting docker, big and small, follow the same 3 steps:
+
+* Step 1: Open a pull request. Anyone can do this.
+
+* Step 2: Discuss the pull request. Anyone can do this.
+
+* Step 3: Accept or refuse a pull request. The relevant maintainer does this (see below "Who decides what?")
+
+
+## Who decides what?
+
+So all decisions are pull requests, and the relevant maintainer makes the decision by accepting or refusing the pull request.
+But how do we identify the relevant maintainer for a given pull request?
+
+Docker follows the timeless, highly efficient and totally unfair system known as [Benevolent dictator for life](http://en.wikipedia.org/wiki/Benevolent_Dictator_for_Life),
+with yours truly, Solomon Hykes, in the role of BDFL.
+This means that all decisions are made by default by me. Since making every decision myself would be highly unscalable, in practice decisions are spread across multiple maintainers.
+
+The relevant maintainer for a pull request is assigned in 3 steps:
+
+* Step 1: Determine the subdirectory affected by the pull request. This might be src/registry, docs/source/api, or any other part of the repo.
+
+* Step 2: Find the MAINTAINERS file which affects this directory. If the directory itself does not have a MAINTAINERS file, work your way up the the repo hierarchy until you find one.
+
+* Step 3: The first maintainer listed is the primary maintainer. The pull request is assigned to him. He may assign it to other listed maintainers, at his discretion.
+
+
+### I'm a maintainer, should I make pull requests too?
+
+Yes. Nobody should ever push to master directly. All changes should be made through a pull request.
+
+### Who assigns maintainers?
+
+Solomon.
+
+### How is this process changed?
+
+Just like everything else: by making a pull request :)

hack/README.md

@@ -0,0 +1,24 @@
+# Hacking on Docker
+
+The hack/ directory holds information and tools for everyone involved in the process of creating and
+distributing Docker, specifically:
+
+## Guides
+
+If you're a *contributor* or aspiring contributor, you should read CONTRIBUTORS.md.
+
+If you're a *maintainer* or aspiring maintainer, you should read MAINTAINERS.md.
+
+If you're a *packager* or aspiring packager, you should read PACKAGERS.md.
+
+If you're a maintainer in charge of a *release*, you should read RELEASE-CHECKLIST.md.
+
+## Roadmap
+
+A high-level roadmap is available at ROADMAP.md.
+
+
+## Build tools
+
+make.sh is the primary build tool for docker. It is used for compiling the official binary,
+running the test suite, and pushing releases.

hack/README.rst

@@ -1,28 +0,0 @@
-This directory contains material helpful for hacking on docker.
-
-make hack
-=========
-
-Set up an Ubuntu 12.04 virtual machine for developers including kernel 3.8
-go1.1 and buildbot. The environment is setup in a way that can be used through
-the usual go workflow and/or the root Makefile. You can either edit on
-your host, or inside the VM (using make ssh-dev) and run and test docker
-inside the VM.
-
-dependencies: vagrant, virtualbox packages and python package requests
-
-
-Buildbot
-~~~~~~~~
-
-Buildbot is a continuous integration system designed to automate the
-build/test cycle. By automatically rebuilding and testing the tree each time
-something has changed, build problems are pinpointed quickly, before other
-developers are inconvenienced by the failure.
-
-When running 'make hack' at the docker root directory, it spawns a virtual
-machine in the background running a buildbot instance and adds a git
-post-commit hook that automatically run docker tests for you each time you
-commit in your local docker repository.
-
-You can check your buildbot instance at http://192.168.33.21:8010/waterfall

hack/RELEASE-CHECKLIST.md

@@ -55,11 +55,9 @@ EXAMPLES:
 
 ### 4. Run all tests
 
-```bash
-go test
-```
+FIXME
 
-### 5. Commit and create a pull request
+### 5. Commit and create a pull request to the "release" branch
 
 ```bash
 git add CHANGELOG.md
@@ -72,7 +70,7 @@ git push origin bump_$VERSION
 ### 7. Merge the pull request and apply tags
 
 ```bash
-git checkout master
+git checkout release
 git merge bump_$VERSION
 git tag -a v$VERSION # Don't forget the v!
 git tag -f -a latest
@@ -87,13 +85,14 @@ Get them from [the infrastructure maintainers](
 https://github.com/dotcloud/docker/blob/master/hack/infrastructure/MAINTAINERS).
 
 ```bash
-docker build -t releasedocker .
+docker build -t docker .
 docker run  \
 	-e AWS_S3_BUCKET=get-nightly.docker.io \
 	-e AWS_ACCESS_KEY=$(cat ~/.aws/access_key) \
 	-e AWS_SECRET_KEY=$(cat ~/.aws/secret_key) \
 	-e GPG_PASSPHRASE=supersecretsesame \
-	releasedocker
+	docker
+	hack/release.sh
 ```
 
 It will build and upload the binaries on the specified bucket (you should

hack/Vagrantfile

@@ -1,36 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-BOX_NAME = "ubuntu-dev"
-BOX_URI = "http://files.vagrantup.com/precise64.box"
-VM_IP = "192.168.33.21"
-USER = "vagrant"
-GOPATH = "/data/docker"
-DOCKER_PATH = "#{GOPATH}/src/github.com/dotcloud/docker"
-CFG_PATH = "#{DOCKER_PATH}/hack/environment"
-BUILDBOT_PATH = "/data/buildbot"
-
-Vagrant::Config.run do |config|
-  # Setup virtual machine box
-  config.vm.box = BOX_NAME
-  config.vm.box_url = BOX_URI
-  config.vm.share_folder "v-data", DOCKER_PATH, "#{File.dirname(__FILE__)}/.."
-  config.vm.network :hostonly, VM_IP
-  # Stop if deployment has been done
-  config.vm.provision :shell, :inline => "[ ! -f /usr/bin/git ]"
-  # Touch for makefile
-  pkg_cmd = "touch #{DOCKER_PATH}; "
-  # Install docker dependencies
-  pkg_cmd << "apt-get update -qq; apt-get install -y python-software-properties; " \
-    "add-apt-repository -y ppa:dotcloud/docker-golang/ubuntu; apt-get update -qq; " \
-    "apt-get install -y linux-image-generic-lts-raring lxc git aufs-tools golang-stable make; " \
-    "chown -R #{USER}.#{USER} #{GOPATH}; " \
-    "install -m 0664 #{CFG_PATH}/bash_profile /home/#{USER}/.bash_profile"
-  config.vm.provision :shell, :inline => pkg_cmd
-  # Deploy buildbot CI
-  pkg_cmd = "apt-get install -q -y python-dev python-pip supervisor; " \
-    "pip install -q -r #{CFG_PATH}/requirements.txt; " \
-    "chown #{USER}.#{USER} /data; cd /data; " \
-    "#{CFG_PATH}/setup.sh #{USER} #{GOPATH} #{DOCKER_PATH} #{CFG_PATH} #{BUILDBOT_PATH}"
-  config.vm.provision :shell, :inline => pkg_cmd
-end

hack/environment/README.rst

@@ -1 +0,0 @@
-Files used to setup the developer virtual machine

hack/environment/bash_profile

@@ -1,19 +0,0 @@
-# ~/.bash_profile : executed by the command interpreter for login shells.
-
-# if running bash
-if [ -n "$BASH_VERSION" ]; then
-    # include .bashrc if it exists
-    if [ -f "$HOME/.bashrc" ]; then
-        . "$HOME/.bashrc"
-    fi
-fi
-
-# set PATH so it includes user's private bin if it exists
-[ -d "$HOME/bin" ] && PATH="$HOME/bin:$PATH"
-
-docker=/data/docker/src/github.com/dotcloud/docker
-[ -d $docker ] && cd $docker
-
-export GOPATH=/data/docker
-export PATH=$PATH:$GOPATH/bin
-

hack/environment/buildbot.conf

@@ -1,18 +0,0 @@
-[program:buildmaster]
-command=su vagrant -c "buildbot start master"
-directory=/data/buildbot
-chown= root:root
-redirect_stderr=true
-stdout_logfile=/var/log/supervisor/buildbot-master.log
-stderr_logfile=/var/log/supervisor/buildbot-master.log
-
-[program:buildworker]
-command=buildslave start slave
-directory=/data/buildbot
-chown= root:root
-redirect_stderr=true
-stdout_logfile=/var/log/supervisor/buildbot-slave.log
-stderr_logfile=/var/log/supervisor/buildbot-slave.log
-
-[group:buildbot]
-programs=buildmaster,buildworker

hack/environment/master.cfg

@@ -1,43 +0,0 @@
-import os
-from buildbot.buildslave import BuildSlave
-from buildbot.schedulers.forcesched import ForceScheduler
-from buildbot.config import BuilderConfig
-from buildbot.process.factory import BuildFactory
-from buildbot.steps.shell import ShellCommand
-from buildbot.status import html
-from buildbot.status.web import authz, auth
-
-PORT_WEB = 8010         # Buildbot webserver port
-PORT_MASTER = 9989      # Port where buildbot master listen buildworkers
-TEST_USER = 'buildbot'  # Credential to authenticate build triggers
-TEST_PWD = 'docker'     # Credential to authenticate build triggers
-BUILDER_NAME = 'docker'
-BUILDPASSWORD = 'pass-docker'  # Credential to authenticate buildworkers
-GOPATH = '/data/docker'
-DOCKER_PATH = '{0}/src/github.com/dotcloud/docker'.format(GOPATH)
-
-c = BuildmasterConfig = {}
-
-c['title'] = "Docker"
-c['titleURL'] = "waterfall"
-c['buildbotURL'] = "http://localhost:{0}/".format(PORT_WEB)
-c['db'] = {'db_url':"sqlite:///state.sqlite"}
-c['slaves'] = [BuildSlave('buildworker', BUILDPASSWORD)]
-c['slavePortnum'] = PORT_MASTER
-
-c['schedulers'] = [ForceScheduler(name='trigger',builderNames=[BUILDER_NAME])]
-
-# Docker test command
-test_cmd = "GOPATH={0} make -C {1} test".format(GOPATH,DOCKER_PATH)
-
-# Builder
-factory = BuildFactory()
-factory.addStep(ShellCommand(description='Docker',logEnviron=False,
-    usePTY=True,command=test_cmd))
-c['builders'] = [BuilderConfig(name=BUILDER_NAME,slavenames=['buildworker'],
-    factory=factory)]
-
-# Status
-authz_cfg=authz.Authz(auth=auth.BasicAuth([(TEST_USER,TEST_PWD)]),
-    forceBuild='auth')
-c['status'] = [html.WebStatus(http_port=PORT_WEB, authz=authz_cfg)]

hack/environment/post-commit

@@ -1,21 +0,0 @@
-#!/usr/bin/env python
-
-'''Trigger buildbot docker test build
-
-   post-commit git hook designed to automatically trigger buildbot on
-   the provided vagrant docker VM.'''
-
-import requests
-
-USERNAME = 'buildbot'
-PASSWORD = 'docker'
-BASE_URL = 'http://localhost:8010'
-path = lambda s: BASE_URL + '/' + s
-
-try:
-    session = requests.session()
-    session.post(path('login'),data={'username':USERNAME,'passwd':PASSWORD})
-    session.post(path('builders/docker/force'),
-        data={'forcescheduler':'trigger','reason':'Test commit'})
-except:
-    pass

hack/environment/requirements.txt

@@ -1,6 +0,0 @@
-sqlalchemy<=0.7.9
-sqlalchemy-migrate>=0.7.2
-buildbot==0.8.7p1
-buildbot_slave==0.8.7p1
-nose==1.2.1
-requests==1.1.0

hack/environment/setup.sh

@@ -1,45 +0,0 @@
-#!/bin/bash
-
-# Setup of buildbot configuration. Package installation is being done by
-# Vagrantfile
-# Dependencies: buildbot, buildbot-slave, supervisor
-
-USER=$1
-GOPATH=$2
-DOCKER_PATH=$3
-CFG_PATH=$4
-BUILDBOT_PATH=$5
-SLAVE_NAME="buildworker"
-SLAVE_SOCKET="localhost:9989"
-BUILDBOT_PWD="pass-docker"
-IP=$(sed -nE 's/VM_IP = "(.+)"/\1/p' ${DOCKER_PATH}/hack/Vagrantfile)
-export PATH="/bin:sbin:/usr/bin:/usr/sbin:/usr/local/bin"
-
-function run { su $USER -c "$1"; }
-
-# Exit if buildbot has already been installed
-[ -d "$BUILDBOT_PATH" ] && exit 0
-
-# Setup buildbot
-run "mkdir -p $BUILDBOT_PATH"
-cd $BUILDBOT_PATH
-run "buildbot create-master master"
-run "cp $CFG_PATH/master.cfg master"
-run "sed -i 's/localhost/$IP/' master/master.cfg"
-run "sed -i -E 's#(GOPATH = ).+#\1\"$GOPATH\"#' master/master.cfg"
-run "sed -i -E 's#(DOCKER_PATH = ).+#\1\"$DOCKER_PATH\"#' master/master.cfg"
-run "buildslave create-slave slave $SLAVE_SOCKET $SLAVE_NAME $BUILDBOT_PWD"
-
-# Allow buildbot subprocesses (docker tests) to properly run in containers,
-# in particular with docker -u
-run "sed -i 's/^umask = None/umask = 000/' slave/buildbot.tac"
-
-# Setup supervisor
-cp $CFG_PATH/buildbot.conf /etc/supervisor/conf.d/buildbot.conf
-sed -i -E "s/^chmod=0700.+/chmod=0770\nchown=root:$USER/" /etc/supervisor/supervisord.conf
-kill -HUP $(pgrep -f "/usr/bin/python /usr/bin/supervisord")
-
-# Add git hook
-cp $CFG_PATH/post-commit $DOCKER_PATH/.git/hooks
-sed -i "s/localhost/$IP/" $DOCKER_PATH/.git/hooks/post-commit
-