Developing and testing patches for Rook

Backgroud

The purpose of this posting is to detail the steps needed to develop and test Rook code using the provided CI scripts. My previous blog posting did not use the CI scripting provided by rook and may not provide sufficient testing coverage to pass the upstream CI.

Pre-built infrastructure

Test Hosts Configuration
This CI development environment uses four (4) libvirt virtual machines as test hosts. The hardware configurations of the virtual machines remains the same as in the previous blog posts. These machines are built from the same clone image and configured with as follows

• Ubuntu 16 (some commands may change for Ubuntu 18)
• Local user with admin rights, updated to use sudo with no password
• Pre-populated SSH host keys and StrictHostchecking set to no in the .ssh/config file
• /etc/hosts file pre-populated with the host name and IP of each test host. 
• Statically defined IP address as per this blog post

Hostnames and IPs

The virtual machines in this environment will be configured with the following hostnames and IPs

192.168.122.31 kube-ub-master
192.168.122.32 kube-ub-node1
192.168.122.33 kube-ub-node2
192.168.122.34 kube-ub-node3

Configuring the master server

The Rook CI scripts are designed to run from the kubernetes master host. These configuration steps are ran as the configured admin user on that host.

Update DNS resolver and hostname

On Ubuntu 16, the resolv.conf file is managed by systemd. This configuration can result in the coredns container not starting due to a lookup loop.

sudo sed -i -e 's/dns=.*/dns=default/' /etc/NetworkManager/NetworkManager.conf
sudo systemctl disable systemd-resolved.service
sudo systemctl stop  systemd-resolved.service
sudo rm /etc/resolv.conf
echo nameserver 8.8.8.8 | sudo tee /etc/resolv.conf
echo kube-ub-master | sudo tee /etc/hostname

Install go 1.11

The CI requires go version 1.11 to be available on the system in order to build the local images and to run the go based test infrastructure.

wget https://dl.google.com/go/go1.11.6.linux-amd64.tar.gz
cd /usr/local
sudo tar -zxf ~kschinck/go1.11.6.linux-amd64.tar.gz
cd ~
echo export PATH=/usr/local/go/bin:\$PATH | tee -a ~/.bashrc
source .bashrc

Install and configure docker

Configure docker to use the registry on the master node

sudo apt-get install -y docker.io git ansible curl
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "insecure-registries" : ["192.168.122.31:5000"]
}
EOF
sudo systemctl start docker
sudo systemctl enable docker

Add user to the docker group

To run the CI scripts, the docker user needs to able to run docker commands without using sudo.

sudo usermod -a -G docker myuser

The new group assignment can be picked up by either logging out and back in or running the newgrp command.

Start the docker registry container

The docker registry container needs to be running in order to host the locally build versions of the Rook images.

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Verify Registry Access

The curl command can be used to verify access to the local registry.

curl http://192.168.122.31:5000/v2/_catalog
{"repositories":[]}

Configure Worker Nodes

The worker nodes need to be configure similar to the master node with the exception of needing golang support added.

for HOST in kube-ub-node1 kube-ub-node2 kube-ub-node3
do
  cat<<EOF | ssh $HOST
sudo sed -i -e 's/dns=.*/dns=default/' /etc/NetworkManager/NetworkManager.conf
sudo systemctl disable systemd-resolved.service
sudo systemctl stop  systemd-resolved.service
sudo rm /etc/resolv.conf
echo nameserver 8.8.8.8 | sudo tee /etc/resolv.conf
echo $HOST | sudo tee /etc/hostname
sudo apt-get install -y docker.io 
cat <<EOG | sudo tee /etc/docker/daemon.json
{
  "insecure-registries" : ["192.168.122.31:5000"]
}
EOG
sudo systemctl start docker
sudo systemctl enable docker
echo ‘export GOPATH=${HOME}/go’ | tee -a ~/.bashrc
source .bashrc
echo $PATH
echo $GOPATH
mkdir -p ${GOPATH}/src/github.com/rook
cd ${GOPATH}/src/github.com/rook
git clone http://github.com/myuser/rook.git
cd rook
git checkout mypatch
EOF
done

Download Source Code

At this point we are ready to prepare for and download the forked git repository.The repository is configured according to the Rook development guidelines. 

These commands will prepare the needed directory path, clone the git repo, switch to the development branch and build the images.

cat <<EOF>>~/.bashrc
export GOPATH=${HOME}/go
EOF
export GOPATH=${HOME}/go
mkdir -p ${GOPATH}/src/github.com/rook
cd ${GOPATH}/src/github.com/rook
git clone http://github.com/myuser/rook.git
cd rook
git checkout mypatch
make

Tag and push built image

The make command will build and upload a rook/ceph image. In order to use this image, this image will need to be tagged and pushed back to the local registry.

docker tag `docker images | grep "rook/ceph" | awk '{print $1":"$2}'` 192.168.122.31:5000/rook/ceph:latest
docker push 192.168.122.31:5000/rook/ceph:latest

The local test framework files need to be update to use the new image. The following command will update the image reference to the local registry.

sed -i -e 's/image: rook.*$/image: 192.168.122.31:5000\/rook\/ceph:latest/' ./tests/framework/installer/ceph_manifests.go

Deploy master  node using CI script

cd tests/scripts
./kubeadm.sh up

The end of this command will display the kubectl command used to configure additional worker nodes. This needs to be copied and used in below worker node configuration setp

Deploy worker nodes using CI script

This command will add each worker host to the kubernetes cluster.

for HOST in kube-ub-node1 kube-ub-node2 kube-ub-node3
do
  cat<<EOF | ssh $HOST
  cd cd ${GOPATH}/src/github.com/rook/rook/tests/scripts
  ./kubeadm.sh install node TEXT_TAKEN_FROM_THE_MASTER_INSTALL_OUTPUT
EOF




Setup the kubectl command for debugging

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

At this point, the admin will be able to use the kubectl command to interact with the deployed kubernetes cluster.

Run CI Test

Rook uses the provide test framework of go to run the validation commands. The test command is ran from the top of the rook repository directory. It will deploy a variety of configurations and validate the operation of the cluster. Output is saved under the _output directory.
This command will run the standard tests called SmokeSuite and save the output to a file in the /tmp directory.

time go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration 2>&1 | tee /tmp/integration.log

The results can be reviewed from /tmp/integration.log file.

The kubernetes cluster should be in a clean state if the tests finish successfully.

Repeating the CI Execution

If an operator is to be update updated, the previous uploaded images should be deleted from the local repository prior to making updates, rebuilding the image, tagging and pushing.

If the CI configuration is updated, it is sufficient to rerun the go test command line.