Backgroud
The purpose of this posting is to detail the steps needed to develop and test Rook code using the provided CI scripts. My previous blog posting did not use the CI scripting provided by rook and may not provide sufficient testing coverage to pass the upstream CI.
Pre-built infrastructure
Test Hosts Configuration
This CI development environment uses four (4) libvirt virtual machines as test hosts. The hardware configurations of the virtual machines remains the same as in the previous blog posts. These machines are built from the same clone image and configured with as follows
- Ubuntu 16 (some commands may change for Ubuntu 18)
- Local user with admin rights, updated to use sudo with no password
- Pre-populated SSH host keys and StrictHostchecking set to no in the .ssh/config file
- /etc/hosts file pre-populated with the host name and IP of each test host.
- Statically defined IP address as per this blog post
Hostnames and IPs
The virtual machines in this environment will be configured with the following hostnames and IPs
192.168.122.31 kube-ub-master
192.168.122.32 kube-ub-node1
192.168.122.33 kube-ub-node2
192.168.122.34 kube-ub-node3
Configuring the master server
The Rook CI scripts are designed to run from the kubernetes master host. These configuration steps are ran as the configured admin user on that host.Update DNS resolver and hostname
On Ubuntu 16, the resolv.conf file is managed by systemd. This configuration can result in the coredns container not starting due to a lookup loop.sudo sed -i -e 's/dns=.*/dns=default/' /etc/NetworkManager/NetworkManager.conf
sudo systemctl disable systemd-resolved.service
sudo systemctl stop systemd-resolved.service
sudo rm /etc/resolv.conf
echo nameserver 8.8.8.8 | sudo tee /etc/resolv.conf
echo kube-ub-master | sudo tee /etc/hostname
Install go 1.11
The CI requires go version 1.11 to be available on the system in order to build the local images and to run the go based test infrastructure.wget https://dl.google.com/go/go1.11.6.linux-amd64.tar.gz
cd /usr/local
sudo tar -zxf ~kschinck/go1.11.6.linux-amd64.tar.gz
cd ~
echo export PATH=/usr/local/go/bin:\$PATH | tee -a ~/.bashrc
source .bashrc
Install and configure docker
Configure docker to use the registry on the master nodesudo apt-get install -y docker.io git ansible curl
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"insecure-registries" : ["192.168.122.31:5000"]
}
EOF
sudo systemctl start docker
sudo systemctl enable docker
Add user to the docker group
To run the CI scripts, the docker user needs to able to run docker commands without using sudo.sudo usermod -a -G docker myuser
The new group assignment can be picked up by either logging out and back in or running the newgrp command.
Start the docker registry container
The docker registry container needs to be running in order to host the locally build versions of the Rook images.docker run -d -p 5000:5000 --restart=always --name registry registry:2
Verify Registry Access
The curl command can be used to verify access to the local registry.curl http://192.168.122.31:5000/v2/_catalog
{"repositories":[]}
Configure Worker Nodes
The worker nodes need to be configure similar to the master node with the exception of needing golang support added.for HOST in kube-ub-node1 kube-ub-node2 kube-ub-node3
do
cat<<EOF | ssh $HOST
sudo sed -i -e 's/dns=.*/dns=default/' /etc/NetworkManager/NetworkManager.conf
sudo systemctl disable systemd-resolved.service
sudo systemctl stop systemd-resolved.service
sudo rm /etc/resolv.conf
echo nameserver 8.8.8.8 | sudo tee /etc/resolv.conf
echo $HOST | sudo tee /etc/hostname
sudo apt-get install -y docker.io
cat <<EOG | sudo tee /etc/docker/daemon.json
{
"insecure-registries" : ["192.168.122.31:5000"]
}
EOG
sudo systemctl start docker
sudo systemctl enable docker
echo ‘export GOPATH=${HOME}/go’ | tee -a ~/.bashrc
source .bashrc
echo $PATH
echo $GOPATH
mkdir -p ${GOPATH}/src/github.com/rook
cd ${GOPATH}/src/github.com/rook
git clone http://github.com/myuser/rook.git
cd rook
git checkout mypatch
EOF
done
Download Source Code
At this point we are ready to prepare for and download the forked git repository.The repository is configured according to the Rook development guidelines.
These commands will prepare the needed directory path, clone the git repo, switch to the development branch and build the images.
cat <<EOF>>~/.bashrc
export GOPATH=${HOME}/go
EOF
export GOPATH=${HOME}/go
mkdir -p ${GOPATH}/src/github.com/rook
cd ${GOPATH}/src/github.com/rook
git clone http://github.com/myuser/rook.git
cd rook
git checkout mypatch
makeTag and push built image
The make command will build and upload a rook/ceph image. In order to use this image, this image will need to be tagged and pushed back to the local registry.docker tag `docker images | grep "rook/ceph" | awk '{print $1":"$2}'` 192.168.122.31:5000/rook/ceph:latest
docker push 192.168.122.31:5000/rook/ceph:latest
The local test framework files need to be update to use the new image. The following command will update the image reference to the local registry.
sed -i -e 's/image: rook.*$/image: 192.168.122.31:5000\/rook\/ceph:latest/' ./tests/framework/installer/ceph_manifests.go
Deploy master node using CI script
cd tests/scripts
./kubeadm.sh up
The end of this command will display the kubectl command used to configure additional worker nodes. This needs to be copied and used in below worker node configuration setpDeploy worker nodes using CI script
This command will add each worker host to the kubernetes cluster.for HOST in kube-ub-node1 kube-ub-node2 kube-ub-node3
do
cat<<EOF | ssh $HOST
cd cd ${GOPATH}/src/github.com/rook/rook/tests/scripts
./kubeadm.sh install node TEXT_TAKEN_FROM_THE_MASTER_INSTALL_OUTPUT
EOF
Setup the kubectl command for debugging
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
At this point, the admin will be able to use the kubectl command to interact with the deployed kubernetes cluster.
Run CI Test
Rook uses the provide test framework of go to run the validation commands. The test command is ran from the top of the rook repository directory. It will deploy a variety of configurations and validate the operation of the cluster. Output is saved under the _output directory.This command will run the standard tests called SmokeSuite and save the output to a file in the /tmp directory.
time go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration 2>&1 | tee /tmp/integration.log
The results can be reviewed from /tmp/integration.log file.
The kubernetes cluster should be in a clean state if the tests finish successfully.
Repeating the CI Execution
If an operator is to be update updated, the previous uploaded images should be deleted from the local repository prior to making updates, rebuilding the image, tagging and pushing.
If the CI configuration is updated, it is sufficient to rerun the go test command line.
