============================ Installation and development ============================ The ``ci-tron`` container can in turn be run from a host OS, booted directly using `boot2container `_ or netbooted over the internet using `iPXE boot server `_. Requirements ============ The CI-tron gateway container is meant to be run on the gateway machine of a CI farm, and comes with the following requirements: - **Hardware**: Two network interfaces, one connected to the internet while the other one is connected to a switch where all the test machines to be exposed by the gateway are connected. - **Volumes**: The container requires a volume to store persistent data (mounted at ``/config``), and optionally a temporary volume which acts as a cache across reboots (mounted at ``/cache``). - **Container**: The container needs to be run as privileged, and using the host network Building the container ====================== The container image is built using `rootless podman `_ , and is provisioned using Ansible recipes (see the ``ansible/`` subproject). The following is a (likely incomplete) list of dependencies for a system running Debian/Ubuntu: .. code-block:: bash $ apt install buildah \ git \ jq \ make \ netcat-openbsd \ podman \ python \ qemu-utils \ qemu-system \ skopeo \ socat \ wget Note: In order to reduce round-trips to an external registry, a local registry is automatically started when running certain make targets. The container image can be built with: .. code-block:: bash $ make gateway Build options ------------- - ``V=1`` Turn on more verbose logging messages in the build process - ``ANSIBLE_EXTRA_ARGS="-vvv ..."`` Pass any custom flags to ``ansible-playbook``. Helpful for re-running only tagged roles in the ansible build, for example. You can also use this to override variables used by the ansible playbook, for example: ``ANSIBLE_EXTRA_ARGS="-e foo=bar"`` - ``IMAGE_NAME=localhost:8088/my/image`` The container name to tag the image with. *WARNING:* The image will automatically be pushed to the registry that got tagged! Defaults to ``localhost:8088/gfx-ci/ci-tron/gateway:latest``. Once completed, a container image will be generated, for example, .. code-block:: text Successfully tagged localhost:8088/gfx-ci/ci-tron/gateway:latest 60cc3db9bedd2a11d8d61a4433a2a8e8daf35a59d6229b80c1fdcf9ece73b7ab Notice that it defaults to a ``localhost`` registry. This is to save on bandwidth, since the ci-tron container is quite big. Running the infrastructure ========================== While not strictly required, enabling nested KVM on the host will dramatically improve the performance of a DUT VM booted from the VPDU. It can be enabled by configuring the relevant module parameter: .. code-block:: bash $ cat /etc/modprobe.d/kvm.conf options kvm-intel nested=Y options kvm-amd nested=1 Nested KVM support can be verified on the host by reading the "nested" parameter from sysfs: .. code-block:: bash $ cat /sys/module/kvm_*/parameters/nested Y We are now ready to start our virtual gateway machine, which will boot directly into the container we built in the previous section: .. code-block:: bash $ make vivian [GITLAB_REGISTRATION_TOKEN=...] [GITLAB_URL=https://gitlab.freedesktop.org] Note 1: The ``GITLAB_REGISTRATION_TOKEN`` should be retrieved from the ``GITLAB_URL`` GitLab instance (`documentation `_). Note 2: options to vivian can be passed by setting ``VIVIAN_OPTS``, for example: ``$ make VIVIAN_OPTS="--ssh-port=50022" ... vivian``. See ``vivian/vivian -h`` for the full list of options it supports. For information on starting virtual DUTs in the VM, see the section on "Spawning virtual DUTs" below. The virtual testing recipes will fetch a Linux kernel and a boot2container ramdisk, and start the system. After the kernel boots and loads the ramdisk, the ramdisk will then pull the ci-tron container, and hand control to it. **N.B:** Due to the stateful nature of the permanent partition in the VM's tmp disk, it is wise to occasionally delete said disk and check a fresh boot continues to work as expected. This can be done using ``make clean``. By default, the boot logs will be redirected to your console until the SSH service comes up. At this point, vivian will switch the console to an SSH-based one. In the boot logs, you should see a Linux kernel booting, then boot2container downloading the container we built, then systemd logs, and finally a dashboard which looks mostly green. On some distros, you may receive the following error message when vivian tries to run qemu: .. code-block:: bash failed to create tun device: Operation not permitted qemu-system-x86_64: -nic bridge,br=vivianbr0,mac=DE:AD:BE:EF:00:12,model=virtio-net-pci: bridge helper failed Re-try after enforcing the user ownership on ``/usr/lib/qemu/qemu-bridge-helper`` on your test system: .. code-block:: bash # chmod u+s /usr/lib/qemu/qemu-bridge-helper There are several options for getting a shell on the gateway: - create one from the dashboard by pressing ``Ctrl-b c`` in tmux - connect via SSH using ``make vivian-connect`` - use netcat to connect to the qemu console with ``nc localhost 4321`` Spawning virtual DUTs ===================== Right now, our gateway has no idea about any potential machine connected to its private network interface. Let's boot one! If the gateway is running in vivian, then virtual DUTs can be booted by using the `vivian-dut` make target. The `OUTLET` variable can be set to select the outlet on the VPDU: .. code-block:: bash $ make OUTLET=4 vivian-add-dut Alternatively, you can use the dashboard to add/start a virtual DUT by selecting one with the arrow keys and submitting the **DISCOVER** button. Once a DUT VM has started, output from it is redirected into a log file located at ``/cache/vpdu/dut-log--.log``. This log file is also tail'd in a new tmux window on the gateway's console dashboard (``CTRL + B`` in tmux then 0..n to select the window by ID). If all went well, you should now see a machine appear in the "Machines" column of the dashboard, with the state ``DISCOVERING``. After a short while, it should transition into ``TRAINING``. During the ``TRAINING`` step, the machine will be booted in a loop for a set amount of times in order to test the boot reliability. After the boot loop is complete, the machine will be marked as ready for testing and its state should change to ``IDLE``. Your first virtual test machine is now available for testing both locally, and on your chosen Gitlab instance. Running a job on the virtual gateway ==================================== Now that you have at least one machine available, you may run jobs on it using the following command: .. code-block:: bash $ executorctl -e http://localhost:8000 run -c 10.0.2.2 -t virtio:family:VIRTIO $JOB_FILE If all went well, congratulations! You seem to have a functional setup! Most of the steps above are amenable to further configuration. You are now in a position to play around and modify defaults to your testing requirements. Hacking on the infrastructure ============================= After making changes to Ansible or any other component of the gateway image, it is recommended to test the changes by either re-generating the gateway image, or running the ansible playbook against an already-running vivian instance: .. code-block:: bash $ make vivian-provision If you want to iterate faster, you may limit the ansible job to some roles by using the TAGS= parameter: .. code-block:: bash $ make TAGS=dashboard,minio,executor vivian-provision You can get a full list of available tags by running: .. code-block:: bash $ ansible-playbook --list-tags ansible/gateway.yml Production deployment ===================== The ci-tron infrastructure is meant to be netbooted over HTTPS, with the boot configuration being stored in a git repository for easy of update. All of the steps are outlined at `iPXE boot server `_.