Running Ansible Against Chroot Directories

Several cluster managers, including BCM, Warewulf, and Scyld use chroot's or chroot like methods for provisioning. This method allows the systemd administator to build a golden image with the requisite packages and configurations, then the cluster manager will usually via it's own methods handle final configurations during provisioning.

Some of these cluster managers have documented methods for using Ansible, such as BCM's use ssh chroot directories (I'll break this method down in a future article), or document a post boot script to run Ansible once the node is up.

Several conversations I have had and witnessed show most are unfamiliar with running Ansible directly against an existing chroot directory using an existing Ansible connector. This connector is not very well documented, but allows one to target an image directly as if it were a target host for an ssh connection.

I'll cover how to get started as well as some of the gotcha's I've found running this in produciton environments.

Chroot Setup

I'll be running on an up to date rocky9 host, but this should work on other EL distributions with equivalent commands for Debian based systemd.

An example chroot install, pulled from the warewulf documentation, will work here:

yum install --installroot /tmp/newroot --releasever 9 basesystem bash \
    chkconfig coreutils e2fsprogs ethtool filesystem findutils \
    gawk grep initscripts iproute iputils net-tools nfs-utils pam \
    psmisc rsync sed setup shadow-utils rsyslog tzdata util-linux \
    words zlib tar less gzip which util-linux openssh-clients \
    openssh-server dhclient pciutils vim-minimal shadow-utils \
    strace cronie crontabs cpio wget rocky-release ipmitool yum \
    NetworkManager

It is then possible to install packages and navigate as the root filesystem using the chroot command

[root@rocky9 ~]# chroot /tmp/newroot/
bash-5.1# yum search slurm
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:02:22 ago on Tue May 14 17:03:08 2024.
================================ Name & Summary Matched: slurm ================================
pcp-pmda-slurm.x86_64 : Performance Co-Pilot (PCP) metrics for the SLURM Workload Manager
bash-5.1#

Also, in order to resolve dns, we will need to add a default entry to /etc/resolv.conf. Normally this gets handled by the cluster managers, but in our "minimal" image we will need to provide it ourselves.

# From within the chroot shell
echo "nameserver 1.1.1.1" >> /etc/resolv.conf"

Ansible Setup

The chroot plugin will already be included with an ansible install, which we'll create using the virtualenv module.

# Create the virtual environment
python3 -m venv venv
# Activate the environment
source ./venv/bin/activate
# Upgrade pip to savoid some warning
pip install --upgrade pip
# Install ansible
pip install ansible

Next we'll create a simple inventory file inventory.cfg, pointing to the chroot directory created above and specifying the following:

/tmp/newroot ansible_connection=chroot

Running Ansible

Finally we'll create a test playbook to install the nmap package into the image.

For Example:

- hosts: all
  tasks:
    - name: Install nmap  
      ansible.builtin.package:
        name: nmap
        state: present

Then its a matter of running the playbook.

(venv) [root@rocky9 ~]# ansible-playbook -i inventory.cfg playbook.yml

PLAY [all] *******************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************
ok: [/tmp/newroot]

TASK [Install nmap] **********************************************************************************************
changed: [/tmp/newroot]

PLAY RECAP *******************************************************************************************************
/tmp/newroot               : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

And there we go. Direct management of a chroot image using Ansible.

Caveats

There are a few caveats that one should be aware of when using the plugin.

Systemd Services

Systemd cannot be directly managed using ansible as it is not running in this environment. You will instead have to apply some logic to the playbooks or roles to take this into account. Thankfully, ansible provides a variable ansible_is_chroot, which can be referenced within the playbooks.

As an example to enable slurmd:

- name: Start slurmd service
  ansible.builtin.service:
    name: slurmd
    state: started
    enabled: true
  when: not ansible_is_chroot

- name: Enable slurmd in Chroot
  ansible.builtin.shell: systemctl enable slurmd
  when: ansible_is_chroot

Warewulf "containers"

While Warewulf containers are built using Dockerfiles, it is possible to run Ansible against them after they have been downloaded, and before they are built into a tarball.

The images are typically located

Run as root

Unfortunately, you have to run ansible as the root user when using the chroot connector. There is no option to use become.

After digging through the internals I have found a way to inject a call to sudo into the commands, however it is cludgey confusing when the sudo prompt appears in the middle of execution. I hope to fork the plugin one day and add support for become so ansible can be run outside of root.

Conclusion

We use the plugin a lot since we manage several Bright Clusters. It has made it easier to maintain consistency in our images, reuse roles across clients, and make it easier to track what has been done to our images.

Next I plan to show how you can use ssh to connect to a chroot based image, bypassing the builtin chroot plugin.