For my work on OpenShift I wanted a way to use my local workstation as a test cluster with vms for a master and multiple nodes. Ideally it would be possible to quickly teardown and rebuild the whole cluster, but I also want reliable hostnames (and IPs) across each rebuild. This post outlines a way to do this with Fedora (25 as of writing) and Vagrant.

The key to getting Fedora configured such that the hostnames and DNS will work is this post by Dominic Cleal. Follow these steps and your vms can request any *.example.com hostname, libvirt’s DNS server will assign IPs and your host OS can resolve them as well, however they’re still not going to always map to the same IP.

For that, we’ll create a custom libvirt network /etc/libvirt/qemu/networks/openshift.xml. (this likely obsoletes Dominic’s step to edit the default network, however it seems to work fine if you have both)

<network>
  <name>openshift</name>
  <uuid>fc43091e-ce20-4af4-973b-99468b9f3d8a</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:29:3d:d7'/>
  <domain name='example.com'/>
  <dns>
    <host ip='192.168.133.10'>
      <hostname>master1.example.com</hostname>
    </host>
    <host ip='192.168.133.20'>
      <hostname>node1.example.com</hostname>
    </host>
    <host ip='192.168.133.21'>
      <hostname>node2.example.com</hostname>
    </host>
    <host ip='192.168.133.22'>
      <hostname>node3.example.com</hostname>
    </host>
  </dns>
  <ip address='192.168.133.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.133.2' end='192.168.133.254'/>
      <host mac='52:54:00:96:a1:3a' name='master1.example.com' ip='192.168.133.10'/>
      <host mac='52:54:00:0d:57:fa' name='node1.example.com' ip='192.168.133.20'/>
      <host mac='52:54:00:0d:57:bd' name='node2.example.com' ip='192.168.133.21'/>
      <host mac='52:54:00:0d:57:bc' name='node3.example.com' ip='192.168.133.22'/>
    </dhcp>
  </ip>
</network>

WARNING: It’s been a really long time since I set this up, the following steps to create and later edit the libvirt network XML may not be 100% accurate.

Create the new network with:

virsh net-define /etc/libvirt/qemu/networks/openshift.xml

After this I believe any changes need to be done with virsh net-edit:

virsh net-edit openshift
virsh net-destroy openshift
virsh net-start openshift
virsh net-autostart openshift

Now we have a handful of hostnames VMs can request, and will always be assigned a reliable IP address so we’ll never run into issues with certs or inventory.

To make use of these hostnames in a Vagrant file:

require 'fileutils'
require 'resolv'

Vagrant.configure('2') do |config|

  config.vm.provider :libvirt do |libvirt|
    libvirt.management_network_name = "default"
    libvirt.management_network_address = "192.168.122.0/24"
  end

  config.vm.synced_folder ".", "/var/vagrant", type: "rsync"

  config.vm.provision "shell" do |s|
    ssh_pub_key = File.readlines("/home/dgoodwin/.ssh/id_rsa.pub").first.strip
    s.inline = <<-SHELL
    mkdir /root/.ssh
    chmod 700 /root/.ssh
    echo #{ssh_pub_key} >> /home/vagrant/.ssh/authorized_keys
    echo #{ssh_pub_key} >> /root/.ssh/authorized_keys
    SHELL
  end

  config.vm.define :m1 do |master|
    master.vm.box = 'rhel-server-7.3'
    master.vm.hostname = "m1.example.com"

    master.vm.provider :libvirt do |domain|
      domain.memory = 4096
      domain.cpus = 1
      domain.storage :file, :size => '10G'
    end

    master.vm.provision :shell do |s|
      s.inline = <<-SHELL
      echo "127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4" > /etc/hosts
      echo "::1         localhost localhost.localdomain localhost6 localhost6.localdomain6" >> /etc/hosts
      echo "DHCP_HOSTNAME=m1.example.com" >> /etc/sysconfig/network-scripts/ifcfg-eth0
      systemctl restart NetworkManager
      SHELL
    end

  end
end

You would similarly add entries for n1/n2 and any other hosts you wanted to make use of in permanent locations in the vagrant file, just alter the fake MAC and claim the appropriate hostname.

Now you can vagrant up all guests, or just a subset, reference them from a constant Ansile inventory, vagrant down when you don’t need them, or vagrant destroy when you’re ready to rebuild. The guests will always come back at the same IP and hostname.

Important Note for VPN Users

If like me you also use your workstation on a VPN, you’ll now have a problem. I use openvpn to connect to the work VPN, which of course requires a new DNS server to resolve hostnames within the VPN. openvpn assumes it can do this, but this overwrites our NetworkManager customizations for DNS. After connecting you’ll now find your host OS (where you’re probably doing development) can no longer resolve your guest hostnames.

My solution for this isn’t pretty, but it works. After connecting to the VPN I run this script, the restart of NetworkManager reconfigures things to use libvirt for DNS again, and I add in config to use Red Hat’s DNS servers.

(root@wrx ~) $ cat vpn.sh
#!/bin/sh
cp ~/redhat_dnsmasq.conf /etc/NetworkManager/dnsmasq.d/redhat_dnsmasq.conf
systemctl restart NetworkManager
cp ~/resolv.conf /etc/resolv.conf
(root@wrx ~) $ cat redhat_dnsmasq.conf
server=/redhat.com/[DNS-SERVER1]
server=/redhat.com/[DNS-SERVER2]
(root@wrx ~) $ cat resolv.conf
# Generated by NetworkManager
search local
nameserver 127.0.0.1

When leaving the VPN, I similarly run:

rm /etc/NetworkManager/dnsmasq.d/redhat_dnsmasq.conf && systemctl restart NetworkManager

Pretty sure this could be automated but my attempts to do so with openvpn hooks all failed. Need to give this another try someday and will update appropriately if I can.

fedora vagrant development