# Fedora, Vagrant, Libvirt, and Functional DNS

For my work on OpenShift I wanted a way to use my local workstation as a test cluster with vms for a master and multiple nodes. Ideally it would be possible to quickly teardown and rebuild the whole cluster, but I also want reliable hostnames (and IPs) across each rebuild. This post outlines a way to do this with Fedora (25 as of writing) and Vagrant.

The key to getting Fedora configured such that the hostnames and DNS will work is this post by Dominic Cleal . Follow these steps and your vms can request any *.example.com hostname, libvirt’s DNS server will assign IPs and your host OS can resolve them as well, however they’re still not going to always map to the same IP.

For that, we’ll create a custom libvirt network /etc/libvirt/qemu/networks/openshift.xml. (this likely obsoletes Dominic’s step to edit the default network, however it seems to work fine if you have both)

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35   openshift fc43091e-ce20-4af4-973b-99468b9f3d8a master1.example.com node1.example.com node2.example.com node3.example.com 

WARNING: It’s been a really long time since I set this up, the following steps to create and later edit the libvirt network XML may not be 100% accurate.

Create the new network with:

 1  virsh net-define /etc/libvirt/qemu/networks/openshift.xml 

After this I believe any changes need to be done with virsh net-edit:

 1 2 3 4  virsh net-edit openshift virsh net-destroy openshift virsh net-start openshift virsh net-autostart openshift 

Now we have a handful of hostnames VMs can request, and will always be assigned a reliable IP address so we’ll never run into issues with certs or inventory.

To make use of these hostnames in a Vagrant file:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43  require 'fileutils' require 'resolv' Vagrant.configure('2') do |config| config.vm.provider :libvirt do |libvirt| libvirt.management_network_name = "default" libvirt.management_network_address = "192.168.122.0/24" end config.vm.synced_folder ".", "/var/vagrant", type: "rsync" config.vm.provision "shell" do |s| ssh_pub_key = File.readlines("/home/dgoodwin/.ssh/id_rsa.pub").first.strip s.inline = <<-SHELL mkdir /root/.ssh chmod 700 /root/.ssh echo #{ssh_pub_key} >> /home/vagrant/.ssh/authorized_keys echo #{ssh_pub_key} >> /root/.ssh/authorized_keys SHELL end config.vm.define :m1 do |master| master.vm.box = 'rhel-server-7.3' master.vm.hostname = "m1.example.com" master.vm.provider :libvirt do |domain| domain.memory = 4096 domain.cpus = 1 domain.storage :file, :size => '10G' end master.vm.provision :shell do |s| s.inline = <<-SHELL echo "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4" > /etc/hosts echo "::1 localhost localhost.localdomain localhost6 localhost6.localdomain6" >> /etc/hosts echo "DHCP_HOSTNAME=m1.example.com" >> /etc/sysconfig/network-scripts/ifcfg-eth0 systemctl restart NetworkManager SHELL end end end 

You would similarly add entries for n1/n2 and any other hosts you wanted to make use of in permanent locations in the vagrant file, just alter the fake MAC and claim the appropriate hostname.

Now you can vagrant up all guests, or just a subset, reference them from a constant Ansile inventory, vagrant down when you don’t need them, or vagrant destroy when you’re ready to rebuild. The guests will always come back at the same IP and hostname.

## Important Note for VPN Users

If like me you also use your workstation on a VPN, you’ll now have a problem. I use openvpn to connect to the work VPN, which of course requires a new DNS server to resolve hostnames within the VPN. openvpn assumes it can do this, but this overwrites our NetworkManager customizations for DNS. After connecting you’ll now find your host OS (where you’re probably doing development) can no longer resolve your guest hostnames.

My solution for this isn’t pretty, but it works. After connecting to the VPN I run this script, the restart of NetworkManager reconfigures things to use libvirt for DNS again, and I add in config to use Red Hat’s DNS servers.

  1 2 3 4 5 6 7 8 9 10 11 12  (root@wrx ~) $cat vpn.sh #!/bin/sh cp ~/redhat_dnsmasq.conf /etc/NetworkManager/dnsmasq.d/redhat_dnsmasq.conf systemctl restart NetworkManager cp ~/resolv.conf /etc/resolv.conf (root@wrx ~)$ cat redhat_dnsmasq.conf server=/redhat.com/[DNS-SERVER1] server=/redhat.com/[DNS-SERVER2] (root@wrx ~) \$ cat resolv.conf # Generated by NetworkManager search local nameserver 127.0.0.1 

When leaving the VPN, I similarly run:

 1  rm /etc/NetworkManager/dnsmasq.d/redhat_dnsmasq.conf && systemctl restart NetworkManager 

Pretty sure this could be automated but my attempts to do so with openvpn hooks all failed. Need to give this another try someday and will update appropriately if I can.