House Keeping

First, some housekeeping. I have not been posting very actively. This is not an apology with a promise to do better. This is an apology with an adjustment of expectations. My life tends to sporadically get very busy for weeks or months at a time with a few days or weeks in between of nothing. This makes it difficult to find time to write on any regular basis. With that said, I want to let you, dear reader, know now that my blog is not updated regularly or, sometimes, even very often. Just because there is a stream of posts that come out back-to-back over the course of a month does not mean that pace of content will continue. If months, or heck, even a year goes by without anything new here, I’m probably still alive. The best thing to do is to add me to your RSS reader (yes, I said it, use an RSS reader, it’ll change your life) and get notified when I post instead of having to check back here all the time in case I put something new up.

Introduction

I ran across a terribly confounding issue today with the vagrant-libvirt provider while looking for a solution to the lack of virtualization software on my M1 Macbook. To summarize, Homebrew on an M1 Macbook seems to be a little bit split brained when it comes to the packaged architectures it provides, despite having Homebrew installed in the, native, ARM configuration. To my surprise, this issue also seems to be little documented online at this time. This post is my attempt to contribute to what’s out there on the subject. Hopefully, it saves the next poor soul dealing with this a great deal of time. As always, if you are that poor soul, please reach out to me. I love hearing from my readers!

The Problem

Apple has done a wonderful thing, they have brought ARM adoption to the mainstream and this has brought a flurry of adoption in the software ecosystem around this (superior) architecture. For 99.9% of workloads, an Macbook Pro with either an M1, or now the M2, processor is a wonderful bit of kit. It’s fast, it provides fantastic battery life and it runs much cooler which means less fan noise for cooling! Unfortunately, a processor architecture shift like this being (thankfully) such a rare occurrence in the modern computing world means that change is slow. Apple did a fantastic job with Rosetta 2 and for most things I hardly noticed anything had changed… mostly

The problems only arise when you’re trying to do relatively low level work, such as virtualization. Xhyve is, currently, the only truly viable virtualization platform for Apple silicon at the time of this writing. Both Virtualbox and VMWare Fusion have little or no support for it yet, and understandably so… but progress is slow. This means that developers have only a few options for creating development environments. The most common replacement for Vagrant for local development and testing environments has been docker. And this works very well for some or many cases but what if we need to test something that can’t run in a container or we need to test something that isn’t containerized in production, is testing in a container really a simulation of production in this case?

My opinion is that the answer to these questions is no. Sometimes, you still need a VM to properly test software or a configuration change before pushing to prod. So, what do you do? My solution was actually, on the surface, quite simple. Use the vagrant-libvirt provider to connect to my KVM server and run my tests there. Sure, it’s not really “local” dev in that case, but the latency is generally low enough and, honestly, there’s more horsepower on that server, that it’s probably going to be a win-win. Here, however, is where the problems start.

The general idea was to install Vagrant and libvirt from Homebrew and then use the vagrant command-line utility to download the vagrant-libvirt provider. Unfortunately, if you have Homebrew setup to download native ARM builds, you will download the ARM build of libvirt but you still get the amd64 build of Vagrant.

You are probably thinking, as I was, that this wouldn’t be an issue. Well, no…

The problem is that you have an AMD64 Vagrant trying to use ARM64 libraries… that ain’t gonna work.

The Solution

If you have read this far, I applaud you. I ramble a lot and, before I ramble more here’s the TL;DR

Solution TL;DR

• Uninstall Vagrant from homebrew
• Download Vagrant from github and bundle install it
• ???
• Profit

Solution in Detail

Since there is no native ARM binary of Vagrant (how’s about we get on that, there, Hashicorp, it’s only been… 3 years?) you are going to have to build it from source and when I say from source, I mean with Ruby (audibly gags).

Step 1 - Dependencies

You’re going to need a few things:

• Ruby >= 2.7 (I use rbenv. YMMV)

Step 2 - Build/Install Vagrant from Source

Once you’ve got all of the dependencies installed and set up (no, I’m not going to show you how. This post is getting long enough), navigate in to the directory in to which you cloned the vagrant repo and run:

$ bundle install

If all goes well, then you got everything you needed to install vagrant and should now see a bin directory inside the repo. But wait, there’s more. We now need to create the binstubs (I’m sorry… what… Ruby, install didn’t just “work”?)

$  bundle --binstubs exec

This will produce an exec directory and exec/vagrant. This is your new, bespoke, Vagrant binary. For simplicity sake, I symlinked it to my $HOME/bin directory. Anywhere in your $PATH is fine, though… or just hate yourself and execute it from the exec directory manually, whatever goats your float.

Step 3 - Install the vagrant-libvirt Provider

If everything went smoothly up to here, the following command should “Just Work”

$ vagrant plugin install vagrant-libvirt

There was an additional, longer, command that I saw recommended in a few places which passed some custom environment variables in to the process. I’ll share an example of that below in case you need it:

CONFIGURE_ARGS="with-libvirt-lib=$(brew --prefix libvirt)/lib with-libvirt-include=$(brew --prefix libvirt)/include" vagrant plugin install vagrant-libvirt

Writing your vagrant file

The last part is easy compared to what we went through above. Setting up the Vagrantfile. Here’s an example of what I use for an Ubuntu 22.04 VM:

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  config.vm.box = "generic/ubuntu2204"

  config.vm.provider :libvirt do |libvirt|
    libvirt.host = "my_hostname"
    libvirt.username = "my_user"
    libvirt.driver = "kvm"
    libvirt.connect_via_ssh = true
    libvirt.id_ssh_key_file = "/Users/my_user/.ssh/id_ed25519"
  end

end

Now, running vagrant up should download the box file locally, copy it to the KVM server and spin up your VM on the remote host. EZ-PZ.

Conclusion

OMG WTF WHY IS EVERYTHING LIKE THIS. CAN’T WE JUST HAVE SOFTWARE THAT WORKS!!!11

But, all kidding aside, this was annoying. I wish we could just have nice things but sometimes we have to make them ourselves.