Posts on The Platfrastructure Life

What do I use?

platfrastructure@xy0.org (Ted Wood) — Wed, 24 Jun 2026 08:00:09 -0400

Inspired by Kevin Quirk I have decided to make one of those Uses pages in which I try to catalog the things I use.

If you read it and have thoughts or comments, feel free to drop me a line via email. I am always interested to hear what my readers have to say.

New Beginnings

platfrastructure@xy0.org (Ted Wood) — Thu, 22 Jan 2026 12:40:00 +0000

Happy Thursday and welcome to another day in my life. This is a bittersweet post today; I am changing jobs. In 2016 I accepted an offer to join the System Engineering team with Mailchimp. The team at the time was relatively small, maybe 20-30 engineers within operations at the time. The company was hiring and growing very rapidly during this period. Within 6 months of being hired I was one of the more senior engineers on the team. A few years later, around 2019, I was promoted to Senior Engineer and changed teams, internally to work with our internal dev tooling team. This team would later become what modern organizations would consider developer experience. My day-to-day shifted from break fixes and standard operations type work to more development and infra tooling work.

Shortly after this transition the world was tossed upside down with COVID and the associated lock-downs. The office closed and everyone began to work from home. This was fine for my team at the time because we were already one of the few mostly remote teams already. The rest of the company, however, struggled to adapt.

Fast-forward to 2021 and it’s announced that Mailchimp is to be acquired by Intuit. There was a lot of concern among the general rank and file that this was going to lead to the destruction of our carefully curated culture. Many people left during this time and things were generally a little chaotic. But, for the next couple of years, things hummed along with business as usual. That was until July 2024.

In July 2024, Intuit announced that they would be doing a fairly sizeable layoff; reducing their staff by about 10%. This sent a shock through the Mailchimp employees as it was seen by many that the layoff disproportionately affected Mailchimp’s numbers. This was purely speculation and I won’t comment on it directly but to say that it was seen as a change in posture within the company and not one that many long-time Mailchimp employees liked. In addition to the layoffs, many people voluntarily left the company in the following days and weeks as a result of their shift in feelings towards the new parent company.

For the next two years, things continued along a new trajectory. Intuit, being fully invested in the AWS ecosystem, was pushing hard on Mailchimp engineering to consolidate our services in to their environment. Most of the work being done shifted from new features to lift-and-shift migrations and re-architecting of core services. After a while, the writing began to appear on the wall that those of us who worked on what was now seen as the “legacy” Mailchimp environments were going to be out of a job in short order. It was time to start figuring out what to do next.

And that leads me to today, January 22, 2026. After 10 years with Mailchimp/Intuit I have tendered my resignation. I leave the company with no ill will and wish the best to those I’ve worked with over the years who will remain. The company is shifting as all companies do and some will have to find new opportunities. I hope that many of them are able to find internal mobility moves when the time comes but I recognize that some of them will have to find new opportunities outside the company. This is just a reality of business and I wish everyone the best.

I look forward to starting the next chapter in my story. I’ve accepted an SRE position with Censys. It seems like a really good fit in terms of matching up my existing skills to their needs while also opening a lot of doors to grow in areas that I would have struggled to do within Intuit. I’m excited to get to know the new team and get the opportunity to finally work remotely, full-time.

VIM-like navigation in MacOS

platfrastructure@xy0.org (Ted Wood) — Thu, 20 Nov 2025 09:46:34 -0500

Introduction

I have been a heavy VIM user (and now Neovim user) for many, many years now. It’s to the point where the vim style of navigation and motions are deeply ingrained in my brain and muscle memory. So, naturally, I want to leverage this super power in all parts of my computer experience. This has lead me on a long journey to continually find ways to adapt every aspect of my setup to support at least some semblance of VIM-like feel.

I use a variety of operation systems from MacOS to Linux (I use Arch, btw) to, even still, Windows. Some of these operation systems make it easier than others to accommodate my preferences. Others, such as MacOS (the subject of this post), not so much.

This post will focus on this difficult category of MacOS interface customization. Over the years I have sourced a number of blogs and StackOverflow posts to inform my setup but, most recently I stumbled upon this post by Mastodon user heywoodlh. Many of the topics discussed herein are derived from this post so kudos to them for the inspiration.

Key Components

To make this all work, we’re going to need a handful of applications:

Vimium - Keyboard navigation for your browser
Shortcat - Keyboard navigation for your OS
cliclick - CLI driven mouse actions
hammerspoon - Automation tool for MacOS
skhd - Custom shortcuts (used here for calling cliclick with keyboard shortcuts)

Vimium

Vimium provides a way to navigate webpages within your browser using a very vim-native feel. Some of the key features of this plugin that I use every day are:

hjkl page navigation
link highlighting
URL yanking
searching a page with /, n and N
gg and G to jump to the top and bottom of a page

These are just some examples of what you can do with the plugin. Be sure to visit the app page linked above for all the different commands that can be used. Additionally, Vimium provides a powerful filtering option to allow you to exclude certain webpages from vim navigation. This is particularly helpful for websites, such as Gmail, that have their own keyboard shortcuts that may conflict or for pages that do not behave as you expect with the plugin.

Shortcat

The article by heywoodlh, referenced above, discusses using a program called Vimac to generate overlays of keyboard-selectable “buttons” within the focused application. Unfortunately, in the intervening years since the post was written, this product has been deprecated and development work has moved to a paid application called Homerow. While I’m sure that Homerow is a fantastic product, and I did try it out briefly, it is a paid application and Shortcat is free (as in beer) and MacOS is my work environment on to which I do not wish to invest money of my own in. Homerow has some features that are missing from Shortcat, such as keyboard driven scrolling, so I would encourage you to check it out if that is appealing to you.

Shortcat provides the same keyboard-selectable overlay that is provided on a webpage by Vimium but does so using the accessibility features of MacOS to provide them within the focused application window. This can be very handy for navigating applications that do not have their own shortcuts or have shortcuts that are unintuitive and can’t be changed (I’m looking at you Slack).

cliclick

Cliclick is a new piece of software to me. It is a CLI tool that allows the execution of mouse and keyboard related actions from the terminal. With this tool, you can fill in any of the gaps in shortcut-based navigation provided by the previous two applications by directly moving and clicking the mouse through the command line. However, doing this through the terminal is quite inconvenient and we need some way to do it with a keyboard shortcut to really tie it all up with a nice bow.

Hammerspoon

Unfortunately, Cliclick does not support scroll wheel activation. It looks like there is a Github issue for this but it has been open for years with no resolution. This is where Hammerspoon comes in. Hammerspoon is a tool for automating tasks on MacOS. It is configured with a Lua script that can be used to execute arbitrary commands and actions. We will use this to scroll the window when we need to. There may even be way to replace cliclick with Hammerspoon entirely but I have not investigated that yet.

skhd

This handy application is the real gem of this entire setup in my opinion and is the key pieces that I took away from the post by heywoodlh. skhd is a generic keyboard shortcut application for MacOS that is configured with a config file ($HOME/.skhd). Using skhd we can execute our cliclick commands from keyboard shortcuts. Heywoodlh provides an excellent configuration example that I’ve directly copied and used in my own configuration and will share here:

ctrl - k : cliclick "m:+0,-20" #up
ctrl - j : cliclick "m:+0,+20" #down
ctrl - l : cliclick "m:+20,+0" #right
ctrl - h : cliclick "m:-20,+0" #left

ctrl + shift - k : cliclick "m:+0,-40" #up (faster)
ctrl + shift - j : cliclick "m:+0,+40" #down (faster)
ctrl + shift - l : cliclick "m:+40,+0" #right (faster)
ctrl + shift - h : cliclick "m:-40,+0" #left (faster)

ctrl - 0x21 : cliclick ku:ctrl c:. #click with ctrl+[
ctrl - 0x1E : cliclick ku:ctrl rc:.  #right click with ctrl+]

ctrl + shift - 0x21 : /opt/homebrew/bin/hs -c 'scroll.scrollUp()' #scroll up with ctrl+shift+[
ctrl + shift - 0x1E : /opt/homebrew/bin/hs -c 'scroll.scrollDown()' #scroll down with ctrl+shift+]

Conclusion

That’s it for now. That’s my basic setup. There are a few other small changes I’ve made to the MacOS keyboard defaults such as swapping capslock with escape, for easier access to the escape key from the homerow, but that’s about it.

I hope you find the information above useful and if you have other suggestions on ways to improve this setup or want to share your own setups that help improve your keyboard driven experience on MacOS, please send me an email.

Meshtastic Hardware Overview

platfrastructure@xy0.org (Ted Wood) — Tue, 18 Nov 2025 07:32:32 -0500

I get asked fairly frequently what sort of hardware I recommend for someone just getting started with Meshtastic. I get asked often enough that I’ve decided I should probably write this down to make it easy to share and to make the information a bit more discoverable.

A Question

The question I get is usually something along the lines of the following:

I am new to this whole meshtastic thing but I want to buy a radio or two to get started. What do you recommend for me?

The first answer to this question is actually another question:

What is your use case?

This is a question simple but within it hides a lot of nuance. That nuance is what I’m looking for. Does the person want to carry the node in their bag or pocket? Do they want to install it in a permanent location? Maybe they want to put it on the roof of their car. Do they have access to grid power or will they want to run it on solar? All of these are important things to consider when choosing the right radio.

Portable Use

In this first of our Radio Recommendation series, we will cover portable use cases. I am going to cover this first as it is often the most common, especially for someone just getting started in the hobby. Portable radios should focus on two key aspects: size and battery life. In the following sections I’m going to discuss specific radios that I believe fit this use case. I will discuss what they are and hit on a few key pros and cons of each device. As with all battery powered devices, runtime will vary with use. All battery life numbers provided will be a rough estimate based on usage. Your exact results will vary somewhat but the same conditions are taken in to account for each estimate below so expectations should be relative to another in the list.

I’m not listing these in any particular order. This was intentional to avoid any personal bias I may have towards the manufacturer or device. While I do provide my opinion on what the device is best for, I encourage you to read the summary of each and decide what’s best for you.

SenseCAP T1000-E

Best For - Every day use in relatively RF dense locations where an antenna would draw unwanted attention or cause logistical issue or protection from the elements is warranted.

The SenseCAP Card Tracker T1000-E is approximately the size and shape of a typical office badge (about the size of a credit card, but thicker). This radio is nice for people who want an ultra portable and discreet radio that won’t draw attention when clipped on to a bag or belt loop. It’s IP65 rated meaning it is water resistant. You can safely use this device outdoors without fear of damage from rain or dirt.

Specifications

Price: $40 USD
Radio: Semtech LR1110
CPU: Nordic nRF52840
Battery: 700mAh
Runtime: ~2 days

Pros

Small/Lightweight
Discreet
IP65 rated
Internal GPS receiver
Good battery life

Cons

Internal, fixed Antenna (less range)
Proprietary Charger
Not user serviceable
No built-in display
No WiFi

Heltec T114 V2

Best For - Every day use where an external antenna would not be an issue, protection from the elements can be provided by other means and RF density is somewhat lower (think suburban environment vs. urban/city)

The Heltech T114 V2 is a small, pocket friendly radio, about the size of a pack of Tic-Tacs or an AirPods case. This makes it great for sticking in your pocket or in the side pocket of your backpack. It uses an external SMA connector for the antenna meaning there are variety of options available to you depending on your specific situation. Additionally, the T114 comes as a “kit” of sorts with the board, antenna wire, battery, GPS module, case and basic antenna. This means the unit is fully user servicable. This also means, however, that it is not IP rated. While it has faired relatively well in light rain during my testing, I would not want to leave it outdoors completely unprotected from the elements for a long time.

Specifications

Radio: Semtech SX1262
CPU: Nordic nRF52480
Battery: Up to ~1000mAh will fit in the official T114 enclosure
Runtime: ~3 days with a 1000mAh battery

Pros

Small size
User serviceable
Replaceable battery
Internal GPS receiver
Good battery life
built-in display

Cons

Somewhat less discreet with it’s external antenna
Not IP rated
Included antenna is mediocre
No WiFi

Muzi Works R1 Neo

Best For: The user who needs a premium, bulletproof product and does not mind paying a bit extra for this.

The Muzi Works R1 Neo is a newer entrant in the space. The R1 Neo is a new generation of the original Muzi Works R1.This new model is IP68 rated. It features an enclosure that is a mixture of machined aluminum and 3D printed PETGCF making it one of the most durable radios on this list, it is also the only one on this list to be rated for prolonged submersion in water. In addition, the R1 Neo includes the largest battery of any radio on this list. All of these features come at a price, however, as it is the most expensive radio on this list. But, for that you get a no compromise, portable radio that can handle just about anything you throw at it.

Specifications

Radio: Semtech SX1262
CPU: Nordic nRF52840
Battery: 1500mAh
Runtime: ~5 days

Pros

IP68 rated
High quality external antenna included
Internal GPS receiver
Slim/sleek, aluminum and PETGCF case
Onboard realtime clock
Builtin notification buzzer
Hardware GPS disable switch
Available belt clip holster

Cons

High price
No built-in display
No WiFi

Notable Mentions

Heltec Lora32 V3/V4

The Heltec Lora32 V3 and V4 are a great option if you want a low cost option to get your feet wet in the hobby. However, neither is included in my list of recommended options above for portable use as they do not meet one of the key criteria for portable nodes; good battery life. These boards are based on the ESP32-S3 and consume a substantial amount of battery relative to their Nordic nRF52xxx counterparts. In testing, a Lora32 V3 using a 1000mAh battery will run for optimistically run for 5hrs. Compare this to the 3 days of runtime you can get from the same battery on an nRF52xxx based radio and you can see why I did not include it here. There are two key advantages to this board, however, over the others. The first is price; the Lora32 V3 can be found on Amazon for as little as $20 making it one of the cheapest boards you can find. Additionally, the ESP32-S3, unlike the nRF52xxx family, supports WiFi. This could be useful if, for example, you wanted to place a radio somewhere with wifi connectivity to act as an MQTT gateway (a topic for another day).

Portable Summary

In summary, these are only a small handful of devices available on the market today but I feel that these cover the majority of use “portable” use cases and cover a variety of features at different price points.

Multi-instance Meshtasticd Setup

platfrastructure@xy0.org (Ted Wood) — Fri, 14 Nov 2025 11:12:53 -0500

Introduction

Running meshtastic on a Raspberry Pi is common place in installations where you are both stationary and want to run “high power”. These types of setups are convenient for a number of reasons:

You can power them over PoE
They typically have higher power output than battery powered devices
You can attach multiple radios to a single control plane
You can run “virtual” radios to provide more than one meshtastic API endpoint per device

This article focuses on the latter two points.

Why?

When running Meshtastic on a Raspberry Pi, you need to run the meshtasticd service. This service is responsible for managing the radios and providing the API endpoints for your client application. Each instance of meshtasticd is only capable of communicating with a single radio and providing a single API endpoint, which means only one client can connect at a time. But the Pi has far more horsepower than is necessary to run just a single radio. What if you want to run multiple radios or you want to have multiple client applications connecting to a single radio? Enter meshtasticd multi-instance “mode”.

This is less of a “mode” and more of an architectural concept, but for the sake of brevity I’m going to refer to it as mode throughout this document. The basic idea is that you will run multiple instances of meshtasticd side-by-side on the same Raspberry Pi. Using the recently introduced ability for network connected meshtastic radios to communicate over UDP multicast, we can do some really neat things. For example:

Connect two radios to one Pi and use them to “bridge” two different mesh presets together.
Run virtual radios to provide multiple API endpoints for a single radio.

Prerequisites

Hardware

Raspberry Pi (3/4/5 or Zero/2/2W)
SD Card (8GB or larger)
Meshtastic radio modem (MeshAdv, MeshToad, etc.)

In addition to the list above, you’ll need a way to power the Pi. This can be done with a dedicated power supply or PoE. Just make sure you are using a power supply that can provide at least 5V 2A. This is more than the USB port on your computer can provide so you’ll need to use a dedicated power supply or PoE.

You will also want an antenna for each radio. Never run your radio without an antenna. Especially 1w+ radios. Doing so can cause irreparable damage to your radio.

Software

Since the focus on this article is around running multiple instances of meshtasticd I’m going to assume you already have a few things in place:

A working Linux installation
A working single-instance meshtasticd setup w/ at least one radio

The Setup

meshtasticd is typically run as a background service. On modern systems, this is done using systemd. The file that controls the default meshtasticd instance is located at /etc/systemd/system/meshtasticd.service. This file is responsible for starting the meshtasticd service and keeping it running. It is also responsible for restarting the service if it crashes. Unfortunately, this file (called a unit file) is only designed to run a single instance of meshtasticd. In order to run multiple instances, we need to create multiple unit files. In addition, there are two parameters that must be passed to both the new meshtasticd processes in order to allow them to simultaneously coexist on the same system. The following sections will dissect the standard unit file and explain how to replicate this file with the necessary changes to run an additional instance

The Unit File

The default unit file for meshtasticd looks like this:

[Unit]
Description=Meshtastic Native Daemon
After=network-online.target
StartLimitInterval=200
StartLimitBurst=5

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
User=meshtasticd
Group=meshtasticd
Type=simple
ExecStart=/usr/bin/meshtasticd
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

We’re not going to go in to detail about each of the parts of this file in this article. If you want to learn more, I recommend systemd.unit(5) for a detailed explanation of the unit file format. We’re going to be focusing on the [Service] section, specifically the ExecStart line. This line is responsible for starting the meshtasticd process. In order to run multiple instances, we need to create multiple unit files with slightly different ExecStart lines. We need to add the following three parameters to the ExecStart line:

--port <portnum> - This parameter specifies the port number that the API server will listen on. The default port is 8080. You can use any port number you want as long as it is not already in use.
--config <config_file> - This parameter specifies the config file to use. We will create a new config file for each instance.
--fsdir <fsdir> - This parameter specifies the directory to use for storing the mesh data. We will create a new directory for each instance.

When we add these parameters, we’re going to end up with a unit file that looks something like this, with <port_number>, <instance_name>, and <fsdir> replaced with the appropriate values:

[Unit]
Description=Meshtastic Native Daemon
After=network-online.target
StartLimitInterval=200
StartLimitBurst=5

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
User=meshtasticd
Group=meshtasticd
Type=simple
ExecStart=/usr/bin/meshtasticd --port <port_number> --config /etc/meshtasticd/<instance_name>.conf --fsdir /var/lib/<instance_name>
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Copy the File

Copy the original unit file to a new file with a different name. Replace <instance_name> with the name you want to give this instance.

sudo cp /etc/systemd/system/meshtasticd.service /etc/systemd/system/meshtasticd-<instance_name>.service

Reload the systemd daemon to pick up the new unit file.

sudo systemctl daemon-reload

Edit the new unit file to add the parameters we discussed earlier. Replace <port_number> and <instance_name> with the appropriate values.

sudo systemctl edit --full meshtasticd-<instance_name>.service

Supporting Files

In addition to the unit file, we need to create a config file and a directory for each instance. The config file should be created in /etc/meshtasticd/<instance_name>.conf. The directory should be created in /var/lib/<instance_name>. You can, more or less, copy the original config file to the new location and rename it. You should then edit the new config file as you would like for your second radio.

Testing

Once you have created your unit file and config file, you can start the new instance up with:

sudo systemctl start meshtasticd-<instance_name>.service

Once it is running to your liking, you can enable it to start on boot with:

sudo systemctl enable meshtasticd-<instance_name>.service

Conclusion

That’s it! You should now have multiple instances of meshtasticd running on your Raspberry Pi. You can connect to each instance using the port number you specified in the unit file. You can also connect to each instance using the --port parameter when running the meshtastic command line tool. The best part is, you’re not limited to just two instances. The process outlined above can be repeated as many times as you like.

Once you have your multi-instance meshtasticd setup, you may wish to set up UDP multicast between your radios. There is very little documentation on this feature at the time of writing but a quick summary of the setup is as follows:

Configure each radio with --set network.enable_protocols 1 using the meshtastic command line tool.
Ensure that all radios are on the same network (localhost works)

OFAC, GeoIPs and Blocking Unwanted Network Traffic with Python

platfrastructure@xy0.org (Ted Wood) — Fri, 20 Dec 2024 10:26:04 -0500

Background

It’s the Holiday season and my employer has some pretty strict an clea rules around changes that can be made during “peak seasons” which, for our product, starts after the US Thanksgiving Holiday and continues until after the 1st of the new year. This has left me with more time than usual to scratch some of the intellectual itches I have accumulated over the past year. This year I decided to work on writing some tools to help automate some of the more tedious, routine tasks that my team is responsible for. This post, however, has nothing at all to do with that work. But it’s important background to know before we dive in to what this post actually is about.

My wife and I purchased a new home in March of 2024. One of the key attractions of the house was that it had a large (~800sq/ft) unfinished and completely open, daylight basement. As I work from home part time, this was very interesting to me as a future home office space. Fast forward to July of this year and we “broke ground” on finishing the basement and turning the concrete and cinder block dungeon I had been working in the last few months in to a proper space in which I could live and work. It’s now about 95% complete and it was time to start re-wiring my home network that included some new (to me) equipment, a Ubiquiti Security Gateway (USG).

I promise I’m Getting to the Point

While setting up the USG I noticed that there was an option to block IP ranges by country of origin. Being the pragmatic engineer that I am, I understand that it’s simply a reality of The Internet that certain GeoIP ranges produce higher volumes of unwanted or malicious traffic and that most of these countries are not countries from which I never expect to see inbound traffic. But I did not want to blindly block every country’s IP range besides the US after all, I do host a small number of service from my home IP address and I don’t know the exact origin of every user of those services. Enter the Office of Foreign Assets Control (OFAC) or, more specifically, OFAC’s Sanctions List Service (SLS).

OFAC and their SLS

For those unfamiliar with OFAC or their SLS, here are some brief excerpts from their website:

The Office of Foreign Assets Control administers and enforces economic sanctions programs primarily against countries and groups of individuals, such as terrorists and narcotics traffickers. These sanctions can be either comprehensive or selective, using the blocking of assets and trade restrictions to accomplish foreign policy and national security goals.

Basically, they’re the organization, within the United States, which enforces the sanctions imposed by executive and legislative branches of government. As part of their efforts in enforcing these sanctions, they provide a service they call the Sanctions List Service (SLS). Again, quoting from their website:

OFAC’s Sanctions List Service (SLS) provides users with easy access to the most up-to-date Sanctions Lists and Sacntions list data ready for immediate download

Effectively, and API for fetching data about current government sanctions.

GeoIPs

The Internet is a big place, on the IPv4 space alone there are 4,294,967,296 unique IP addresses. With the emergence of widespread usage of IPv6, this number has increased to 340 undecillion (yes, that’s apparently an actual number. That’s 340 with 36 zeroes after it or 3.4x1038 for you nerds out there.). That’s a lot of unique devices and a lot of people you probably don’t know.

The Internet is also a (relatively) organized place. Countries are assigned blocks of IP addresses by the Internet Assigned Number Authority (IANA). To save you from having to read it here, if you want to learn more about IANA and GeoIPs, head over to Geo Targetly to learn more, it’s way more than I want to cover here. For our purposes here, you just need to know that IANA assigned blocks of IPs to countries and, to a lesser extent, cities. This information is public and can be found from a number of sources online. One popular source is MAXMIND. They provide a free and paid tier service for downloading up-to-date GeoIP information. They even have official libraries for many of the major programming languages.

Putting the Pieces Together

So, let’s see, I wanted to block IPs from countries that have no business on my network… but I don’t want to blindly block every country, just the ones that we’ll call “adversarial” with respect to my relative geographical location. We should be able to do this by sourcing OFAC’s most recent sanctions list and using that to lookup IP addresses in the MaxMind database, right? Sort of….

Now, I may be wrong but I spent quite a while looking for a way to ask the question:

What IP addresses belong to country X?

The issue stems from the format of the free tier MaxMind GeoLite2-Country database. It is structured with the intention of looking up information by the IP address… which is fine, and is definitely in and of itself useful, but it’s not what we want to do here. After much trial and error, it did not seem like it was going to be possible to efficiently use the GeoIP2 library, with the free tier database, to ask this question.

Fortunately, there is a second Python library available called MaxMindDB. This library is very similar to the official GeoIP2 library except that they’ve implemented an __iter__() method to list out the contents of the entire database file. It’s not ideal, but maybe we can work with this.

Meat and Potatoes

Using the above mentioned MaxMindDB library and it’s __iter__() method, we can construct our own dataset in a format that lets us look up the IP addresses for countries listed in the OFAC SLS list. It’s not quite done yet, but I think it’s “done enough” to officially announce publicly that I’ve written a tool/library to do just this.

The Country IP Lookup project can consume either the MaxMind Database permalink URL (requires a registered MaxMind account) or a locally cached copy of one of the MaxMind Country databases and return the data as follows:

{
  "RU"": ["a.a.a.a","b.b.b.b", "c.c.c.c"],
  "US": ["x.x.x.x", "y.y.y.y", "z.z.z.z"],
  etc...
}

With this data, you can now take the ISO country codes provided by the OFAC SLS and return a list of IPs and IP ranges. Additionally, this library can act as a standalone tool that accepts a country ISO code and returns the IPs and IP ranges as a newline delimited list, useful for incorporation in shell scripts and other non-python utilities:

./lookup.py
  usage: lookup [-h] -c COUNTRY [--config CONFIG_FILE] [-d DB_CACHE]

Conclusion

I was initially very frustrated when I discovered that there didn’t seem to be an existing way to answer my seemingly simple question. But after tackling this problem, I am kind of glad that there wasn’t. It forced me to think outside the box and put some of the skills I’ve obtained over the last years both as a security focused systems administrator and now software developer to work. But I also know that I’m a lifelong learner and that I don’t know everything there is to know about Python or the subjects that I touched on in this post. It’s entirely possible that there actually is a way to produce this same result with the existing libraries and I just couldn’t figure it out. But now I know that I have a way to do it and it’s something I built myself and know how to use, end-to-end. And I know that if there’s ever a time when I want it to work differently, I’m empowered to get my hands dirty again and make it work differently.

As for the practicality of this experiment…. Is this necessary? Probably not. Do I think my home network is going to be the target of some state sponsored cyber attack? I certainly hope not. But was it interesting to learn all this?

You bet.

Portable Dev Environments with Devpods

platfrastructure@xy0.org (Ted Wood) — Thu, 20 Jun 2024 14:32:12 -0400

If you’re not already following Mischa van den Burg on Youtube, go do it now. Mischa has some really good content for developers and engineers alike. His most recently video (as of the time of this writing) is on the topic of using devpods.sh to create a portable development environment. This got me thinking about how I create my development environments and if there was a possibility that I could leverage any of the ideas in my day-to-day work.

Present Day

My current development environment is pretty straight forward in terms of tools. I use Neovim (Actually I use LazyVim but that’s a topic for another day) and that’s about it. I have a bunch of customizations and plugins for it, but at the end of the day, it’s just Neovim doing the heavy lifting.

The Motivation

To put it bluntly; shit happens. Since starting with Mailchimp in 2016 I’ve had six laptops. Some of these were replaced after the standard three years but, in reality most were replaced due to theft, hardware failure or company acquisition. Each time, it would take me at least a day or more to get back up and running at 100%. This is wasted time. I need a better way to manage things.

Early attempts

Early in my quest to improve this situation I experimented with several options, some of which I still use to this day. Take, for instance, my dotfiles. I use dotbot to manage and bootstrap my dotfiles in a way that keeps them organized and makes re-installing them a breeze. But this leaves a few loose ends in the form of installed software packages. I tried to mitigate this by generating a Brewfile of all the software I have installed, but that was never quite comprehensive enough and seemed to always require a little extra work every time I had to rebuild the world.

I eventually stumbled upon Nix. The idea of having a standardized way to declare packages and configuration in a single place seemed like it would be a great idea… in theory, at least. In practice, I found Nix to be more than I needed and I was spending more time fighting the configuration than it was worth. Additionally, using Nix to configure applications, while genuinely an interesting and good idea in some cases, can be problematic when you’re trying to learn a new program, plugin or configuration. Most online resources will assume you’re configuring the application directly and, while I’m sure someone who is very familiar with Nix configuration would have little trouble translating, I found it very difficult to learn to configure new software in Nix as I was usually unfamiliar with both.

Devcontainers

Finally, we arrive at the meat of the topic, devcontainers and, more specifically devpods.sh. I’ve been using containers for a long time in my development workflow. Usually, it’s for building and testing my code to ensure I’m doing so in a sterile environment that I know will work in CI (since CI does everything in a container too). I’ve even gone so far as to try and execute builds and tests on a remote docker host with varying degrees of success. But it never occurred to me to move my entire development environment in to a remote container for the simple reason that it always seemed like it would be more trouble than it was worth. Now, using devcontainers and devpods, I can easily define a fully functional environment in just a few steps and while I don’t believe I’ve yet fully realized the potential here, I am already definitely seeing how it could be a powerful option.

Now, the only thing I need to install on my physical machine is devpods itself, configure a local or cloud backend provider and clone my code. The rest of it, my editor, my test suites and my system tools are all ecapsulated inside of the devcontainer itself and a simple (reusable) setup script used during initialization of the environment.

The best part of all of this is that it’s platform agnostic. I no longer need to worry about what OS I’m on or if it supports Docker or how much battery life will suffer from running all these containers. If I’m on a low powered device such as an tablet or Chromebook, I can pull up a terminal Linux terminal, install the AppImage for devpods, configure Google Cloud, Digital Ocean or even my home K3S cluster as the backend provider, clone my code and I have the same exact development environment as I would on my Macbook Pro or desktop computer.

Conclusion

I know I didn’t really cover any technical details in this post and that was actually intentional. Mischa has covered the details of everything I’ve talked about here on his Youtube channel and his Github. So I highly encourage you to learn more by going and checking out his content and experimenting with this yourself.

Using direnv to improve your workflow

platfrastructure@xy0.org (Ted Wood) — Wed, 01 May 2024 12:52:51 -0400

Introduction

For the uninitiated, direnv is a tool that can load and unload environment variables depending on your current working directory. On it’s surface, this seems like a fairly trivial thing but if you take a closer look, direnv packs some really neat features which can really improve your development workflow.

Take for example the “simple” task of including an extra directory in your $PATH while working inside your project directory. Without direnv you’re looking at something like this:

$ cd project_dir/
$ export OLD_PATH=$PATH
$ export PATH=./bin:$PATH

<do some stuff>

$ cd ..
$ export PATH=$OLD_PATH

Envrc

Now, we could certainly write two scripts to do this for us and run them whenever we work in our project environment but we still have to remember to run them when we enter and exit a project environment. What if we had something to handle that for us; enter .envrc.

Instead of having a set of scripts to set up and tear down your environment manually, let’s put the necessary bits in to a file named .envrc within the root of our project directory:

export PATH=./bin:$PATH

Now tell direnv it’s ok to load the file with direnv allow and that’s it! Whenever you enter this directory in the future, direnv will change the value of $PATH for you. But wait, IT GETS BETTER!

The above example works but direnv has some extra tricks up it’s sleeve to make this process even easier and less error prone.

DIRENV-STDLIB

Direnv provides a stdlib of common functions that make working with direnv easier and less prone to common mistakes. Using the stdlib, let’s improve our example above. Replace the contents of the .envrc with the following:

PATH_add bin

You will need to re-allow direnv with direnv allow after any modifications to a .envrc file. Once you’ve done this, you can leave and re-enter the directory and your path will be updated to include /my/project/bin. Using PATH_add has several advantages over directly setting your $PATH. Primary among them being that it avoids common mistakes such as export PATH=bin which would remove all other values of $PATH (effectively breaking your environment).

Advanced Trickery

We have seen how direnv can modify environment variables and we’ve learned that direnv stdlib has some builtin functionality to make our lives either. Now, let’s look at some of the more interesting aspects of direnv and how they can be used to improve your development environment.

Nested Environments

Let’s say we have a project with some subdirectories, maybe it looks something like this:

project
├── bin
│   └── file2.sh
└── foo
    └── bin
        ├── file.sh
        └── file2.sh

We want to add project/bin to our path but if we go in to project/foo/ we want to also add project/foo/bin as well. If we create a .envrc inside of project/ and inside of project/foo/ each containing PATH_add bin we are going to have a problem. In this scenario, direnv will load project/foo/.envrc when we navigate to project/foo/ but in doing so, it is going to unload the rc file from project/. If you guessed the stdlib has a solution for this, you’re absolutely right; it has two options, in fact: source_up .envrc and source_up_if_exists .envrc. Including one of these options in the rc file for project/foo/ will enable us to include both files. The following is an example of the project/foo/.envrc:

source_up .envrc
PATH_add bin

It’s important to remember that these rc files are read from top to bottom in this case. If you had project/bin/my_app and project/foo/bin/my_app and wanted the most specific to be used when running my_app, you need to ensure you include PATH_add bin after sourcing the previous directories rc since doing so is equivalent to:

PATH_add ../bin
PATH_add bin

and PATH_add prepends the specified directory to the front of $PATH.

Custom functions

In addition to the stdlib, direnv provides a way to create custom functions inside of ~/.config/direnv/direnvrc. Inside this file you can define standard bash functions that you want to have available to any .envrc file, allowing you to extend the functionality of direnv. Let’s now take look at one such use case for custom functionality within direnv.

Per Directory Command Aliases (sort of)

Unfortunately, as of today, there is no native method for loading additional shell aliases inside of a .envrc file. However, using custom functions we can kind of fake it. While this does not, strictly speaking, create shell aliases, it does provide a method for creating alias-like commands inside of your .envrc file. The one downside to this approach is that it will create and leave files behind within your project directory. These are easy enough to ignore with a .gitignore file and I’ve seen some examples of ZSH scripts which can handle the load/unload events outside of direnv but I will not be looking at those today. If you want to read more, check out the comments on this PR.

The meat of this is in the custom function. You’ll place the following function inside of your ~/.config/direnv/direnvrc file:

export_alias() {
  local name=$1
  shift
  local alias_dir=$PWD/.direnv/aliases
  local target="$alias_dir/$name"
  local oldpath="$PATH"
  mkdir -p "$alias_dir"
  if ! [[ ":$PATH:" == *":$alias_dir:"* ]]; then
    PATH_add "$alias_dir"
  fi

  echo "#!/usr/bin/env bash" > "$target"
  echo "PATH=\"$oldpath\"" >> "$target"
  echo "$@" >> "$target"
  chmod +x "$target"
}

Using Terraform with libvirt in 2022

platfrastructure@xy0.org (Ted Wood) — Sat, 24 Dec 2022 13:07:00 +0000

Introduction

Following on with the theme of “looking for things to use my new server for” I decided to look and see what other options were out there for managing and provisioning VMs. While Vagrant and Ansible together are fantastic at this, Vagrant really isn’t a good option for “production” VMs (production in quotes here because… it’s my lab server and production is quite relative in that context). I use a lot of Terraform in my day job so I thought I’d look to see if there was a libvirt or kvm provider for it, turns out there is but it’s a little… well, we’ll call it undermaintained as it’s not quite dead but the development of it seems to have slowed considerably in the last year. You can find the provider here.

Now, credit where credit is due. This provider works very well, it’s just that it appears to be a side project for a single developer and we all know we have infinite time for these kinds of things so it’s probably just something that’s fallen on the back burner for one reason or another. That said, with a little tinkering and outside the box thinking, we can still use this provider today, here’s how.

Getting Started

You’ll need a couple of things to get started:

Terraform

I’ve come to like tfenv for installing and managing Terraform executables on my system. I recommend installing it and then using it to install the latest Terraform version. Installation steps are in the README.md found at the link above but here’s a summary for the lazy folks who are also on MacOS.

$ brew install tfenv

I forget whether or not this adds it to your ~/.bash_profile automatically or not so read the message brew gives you when the install is finished. You’ll need to add or confirm that the following line was added to it.

$ grep tfenv ~/.bash_profile

If not, add it

$ test -d $HOME/.tfenv || mkdir $HOME/.tfenv
$ echo 'export PATH="$HOME/.tfenv/bin:$PATH"' >> ~/.bash_profile

Now install the latest Terraform version and activate it:

$ tfenv install
$ tfenv use 1.3.6

If you’re following this tutorial in the future, you may have to run tfenv list to see what “latest” version was that was installed and substitute that version for the one shown above.

Terraform-provider-libvirt

The libvirt provider is configured and used just like any other Terraform provider. Here is an example directly from the project README:

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
    }
  }
}

provider "libvirt" {
  # Configuration options
}

You’ll notice, however, that if you go through the examples in the repo that they’re organized by Terraform version and that the newest version they have listed is 0.13… well, we’ve moved a bit beyond that so you’re going to find that some of the examples don’t work perfectly, thankfully, yours truly has spent some time with this and figure out at least one issue and how to work around it:

Inside of the 0.13 Ubuntu example there are two template files used, one for the network config and one for the cloud-init config. The example uses the, now deprecated, template_file data type. We can safely remove these data objects and replace the references to them in the later libvirt_cloudinit_disk resource with the following:

resource "libvirt_cloudinit_disk" "commoninit" {
  name           = "commoninit.iso"
  user_data      = templatefile("${path.module}/cloud_init.cfg", {})
  network_config = templatefile("${path.module}/network_config.cfg", {})
  pool           = libvirt_pool.ubuntu.name
}

This should now allow the Ubuntu 0.13 example to work with >= 1.0.

Libvirt permissions problems

The last issue I encountered with the Ubuntu example was:

libvirt_pool.ubuntu: Refreshing state... [id=1422d683-633e-4c85-abfc-c182bbd46b43]
libvirt_cloudinit_disk.commoninit: Refreshing state... [id=/tmp/terraform-provider-libvirt-pool-ubuntu/commoninit.iso;12cd7610-9528-439c-8884-c702aa29984a]
libvirt_volume.ubuntu-qcow2: Refreshing state... [id=/tmp/terraform-provider-libvirt-pool-ubuntu/ubuntu-qcow2]

when attempting to apply. Everything appeared to get created properly on the target server but the domain wouldn’t start up. Well, it turns out this is not a libvirt provider issue but, instead, an issue with the qemu configuration shipped with Ubuntu 22.04 by default. I’m not 100% sure if this is because I don’t have selinux or apparmor installed on the server and the qemu package for Ubuntu expects that it’s enabled or if this is legitimately a bug in the pre-packaged qemu installation for Ubuntu, but either way, it’s an easy fix.

If you don’t have apparmor or selinux installed on your system, simple edit /etc/libvirt/qemu.conf and search for the security_driver directive. If you have selinux installed, set it to “selinux”. If you have apparmor installed, set it to “apparmor” and if you have neither installed, set it to “none”. Once you’ve set that parameter, save and close the file and restart the libvirtd service:

$ sudo systemctl restart libvirtd.service

Once the service is restarted, issuing virsh start <domain> should along with subsequent terraform apply runs made to create additional new domains.

Additional Notes and Conclusion

One additional thing to make note of, and I’m not exactly sure yet if this is a shortcoming of the provider or if I just need to look around for something I haven’t yet found in the docs, is that terraform destroy did not completely clean up the created domain. I found that after terraform apply ran to create a domain, subsequent destroy and apply commands would fail with the system complaining that the target domain already existed. I’ve yet to find a way to have Terraform fully clean this up and have had to either use Ansible or manually ssh to the KVM host and undefine the domain.

And in conclusion, I think the libvirt provider for Terraform needs some work. I am certainly glad it exists but with how little attention the project seems to be receiving, currently, and how many issues I ran in to it while trying to configure a simple example domain, I am not sure I’d want to use it for production. The right answer here is probably to use Packer to build a static VM image, Ansible to build and deploy the domain and image and Cloud-init to do any first-run steps that can’t be built in to the vm image during build with packer.

How to use Vagrant on an M1 Macbook with a remote kvm server with Vagrant-libvirt

platfrastructure@xy0.org (Ted Wood) — Fri, 23 Dec 2022 22:45:55 +0000

House Keeping

First, some housekeeping. I have not been posting very actively. This is not an apology with a promise to do better. This is an apology with an adjustment of expectations. My life tends to sporadically get very busy for weeks or months at a time with a few days or weeks in between of nothing. This makes it difficult to find time to write on any regular basis. With that said, I want to let you, dear reader, know now that my blog is not updated regularly or, sometimes, even very often. Just because there is a stream of posts that come out back-to-back over the course of a month does not mean that pace of content will continue. If months, or heck, even a year goes by without anything new here, I’m probably still alive. The best thing to do is to add me to your RSS reader (yes, I said it, use an RSS reader, it’ll change your life) and get notified when I post instead of having to check back here all the time in case I put something new up.

Introduction

I ran across a terribly confounding issue today with the vagrant-libvirt provider while looking for a solution to the lack of virtualization software on my M1 Macbook. To summarize, Homebrew on an M1 Macbook seems to be a little bit split brained when it comes to the packaged architectures it provides, despite having Homebrew installed in the, native, ARM configuration. To my surprise, this issue also seems to be little documented online at this time. This post is my attempt to contribute to what’s out there on the subject. Hopefully, it saves the next poor soul dealing with this a great deal of time. As always, if you are that poor soul, please reach out to me. I love hearing from my readers!

The Problem

Apple has done a wonderful thing, they have brought ARM adoption to the mainstream and this has brought a flurry of adoption in the software ecosystem around this (superior) architecture. For 99.9% of workloads, an Macbook Pro with either an M1, or now the M2, processor is a wonderful bit of kit. It’s fast, it provides fantastic battery life and it runs much cooler which means less fan noise for cooling! Unfortunately, a processor architecture shift like this being (thankfully) such a rare occurrence in the modern computing world means that change is slow. Apple did a fantastic job with Rosetta 2 and for most things I hardly noticed anything had changed… mostly

The problems only arise when you’re trying to do relatively low level work, such as virtualization. Xhyve is, currently, the only truly viable virtualization platform for Apple silicon at the time of this writing. Both Virtualbox and VMWare Fusion have little or no support for it yet, and understandably so… but progress is slow. This means that developers have only a few options for creating development environments. The most common replacement for Vagrant for local development and testing environments has been docker. And this works very well for some or many cases but what if we need to test something that can’t run in a container or we need to test something that isn’t containerized in production, is testing in a container really a simulation of production in this case?

My opinion is that the answer to these questions is no. Sometimes, you still need a VM to properly test software or a configuration change before pushing to prod. So, what do you do? My solution was actually, on the surface, quite simple. Use the vagrant-libvirt provider to connect to my KVM server and run my tests there. Sure, it’s not really “local” dev in that case, but the latency is generally low enough and, honestly, there’s more horsepower on that server, that it’s probably going to be a win-win. Here, however, is where the problems start.

The general idea was to install Vagrant and libvirt from Homebrew and then use the vagrant command-line utility to download the vagrant-libvirt provider. Unfortunately, if you have Homebrew setup to download native ARM builds, you will download the ARM build of libvirt but you still get the amd64 build of Vagrant.

You are probably thinking, as I was, that this wouldn’t be an issue. Well, no…

The problem is that you have an AMD64 Vagrant trying to use ARM64 libraries… that ain’t gonna work.

The Solution

If you have read this far, I applaud you. I ramble a lot and, before I ramble more here’s the TL;DR

Solution TL;DR

Uninstall Vagrant from homebrew
Download Vagrant from github and bundle install it
???
Profit

Solution in Detail

Since there is no native ARM binary of Vagrant (how’s about we get on that, there, Hashicorp, it’s only been… 3 years?) you are going to have to build it from source and when I say from source, I mean with Ruby (audibly gags).

Step 1 - Dependencies

You’re going to need a few things:

Ruby >= 2.7 (I use rbenv. YMMV)

Step 2 - Build/Install Vagrant from Source

Once you’ve got all of the dependencies installed and set up (no, I’m not going to show you how. This post is getting long enough), navigate in to the directory in to which you cloned the vagrant repo and run:

$ bundle install

If all goes well, then you got everything you needed to install vagrant and should now see a bin directory inside the repo. But wait, there’s more. We now need to create the binstubs (I’m sorry… what… Ruby, install didn’t just “work”?)

$  bundle --binstubs exec

This will produce an exec directory and exec/vagrant. This is your new, bespoke, Vagrant binary. For simplicity sake, I symlinked it to my $HOME/bin directory. Anywhere in your $PATH is fine, though… or just hate yourself and execute it from the exec directory manually, whatever goats your float.

Step 3 - Install the vagrant-libvirt Provider

If everything went smoothly up to here, the following command should “Just Work”

$ vagrant plugin install vagrant-libvirt

There was an additional, longer, command that I saw recommended in a few places which passed some custom environment variables in to the process. I’ll share an example of that below in case you need it:

CONFIGURE_ARGS="with-libvirt-lib=$(brew --prefix libvirt)/lib with-libvirt-include=$(brew --prefix libvirt)/include" vagrant plugin install vagrant-libvirt

Writing your vagrant file

The last part is easy compared to what we went through above. Setting up the Vagrantfile. Here’s an example of what I use for an Ubuntu 22.04 VM:

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  config.vm.box = "generic/ubuntu2204"

  config.vm.provider :libvirt do |libvirt|
    libvirt.host = "my_hostname"
    libvirt.username = "my_user"
    libvirt.driver = "kvm"
    libvirt.connect_via_ssh = true
    libvirt.id_ssh_key_file = "/Users/my_user/.ssh/id_ed25519"
  end

end

Now, running vagrant up should download the box file locally, copy it to the KVM server and spin up your VM on the remote host. EZ-PZ.

Conclusion

OMG WTF WHY IS EVERYTHING LIKE THIS. CAN’T WE JUST HAVE SOFTWARE THAT WORKS!!!11

But, all kidding aside, this was annoying. I wish we could just have nice things but sometimes we have to make them ourselves.

Final Day in SF: Post Next 2019

platfrastructure@xy0.org (Ted Wood) — Sat, 13 Apr 2019 17:57:17 -0400

This post has nothing to do with the conference or technology. It will also be brief. I wanted to give a huge shout out to the San Francisco skating community. I was a stranger in their community and they welcomed me with open arms. Having spent a week in San Francisco I came to feel like most of the people there were cold. Rarely did I experience a warm smile on the street, a friendly face. It was very refreshing to find that not everyone in the city is like this. Maybe it’s something about people drawn to inline skating, I’m not sure, but I will be forever thankful of the San Francisco skate community for the warm smiles, the hugs and the laughter we shared on my final day in the city.

Friday was a work day for me. Nothing interesting to report there. The fun started around 6PM local time. Strapped on my skates and headed out to explore. I spent about an hour skating around, finding alleyways and hidden parts of the city not normally found unless on foot. The city has a lot of interesting spots to explore on skates. I highly encourage anyone visiting to try it sometime. Just put your skates on and let the whims take you where they will. I’m sure you won’t be disappointed.

After about an hour of exploration I headed over to the Ferry Building; the meeting spot for the Friday night skate. The skate wasn’t expected to start for a while so there was only one other skater there. We chatted and instantly became friends. He showed me a great food spot not far from the meetup spot where I was able to grab a quick bite before the skate started.

People started to filter not long after we got back from the food run. We danced in the square for an hour waiting for the rest of the gang to show up. Once the time arrived we set out for what was sure to be a fun journey across the city. Taking about three hours, we covered 12 miles of San Francisco. We hit several scenic sites, some great photo spots. Got to bomb a few hills and storm a few tunnels.

After the main event concluded, about half of the group stuck around for a game of roller hockey in the square. This was some of the most fun I’ve had in a long time. We played for about an hour, concluding around 1AM. After the conclusion of the hockey game, I spent another hour exploring the city some more. Let me tell you, at 1AM, San Francisco, specifically the financial district, is a skaters paradise. Some really great spots to explore and almost no cars or pedestrians to worry about.

If you are ever in San Francisco on a Friday night, I highly encourage you to check out the night skate. The group is very welcoming and friendly. The route is about as easy as any route in a city of hills can be and the after skate activities are second to none.

Google Next 2019: Day 4

platfrastructure@xy0.org (Ted Wood) — Fri, 12 Apr 2019 01:22:54 -0400

Day four, the final day. Well, the final day of the actual conference, anyway. The day started like the others. Wake up, go for a run, grab some coffee on the walk to the conference. Probably the best part of the day, to be honest. Not that today was particularly bad or anything but it’s hard to beat coffee and stroll through downtown SF.

The first talk on the schedule today was a security track about identity management in GCP. Unfortunately, due to timezone struggles, I slept in too late and didn’t make it. I’m sure the talk was great but I have no ability to report anything about it.

I had two other talks scheduled for the day. One of them was at 11:40PM, GCP: Move Fast and Don’t Break Things. The second was 1:15PM titled Repeatable GCP Environments at Scale With Cloud Build Infra-As-Code Pipelines. The astute reader at this point is probably wondering when I got lunch in between this… well, I had to sacrifice one of them. I went to the talk on repeatable builds.

The repeatable builds talk was actually really good. They talked about using Terraform, as well as Deployment Manager, to define your infrastructure in code and use cloud builds with forseti policy enforcement to do some sanity checking and build tests before deploying to production. Overall, I came away with a few nice tidbits of information that will hopefully help improve our overall GCP experience.

After the talks were over the team went out for drinks with some Googlers. Then a few of us went over to Berkeley, CA for dinner.

I have one full day left in SF. I have to work from the office for the day but I hope to have enough time to go see a few things before I leave.

Google Next 2019: Day 3

platfrastructure@xy0.org (Ted Wood) — Wed, 10 Apr 2019 20:25:16 -0400

Day three was jam packed with some really interesting talks. It started off with a talk titled Develop Faster on Kubernetes with Google Container Tools and Cloud Build. After lunch, I attended a few of the smaller sessions in more informal fireside chat spaces. These later talks mainly focused on building devops teams and utilizing good devops practices within an organization.

The theme of the day for me, it seemed, was how to utilize Google tools and infrastructure as code to streamline infrastructure automation. I’m sure this was somewhat intentional at a subconscious level as that’s been my primary area of focus at work recently. To this end, Develop Faster on Kubernetes with Google Container Tools and Cloud Build was particularly interesting. They discussed using Skaffold in conjunction with the Skaffold integration in VSCode and Cloud Build to allow developers to quickly iterate on a project without the need to constantly run kubectl and docker commands to rebuild their changes (Now I just need to find something like this for vim ;) ).

The fireside chats later in the day covered a couple of different topics. The first one was mainly about how to build a devops team and form a culture that promotes collaboration across teams. The next session covered Service Levels and Error Budgets. I found this talk particularly interesting as it’s something we’ve been working on at Mailchimp for a while now; finding a balance between pushing new production features while also maintaining a very high level of service availability for our global customer base. One point that I found most interesting was that Stackdriver provides convenient tools for calculating your error budgets based on collected metrics. I could definitely see this being useful as we continue to improve our processes around this.

Finally, the day wrapped up with another small talk called Batteries Included Devops. This talk felt like a higher level continuation of the previous talk, focusing heavily on how Google Cloud provides tools out of the box for making the right devops practices convenient or automatic.

After the talks completed I headed out for a nice Burrito with several other Mailchimp engineers over in the Mission District. It was chilly out but nice to hang out in the park with my coworkers and talk about stuff that wasn’t work related; a rare treat.

I look forward to tomorrow. Another jam packed day full of learning and fun.

Google Next 2019: Day 2

platfrastructure@xy0.org (Ted Wood) — Wed, 10 Apr 2019 12:16:56 -0400

Day two started in the best way possible, with a sunrise skate around the bay. I met up with two of my friends in the area and spent about an hour skating around the west bay. Starting at the Ferry Building and just rolling around, enjoying the weather. I can’t think of any better way to start the day. But it couldn’t last forever, I am here for Google Next after all.

The day started with the first keynote of the conference. I missed a good bit of it because I was skating (oops!) but the last twenty minutes of it were filled with buzzwords and the usual enterprise marketing speak, so I assume I didn’t miss much.

After the keynote ended I wandered around the vendor hall for a little while. Bumped in to a few friends that were running the booths for their companies and had a good opportunity to catch up with them. I also took a moment to enter in to a few raffles; you never know, you might get some free stuff, right!? After wandering around the vendor hall for a little while, I went looking for lunch. The lunches at Google Next are generally box lunches; sandwiches, salads, etc. This year was no different; I had a chicken sandwich.

After lunch I went to the first actual talk session. This was on CI/CD in Kubernetes. I was left very disappointed. I was hoping this talk would be a bit more technical than it was. The speaker ended up mentioning a lot of tools but the part I was hoping to learn, how to link them together in ways that solve complex issues around CI/CD pipelines, was really not covered in any great depth.

The last talk I attended for the day was much better than the last talk. A discussion on access controls in GCP. I was very pleased with the level of technical detail that the speaker included. It got off to a slow start but I appreciated that the speaker was taking the time to cover the basic concepts for some of the less experienced members of the audience. One of the most interesting takeaways from the talk was the information about context aware access controls that are going to be going in to general availability soon. The ability to write access policies that are more or less restrictive based on the context of the access request will be infinitely useful. One of the most interesting uses for my areas of responsibility would be the ability to lock down ssh access to compute nodes based on the on-call schedule. In general, we want to discourage artisanal modification of our compute nodes, but sometimes it’s necessary for an engineer to do it in emergency, on-call situations.

The day wrapped up with some great socialization with my coworkers. I believe that it’s vital to have a very strong relationship with your team members in order for you to perform at your best. This is one of my favorite aspects of conferences. We are all normally so wrapped up in our work that it can be really hard to get to know your team during the normal work work.

That’s all until tomorrow!

Google Next 2019: Day 1

platfrastructure@xy0.org (Ted Wood) — Mon, 08 Apr 2019 18:22:14 -0400

Today was a normal work day for me, but normal only in the sense that I was doing work. The conference itself doesn’t actually start until tomorrow. Mailchimp has a couple of remote offices and one of them is in Oakland, CA. I came in on Sunday with the goal of spending Monday at our California office and getting to know my California coworkers. To that end, I think the day has been a success. I was able to have lunch with one of my Oakland teammates and I’m hoping there will be a few people who want to grab dinner later today.

I have high hopes for Google Next this year. Last year was good but I was still very new to Google’s cloud products. Since last year, Mailchimp has started to do more experimentation with their cloud products and services and I feel that I’ve also grown a bit in my technical understanding in various areas. I’m hoping that this will allow me to absorb more, understand more deeply, the topics that are discussed.

For now, I’m going to go find some food. I’ll be back tomorrow with a day two report!

Google Next 2019: Day 0

platfrastructure@xy0.org (Ted Wood) — Sun, 07 Apr 2019 16:54:03 -0400

I landed in San Francisco, CA today for what is sure to be an interesting week. This week is Google’s annual Google Cloud Next conference. I would be remiss if I didn’t take the opportunity to enjoy a little personal time on the west coast. As usual I brought my skates with me and will be enjoying the Sunday night skate at Golden Gate Park as well as the Friday night skate before I head home.

This year’s hotel is a bit of an experiment. I’m not staying at one of the large chain hotels this year. I opted for a smaller independent hotel in China Town. Of course, I got in super early so I haven’t actually seen my room yet but I did get to have some good dim sum for lunch. So we’re off to a good start.

Sports in my Life

platfrastructure@xy0.org (Ted Wood) — Wed, 20 Mar 2019 17:21:45 -0400

Little known fact about me, I like sports. It feels kind of risqué admitting it to an audience no doubt made up of mostly technologists, geeks and nerds; the very people with whom I regularly make “sports ball” jokes with, but it’s true. But I’m not your typical sports fan. Sure, I watch the Superbowl. Yes, I enjoy going to see a baseball game with friends once in a while (maybe < once a year in reality) but those aren’t my passion. I like individual sports. My personal favorites to watch are speed skating and cycling. Coincidentally, those also happen to be two that I compete in. SURPRISE, I don’t sit in front of my computer all day. In fact, of all the things I do in life, that’s probably my least favorite.

One of the earliest memories I have as a child is my parents teaching me to ride a bike. Some years later, maybe when I was somewhere between 8-11 years old, I discovered the world of inline skating. I’ve felt at home both on two and four wheels ever since. It wasn’t until I was about 17 years old, however, that I really found an interest in riding bikes (more than just as a way to get to a friend’s house at least). I spent the junior and senior years of high school doing basically two things outside of school: biking and discovering the Internet. I would come home from school, immediately get on my bicycle and ride for hours. I’d come home in time for family dinner, do my homework (maybe) and then I was upstairs in my “cave”, online on IRC, message boards, you name it. I would typically be up until 2, 3AM, sometimes later. Up for school the next day at 6AM, repeat.

It wasn’t until last year, however, that I thought that maybe I should try this racing thing. I’d always just enjoyed skating or biking for their own right. I never felt compelled to prove I was better than another person. Something in me has changed in the last year. While I still don’t feel that competitive drive to beat another person, I want to beat myself; to prove that I can consistently improve on my own times and my own achievements. This has driven me to make triathlon my sport of choice this year.

I view triathlon as a sport in which you prove your capabilities only to yourself. Sure you can be faster than another person, you can win the race. But so many people compete not against others but against the clock , against themselves in this sport. It feels like a natural fit. This year, I have no real goals other than to be consistent. I’m on a structured training plan which has already started to really help me improve. I also have a couple of races lined up for the year, just something to motivate me to stay on target with my training.

So, I’m going to try to keep the tags straight on the posts but you might start seeing some non-tech posts popping up every now and again. Hopefully you’ll stick around for the journey.

Soma.fm and Mplayer

platfrastructure@xy0.org (Ted Wood) — Thu, 20 Dec 2018 14:14:08 -0500

I like to listen to soma.fm at work. I also like to do everything (as much as possible) in the command line so I don’t have to context switch between applications too much. For years I’ve used mplayer to listen in. I eventually got tired of forgetting the playlist URL and wrote the following bash function so that I’d only have to remember the name of the playlist I wanted.

soma() {
  mplayer -playlist http://somafm.com/"$1"
}

This worked well for a couple of years, really. I usually only listen to one or two stations and can easily remember the name of the station’s playlist file. Today I decided that I wanted to be able to explore new stations with this shell function as well. Enter the new bash function and a bash completion script.

The new bash function parses https://somafm.com/listen for the MP3 streams and prints them to STDOUT:

somalist() {
  curl -s https://somafm.com/listen/ | awk -F '[<>]' '/MP3:/ { print $4 }' | awk -F '"' '{print $2}' | tr -d \/
}

This is not great but it works, for now. Parsing html with awk/bash/sed or any other generic interpreter is always hit or miss. Right now this works. In the future, maybe it won’t work so great. I’m prepared to accept this and fix it as needed. Now that we have a command that can play our playlists and one that can list them, we need tie it all together with bash completion.

Bash completion scripts are what automatically tab complete options and arguments. These can be static or dynamic. In our case, we want to provide the output of somalist() to the user when they type soma and press the <tab> key. On MacOS the following file should be placed in /usr/local/etc/bash_completion.d (assuming you’re using Homebrew). If you’re on Linux, it will likely be /etc/bash_completion.d. Consult your distribution specific documentation for more details:

#!/usr/bin/env bash

_soma_completion() {
    if [ "${#COMP_WORDS[@]}" != "2" ]; then
        return
    fi

    COMPREPLY=($(compgen -W "$(somalist)" -- "${COMP_WORDS[1]}"))
}

complete -F _soma_completion soma

Source this file with source /usr/local/etc/bash_completion.d/soma and try it out by typing soma followed by the key. As an added bonus, it will give you the most closely matching result if you type a partial name in.

Conclusion

This works for me. It’s a little fragile but I found it useful enough that I’d share it. Check out my Github if you’re interested in some of the other scripts I whip up and updates to this if I decide to make improvements in the future.

Building GRPC on CentOS 6

platfrastructure@xy0.org (Ted Wood) — Fri, 02 Mar 2018 11:04:27 -0500

When transitioning from a monolithic application towards a more service oriented architecture you’re likely to either have a period where you have a hybrid setup until all services are split out of the monolith or make the conscious decision to leave some things be either temporarily or for the long term. Does this mean you should have to sacrifice consistency in your API? NO!

In my case, I was dealing with servers as part of a monolithic application running on CentOS 6. We chose to use GRPC for our services but it requires a newer compiler than provided in the normal CentOS repos. This post will outline my solution to the problem and hopefully help others who find themselves in similar positions.

The Compiler

CentOS 6 was released about 7 years ago. July 10 2011 to be exact. That means it’s Ancient in technology years. These stable OSes are great for running things that you don’t plan on changing for a very, very long time but not so great for developing software using the latest tools and libraries. Such is the case if you want to use GRPC within your application to talk to some of your more modern services. The main issue is that the compiler available in CentOS 6 isn’t C++11 compatible. As a result, the compile flags will not be recognized and it will fail to build. The first thing we need to do then is find a C++11 compatible compiler for CentOS 6.

The Repository

After much searching I came across a repository on people.centos.org owned by Tru Huynh, a developer with the CentOS project since 2005 and member of their infrastructure team. This repository contains the devtools-2 packages which, coincidentally, include a version of gcc which is C++11 compatible. To install this repository and the compiler run the following:

wget http://people.centos.org/tru/devtools-2/devtools-2.repo -O /etc/yum.repos.d/devtools-2.repo
yum install devtoolset-2-gcc devtoolset-2-binutils devtoolset-2-gcc-c++

Using the New Compiler

Once you’ve installed the packages above you’ll need to be sure you’re in the right environment. There are two ways to do this. The quick way to do it (if you’re just testing) is to run:

scl enable devtoolset-2 bash

This will drop you in to a shell with the right environment to use the new compiler. If you’re doing this as part of a script or in a build server where you’ll always want to use this compiler, set the following in your environment (either through .bash_profile, .bashrc, or similar):

echo "WARNING: devtoolset-2 is enabled!"
source /opt/rh/devtoolset-2/enable

That’s it! You can now build GRPC (or wahtever you want) using this compiler and it will work on CentOS 6.

Conclusion and Notes

One thing to note about this repository. It is not an official repo and the packages within it are not signed. The SRPMS are published if you want to look over the work yourself or make any modifications. I’d suggestion you review these tools careful before you decide to use them yourself.

Repairing a Raspberry Pi Boot Partition

platfrastructure@xy0.org (Ted Wood) — Mon, 26 Feb 2018 18:08:28 -0500

The power at my home went out the other day. When it came back on, my ADSB receiver running on a Raspberry Pi, however, did not. It would flash green for a moment and then stop all activity. Plugging it up to a monitor confirmed my fear, it booted to the dreaded rainbow screen and no further. This meant the filesystem was likely corrupt. This post will walk you through my troubleshooting steps and ultimate resolution of this issue.

Troubleshooting

The first thing you want to do with a corrupted SD image is isert it in a working environment and run fsck. On a default Raspbian install there will be two partitions on the SD card. One is formatted fat32, the other will likely be ext4. The following steps assume you’re running linux and you have the SD card inserted in to your computer. I’ll walk you through unmounting both partitions but if you know how to do that, go ahead and do so now.

Unmount the SD Card

You’ll first want to unmount the card (but leave the card inserted) to do this, run the following command:

sudo umount /dev/mmcblkp*

Once unmounted we can proceed to repairing the filesystem.

Repair the est4 partition

Repairing the ext4 partition requires that we have e2fsck installed. Check your package manager if you don’t have the command available.

e2fsk -f -y /dev/mmcblkp2

Once complete, try removing the SD card and booting the Raspberry Pi. If it still won’t boot. Re-insert the card in your computer, unmount it like we did earlier and proceed to the next section.

Repair the boot partition

The boot partition is formatted in fat32. To repair this, we’ll use the dosfsck program. Again, check your package manager to find out what package provides this tool on your system if you don’t have the command available.

dosfsck -t -a -w /dev/mmcblkp1

Make note of any files that this commands reports as “unreadable”. We’ll need to replace these a little later on. If you don’t see this message, you should try booting your card now. It should work. If it doesn’t you may need to replace the entire filesystem. The steps for doing so will be similar to the ones in the next section but you’ll replace every file instead of just the unreadable ones you would have noted during the previous command.

Replace Corrupted Files in /boot

The last repair step is replacing corrupted files in your /boot partition. If you’re here, something went very wrong. Thankfully, repairing things isn’t as bad as some might have you believe. We’ll need to download a copy of the raspbian image file, mount it to our local system and copy the corrupted files from the known good image on to our SD card.

Download Raspbian

The first thing you need to do is download a copy of Raspbian if you don’t already have a copy. You can download that here.

Mount the Image

Mounting an img file is a lot like mounting an ISO file if you’ve ever done that. The difference is that we need to provide the partition offset manually to the mount command.

First, we create a path to mount the img file to.

mkdir /tmp/raspbian

Next, we need to get some info about the file from fdisk. Replace <raspbian_image_path> below with the actual path to your img file.

fdisk -l <raspbian_image_path>

You should see an output that looks similar to this:

$ fdisk -l 2017-11-29-raspbian-stretch-lite.img
Disk 2017-11-29-raspbian-stretch-lite.img: 1.7 GiB, 1858076672 bytes, 3629056 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x37665771

Device                                Boot Start     End Sectors  Size Id Type
2017-11-29-raspbian-stretch-lite.img1       8192   93236   85045 41.5M  c W95 FAT32 (LBA)
2017-11-29-raspbian-stretch-lite.img2      94208 3629055 3534848  1.7G 83 Linux

You want to multiply the Sector size by the start-block for img1. In the example above this would be 512 * 8192 = 4194304. 4194304 is our offset. Now we’ll use that to mount the img file to /tmp/raspbian. Once again, replacing <path_to_image> with our actual file path.

mount -o loop,offset=4194304 <path_to_image> /tmp/raspbian

You should now be able to list the contents of the boot partition in the image file with ls /tmp/raspbian. Now insert your SD card and copy the files from the mounted image to the /boot partition on your SD card. Once you’re done, unmount the SD card and attempt to boot the Pi again.

One thing I noticed when copying the files over was that one of the dtb files was missing from the image. I just skipped over this and replaced what I could. For me, this was fine. YMMV.

Hopefully this helps anyone stuck with a corrupt SD card image get back up and running again. I imagine this same method can be used to repair the ext4 partition as well. Just adjust your offset calculation to use the start-block for the img2 partition instead in order to access the files.

Gophercon Day 3

platfrastructure@xy0.org (Ted Wood) — Sun, 16 Jul 2017 08:28:05 -0600

Yes, you’re not lost. I skipped day 2. Not physically but for the purposes of this blog I’ll be skipping it. Why you ask? It’s simple, I am a day late updating this thing and nothing memorable must have happened because I don’t remember anything specific about the day. The talks were good; I was able to make it in to the ones I wanted this time. I won a plushie Gopher doll at the Google booth. But otherwise, uneventful.

Day three, however, was community day, and my experience was extremely positive and memorable. Community day for me turned in to a Go hack-a-thon. I met up with Kris Nova and others to work on Kris' new project Kubicorn.

Kubicorn is a project that helps a user manage cloud infrastructure for Kubernetes. With kubicorn a user can create new clusters, modify and scale them, and take a snapshot of their cluster at any time. In other words, it’s a simplified re-thinking of how we define, deploy and backup our Kubernetes cluster infrastructure. Rather than using JSON first, we define profiles as struct literals and marshal in to an object. Striving for clean code and an empathetic user experience.

This was my first experience working on a real project with Go. Getting in on a project at such an early stage was a unique experience I was not well prepared for. I am still very new to Go and have only a little experience contributing actual code to a software project. Being able to work with other developers in person as a neophyte was a huge boost. Normally I would not have had the confidence to contribute to anything as technical as this project but having other experienced developers nearby to assist with technical and design questions allowed me to produce several pull requests against the project, as minor as those PRs may have been.

As Gophercon comes to a close, I want to give a huge thank you to the conference organizers. This conference was the best one I’ve attended this year. It’s still growing and with that comes some struggles and the GopherCon team did a very good job of making the conference an experience to remember. I will definitely look forward to being able to attend the conference again in the future. Unfortunately, not next year (the dates conflict with another recurring conference I have attended for over 10 years) but perhaps in 2019.

GopherCon Day 1

platfrastructure@xy0.org (Ted Wood) — Fri, 14 Jul 2017 15:04:47 -0600

GopherCon 2017 day 1 has concluded. From my perspective, it was a good day. The talks were good. The hallway track was, as always, great. The food, fantastic. The crowds, overwhelming.

The day started off with the keynote by Tammy Butow. From there, Russ Cox provided a look at the future of the Go language. After the break for coffee the day resumed with an attempt to see Liz Rice’s talk on Syscalls. I say an atempt was made because the crowds at GopherCon this year were unexpected both by me and the GopherCon team. Popular track rooms quickly filled and many people were unable to attend. This trend continued throughout the day. Thankfully, with better planning, the afternoon tracks went better.

At 2PM Kelsey Hightower spoke about the possibilities of self deploying Go applications on Kubernetes. This talk was informative and, as alwys, entertaining. I question the practicality of his proposals in production but the idea of an application being able to self deploy to a Kubernetes cluster is definitely worth further exploration.

That concluded the talks I attended for the day. There was another failed attempt in the afternoon to attend the talk on Go Anti-Patterns, again due to over crowding.

I look forward to the rest of the week.

GopherCon Day 0

platfrastructure@xy0.org (Ted Wood) — Wed, 12 Jul 2017 09:23:12 -0400

I landed in Denver, CO yesterday for GopherCon 2017. I’ve never been to Denver before. I didn’t know what to expect upon landing. I had heard things, of course. Mostly through what I hear in the media. So, naturally, the only thing I knew was that Marjuana was legal and widely available through shops all over the state. What I wasn’t expecting, however, was what I saw upon approach to land at the airport. Miles and miles of flat farm land with a beautiful backdrop of snow capped mountains. The second thing that struck me was how laid back the airport atmosphere was. It’s quite possibly the least busy airport I’ve been to in recent memory. The final thing that struck me which I’ll mention here is the city. Denver is one of the most beautiful cities in the I’ve yet visited in the United States. I was fortunate to have time after dinner to go for a walk and look around. The city is clean. The people are friendly and the architecture of the buildings around the capital is gorgeous, rivaled only by those I’ve seen in Washington, DC.

If the first night in town is any indication of what’s to come, I look forward to the rest of my stay with great excitement.

Automating the Site with Hugo and Terraform

platfrastructure@xy0.org (Ted Wood) — Tue, 23 May 2017 23:09:14 -0400

Introduction

I’d like to start with a some background. This site is hosted on Digital Ocean. This site is built using Hugo. I deploy this site using Terraform. It runs on the lowest end $5/month droplet from Digital Ocean.

If This sounds appealing to you, read further. We’ll be discussing how you too can have a blog site without the need for heavy CMS software like Wordpress, etc.

Summary

In this post we’ll work through the process of running a simple blogging site using Hugo. We will also discuss one possible way (the way I am using) to deploy the site using Terraform via the Digital Ocean provider.

All code for this website can be found on the project page located on Github.

Terraform

Terraform is an automated server deployment utility. You define how to construct your server inside of a configuration file (or multiple configuration files in the same directory). It has providers for many different cloud service providers. For the purposes of this blog post I’ll be focusing on the provider for Digital Ocean.

Installing Terraform

Installing Terraform is fairly straight forward. On most platforms it’s a matter of downloading the platform specific zip/tar file and extracting the binary. This binary can then be placed anywhere you like. I recommend with $HOME/bin or /usr/loca/bin for convenience and consistency.

Terraform Files

Defining your system is done in files with a .tf extension. For this site, I have split the files up in to two different .tf files. The first file, webserver.tf, is where the bulk of the configuration occurs. I have two main sections within this file. The first section describes the DNS resource. This uses the Digital Ocean provider to dynamically create a DNS A record for the site once everything is setup and ready to go. The second section defines the actual droplet. What image to use, droplet size, how to remotely connect and what to do to configure the server.

The second file, provider.tf, contains the provider specific information. This includes Digital Ocean API keys, private and public key file names and other various information which I would not want made public to the world. I am a firm believer in version controlling everything possible. By splitting up the sensitive information I am able to put the bulk of the config on GitHub and add an entry in to the .gitignore file for the provider.tf, avoiding any possibility of accidentally uploading sensitive information to a public repository.

You can find the Terraform file that deploys this website can be found here.

Hugo

Hugo is a static website generator written in Go, a powerful and modern programming language. With Hugo, all of the page content is written in Markdown. The layout of the page is defined by it’s type. The configuration files can be defined in TOML, YAML or JSON.

Installing Hugo

Installing Hugo is slightly more complicated than Terraform, but still fairly simple. If you are on MacOS I recommend installing with Homebrew. All other platforms should install using the platform specific instructions provided on the install page.

Build your first Hugo site

Once installed, creating the skeleton for your first site is as simple as running hugo new site <name_of_site>. Change directories in to your site and edit the config.toml file. You can find the config toml used for this site here. This config file defines the value of things such as your site name and theme as well as the menu for your site.

Once you have some of the config.toml figured out you should select a theme. Right out of the box, your site doesn’t know how to serve up the content files we will be adding. Adding a basic theme from the gohugo themes site is a quick way to get your content up. If you want to do so later on there is great documentation on making your own themes. For this tutorial, we’ll use the after-dark theme that I’m using here.

After extracting the contents of the downloaded theme file to the themes directory, we can return to the project root and run hugo new post/first.md. This will create a new post for us under content/post named “first.md”. This file is already populated with what is called “front matter”. Front matter is meta data about the page. It is defined using the archetypes found in the “archetypes” folder. If you now, from the project root, run hugo serve you will start a web server on your system and should be able to browse the site by visiting http://localhost:1313.

But wait, there’s no content! Open content/post/first.md in your favorite text editor and edit the “draft = true” to say “draft = false” instead. Altneratively you can run the hugo serve command with the -D flag to render pages marked as drafts.

If the above worked, the page you now see tells you a little about configuring archetypes for the after-dark theme. Follow the instructions and edit the post archetype to remove that message but leave the front matter in place.

Putting the Two Together

Putting these two together requires just a little bit of glue but it’s more than I care to explain in a blog post so I’ve made my entire setup publicly available on Github. Most of it is documented and the scripts make getting things setup fairly simple. It should be sufficiently detailed to start you down the road of deploying your hugo site using terraform, at least.

Conclusion

While I’ve used other deployment and configuration utilities in the past. This is my first experience with Terraform. Overall, my impressions are positive. It’s a good, general purpose, cloud deployment tool which makes getting started down the path of infrastructure as code easy and relatively painless.

As you continue to expand upon the topics discussed here and run your site long term, you will likely find that configuring specific aspects of your server is tedious with Terraform. Shell scripts can only get you so far. I would point anyone interested in learning more towards some of the industry standard configuration management tools available today for free. Ansible and SaltStack are simple, popular options that are well suited for both small and large scale use. If you find yourself needing more than either of those can provide, Puppet is another good option. You can also use Puppet for smaller scale deployments but some features are restricted without a complete puppet master/client setup.

first

platfrastructure@xy0.org (Ted Wood) — Tue, 23 May 2017 22:03:36 -0400

This is my first hugo post. There will be a post to follow about how I built it.