Posts on Andrew Helwer

Laptops all have built-in security tokens these days

Fri, 08 May 2026 00:00:00 +0000

I’ve been a fan of security tokens for a decade now and have accrued quite a collection. This redundancy isn’t a bad thing, as security tokens are easily misplaced and the only way to recover from a lost token is using a second token that is also registered with the service you’re trying to access. I use security tokens whenever I can! SSH authentication, universal two-factor (U2F) authentication, passwordless local login, sudo command elevation, and git commit signing are all things I use security tokens for every day. When I take my laptop traveling, there also travels a yubikey. However, it took me an oddly long time to realize that I’m a relic of a bygone era. Laptops and smartphones all have built-in security tokens these days! I’ve been carrying around yubikeys when an even better one is built right into my macbook. This post is about how I use security tokens, and how I configured my laptop’s secure element to replace my yubikey collection.

Let go of StackOverflow; communities must take ownership

Tue, 25 Nov 2025 00:00:00 +0000

Despite the title this is less a directive and more a description of what has already happened, and how I came to realize it. Still, there are many who remember StackOverflow as a revelation in the mid-2010s and think some vestige of those times remain. They think if they make an effort to distill their confusion into a question and post it there they will be rewarded. Perhaps some scarce few will, but a better way exists.

A supposedly worthwhile contract I'll never do again

Fri, 04 Jul 2025 00:00:00 +0000

Here I’ll talk about a type of TLA⁺ contract I’ve worked on a few times, and why it didn’t work out as well as hoped. I’m not trying to torpedo other peoples’ contracts here - I just hope to share this experience so others can structure their TLA⁺ contracts differently, hopefully leading to greater success for both parties and industry usage of TLA⁺ as a whole.

The proposal

The contract proposal goes like this: a client wants to build a distributed system, and has read that TLA⁺ is effective at modeling such things. Conventional wisdom holds that it’s always cheaper to catch bugs earlier in the development pipeline, and what’s earlier than the design stage? However, they don’t have any in-house TLA⁺ knowledge and are unsure about whether they want to invest in training or hiring. Instead, they email me a proposal: could I do a short 1-3 month contract where I specify their system in TLA⁺ for them? The benefits of this setup are generally reasoned as follows:

The current state of TLA⁺ development

Thu, 15 May 2025 00:00:00 +0000

The 2025 TLA⁺ Community Event was held last week on May 4th at McMaster University in Hamilton, Ontario, Canada. It was a satellite event to ETAPS 2025, which I also attended, and plan to write about in the near future. I gave a talk somewhat-hucksterishly titled It’s never been easier to write TLA⁺ tooling! which I will spin into a general account of the state of TLA⁺ development here. The conference talks were all recorded, so if you’d like this blog post in video form you can watch it below:

TLA⁺ is more than a DSL for breadth-first search

Wed, 18 Sep 2024 00:00:00 +0000

Although it isn’t usually taught that way, a lot of TLA⁺ newcomers develop the understanding that TLA⁺ is just a fancy domain-specific language (DSL) for breadth-first search. If you want to model all possible executions of a concurrent system - so the thinking goes - all you have to do is define:

The set of variables modeling your system
The values of those variables in the initial state(s)
Possible actions changing those variables to generate successor states
Safety invariants you want to be true in every state

The model checker will then use breadth-first search (BFS) to churn through all possible states (& thus execution orders) of your system, validating your invariants. If one of your invariants fails, BFS gives you the shortest execution path reaching that state. It even checks for deadlock by finding states with no possible successor states.

TLA⁺ Unicode support

Tue, 28 May 2024 00:00:00 +0000

TLA⁺ was developed by Leslie Lamport, originator of \(\LaTeX\), so it’s unsurprising that TLA⁺ syntax looks pretty \(\LaTeX\)-y. It’s a very mathy language, with much use of symbols like (among others) \A,\E, /\, \/, and \in denoting \(\forall\), \(\exists\), \(\land\), \(\lor\), and \(\in\) respectively. The language tools include a tla2tex command to format TLA⁺ specs into \(\LaTeX\) for integration in research papers. However, research papers are not where I spend the most time looking at TLA⁺. Here’s the tale of how I brought those beautiful symbols into the code editor, and how I learned to work with others in FOSS along the way!

Wrangling monotonic systems in TLA⁺

Wed, 01 Nov 2023 00:00:00 +0000

TLA⁺ sees a lot of use modeling distributed systems. The ability to explore all possible interleavings of events makes concurrency simple to reason about. For this TLA⁺ uses something called finite model-checking, which is really just a breadth-first search through the entire state space. The key here - and this really must be emphasized - is that the model is finite. There can’t be an infinite number of states, or of course the model checker will run forever.

FOSS I Love

Sat, 29 Apr 2023 00:00:00 +0000

Here I’ll write an ode to two closely-linked FOSS projects that have, recently, absolutely floored me with their competence & quality: Sunshine and Moonlight. These are used to stream video games over a (usually local) network. Sunshine is a game streaming server: it runs on your PC as it chugs away doing all the heavy lifting of running the game itself, while Moonlight is a game streaming client that runs on whatever thin hardware exists where you want to play the game! I run Sunshine on my home office workstation and Moonlight on a NVIDIA Shield Android TV connected to a 55" screen in the living room. Some games are just best played on the couch! It’s also important if you have someone you want to share the experience with instead of sequestering yourself alone in your dark Gamer Den.

Inlining SVGs for Dark Mode

Thu, 06 Apr 2023 00:00:00 +0000

I will here indulge in the traditional practice of using my blog to talk about how I’m using my blog. This page is built with the Hugo static site generator. I recently updated it to use the latest version of the beautifulhugo theme, which unbeknownst to me included a dark mode colorscheme. Recent browsers use the prefers-color-scheme option to automatically choose light or dark mode CSS styles, if the website supports it. And my website did support it, not that I knew it until people started commenting that my syntax-highlighted code blocks were unreadable! I figured out how to toggle light/dark mode in Firefox (ctrl+shift+I to open the inspector pane then click the sun/moon icons), perused my website, and found an even greater problem: my treasured vector diagrams that I put so much time & effort into were completely invisible against a dark background! Here’s a quick post about supporting dark mode on my blog by inlining SVGs and setting their color with the currentColor CSS variable.

Using TLA⁺ at Work

Wed, 05 Apr 2023 00:00:00 +0000

Here’s a short report of a time I used TLA⁺ at work, with interesting results. TLA⁺ is a formal specification language that is particularly effective when applied to concurrent & distributed systems. TLA⁺ made it tractable for an ordinary software engineer to reason about a tricky distributed systems problem, and it found a bug introduced by an “optimization” I tried to add (classic). The bug required 12 sequential steps to occur and would not have been uncovered by ordinary testing.

Pseudocode Showdown

Thu, 30 Mar 2023 00:00:00 +0000

Last weekend I had a conversation with an undergraduate student new to computer science, who was reading CLRS. “I wish” they said, “that all the pseudocode in my algorithms textbook was just written in Python.” “Ah” I said, “but textbook authors sometimes want their work to endure beyond a decade.” “But Python’s been around for a long time” came the reply, “and it’s very readable, and you can’t execute pseudocode anyway so what’s the harm?”

Google Groups has been left to die

Wed, 08 Mar 2023 00:00:00 +0000

An unusual topic for this blog, but worth boosting to a larger audience: Google Groups is dying. Its epitaph is not yet inscribed on the Killed by Google website, but the end is easily seen from here (although it should also be noted its death was called as early as 13 years ago). The deficiencies in Google Groups search, supposedly Google’s forte, have long been noted. Lately though basic features have just stopped working.

Can sanitizers find the two bugs I wrote in C++?

Tue, 07 Feb 2023 00:00:00 +0000

A few days ago I published a short post about two bugs I wrote while developing the C++ external scanner for my TLA⁺ tree-sitter grammar. Reactions were mixed! Many people were supportive, but there were of course the usual drive-by claims by developers that the bugs were trivial, they would’ve found & fixed them inside of 20 minutes, and I was laughably incompetent for having written them in the first place. Maybe so! I’m a fan of formal methods primarily so I don’t have to be a genius to write correct code. In that same vein of building tools to save us from ourselves, one user suggested building the tree-sitter grammar with the LLVM address & undefined behavior sanitizers enabled. I’d used valgrind a long time ago but had never played around with sanitizers. I was also doing some closely-associated work to build the grammar for fuzzing with LLVM’s libFuzzer, so it seemed a fun detour to check whether those sanitizers would have saved me days of debugging pain!

Two C++ bugs I wrote

Sat, 04 Feb 2023 00:00:00 +0000

Here’s a short post about two bugs I wrote while writing C++ code for the external scanner of my TLA⁺ tree-sitter grammar. External scanners use handwritten C or C++ code to parse the non-context-free parts of your language. I’ll try not to dump on C++ too hard but both of the bugs are highly ridiculous and exemplify why I hope to write as little of the language as possible for the rest of my career. These aren’t bugs with C or C++ themselves (although honestly this point could be argued) but I share them in the hopes someone finds entertainment in my misery.

Writing a TLA⁺ tree-sitter grammar

Wed, 11 Jan 2023 00:00:00 +0000

2021 saw the completion of my first substantial free software project: a TLA⁺ grammar for tree-sitter, the error-tolerant incremental parser generator. The project stabilized & found users over the course of 2022, then over the holidays I used it to build the TLA⁺ Unicode Converter. The new year is a time to reflect on the past and look to the future, so here in early 2023 seems ideal to publish my experience.

Both TLA⁺ and tree-sitter itself ensured the project was fascinating on a technical level; even more interesting were the social and psychological aspects of my first real involvement with the free software community! I’ll go over why I wanted to create this project and the main technical challenges I faced doing so, then discuss the conditions that enabled me to create it and how free software development changed the way I think. If you’d rather get the technical part in video form (with demos!), you can watch the presentation I gave at TLA⁺ Conf 2021 (slides: pdf, odp):

What's the difference between a computer and a rock?

Wed, 19 Oct 2022 00:00:00 +0000

Author’s note: I wrote this in 2017 and it languished in the drafts since then because I couldn’t think of a good conclusion. Really this is typical of the subject matter and all this piece offers is something to chew on, if you have the interest in it or have never before considered the question. At the time I was helping tutor a TLA⁺ seminar and had the opportunity to ask Leslie Lamport (who has thought about computation quite a bit!) what he thought of the question with all his accumulated wisdom, and he replied “it’s one of those meaningless distinctions, like the difference between alive and not-alive.” So there you have it. Hopefully there are enough decent jokes in this post to make it worthwhile.

The Missing Prelude to The Little Typer's Trickiest Chapter

Thu, 13 Oct 2022 00:00:00 +0000

It’s hard to find a textbook series garnering more effusive praise than The Little Schemer, The Little Prover, The Little Typer & co. The Little Typer introduces dependent type theory and is the first of the series I’ve read. I quickly grew to appreciate & enjoy its dialogue-based presentation - I’m a real convert! I might release future didactic blog posts as a dialogue rather than straight recitation of material in block paragraphs. Humans seem to naturally find conversations more interesting than the traditional lecture format of a single voice droning on, which might explain why comment sections often get more traction than the actual article being commented upon! Or the popularity of topic podcasts versus lecture series.

Regexes in the Z3 Theorem Prover

Wed, 19 Jan 2022 00:00:00 +0000

Republished from Teleport’s official blog (link). I received compensation from Teleport for writing this post.

Z3 is a satisfiability modulo theories (SMT) solver developed by Microsoft Research. With a description like that you’d expect it to be restricted to esoteric corners of the computerized mathematics world, but it’s made impressive inroads addressing conventional software engineering needs: analyzing network ACLs and firewalls in Microsoft Azure, for example. Z3 is used to answer otherwise-unanswerable questions like “are these two firewalls equivalent?” or “does this set of network ACLs violate any security rules?”

Two pictures of quantum computation

Thu, 10 Dec 2020 00:00:00 +0000

Interpretations of quantum mechanics are boring. Boring! Maybe the universe has a strict partition between quantum and non-quantum. Maybe there are a bunch of parallel universes with limited crosstalk. Or maybe it’s whatever the Bohmian mechanics people are talking about. Shut up and calculate, I think. I don’t say this out of some disdain for idle philosophizing or to put on airs of a salt-of-the-earth laborer in the equation mines. It’s just there are so, so many interesting things you can learn about in quantum theory without ever going near the interpretation question. Yet it’s many peoples’ first & last stop. Rise above!

How do you reason about a probabilistic distributed system?

Fri, 11 Sep 2020 00:00:00 +0000

In which I am stunted upon by coin flips

Wasn’t too long ago that I felt pretty good about my knowledge of distributed systems. All someone really needed in order to understand them, I thought, was a thorough understanding of the paxos protocol and a willingness to reshape your brain in the image of TLA⁺. Maybe add a dash of conflict-free replicated datatypes, just so you know what “eventual consistency” means. Past that it’s just some optimizations and mashups which come easily to your TLA⁺-addled brain.

Meditation

Sat, 29 Aug 2020 00:00:00 +0000

This isn’t going to be a post about how adopting a several-thousand-year-old practice can make you a better servant of capital. Instead, let’s talk about when I feel the lowest of the low. It comes after spending any number of hours on my computer, maybe even a full day, endlessly circling around different websites in search of stimulation, the quick jolt that comes with learning an interesting fact or watching a funny short video or seeing someone get dunked on for having a bad political opinion. I’ll eventually wake into a state that could be identified as conscious thought, look back on the many (genuinely exciting!) things I wanted to learn or do that day, and contrast it with what I actually did. What did I actually do? I honestly don’t even know. I doubt I could list even three things I’d read or experienced the entire preceding eight hours. At this moment I feel very bad. I know one of the ~30,000 days in my life is gone. And I didn’t get anything out of it. Actually, I’m worse off, because spending your time in this way just begets even more time spent in this way.

Taking my home work setup seriously

Fri, 14 Aug 2020 00:00:00 +0000

The headlines don’t lie. Microsoft, Google, Amazon, Facebook, and a whole host of other tech companies have announced employees will be working from home until early-mid 2021. There are reasons to believe this will be pushed back; even if the staggeringly ambitious timelines for vaccine development are met, a vaccine might not be a silver bullet and the pandemic could require management for the next 2-3 years. As a software engineer in big tech’s orbit, this means it’s time to settle in for the long haul and take my home work setup seriously.

Doing a math assignment with the Lean theorem prover

Sun, 05 Apr 2020 00:00:00 +0000

Note: this post was written for Lean 3; the latest version, Lean 4, is a very different language.

Turn back the clock to 2009: a confused physics major newly infatuated with math and computer science, I enrolled in MATH 273: Numbers and Proofs at the University of Calgary. This wasn’t my first encounter with mathematical proof; in first-year calculus I’d mastered rote regurgitation of delta-epsilon proofs. Despite writing out several dozen, their meaning never progressed beyond a sort of incantation I can summon to this day (for every \( \epsilon > 0 \) there exists a \( \delta > 0 \) such that…). We were told on the first day of MATH 273 that the purpose of proof is to compel belief. This was a bright start but sadly marked the beginning of a deeply embarrassing semester reaching its nadir when I asked the professor “how do we prove a definition” fully two-thirds of the way through the course. I got a B-.

Simulating physical reality with a quantum computer

Sat, 21 Dec 2019 00:00:00 +0000

Quantum Computers: Not Just for Breaking RSA

There’s no denying it, Shor’s algorithm was a blockbuster result. The thought of an exotic new computer breaking all widely-used public-key crypto plays well with the public imagination, and so you’d be forgiven for believing quantum computing is ultimately a sort of billions-dollar make-work project for software engineers: forcing our profession to relive a Y2K-like mass upgrade of old systems to new, quantum-safe encryption algorithms. Great for consultants, bad for people who want resources invested toward actual social good.

Walking the faster-than-light tightrope

Fri, 07 Dec 2018 00:00:00 +0000

Measurement and signaling in the nonlocal world

Popular understanding of quantum mechanics usually focuses on three learning objectives:

At small scales, particle properties (position, momentum, spin, etc.) are in superposition - they don’t have a definite value, but instead are “smeared” across multiple possible values.
Measuring a superposed particle property makes it collapse probabilistically to a specific value. We don’t simply discover the property’s pre-existing value; rather the property is forced to take on a definite value by the act of measurement.
Particles can be entangled, which means operations on one affect the other instantaneously across arbitrarily-large distances (known as nonlocality); however, this has restrictions and cannot be used for faster-than-light (FTL) communication.

Enraged Bohmian Mechanics enthusiasts approach the comment section (Source: Wellcome Collection)

Checking Firewall Equivalence with Z3

Tue, 13 Feb 2018 00:00:00 +0000

Lessons I’ve learned from software engineering are uniformly cynical:

Abstraction almost always fails; you can’t build something on top of a system without understanding how that system works.
Bleeding-edge methods are a recipe for disaster
Everything good is hype and you’ll only ever get a small fraction of the utility being promised.

Imagine my surprise, then, when the Z3 constraint solver from Microsoft Research effortlessly dispatched the thorniest technical problem I’ve been given in my short professional career.

Formal Verification, Casually Explained

Mon, 12 Feb 2018 00:00:00 +0000

Why are we here?

What guarantees does formal verification provide? This question rests at the apex of a hierarchy of inquiry extending all the way down to how we can know anything at all!

What do we mean by software correctness?

There are precisely two different ways for a piece of software to be correct:

The supreme deity of the universe descends from the heavens and decrees, with all the weight of Objective Truth, that a certain piece of software is correct.
We have a list of things we want the software to do, and use logic to prove the software does these things.

Crafty readers will come up with, like, twenty different caveats to the second definition. I encourage it! Take a minute to unearth some of the hidden assumptions. Good? Good. I’ll talk about those hidden assumptions in greater detail below, but the point I want to make is this: there is no such thing as inherently correct software. Your wacky program could say 1 * 1 = 2, and that isn’t incorrect in any objective sense - if that’s how you want your program to behave, heck, knock yourself out. We call our list of things we want the software to do a specification. A program is only correct (or incorrect) relative to its specification; if the program is correct relative to its specification, we say it implements or refines the specification.