llimllib's notes

my tools in 2026

2026-01-03T18:35:04.419Z

Here's a brief survey of the tools I'm currently using

Computer
Software
Browser
- browser plugins
Note Taking
Mail
Chat
Music
Webcam
Photos
Video

Computer

I use a 14-inch MacBook Pro M1 Max that I bought in 2021. It is, on balance, the best computer I have ever owned. The keyboard is usable (not the best macbook keyboard ever, but... fine), the screen is lovely, it sleeps when it's supposed to sleep and the battery still lasts a long time.

The right shift key is broken, but using the left one hasn't proved to be much of a challenge.

I usually replace my computers every 5 years, but I don't see any reason why I'd need a new one next year.

Software

editor

The most important piece of software on my computer is neovim. I've been using vim-ish software since 2003 or so, and on balance I think the neovim team has done a great job shepherding the software into the modern age.

I get a bit irritated that I need to edit my configuration more often than I used to with Vim, but coding tools are changing so fast right now that it doesn't seem possible to go back to the world where I edited my config file once every other year.

neovim plugins

My vim config is available here. I won't list all the plugins; there aren't that many, and most of them are trivial, rarely-used or both. The ones I need are:

telescope.nvim
- I have "open by filename search" bound to ,ff and "open by grep search" bound to ,fg, and those are probably the two most common tools I use in vim. Telescope looks great and works fast (I use telescope-fzf-native for grep)
codecompanion
- I added this in January last year, and it's become the default way I interact with LLMs. It has generally very nice features for working with buffers in vim and handles communications with LLMs cleanly and simply. You don't need Cursor et al to work with LLMs in your editor!
  - I do use claude code for agentic missions, as I have not had much success with agentic mode in codecompanion - I use it more for smaller tasks while I'm coding, and it's low-friction enough to make that pretty painless
sonokai colorscheme
- I use a customized version, and it makes me happy.

Now that vim has native LSP support, I could honestly probably get away with just those plugins. Here's a screenshot of what nvim looks like in a terminal window for me, with a telescope grep open:

terminal

I use kitty and my config is here.

I'd probably switch to ghostty if it supported opening hyperlinks in a terminal application, but it doesn't.

I use this feature constantly so I want to explain a bit about why I find it so valuable. A common workflow looks like this:

I cd to a project directory
I either remember a file name or a string I can search for to find where I want to work
- In the former case, I do fd <filename>
- In the latter, I do rg <string>
Then I click on the filename or line number to open that file or jump to that line number in a file

Here's a video demonstrating how it works:

download

The kitty actions are configured in this file - the idea is that you connect a mime type or file extension to an action; in this case it's $EDITOR <filename> for a filename without a line number, and $EDITOR +${FRAGMENT} <filename> if it has a line number.

terminal software

I love mise for managing versions of programming environments. I use it for node, terraform, go, python, etc etc
- I have mise.toml files in most of my projects which set important environment variables, so it has replaced direnv for me as well
fd for finding files, a better find
ripgrep for grepping
atuin for recording my command line history
GNU Make and occasionally Just for running tasks
- make is broken and annoying in old, predictable ways; but I know how it works and it's available everywhere
  - here's an example makefile I made for a modern js project
- just is modern in important ways but also doesn't support output file targets, which is a feature I commonly use in make; see the above file for an example
gh for interacting with github. I particularly use my pr alias for gh pr create quite a lot to open pull requests
jq for manipulating json
llm for interacting with LLMs from the command line; see An AI tool I find useful for an example

Browser

I switched from Firefox to Orion recently, mostly because I get the urge to switch browsers every six months or so when each of their annoyances accumulate.

Orion definitely has bugs, and is slow in places, and Safari's inspector isn't nearly as nice as Firefox or Chrome. I wouldn't recommend other people switch, even though I'm currently enjoying it.

browser plugins

Bitwarden - I dunno, it's fine, it mostly doesn't annoy me
uBlock origin - I'm very happy with it as an ad blocker

That's it, I don't use any more. Browser plugins have a terrible security story and should be avoided as much as possible.

Note Taking

I use Obsidian, which I publish to the web with the code here (see generating HTML))

It's not clear to me why I like using obsidian rather than just editing markdown files in vim, but I'm very happy with it.

Mail

I use Apple's Mail.app. It's... fine enough I guess. I also occasionally use the fastmail web app, and it's alright too.

I am very happy with Fastmail as an email host, and glad I switched from gmail a decade or so ago.

Chat

I use Slack for work chat and several friend group chats. I hate it even though it's the best chat app I've ever used.

I desperately want a chat app that doesn't suck and isn't beholden to Salesforce, but I hate Discord and IRC. I've made some minor attempts at replacing it with no success. It's the piece of software I use day to day that I would most love to replace.

I also use Messages.app for SMS and texts

Music

I switched in October from Spotify to Apple Music. I dislike Apple Music, but I also disliked Spotify ever since I switched from Rdio when it died.

I'm still on the lookout for a good music listening app. Maybe I'll try Qobuz or something? I don't know.

I've also used yt-dlp to download a whole bunch of concerts and DJ sets from youtube (see youtube concerts for a list of some of them) and I often listen to those.

Webcam

I have an old iPhone mounted to my desk, and use Reincubate Camo to connect it to video apps.

I occasionally use OBS to record a video or add a goofy overlay to video calls, but not that often.

Photos

I use Adobe Lightroom to import photos from my Fuji X-T30 and Apple Photos to manage photos.

With the demise of flickr, I really have no place to post my photos and I've considered adding something to my website but haven't gotten it done.

Video

I use vlc and IINA for playing videos, and ffmpeg for chopping them up from the command line

2025 in review

2025-12-31T14:54:50.522Z

coding
tools
games
travel
visualization

coding

I didn't do that much work in the open this year, unfortunately. I made a few small tools and polished my workflow scripts quite a bit though:

review

I use my review script to get feedback from an LLM on a pull request before I submit it, or to review other people's PRs. I like this a lot more than tools that annotate a PR in github, because it lets me read it locally and privately, and decide for myself which of its suggestions are valuable.

As I wrote when I released the tool, I find that most of its suggestions are not useful, but having a list of suggestions to evaluate has proved a very valuable resource for me and has prevented me from making quite a few obvious errors.

worktree

My worktree script helps me use git worktrees in a large node.js monorepo at work without so much pain. This year I:

added submodule support
added support for github fork notation (user/repository)
added the ability to store its config in git global config

I did play with jj a bit this year, but I'm so used to git that it's not a pain point for me and it seems like a waste of time, though I remain interested in it.

time

My favorite tiny script of the year is my time script, which I wrote because I often have to compare times in different timezones with different time formats. For example, sometimes I'm reading timestamps from a log message in milliseconds since the epoch, and I need to know when that was both in my local time and in UTC.

It has two functions:

when called without arguments, prints out the current time in a variety of formats, useful for comparing time between timezones or formats:

$ ,time
ts seconds: 1767193930.0
ts ms:      1767193930000
Local:      2025-12-31 10:12:10 EST
UTC:        2025-12-31 15:12:10 UTC

if passed an argument, will do its best to parse the time and display it in those same formats:

$ ,time 1756082819
ts seconds: 1756082819.0
ts ms:      1756082819000
Local:      2025-08-24 20:46:59 EDT
UTC:        2025-08-25 00:46:59 UTC

$ ,time Dec 31 2024
ts seconds: 1735621200.0
ts ms:      1735621200000
Local:      2024-12-31 00:00:00 EST
UTC:        2024-12-31 05:00:00 UTC

gp

I have the rights to push to the main branch of a bunch of git repositories, and I found myself occasionally pushing directly to main by accident instead of creating a branch.

To solve that problem, I replaced my git push alias with a script that checks if I'm pushing to a main branch by accident, and displays a message to the console with the ability to do the push if it was indeed intentional.

tools

I still spend most of my life in neovim in the terminal, and have no desire to change either of those.

I'd like to switch to ghostty, but it doesn't support opening hyperlinks in terminal applications (github discussion). I find this workflow incredibly useful and don't want to live without it, so I'm still using kitty.

I've been using Obsidian to make this website, and remain very happy with it.

I've updated the script that builds the website in minor ways throughout the year. It's slow and I dislike the markdown library it uses, but it also gets the job done and I don't have to think about it.

I added hacky callout support

But it was hacky and painful to do, so it goes

I switched from Firefox to Orion browser, which is working well enough though it definitely has some bugs. I tend to switch browsers every six months as I accumulate dissatisfactions with all of them.

games

For a long time, I've vaguely wanted to play games that are Windows-only, but I also refuse to install Windows and I don't have a Linux gaming computer set up. A couple weeks ago, I finally bit the bullet and purchased CrossOver and have started playing a few games that are Windows-only.

Some games I or my kids enjoyed this year:

We all enjoyed N++, a platformer stripped down to the absolute basics. I remember playing n as a flash game so many years ago, and its older sibling is still a blast.
I enjoyed playing Animal Well
Because Silksong came out this year, I went back and tried to finish Hollow Knight. I managed to get to the last boss, but couldn't beat him, and didn't end up buying Siksong. Maybe next year
My older son really enjoyed Ball x Pit
My younger son got old enough to play Tears of the Kingdom for himself, and enjoys coming up with silly weapon combinations more than actually playing the story, which is fun
We had fun playing some FTL runs together
We all had fun with Dead Cells

travel

Working for a distributed company means traveling for meetups, and we had them in particularly fun locations this year. I was fortunate to travel to Colorado, Montana, and Mexico for work this year.

My family and I spent two weeks in Lecco, Italy, which was gorgeous. I'd never been to Italy before and it was great

visualization

I continued toying around with different visualizations of NBA data this year. My son asked me the other day if it was work, and I told him that I make graphs in the same way that other guys build models or go fishing

I kept up my nba_data archive, and added a few data sources to it. I particularly enjoy Dean Oliver's analytics data, and have found some good uses for it
Lately I've been playing around with tables and trying to make them especially expressive; this one has a bar graph background that changes with the column you've sorted by, and has interactive filtering:

Due to a particularly irritating github pages bug, I moved my observable notebook off of github pages and over to https://billmill.org/nba
I post about this work on my bluesky account, and occasionally post a graph or progress on other work

Advent of Code 2025

2025-12-03T00:43:06.843Z

This year I've decided to do Advent of Code in Gleam, a programming language I know nothing about and have never written a line of code in.

Previously: Advent of Code 2024, 2023 problem log, and incomplete answers back to 2015 in github

day 1
day 2

Day 1

Not the easiest day 1! The problem centered around a ring of the numbers 0-99.

My answer for both parts is here
I struggled with coming up with a simple way to express the crossings of zero for the second part, and I'm not super happy with how that came out
You need a library to read files in gleam! gleam add simplifile to get the standard sync library it seems
echo is super handy for debugging pipelines - just stick it between |> braces and it will show you what's going on
I ought to figure out a better way to handle errors than just unwraping them and picking a random default value, it's definitely going to bite me eventually if I don't panic. I'm surprised there's not an unwrap_or_panic or something like that! Maybe I'll write it as must or something, but for today I just got by

Day 2

It seems that the creator of gleam doesn't like tuples, so today I tried not to use any. I started with a custom type to represent a range:

pub type Range {
  Range(start: Int, end: Int)
}

For part 1, I used a generative approach where I iterated from the start of each range through the end, and checked if the number was of the form <x><x>. The heart of it is this function:

pub fn acc_dubs(front: Int, range: Range, acc: List(Int)) {
  let dub =
    string.append(int.to_string(front), int.to_string(front))
    |> int.parse
    |> result.unwrap(0)
  case dub >= range.start, dub <= range.end {
    True, True -> acc_dubs(front + 1, range, [dub, ..acc])
    True, False -> acc
    False, True -> acc_dubs(front + 1, range, acc)
    False, False -> panic as "impossible"
  }
}

For part 2, I got in loops thinking about how to generate all possible combinations of numbers, so I instead took a filtering approach. This function checks if a number is symmetric in the way the problem states (assuming that there are no numbers with more than 10 digits):

pub fn is_sym(n) {
  let ns = n |> int.to_string
  let l = ns |> string.length
  list.any(
    [
      atoi(string.repeat(string.slice(ns, 0, 1), int.max(l, 2))),
      atoi(string.repeat(string.slice(ns, 0, 2), int.max(l / 2, 2))),
      atoi(string.repeat(string.slice(ns, 0, 3), int.max(l / 3, 2))),
      atoi(string.repeat(string.slice(ns, 0, 4), int.max(l / 4, 2))),
      atoi(string.repeat(string.slice(ns, 0, 5), int.max(l / 5, 2))),
    ],
    fn(x) { x == n && n > 9 },
  )
}

I find that pleasing!

Then I iterated over every number in each range and checked it for symmetry, and summed the result:

pub fn part_b(input) {
  let ranges = parse(input)
  list.fold(
    list.flat_map(ranges, fn(r) { acc_nubs(r.start, r.end, []) }),
    0,
    int.add,
  )
}

pub fn acc_nubs(start, end, acc) {
  case start > end, is_sym(start) {
    True, _ -> acc
    _, True -> acc_nubs(start + 1, end, [start, ..acc])
    _, False -> acc_nubs(start + 1, end, acc)
  }
}

I didn't do any better at handling errors today! I think I will make that an emphasis tomorrow.

Custom types were really simple, and I haven't yet hit any big gleam roadblocks. It takes me a minute to think of how to phrase problems recursively, but I like that it kind of forces you to break your code into small, clear functions.

I wish there were better inline testing functionality in gleam, I don't want to put my tests out into a test directory. I'm kind of annoyed generally by how picky it is about the project structure - but I also see how it fits into the zen of the language, which minimizes your options to keep complexity under control.

Licensing will not save us

2025-11-16T13:33:25.613Z

I enjoy this piece by Erlend Sogge Heggen which argues that we, the open source developers, ought not to freely give away our work because it advantages the capitalists and fascists that are leveraging the fruits of our labor to make untold millions for themselves without giving back to the community they're building off or the world at large.

I've been using the computer long enough to remember how hard it was to get your hands on software that was interesting and useful. Without open source software, I'm certain that computing would not have progressed as far as it has, and that I and many others would not have the careers we enjoy because we wouldn't have found a way in.

The class of business leaders who have built on open source software (and who often started as developers themselves) has taken a heavy toll on the world without returning the value they owe, but I also fear returning to a world where a privileged class of people has access to the source code for every important application, and interested people have to choose whether to break the law to satisfy their curiosity.

More and more developers are playing around with licensing to try and defend themselves from the predatory practices of the tech elite, and I'm here for it - but that cannot be the whole solution.

Only organization and community can protect the fruits of OSS (or a future OSS-like?) labor from explotation. It can't be one community, because there are many different aims, cultures, and viewpoints, but there can be many interlinked communities sharing tools, knowledge and practice.

The sooner we start building the practices, habits and (less importantly) software to make it easy to build, maintain, and own communities of practice, the sooner we will make it possible to share the fruits of our labor in a less-destructive manner. Licensing alone won't save us, we need to build stable social organizations and learn how to empower them.

Writing good technical articles is hard, we should do more of it anyway

2025-09-24T10:55:59.621Z

An article is making the rounds about what it feels like to read a technical article when you don't understand the terminology

Why do people feel OK putting a partially-assed project on github, but have a much higher bar for when they release a technical article?
It's fractally difficult to write technical tutorials and documentation
- Everybody comes to it with a different skillset
- You're writing for everybody in the world speaking every language
- Every single thing you write about
- Computers are so new that there's very little common baseline for everyone
- we don't even know what the fields are yet!
- There's little-to-no feedback
- People feel very free to give harsh criticism
  - That leaves only the thickest-skinned authors able to write
- The worst part for me is the complete lack of feedback
  - You chuck an article out into the ether, and either you hear nothing or the feedback you get ranges from mildly critical to harshly critical
  - There's neither reason nor incentive to leave positive things about it. Who cares that you enjoyed the article
Let's try to create a world of constructive dialogue

An AI tool I find useful

2025-07-27T14:31:07.68Z

One of the tasks that I do most often is to review code. I've written a review command that asks an AI to review a code sample, and I've gotten a lot of value out of it.

I ignore most of the suggestions that the tool outputs, but it has already saved me often enough from painful errors that I wanted to share it in the hope that others might find it useful.

How to install it

install llm and configure it to use the provider and model you'd like to use
- on a mac, brew install llm
optionally install bat
- on a mac, brew install bat
save the review script anywhere on your path and make it executable

How it works

The main job of the script is to generate context from a git diff and pass it to llm for code review.

If you run review with no arguments, it will:

run git diff -U10
- the -U argument changes the amount of context given in a git diff
- any additional arguments you pass will be forwarded directly to git diff
estimate the tokens in the output and check if that fits within the context window
- shrink the context window if necessary and re-run git diff
- if you want to manually expand the context beyond 10 lines
  - you can append to the system prompt with the --context flag
  - you can expand the diff context to 100 by passing -U100 or any number you prefer
pipe the context to llm
- You can configure llm to use whichever model or AI company you prefer
highlight the output with bat if available
- bat is great and I highly recommend using it if you don't already. brew install bat on a mac will install it

The result looks like this in my terminal:

How I use it

My main use of the command is to review a PR I'm preparing before I file it. The biggest value I've gotten out of it is that it frequently catches embarrassing errors before I file a PR - misspellings, DELETEMEs I forgot to remove, and occasionally logic errors.

It also often suggests features that make sense to add before finishing the PR, or as next steps.

It is very important to use it intelligently! The LLM is just an LLM, and it also may be missing context. The screenshot above has two examples of mistaken suggestions that I read and ignored; you have to apply your own understanding and taste to its output.

Keep in mind that it is tasked via its system prompt with finding problems and making suggestions; no matter how good your code is it will try to find and suggest something.

I also use it for reviewing other people's PRs, with review origin/main origin/some-feature-branch. In these cases, I really am just using it for clues as to some things that I may need to investigate with my actual human brain. Please do not just dump llm suggestions into a PR! That's both rude and likely to be unhelpful.

How it differs

That last point brings me to why I prefer this tool to github's own copilot review tool.

I can use my review tool in private, and evaluate its suggestions in private
I can run it repeatedly as I change the code
The separation of my terminal from the code review tool provides a space for me to apply critical thinking

Areas for improvement

A lot of things are hard-coded into the script, because I'm its only user
- If you find use in it, please let me know!
the system prompt seems to work fine, but the range of possible system prompts is so large that I'm sure it could be better

Postscript

Thanks to a suggestion on lobste.rs from davidcrespo, I added the ability to provide context via stdin. Thanks for the suggestion!

A pretty decent retry, and not a library

2025-07-19T15:14:41.434Z

I'm pretty happy with the retry function I've ended up with for my nba stats downloader:

G = TypeVar("G")

def retry(f: Callable[..., G], **kwargs) -> G:
    """
    Retry a function call with backoff and print out the exceptions raised
    unless they are a timeout.
    """
    i = 0
    while 1:
        try:
            return f(**kwargs)
        except Exception as exc:
            if i > 11:
                raise RetryLimitExceeded(f.__name__, i, kwargs, exc)
            timeout = [1, 2, 5, 10, 15, 20, 25, 25, 25, 50, 50, 100][i]
            i += 1
            print(f"failed {f.__name__}({kwargs}), sleeping {timeout}:\n{exc}")
            # print the traceback unless it's a timeout
            if not isinstance(exc, (ReadTimeout, ReadTimeoutError, TimeoutError)):
                print(traceback.format_exc())
            sleep(timeout)
    # The LSP doesn't detect that this is unreachable: appease it
    raise Exception("this should never happen")

Be explicit

One thing I like about it is that it doesn't calculate a backoff value - it picks from an explicit list of backoffs.

Before I wrote this one, I had written a bunch of retry functions and always tried to come up with some clever exponential algorithm and do a bit of math. One day I was working with backoffs for my day job in a database context and I decided to go look at what sqlite uses:

static const u8 delays[] = { 1, 2, 5, 10, 15, 20, 25, 25,  25,  50,  50, 100 };
delay = delays[count];
// ed: That's "sqlite3 OS sleep", not "sqlite 30s sleep"
sqlite3OsSleep(db->pVfs, delay*1000);

It's pleasantly grug-brained; why use clever code when simple code do trick?

So I stole the idea and I use it whenever I need a retry function now.

Don't be generic

The retry function does not allow you to specify your timeouts or configure the list of exceptions that don't get tracebacks or configure the message that gets thrown.

It doesn't support regular args because I only need kwargs in the context I'm using it in.

If I needed to use it again, I'd probably have to modify one or both of those choices. That's OK!

This leads to:

Don't create a library for small stuff

One instinct for a free software developer, when they have a pleasant battle-tested function, is to make it into a library. The Javascript ecosystem is full of libraries this small or smaller - I could go look and I'm sure I'd find seventeen similar examples of retry libraries.

It feels nice, like you're giving back to the community that you've drawn value from.

But it's a mirage!

The value of having a small function like this in your codebase, where you can see it, read it, and tailor it to suit your needs is greater than the value of having a highly configurable library that's difficult to read and debug.

As the code gets more complex, the weight of that complexity can make it such that it starts to make sense to isolate the code into a library and share it.

I'd like to advocate that people usually underweight how much value there is in having the code included in your application by tailoring it to your application exactly rather than abstracting it into a library.

Toolboxes, not libraries

For small code tasks like this, it's better to have a toolbox of examples you can pull from and customize.

If I needed retry in a golang or rust program tomorrow, I could easily remember that I've written this code and translate it for use. If I'd instead used a retry library, I'd have to find one in that ecosystem and figure out which library was best, how to use it, and add an external dependency.

This is much of the value that StackOverflow has provided to the world: a giant set of comments containing small, customizable code examples such as this one that solve small problems, which people can copy and modify as they wish.

Given that StackOverflow seems to be dying, I wonder how we could do a better job supporting this "toolbox" type of code?

Would it make sense for language ecosystems to host toolboxes? What if python had a python.org/examples library where people can paste code examples with a description and usage guide, and perhaps star them when they find them useful?

Or perhaps we could try to have a sort of "codex" by problem type, where retry had a thousand examples in different languages, with different techniques, usages, and lineages?

postscript

Kartik Agaram points to this excellent post about four repositories he's created and uses, that he calls "template repositories", similar to the "toolbox" idea
Konrad Hinsen links this preprint article about what it means to establish trust in scientific code. I like this:

The evolution of software in such a universe is very different from what we see today. There is no official repository, no development timeline, no releases. There is only a network of many variants of some code, connected by forking relations. Centralized maintenance as we know it today does not exist. Instead, the community of scientists using the code improves it in small steps, with each team taking over improvements from other forks if they consider them advantageous. Improvement thus happens by small-step evolution rather than by large-scale design. While this may look strange toanyone used to today’s software development practices, it is very similar to how scientific models and theories have evolved in the pre-digital era

Comparing golang sqlite to C sqlite

2024-10-01T12:19:11.675Z

Following a question on reddit, I got curious about how well golang sqlite bindings (and translations, and WASM embeddings) compare to using C directly.

I'm competent with C, but no expert, so I used the help of an LLM to translate a golang sqlite benchmark to C and tried to make it as equivalent to the golang as possible in terms of the work it was doing.

The test I copied is pretty simple: insert a million users into a database, then query the users table and create user structs for each.

The result is just one test, but it suggests that all sqlite bindings (or translations) have trouble keeping up with sqlite in C. All times here are in milliseconds, and I picked the best of two runs:

library	insert (ms)	ratio	query (ms)	ratio
C sqlite3	737	1.00	84	1.00
mattn/sqlite3	1380	1.87	1036	12.33
crawshaw.io/sqlite	1097	1.49	542	6.45
modernc.org/sqlite	4148	5.63	966	11.50
eatonphil/gosqlite	1017	1.38	651	7.75
ncruces/go-sqlite3	2178	2.96	611	7.27
zombiezen/go-sqlite	1403	1.90	211	2.51
python 3.12.2	1992	2.70	1426	16.98

The tests were run with go 1.23.1 and clang 16.0.0 on an apple m1 mac with 32gb of ram.

This is a deeply non-serious benchmark - I was browsing the web while I was running it, for starters. Still, the magnitude of the results indicates that sqlite in C is still quite a bit faster than any of the alternatives in go.

Somebody on mastodon asked me how this compares to previous versions of go, so I did a brief test of mattn and crawshaw on go 1.19, and found that they were in the range of 10-15% slower, so real progress on making cgo faster has been made in a pretty short timeframe.

The code is available here in a gist.

update

For kicks, I added a python version to the table above; source is in the same gist

2025 Jun 19: noticed that the python ratios were not correct and fixed them; I did not re-run any tests

Federation is extremely expensive

2025-06-19T14:59:05.557Z

There is a certain feeling among the developers of messaging software that if you want to create something worthwhile, you need to create a federated system.

This is how you get ActivityPub, atproto, matrix, xmpp, nostr, and on and on and on. A series of protocols that aim to recreate the successes of email and IRC.

The successes are overblown

Email and IRC have huge problems, many of which are caused by their own decentralization.

Both protocols suffer from the extreme difficulty of upgrades - it's nearly impossible to organize a whole herd of people running servers to upgrade at the same time. The effect is that once a decentralized protocol has stabilized, it is extremely resistant to upgrades.

Both protocols suffer from the challenge of preventing spam. It has proven so difficult for email that is has de facto centralized around Google Mail and a few other major providers.

Email's design is extremely substandard by modern standards, but despite the quasi-centralization of providers, we have no hope of changing it because there are still too many providers and consumers to make modernization possible. This leaves us with the worst of both worlds, a sub-standard protocol with a mostly-centralized structure.

Moderation is ~impossible

A key lesson of the modern internet is that a well moderated service is essential to providing a decent experience for users who are not trolls.

If your service is truly decentralized, consistent moderation is impossible by design.

Taking on federation means taking on the Herculean task of coordinating decentralized moderation. I don't think it's impossible to do an OK job of it, but it's a huge task and an extreme social challenge.

Ownership is important

I think a lot of the desire for federated systems comes down to a desire for ownership; the ability of a person or a group of people to own some piece of the system. It feels like in a decentralized system, even if you don't, you could own a piece of it. You could stand up an email server, or a mastodon server, and do things your own way if you had to.

This is an excellent impulse, and we should nurture it!

But federation is not the only way to achieve ownership. We can make open software that allows people to own their own computation in meaningful ways a lot faster by skipping federation and focusing instead on building software that delivers a stronger user interface and user experience while providing data ownership.

I want to see is a world in which we can deliver more open software to develop communities more rapidly, and I think we might get there faster if we accept that the cost of federation is so high that we ought to consider avoiding it.

postscript

This post follows a conversation I had on a private Slack. Part of the reason why I think that conversation happened on Slack instead of on an open chat system is that we have spent so much effort building federated systems that we've failed to deliver working open software to replace it.

Triple Storyline for Ethical Design

2024-12-07T02:01:11.598Z

A good user experience is only as good as the action it enables. Designing a system that makes it easy to do bad things is bad. A system that automates discrimination is, by Spinoza’s light, evil.

[Erika Hall](A good user experience is only as good as the action it enables. Designing a system that makes it easy to do bad things is bad. A system that automates discrimination is, by Spinoza’s light, evil.) via mastodon

Advent of Code 2024

2024-12-06T21:41:44.068Z

Last year I kept my advent of code log in a note on this site: 2023 problem log

This year I wanted to make it easier for me to generate and share visualizations, and break the code into more logical explanations, so I'm doing it at https://llimllib.github.io/advent2024/ in an observable framework notebook.

The source code is all on github, but hopefully the notebook provides an easy reading experience.

I've tried to do visualizations for anything that can be remotely visualized, and plan to play around with it some more and learn more about framework and plot.

Colophon

2024-08-12T20:54:18.364Z

To add to this blog, I type out a note in Obsidian; I use Obsidian with very few plugins.

download

When it's reasonably done, I back it up with the obsidian git plugin by clicking the "backup" button

That backs it up to a private github repository.

That repository has a github action that runs run.py from my obsidian_notes repository.

That script converts the big pile of Obsidian-flavored markdown into a big pile of HTML.

The github action uploads that big pile of HTML to a Digital Ocean space using s3cmd. (I don't recommend Digital Ocean spaces, but that's what I use for now)

To serve notes.billmill.org, I have caddy installed on my web server, proxying requests to cdn.billmill.org, which is a CNAME to the digital ocean space.

That's it! Feel free to ask me questions on mastodon.

How to debug git with visual studio code on a mac

2024-08-09T13:26:26.812Z

I needed to figure out how to debug the git binary to figure out exactly what it was doing, and I thought I'd share the setup because it took me a little while to get going.

Prerequisites

install visual studio code
install the microsoft C/C++ extension
you will need to have lldb, the llvm debugger available on your system.
- To see if you have lldb available, run which lldb; as long as it prints a path and not lldb not found, you're good to go
- The most common way to get it installed is to install Xcode from the app store or the Xcode Command Line Tools which can be installed with xcode-select --install
- You may also install it with homebrew by doing brew install llvm. Be aware though that brew will not link it by default, so you'll have to figure out how to tell vs code to use it

Building git

Clone the git repository: git clone https://github.com/git/git
Change into the cloned directory: cd git
Set some build variables by creating a file called config.mak. It should have exactly these contents:
```
DEVELOPER=1
CFLAGS+= -O0
```
- A command you might use to make this file: printf 'DEVELOPER=1\nCFLAGS+= -O0' > config.mak
- see the Makefile for more info on what DEVELOPER does
- the CFLAGS variable provides a flag to the C compiler to build a binary with no compiler optimizations; traditionally O3 means "the most optimizations" and O0 means "no optimizations"
  - clang documentation on optimization options
make -j4
- The -j argument to make tells it how much parallelism it can use. It speeds up the build by increasing parallelism; you can use more if you have more CPUs, or less if you have fewer. The build doesn't take very long though so it's not a big deal.
When this completes successfully, you should have a bunch of new binaries in your git directory, such as git, git-add, git-am, etc

Debugging git

make a .vscode directory: mkdir .vscode
open the git directory with visual studio code: code .

create a file in the .vscode directory called launch.json that tells vs code how to start git for debugging. Its contents should be:

{
  "configurations": [
    {
      "name": "debug git",
      "type": "cppdbg",
      "request": "launch",
      "program": "${workspaceFolder}/git",
      "args": ["status"],
      "cwd": "${workspaceFolder}",
      "externalConsole": false,
      "MIMode": "lldb"
    }
  ]
}

Note that we're passing status as an argument; the first command we're going to debug is git status. To debug other commands, you'll want to change the args option
See the documentation for information on these configuration options and what additional flags are available

Open the builtins/commit.c file in the editor
Set a breakpoint on the first line inside the cmd_status function; in my current version of git, that's located at line 1504
- To set a breakpoint, click on the left gutter, just to the left of the line numbers; you should see a red dot appear. Once a breakpoint is set, the debugger will stop when it reaches that location, allowing you to control execution of the program
  - here is VS Code's documentation on how to set breakpoints
- Annoyingly, setting a breakpoint at the function name doesn't seem to work for me, so I have to set a breakpoint inside the function or else the program doesn't break
  - If you really want to do this, in the command palette there's an action called Add Function Breakpoint, which created a function breakpoint that I could type cmd_status into, and that did break on the function invocation
- The builtins directory contains most of the high-level git UI code
- git subcommands will have a function name cmd_<command name>; check out cmd_log or cmd_commit for other examples
Switch to the Run and Debug tab of VS code
click on start debugging, the green arrow located at the top left of the window, or press F5 to do the same thing.
- alternatively, open your command palette with ⇧⌘P, type Debug, and select Debug: Select and Start Debugging, then select debug git

If all goes well, you should now be controlling the execution of git inside VS code! Go ahead and dig in to try and understand what's going on.

I might write more about what it is that's going on in there, but I'm not sure - let me know if you would like to understand some part of it better.

A command to print terminal color codes

2024-07-19T13:05:13.988Z

I write shell scripts that use terminal colors often enough that it's handy to know a bit about them, but not often enough that I remember how they work without looking it up.

So, this morning, I wrote colors, a shell script that prints out tables of the basic ANSI colors and text effects, and demonstrates what they look like on your terminal.

Here's the script, and here's what it looks like on a mostly-stock apple terminal.app with a dark color scheme:

You can pass -v to the script to get a table of the 8-bit colors, in case you want to get real fancy with your script:

Serving a billion web requests with boring code

2024-06-24T17:32:45.949Z

High level
Boring über alles
The boring bits
- Why React?
- golang
The innovation tokens
- Modular backend
- gRPC
Strict backwards compatibility
Faceted search
Database
Miscellaneous tooling
Logging
Documentation
Runtime integrations
And more

When I worked as a contractor to the US government at ad hoc, I was fortunate enough to get the opportunity to design large parts of a relaunch of medicare plan compare, the US government site through which hundreds of thousands of medicare recipients purchase their health care plans each year.

We released after about a year of development, and were able to help millions of people find and purchase health care - if you're in the US, you are pretty likely to know somebody who used this system.

Though the US health care system is incredibly awful in many respects, I'm very proud of what the team I worked with was able to build in a short time frame and under a lot of constraints.

The team of people that I worked with - managers, designers, engineers and business analysts - were excellent from top to bottom. I've never before experienced such a dedicated, collaborative, trusting team, and I learned a tremendous amount from them.

I especially want to share this story because I want to show that quality software can get written under government constraints. It can! And if we all start believing that it can, we're more likely to produce it.

High level

I worked on this system for about two and a half years, from the very first commit through two open enrollment periods.

The API system served about 5 million requests on a normal weekday, with < 10 millisecond average request latency and a 95th percentile latency of less than 100 milliseconds.

It served a very low baseline rate of errors, mostly spurious errors due to vulnerability scrapers. I'm proud that I can count the number of times an engineer was woken up by an emergency page on one hand.

I was amazed at how far you can get by leaning on postgres and golang, keeping your system as organized and simple as possible, and putting in work every day for a long period of time.

Boring über alles

My north star when building the system was to keep it as boring as possible, in the Dan McKinley sense. (Go read "Choose Boring Technology" if you haven't yet, it's required reading. It's better than this article.)

There is a concept of "Innovation tokens" in that article, and I was explicit in choosing the pieces I used to build the site with how I spent them.

The boring bits

postgres - The rock on which I prefer to build my systems
golang - I had experience with it from working on healthcare.gov, and felt extremely confident in its reliability and relative simplicity
react - React is boring in the sense we're discussing here, it's extremely widely deployed and was known well by my teammates

Why React?

There are many valid criticisms of react; this piece is an example, and I was aware of the issues already in 2018 when I was building the site. The main thrust is that it tends towards large application bundles, which take a long time to download and execute, especially on the cheap mobile phones that are the main link to the internet for so many people.

In building a piece of infrastructure for the government, it was especially concerning that the application be available to as many people as possible. We took accessibility seriously both in the sense that the site needed to have proper design for users with disabilities and also in the sense that people with many devices needed to connect to it.

Nevertheless, I chose an SPA architecture and react for the site.

I would have loved to have done differently, but I worried that choosing to use a multi-page architecture or a different library would have slowed us down enough that we wouldn't have delivered on our tight timeline. I didn't have enough trust in any of the alternatives available to me at the time to make me believe we could choose them safely enough.

The result fell prey after a few years to a common failure mode of react apps, and became quite heavy and loaded somewhat slowly.

I still think I made the right choice at the time, but it's unfortunate that I felt I had to make it and I wish I had known of a nice clean way to avoid it.

golang

Golang was overall a joy to build this project in. It runs efficiently both at build time and at run time, and having binary executable artifacts that build quickly makes it easy to deploy rapidly.

Developers new to the language (our team of engineers grew from 2 to 15) were able to get onboard quickly and understand the language with no trouble.

Error handling being both immediate and verbose is, in my opinion, a great feature for building systems that are resilient. Every time you do something that might fail, you are faced with handling the error case, and once you develop patterns they are consistent and predictable. (I know this is a big topic, I should probably write more about it)

The day that we were able to switch to go modules, a big pain point went away for us. We hit a few bumps in the road as very early adopters but it was worth it.

My biggest gripe with the golang ecosystem was that the documentation generation for projects that are not public sucks. For a long time, the documentation generator didn't even support projects that used modules.

That said, I was overwhelmingly happy with my choice here and never regretted it.

The innovation tokens

I made two architectural bets that I was less confident of than the others:

Modular backend - I designed the backend neither as microservices, nor as a monolith, but into three large modules
gRPC - The backend services communicated with each other via gRPC, and with web clients via grpc-gateway

Modular backend

I split the backend up into three parts; they all lived in the same repository but were designed such that they could be pulled apart and given to a new team if necessary.

Each component had its own postgres database (which were physically co-located, but never intertwined) and strictly used gRPC to communicate between themselves.

The split was largely based around data access patterns:

drug pricing (aka druginfo)

One thing the site needed to be able to do was estimate the cost of any packaging variation of any drug at any pharmacy on any health insurance plan. Easy, right?

This combinatorial explosion (I once calculated how many trillions of possibilities this was) necessitated a very carefully-indexed database, a large amount of preprocessing, and a commitment to engineering with performance in mind.

It took a long time and a ton of government health system reverse engineering to figure out how to get even close to right with this part. I'm forever indebted to my colleagues who dove deep into the depths of CMS bureaucracy, and then turned it into documentation and code.

plan search (aka planinfo)

The main purpose of the site was for people to search for and purchase medicare part C and part D health care plans.

Every day, we'd get a new dump of detailed health care plan information from CMS; this module would load the information into a new postgres database, and then we'd deploy a new version pointing at the new day's data.

Both planinfo and druginfo had entirely immutable databases in this way; their only job was to serve an API based on the most recent data for each.

beneficiary information (aka beneinfo)

In the insurance argot, a person on a health care plan is a "beneficiary" of the plan. It sounds a little self-important to me, but that's just what it is I suppose.

(One thing I tried to do throughout my work on this project was to use the proper industry jargon wherever possible rather than something more familiar to myself. I felt it was part of the commitment to boringness to keep the jargon friction down to a minimum.)

The job of the beneinfo module was to store information about plan customers, and was the only part of the application where the database was long-lived and mutable.

We strove to store as little data as possible here, to minimize the risk should there be any data leakage, but there was no way around storing a very scary amount of Personally Identifiable Information (PII). We were as serious as possible about this data, and the risk of losing control of it kept me nervous at all times.

gRPC

Overall, gRPC was not as great for us as I'd hoped when beginning the project.

The best benefit of using it, and the driver behind my choice to use it, was that every interface was specified in code. This was very useful; we could generate tools and interfaces that changed in lockstep.

The biggest pain points were all related to the tooling. I maintained a set of very hairy makefiles with eldritch protoc commands to build all the interfaces into go files that we could use, and debugging those was always painful.

Not being able to curl the system, as we would if it were a JSON API, was a pain in the butt. grpcurl existed, and we used it, but was not nearly as nice.

grpc-gateway was the best part of the ecosystem I used, it served more than a billion requests for us and was never once the source of a problem. It enabled us to do what gRPC ought to have been able to do from the start, serve requests to web clients.

I loved having interface schemas, but we used so few of gRPC's features and the code generation was so complicated that we probably would have been slightly better off without it.

Strict backwards compatibility

We followed a strict backwards-compatibility requirement, and only added and never removed fields from our interfaces. Once a field was exposed in the public API, it would be exposed forever unless it became a security problem (which, thankfully, never happened to us in the years I worked on this project).

We did the same with the databases as the API; columns were added and rarely removed. If a column actually merited removal, the process was to add a column, remove all references to the old one, wait a few weeks to make sure we didn't need to roll back, then finally remove the column from the database.

Our discipline with backwards compatibility gave us freedom to keep up a high rate of changes and maintain confidence that local changes would not have negative downstream consequences.

Faceted search

A core principle of the app was to rely on postgres whenever possible, and also to be stupid instead of clever whenever possible.

Faceted search was an excellent example of both of those properties. We could have reached for elasticsearch, and we also could have tried to use an ORM, or built a little faceting language.

We implemented faceted search by having a well-indexed table of plans, and building a SQL query string by tacking on conditions based on a long series of conditions.

The function buildQuery which implemented the core part of this scheme is a single 250 line function, heavily commented, which lays out the logic in a nearly flat way. The focus is kept squarely on business requirements, instead of on fancy code.

Database

creation

We stored the database schemas in a series of .sql files with leading numbers, so that they could be loaded in order at database creation time.

For planinfo and beneinfo, there were no migrations, because the database was recreated every day. Instead, there was a version number stored in both the database and the application. Given that we (tried very hard to) never make backwards incompatible changes to the database, the apps would check at startup that their database schema version number was greater than or equal to the database version number stored in the database, and refuse to start if they were not.

This was a general pattern: If an app encountered any unexpected or missing configuration, it refused to start and threw noticeable, hopefully clear, errors.

I tried hard to make it so that if the apps actually started up, they would have everything they needed to run properly.

There were occasional instances where we accidentally rolled out backwards-incompatible changes to the database, and in those cases we generally rolled back the data update and rebuilt it.

ETL

The part of the system I'm most proud of, and on which I spent the most effort, is the ETL process.

We had a series of shell scripts for each data source we ingested (there were many), which would pull the data and put it in an s3 bucket.

Then, early in the morning, a cron job would spin up an EC2 instance, which would pull in the latest ETL code and all the data files. It would spin up a new database in our RDS instance, and begin the ETL process.

If things went well, right about the time the east coasters got into work, a new database would be rotating into service.

My recollections are not exact, but it took something like two to four hours to generate a new database with more than 250 million rows out of several gigabytes of text files in various formats.

The code to insert data into the database heavily utilized postgres' COPY statement, avoiding INSERTs as much as possible in favor of generating batches of collections that could be COPYed into the database.

models

We used the xo library to connect to the database and generate models, along with heavily customized templates.

The templates themselves, and the code to create the models from them was hairy. Thankfully it mostly only had to be written once and occasionally edited.

testing

Here was my biggest mistake: I invested a great deal of time and effort in creating sql-mock tests for data that changed regularly. These tests needed constant, tedious maintenance.

I should instead have tested against a live database, especially given that we were working mostly with immutable databases and wouldn't have had to deal with recreating it for each test.

Local database for development

Each table in the database had an accompanying script that would generate a subset of the data for use in local development, since the final database was too large to run on a developer's machine.

This let each developer work with a live, local, copy of the database and enabled efficient development of changes.

I highly recommend building in this tooling from the start, it saves you from either trying to add it in once your database grows large, or having your team connect to a remote database, making development slower.

Miscellaneous tooling

We had a CLI tool, written mostly as a bunch of shell scripts, with a ton of available commands that performed all kinds of utility functions related to observability and operations.

It was mostly written by an excellent coworker, and working with it is where I learned to write effective shell scripts. (Thanks to Nathan and shellcheck for this vital skill).

Having this available from the start served as a really useful place for utility features to coalesce; without it they tend to scatter to more places further afield, or just live in a developer's shell history.

One fun bit of tooling I built was the ability to generate graphs from splunk (our log aggregation service) via slack commands, which was particularly helpful in incident handling, as you could easily share graphs with your coworkers.

Logging

Every request that entered the backend got a request id as soon as it hit the system, and that request id was carried around with it wherever it went. Middleware gave the request its id, and then another middleware constructed a sub-logger with the request id embedded into it that was attached to the request context, so that all logs always had the request ID attached.

The system logged on entry and exit, with as much detail as was safe. Any other logging that was above the debug level was supposed to be exceptional, although we weren't super strict about that.

We used zerolog, and it worked great for us.

Documentation

At some point, I converted the markdown docs I and others had written in github into a book about how the system worked, using sphinx-book-theme.

Miraculously, this book gained traction, and I got great contributions from teammates and everybody knew where to look to find system documentation.

I have started documentation websites for many other projects, and no other ones have ever worked as successfully, and I wish that I had any idea why that was but I don't.

It proudly featured our mascot (the corgi) showing off its most notable feature

Runtime integrations

Our client frequently wanted us to add queries that would operate from the browser, and I was fortunate to be able to push back and turn many of those into build-time requests instead.

One place where our performance got killed at our clients' request was with render-blocking analytics scripts; it seemed every team wanted a different script run to get the analytics that they "needed". I advised them against it and tried to demonstrate the performance and download size problems they incurred, but the client was not interested in my arguments.

And more

There are so many more parts of a system like this that I haven't covered here; I mostly wanted to write down a bit about a bunch of the pieces while I still remember what they are.

I was very fortunate to be able to work with such a positive, creative, and engaged team that made the space for such a successful project to be possible. An article about the social factors and personalities that made the team go, and the site happen, would be a second article as long as this one is.

llimllib's notes

my tools in 2026

Computer

Software

editor

neovim plugins

terminal

terminal software

Browser

browser plugins

Note Taking

Mail

Chat

Music

Webcam

Photos

Video

2025 in review

coding

review

worktree

time

gp

tools

games

travel

visualization

Advent of Code 2025

Day 1

Day 2

Licensing will not save us

Writing good technical articles is hard, we should do more of it anyway

Let's try to create a world of constructive dialogue

An AI tool I find useful

How to install it

How it works

How I use it

How it differs

Areas for improvement

Postscript

A pretty decent retry, and not a library

Be explicit

Don't be generic

Don't create a library for small stuff

Toolboxes, not libraries

postscript

Comparing golang sqlite to C sqlite

update

Federation is extremely expensive

The successes are overblown

Moderation is ~impossible

Ownership is important

postscript

Triple Storyline for Ethical Design

Advent of Code 2024

Colophon

How to debug git with visual studio code on a mac

Prerequisites

Building git

Debugging git

A command to print terminal color codes

Serving a billion web requests with boring code

High level

Boring über alles

The boring bits

Why React?

golang

The innovation tokens

Modular backend

drug pricing (aka druginfo)

plan search (aka planinfo)

beneficiary information (aka beneinfo)

gRPC

Strict backwards compatibility

Faceted search

Database

creation

ETL

models

testing