Chandler Swift's Blog

Notes on installing LineageOS on OnePlus 9 ("lemonade")

Tue, 12 Nov 2024 00:00:00 +0000

I just got a hand-me-down OnePlus 9, and installed LineageOS on it. There were a few things that were non-obvious about the installation (which is a first; normally the LineageOS docs are pretty comprehensive!), so I’ve documented those here.

I’m following the installation guide from the LineageOS Wiki, with a LE2115 device.

`adb`/`fastboot` on NixOS

Basic requirements

Make sure your computer has adb and fastboot.

nix-shell -p android-tools

Correct Firmware Version

Warning: Before following these instructions please ensure that the device is currently using Android 13 firmware. If the vendor provided multiple updates for that version, e.g. security updates, make sure you are on the latest! If your current installation is newer or older than Android 13, please upgrade or downgrade to the required version before proceeding (guides can be found on the internet!).

(also the better part of “Checking the correct firmware”)

I was on OxygenOS 14, which was too new, and needed to downgrade to 13. I had one more teeny little insignificant requirement for this bit: Don’t run random untrusted binaries off the internet! Thankfully, xda-developers had the hookup as usual:

https://xdaforums.com/t/oneplus-9-rom-ota-oxygen-os-repo-of-oxygen-os-builds.4254579/ (5040MiB)

From the “Downgrade zips (will wipe your data)” section, I downloaded this file:

https://oxygenos.oneplus.net/8990_sign_LE2115_11_F_OTA_2310_all_uaFihW_10100001.zip (14MiB)

From the body of the first post:

For OxygenOS 14, the OTA-OnePlus-localUpdate-Oplus_key.apk APK […] must be used. It will become available as “Software Update” in the app drawer.

Push the image, set up the updater…

adb push 8990_sign_LE2115_11_F_OTA_2310_all_uaFihW_10100001.zip /sdcard/
adb install OTA-OnePlus-localUpdate-Oplus_key.apk

…et voila! There it is.

Extracting the image takes about 15 minutes. Installing takes about 2 minutes.

Beware: The phone gets wiped during this install!

Root

I’m using magisk. Directions there are pretty good. boot.img is the thing we used for installing LineageOS earlier.

adb push boot.img /sdcard/

and follow the directions! https://topjohnwu.github.io/Magisk/install.html

Advent of Code 2022: Day 22

Wed, 11 Oct 2023 00:00:00 +0000

Well, it’s been a bit, but I’m thinking I might have a chance at finishing 2022 off before 2023 starts…if only just! Quite a few things have changed since last time, not the least of which is this¹:

chandler@boron:~$ uname -m
riscv64

I’m doing most of my non-employer-affiliated development work on a StarFive VisionFive2 these days (with a blog post likely to come at some point), and the transition to RISC-V ended up being pretty straightforward.

As always, I’ve been using Emacs and magit, but haven’t gone out of my way to introduce any new Emacs features this time.

I finally finished this today, but today was the fifth (I believe) major attempt at cracking this particular problem. I solved the first part pretty quickly, but the second part had me stymied for quite some time. My original goal was to find a general solution for any cube net. I spent an evening with my friend Dustin trying to reason about how cube folding worked; we determined that there were eleven possible ways that a cube could unfold…and then promptly found the Wikipedia page for Nets, describing the eleven nets of a cube. Perhaps our drawing is more stylish?

But we weren’t able to figure out a good algorithm for doing so. I tried a few solutions that people had proposed in the r/adventofcode thread, but found that every solution I tried wasn’t a true solution for every case, at least as I understood it; Dustin and I always found a counterexample that broke the solution for at least one orientation of at least one net.

With that in mind, on attempt four I decided to hardcode the net to keep the problem manageable. Despite that advantage, it still gave me some grief. I spent several hours debugging the code, and fixed two off-by-one errors, one flipped sign (which resulted in an edge being joined backwards, making a bit of a Möbius cube edge), and an issue with incorrectly handling rotations when moving off the edge of the net. Specifically, when the cursor moved off the edge, I updated the target_pos and orientation, and if the target_pos was unobstructed, I advanced, updating pos to target_pos. If it was obstructed, I didn’t advance. However, I didn’t revert orientation, leading to an incorrect orientation if I was doing all of these:

Moving off the edge of the net to “warp” to another cube face
Aiming for a space that was blocked
Changing my orientation

This isn’t frequent, but it does happen, and it cost me some serious debugging time!

Anyway, the code ran a bit long, including a pretty lengthy handling of each transition on each edge of the net. Check out the sweet inline net drawings in the comments, though!

Source code

use std::fs;

#[derive(Debug, Clone, Copy, Eq, PartialEq)]
enum Square {
    Void,
    Wall,
    Space,
}

#[derive(Debug, Eq, PartialEq)]
enum Rotation {
    Clockwise,
    CounterClockwise,
}

fn print_map(map: &Vec<Vec<Square>>, position: (i32, i32), orientation: (i32, i32)) {
    for (i, row) in map.iter().enumerate() {
        for (j, col) in row.iter().enumerate() {
            if (j as i32, i as i32) == position {
                match orientation {
                    (1, 0) => print!(">"),
                    (0, -1) => print!("v"),
                    (-1, 0) => print!("<"),
                    (0, 1) => print!("^"),
                    _ => panic!("Illegal orientation"),
                }
            } else {
                match col {
                    Square::Void => print!(" "),
                    Square::Wall => print!("#"),
                    Square::Space => print!("."),
                }
            }
        }
        println!();
    }
}

// Note that this puts a single row of Void around the map, so that we
// don't overflow
fn parse_map(data: &str) -> Vec<Vec<Square>> {
    let rows: Vec<_> = data.split('\n').collect();
    let max_width = rows.iter().map(|r| r.len()).max().unwrap(); // The first row won't necessarily be full-width, so find one that is
    let mut map = vec![vec![Square::Void; max_width + 2]; rows.len() + 2];
    for (i, row) in rows.iter().enumerate() {
        for (j, c) in row.chars().enumerate() {
            match c {
                ' ' => (),
                '.' => map[i + 1][j + 1] = Square::Space, // This AoC brought to you by Squarespace. With Squarespace's easy-to-use dra
                '#' => map[i + 1][j + 1] = Square::Wall,
                _ => panic!("Unexpected map char {}", c),
            }
        }
    }
    map
}

fn parse_directions(data: &str) -> Vec<(usize, Option<Rotation>)> {
    let mut directions = Vec::new();
    for l_chunk in data.split('L') {
        let r_chunks: Vec<_> = l_chunk.split('R').collect();
        for r_chunk in &r_chunks[..r_chunks.len() - 1] {
            directions.push((r_chunk.parse().unwrap(), Some(Rotation::Clockwise)));
        }
        directions.push((
            r_chunks[r_chunks.len() - 1].parse().unwrap(),
            Some(Rotation::CounterClockwise),
        ));
    }
    let last_index = directions.len() - 1; // You'd think the borrow checker could figure this out, but it specifically asks me to do this
    directions[last_index].1 = None; // Last one doesn't have an associated rotation
    directions
}

fn process(data: &str, cube: bool) -> usize {
    let (map, directions) = data.split_once("\n\n").unwrap();
    let map = parse_map(map);
    let directions = parse_directions(directions);

    let mut pos = (
        map[1].iter().position(|x| *x == Square::Space).unwrap() as i32,
        1,
    );
    let mut orientation = (1, 0); // facing right

    for (distance, rotation) in directions {
        for _ in 0..distance {
            // We can do this because the map has a one-cell buffer on
            // each edge, so we won't have any out-of-bounds-ness here.
            let mut target_pos = (pos.0 + orientation.0, pos.1 - orientation.1);
            let mut target_orientation = orientation.clone();

            if map[target_pos.1 as usize][target_pos.0 as usize] == Square::Void {
                if cube {
                    // Unfortunately, I haven't managed to find a way
                    // to generalize this problem, so we're going to
                    // end up hardcoding a bunch of things
                    // here. Beyond just being a bit ugly and
                    // non-generalizable, this also means my test case
                    // won't work, since that uses a different cube
                    // net.
                    //
                    // That said, here's some more neat looking cube
                    // nets I drew!
                    //     ┌─1─┐
                    //     3   2
                    // ┌─3─┼───┼─2─┬─1─┐
                    // 4   │   │   │   4
                    // └─5─┼───┼─7─┴─6─┘
                    //     5   7
                    //     └─6─┘
                    // ┌───┬───┬───┐
                    // │   │   │   │
                    // └───┴───┼───┼───┬───┐
                    //         │   │   │   │
                    //         └───┴───┴───┘
                    //
                    // All the numbers are slightly different than
                    // they'd otherwise be since I have an extra
                    // padding square surrounding the board.
                    //
                    // Here's the shape we'll be using, with connected
                    // sides labeled:
                    //       ┌─1─┬─2─┐
                    //       6   │   3
                    //       ├───┼─4─┘
                    //       7   4
                    //   ┌─7─┼───┤
                    //   6   │   3
                    //   ├───┼─5─┘
                    //   1   5
                    //   └─2─┘
                    //
                    match target_pos {
                        // side labeled 1
                        (x, 0) if x > 50 && x <= 100 => {
                            target_pos = (1, x + 100);
                            target_orientation = (1, 0);
                        }
                        (0, y) if y > 150 && y <= 200 => {
                            target_pos = (y - 100, 1);
                            target_orientation = (0, -1);
                        }
                        // 2
                        (x, 0) if x > 100 && x <= 150 => {
                            target_pos = (x - 100, 200);
                            // orientation unchanged
                        }
                        (x, 201) if x > 0 && x <= 50 => {
                            target_pos = (x + 100, 1);
                            // orientation unchanged
                        }
                        // 3
                        (151, y) if y > 0 && y <= 50 => {
                            target_pos = (100, 151 - y);
                            target_orientation = (-1, 0);
                        }
                        (101, y) if y > 100 && y <= 150 => {
                            target_pos = (150, 151 - y);
                            target_orientation = (-1, 0);
                        }
                        // 4
                        (x, 51) if x > 100 && x <= 150 => {
                            target_pos = (100, x - 50);
                            target_orientation = (-1, 0);
                        }
                        (101, y) if y > 50 && y <= 100 => {
                            target_pos = (y + 50, 50);
                            target_orientation = (0, 1);
                        }
                        // 5
                        (x, 151) if x > 50 && x <= 100 => {
                            target_pos = (50, 100 + x);
                            target_orientation = (-1, 0);
                        }
                        (51, y) if y > 150 && y <= 200 => {
                            target_pos = (y - 100, 150);
                            target_orientation = (0, 1);
                        }
                        // 6
                        (0, y) if y > 100 && y <= 150 => {
                            target_pos = (51, 151 - y);
                            target_orientation = (1, 0);
                        }
                        (50, y) if y > 0 && y <= 50 => {
                            target_pos = (1, 151 - y);
                            target_orientation = (1, 0);
                        }
                        // 7
                        (x, 100) if x > 0 && x <= 50 => {
                            target_pos = (51, x + 50);
                            target_orientation = (1, 0);
                        }
                        (50, y) if y > 50 && y <= 100 => {
                            target_pos = (y - 50, 101);
                            target_orientation = (0, -1);
                        }
                        _ => panic!("I haven't handled this case: {:?}", target_pos),
                    }
                } else {
                    loop {
                        let next_target_pos =_pos.0 - orientation.0, target_pos.1 + orientation.1);
                        if map[next_target_pos.1 as usize][next_target_pos.0 as usize] == Square::Void {
                            break;
                        }
                        target_pos = next_target_pos;
                    }
                }
            }
            match map[target_pos.1 as usize][target_pos.0 as usize] {
                Square::Wall => (),
                Square::Space => {
                    pos = target_pos;
                    orientation = target_orientation;
                }
                Square::Void => panic!("Square is still Void; we just fixed this!"),
            }
        }
        orientation = match rotation {
            None => orientation, // Ignore the last step
            Some(Rotation::Clockwise) => (orientation.1, -orientation.0),
            Some(Rotation::CounterClockwise) => (-orientation.1, orientation.0),
        };
    }
    let orientation_val = match orientation {
        (1, 0) => 0,
        (0, -1) => 1,
        (-1, 0) => 2,
        (0, 1) => 3,
        _ => panic!("Illegal orientation"),
    };
    (pos.1 * 1000 + pos.0 * 4 + orientation_val) as usize
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim_end(); // There's some intentional whitespace at the beginning
    println!("{}", process(data, true));
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "        ...#
        .#..
        #...
        ....
...#.......#
........#...
..#....#....
..........#.
        ...#....
        .....#..
        .#......
        ......#.

10R5L5R10L4R5L5";

    #[test]
    fn test_parse_directions() {
        assert_eq!(
            parse_directions("10R5L5R10L4R5L5"),
            vec![
                (10, Some(Rotation::Clockwise)),
                (5, Some(Rotation::CounterClockwise)),
                (5, Some(Rotation::Clockwise)),
                (10, Some(Rotation::CounterClockwise)),
                (4, Some(Rotation::Clockwise)),
                (5, Some(Rotation::CounterClockwise)),
                (5, None),
            ]
        );
    }

    #[test]
    fn test_part1() {
        assert_eq!(process(DATA, false), 6032);
    }

    #[test]
    fn test_part2() {
        assert_eq!(process(DATA, true), 5031);
    }
}

Soundtrack

I found Mountain Stage listening to KUMD (now 103.3 The North) in Duluth. This episode aired a few weeks ago.

This episode was recorded on May 7th, 2023 at Peoples Bank Theatre in Marietta, Ohio. The lineup includes Amy Ray Band, The Lone Bellow, Alison Brown, Tony Furtado, and Villages.

Listen to the episode on their podcasts page, or download the episode as a 128kbps MP3!

I was (roughly) today years old when I learned Gnome terminal has had a copy-to-html feature for years, and it works really nicely. (I simplified the output a bit, but the default would have been plenty workable.) ↩︎

Advent of Code 2022: Days 20 and 21

Sat, 14 Jan 2023 00:00:00 +0000

Two more down, today. Magit has been awesome, and I think I made it through the entirety of both of these problems without reflexively opening a terminal to solve some problem I encountered (either regarding git, or basic file manipulation)!

Day 20

A short one, though not as quick as it could have been; I ran into some off-by-one errors, and some issues with the remainder operator when going negative. (Wikipedia has a quick review; Rust’s modulo (%) operator uses the truncated division variant. I wanted Euclidian modulo, which is now available as a method on numeric types! See the RFC, the implementation PR, or the docs.)

Source code

use std::fs;

#[allow(dead_code)]
fn print_current_order(original_order: &Vec<i64>, positions: &Vec<usize>) {
    println!(
        "{}\n",
        positions
            .iter()
            .map(|p| original_order[*p].to_string())
            .reduce(|a, b| a + ", " + &b)
            .unwrap()
    );
}

fn process(data: &str, decryption_key: i64, mix_count: usize) -> i64 {
    let original_order: Vec<i64> = data
        .split("\n")
        .map(|n| n.parse::<i64>().unwrap() * decryption_key)
        .collect();
    let mut positions: Vec<usize> = (0..original_order.len()).collect();

    for _ in 0..mix_count {
        for i in 0..original_order.len() {
            let current_pos = positions.iter().position(|x| *x == i).unwrap();
            positions.remove(current_pos);
            let new_pos = (current_pos as i64 + original_order[i])
                .rem_euclid(positions.len() as i64) as usize;
            positions.insert(new_pos, i);
        }
    }

    let original_zero_pos = original_order.iter().position(|x| *x == 0).unwrap();
    let start = positions
        .iter()
        .position(|x| *x == original_zero_pos)
        .unwrap();

    println!(
        "{} {} {}",
        original_order[positions[(start + 1000) % positions.len()]],
        original_order[positions[(start + 2000) % positions.len()]],
        original_order[positions[(start + 3000) % positions.len()]]
    );

    original_order[positions[(start + 1000) % positions.len()]]
        + original_order[positions[(start + 2000) % positions.len()]]
        + original_order[positions[(start + 3000) % positions.len()]]
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data, 811589153, 10));
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "1
2
-3
3
-2
0
4";

    #[test]
    fn test_part1() {
        assert_eq!(process(DATA, 1, 1), 3);
    }

    #[test]
    fn test_part2() {
        assert_eq!(process(DATA, 811589153, 10), 1623178306);
    }
}

Day 21

This was the quickest first half AoC I’ve done in a while; I’m guessing it took just under fifteen minutes from start to finish. (Not even remotely quick enough to get on the leaderboard, by any means; the hundredth person was done with part 21 in less than five minutes!)

The second half, once I realized it’s not feasible to just try every number, was straightforward enough—just recurse up like I’ve recursed down before, and solve from the bottom up. Easier said than done, and the implementation didn’t end up being terribly elegant.

Source code

use std::collections::HashMap;
use std::fs;

#[derive(Clone, Copy)]
enum Monkey<'a> {
    Const(i64),
    Add(&'a str, &'a str),
    Sub(&'a str, &'a str),
    Mul(&'a str, &'a str),
    Div(&'a str, &'a str),
}

fn eval_monkey(monkeys: &HashMap<&str, Monkey>, addr: &str) -> i64 {
    let monkey = monkeys[addr];
    match monkey {
        Monkey::Const(a) => a,
        Monkey::Add(a, b) => eval_monkey(&monkeys, a) + eval_monkey(&monkeys, b),
        Monkey::Sub(a, b) => eval_monkey(&monkeys, a) - eval_monkey(&monkeys, b),
        Monkey::Mul(a, b) => eval_monkey(&monkeys, a) * eval_monkey(&monkeys, b),
        Monkey::Div(a, b) => eval_monkey(&monkeys, a) / eval_monkey(&monkeys, b),
    }
}

fn get_value_for(
    monkeys: &HashMap<&str, Monkey>,
    monkeys_used_by: &HashMap<&str, &str>,
    addr: &str,
) -> i64 {
    let parent = monkeys_used_by[addr];
    let (parent_first_child, parent_second_child) = match monkeys[parent] {
        Monkey::Const(_) => panic!("We're a child of const"),
        Monkey::Add(a, b) => (a, b),
        Monkey::Sub(a, b) => (a, b),
        Monkey::Mul(a, b) => (a, b),
        Monkey::Div(a, b) => (a, b),
    };
    if parent == "root" {
        if parent_first_child == addr {
            eval_monkey(&monkeys, parent_second_child)
        } else {
            eval_monkey(&monkeys, parent_first_child)
        }
    } else {
        match monkeys[parent] {
            Monkey::Const(_) => panic!("We're a child of const"),
            Monkey::Add(a, b) => {
                if a == addr {
                    get_value_for(&monkeys, &monkeys_used_by, parent) - eval_monkey(&monkeys, b)
                } else {
                    get_value_for(&monkeys, &monkeys_used_by, parent) - eval_monkey(&monkeys, a)
                }
            }
            Monkey::Sub(a, b) => {
                if a == addr {
                    get_value_for(&monkeys, &monkeys_used_by, parent) + eval_monkey(&monkeys, b)
                } else {
                    eval_monkey(&monkeys, a) - get_value_for(&monkeys, &monkeys_used_by, parent)
                }
            }
            Monkey::Mul(a, b) => {
                if a == addr {
                    get_value_for(&monkeys, &monkeys_used_by, parent) / eval_monkey(&monkeys, b)
                } else {
                    get_value_for(&monkeys, &monkeys_used_by, parent) / eval_monkey(&monkeys, a)
                }
            }
            Monkey::Div(a, b) => {
                if a == addr {
                    get_value_for(&monkeys, &monkeys_used_by, parent) * eval_monkey(&monkeys, b)
                } else {
                    eval_monkey(&monkeys, a) / get_value_for(&monkeys, &monkeys_used_by, parent)
                }
            }
        }
    }
}

fn parse(data: &str) -> (HashMap<&str, Monkey>, HashMap<&str, &str>) {
    let mut monkeys: HashMap<&str, Monkey> = HashMap::new();
    let mut monkeys_used_by: HashMap<&str, &str> = HashMap::new();
    for row in data.split("\n") {
        let (monkey, op) = row.split_once(": ").unwrap();
        let op: Vec<&str> = op.split(" ").collect();
        monkeys.insert(
            monkey,
            match op.len() {
                1 => Monkey::Const(op[0].parse().unwrap()),
                3 => {
                    assert_eq!(monkeys_used_by.insert(op[0], monkey), None);
                    assert_eq!(monkeys_used_by.insert(op[2], monkey), None);
                    match op[1] {
                        "+" => Monkey::Add(op[0], op[2]),
                        "-" => Monkey::Sub(op[0], op[2]),
                        "*" => Monkey::Mul(op[0], op[2]),
                        "/" => Monkey::Div(op[0], op[2]),
                        other => panic!("Unknown operator {}", other),
                    }
                }
                _ => panic!("Unknown operator {:?}", op),
            },
        );
    }

    (monkeys, monkeys_used_by)
}

fn process1(data: &str) -> i64 {
    let (monkeys, _monkeys_used_by) = parse(data);
    eval_monkey(&monkeys, "root")
}

fn process2(data: &str) -> i64 {
    let (monkeys, monkeys_used_by) = parse(data);

    get_value_for(&monkeys, &monkeys_used_by, "humn")
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process2(data)); // process1(data) for part 1
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "root: pppw + sjmn
dbpl: 5
cczh: sllz + lgvd
zczc: 2
ptdq: humn - dvpt
dvpt: 3
lfqf: 4
humn: 5
ljgn: 2
sjmn: drzm * dbpl
sllz: 4
pppw: cczh / lfqf
lgvd: ljgn * ptdq
drzm: hmdt - zczc
hmdt: 32";

    #[test]
    fn test_part1() {
        assert_eq!(process1(DATA), 152);
    }

    #[test]
    fn test_part2() {
        assert_eq!(process2(DATA), 301);
    }
}

Soundtrack

(Available at Bandcamp, or on YouTube!)

Advent of Code 2022: Days 18 and 19

Fri, 13 Jan 2023 00:00:00 +0000

I’ve been recovering from a rough bout of COVID-19, which really knocked me out for a few days, and (in combination with a tough couple problems) set my AoC progress back a few more days. But I’m back at it! Today: getting acquainted with magit.

Day 18

Today, I installed and tried out magit. Installation was straightforward: M-x package-install <RET> magit <RET>. Usage was…a bit more complex. I skimmed the manual, but didn’t have time for the in-depth read it probably merits. Instead, I read the Getting Started section and looked up key combinations as I needed them. The built-in help was, for me, more than sufficient for my basic use. I expect that (at least for now) I’ll still be dropping to a command line to do anything more advanced than the usual git add -p; git commit; git push, but the integration for that basic workflow has been easy to use and nicely integrated.

Oh, and I solved some AoC problems, too!

Source code

use std::collections::HashSet;
use std::fs;

// TODO: this should be calculable via something like the number of
// points to the (2/3) power over 2 times a constant I'm too lazy to
// work out for this comment, since that would occur if the points
// formed a rough sphere around the air bubble. But I don't wanna, so
// I'm just setting it to something arbitrary and large.
const MAX_AIR_BUBBLE_SIZE: usize = 10000;

fn is_air_bubble(droplets: &HashSet<(i32, i32, i32)>, start: (i32, i32, i32)) -> bool {
    let mut queue = Vec::new();
    let mut bubble = HashSet::new();
    queue.push(start);
    while queue.len() > 0 {
        if bubble.len() >= MAX_AIR_BUBBLE_SIZE {
            return false;
        }
        let current = queue.pop().unwrap();
        bubble.insert(current);
        for offset in [
            (-1, 0, 0),
            (0, -1, 0),
            (0, 0, -1),
            (0, 0, 1),
            (0, 1, 0),
            (1, 0, 0),
        ] {
            let neighbor = (
                current.0 + offset.0,
                current.1 + offset.1,
                current.2 + offset.2,
            );
            if !droplets.contains(&neighbor)
                && !queue.contains(&neighbor)
                && !bubble.contains(&neighbor)
            {
                bubble.insert(neighbor);
                queue.push(neighbor);
            }
        }
    }
    true
}

fn process(data: &str, include_air_bubbles: bool) -> usize {
    let droplets: HashSet<(i32, i32, i32)> = data
        .split('\n')
        .map(|line| {
            let mut nums = line.split(',').map(|i| i.parse().unwrap());
            (
                nums.next().unwrap(),
                nums.next().unwrap(),
                nums.next().unwrap(),
            )
        })
        .collect();

    let mut surface_area = 0;
    for droplet in droplets.iter() {
        for offset in [
            (-1, 0, 0),
            (0, -1, 0),
            (0, 0, -1),
            (0, 0, 1),
            (0, 1, 0),
            (1, 0, 0),
        ] {
            let neighbor = (
                droplet.0 + offset.0,
                droplet.1 + offset.1,
                droplet.2 + offset.2,
            );
            if !droplets.contains(&neighbor)
                && (include_air_bubbles || !is_air_bubble(&droplets, neighbor))
            {
                surface_area += 1;
            }
        }
    }
    surface_area
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data, false));
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "2,2,2
1,2,2
3,2,2
2,1,2
2,3,2
2,2,1
2,2,3
2,2,4
2,2,6
1,2,5
3,2,5
2,1,5
2,3,5";

    #[test]
    fn trivial_part1() {
        assert_eq!(process("1,1,1\n2,1,1", true), 10);
    }

    #[test]
    fn test_part1() {
        assert_eq!(process(DATA, true), 64);
    }

    #[test]
    fn test_part2() {
        assert_eq!(process(DATA, false), 58);
    }
}

Day 19

A few minor optimizations here, to get the program to complete in a slightly more reasonable time:

Aborting a branch early if it’s not going to produce sufficient numbers of geodes. Specifically, the maximum number of geodes I could possibly produce is the number I have now, plus the number I can gather with the robots I have, plus the number I can gather with additional robots if I made one more per minute. This is the possible_geodes calculated below. If that’s less than the maximum number of geodes I’ve already attained by another path, I can give up; there’s no way this branch will prove fruitful. One note is that originally I’d had my branching functions reversed (so I tried building an ore-gathering robot first, then a clay-gathering robot, and so on). It turns out that spending all your resources building ore-gathering robots is a terrible strategy, while “build the most advanced robot you can” is quite a bit better. So I reversed the order, which got me a pretty high score early on in the process, and lets me prune quite a bit more aggressively later.
Not producing more robots of a given type than I’d need. For example, clay is only used in the production of obsidian-collecting robots, which cost, say, X clay. I can only build one robot per minute, so I use at most X clay per minute, so there’s no point in gathering more than X clay per minute, so I should never construct an X+1th clay-gathering robot.

A few more optimizations I didn’t try, but might have considered if I needed additional performance:

I do a lot of copying around of 32-bit ints in structs. Most of them are tiny; I’d had a lot of them (e.g. the Blueprint costs) as u8s for a while before consolidating everything to u32s to save on tons of extra type casts everywhere, as well as save me some reasoning about whether I’d need the extra space. This was probably the wrong move for performance reasons; now I’m copying a bunch of zero bytes around unnecessarily. I could go back to u8s.
I could probably do some more logicking to more aggressively prune branches that I can eliminate any possibility of bearing fruit. That said, I do feel like I got several of the easiest wins here.
This program is probably a reasonable candidate for some form of parallel processing, and on my 8-core processor I could probably expect a speedup of somewhere between 4 and 8 times, if the locks don’t get too heavily contended.

With these optimizations, and compiling with optimizations (-O), the first part takes just under two seconds to run, and the second part takes just over two minutes.

Source code

use std::fs;

#[derive(Debug, Eq, PartialEq, Copy, Clone)]
struct Blueprint {
    id: u8,
    ore_robot_ore_cost: u32,
    clay_robot_ore_cost: u32,
    obsidian_robot_ore_cost: u32,
    obsidian_robot_clay_cost: u32,
    geode_robot_ore_cost: u32,
    geode_robot_obsidian_cost: u32,
}

#[derive(Debug, Copy, Clone)]
struct Inventory {
    ore: u32,
    clay: u32,
    obsidian: u32,
    geodes: u32,
}

#[derive(Debug, Copy, Clone)]
struct Robots {
    ore_collecting: u32,
    clay_collecting: u32,
    obsidian_collecting: u32,
    geode_cracking: u32,
}

fn parse_blueprint(data: &str) -> Blueprint {
    let (id, remaining) = data[10..].split_once(": ").unwrap();
    let id = id.parse().unwrap();
    let (ore_robot_ore_cost, remaining) = remaining[21..].split_once(" ").unwrap();
    let ore_robot_ore_cost = ore_robot_ore_cost.parse().unwrap();
    let (clay_robot_ore_cost, remaining) = remaining[27..].split_once(" ").unwrap();
    let clay_robot_ore_cost = clay_robot_ore_cost.parse().unwrap();
    let (obsidian_robot_ore_cost, remaining) = remaining[31..].split_once(" ").unwrap();
    let obsidian_robot_ore_cost = obsidian_robot_ore_cost.parse().unwrap();
    let (obsidian_robot_clay_cost, remaining) = remaining[8..].split_once(" ").unwrap();
    let obsidian_robot_clay_cost = obsidian_robot_clay_cost.parse().unwrap();
    let (geode_robot_ore_cost, remaining) = remaining[29..].split_once(" ").unwrap();
    let geode_robot_ore_cost = geode_robot_ore_cost.parse().unwrap();
    let (geode_robot_obsidian_cost, _remaining) = remaining[8..].split_once(" ").unwrap();
    let geode_robot_obsidian_cost = geode_robot_obsidian_cost.parse().unwrap();
    Blueprint {
        id: id,
        ore_robot_ore_cost: ore_robot_ore_cost,
        clay_robot_ore_cost: clay_robot_ore_cost,
        obsidian_robot_ore_cost: obsidian_robot_ore_cost,
        obsidian_robot_clay_cost: obsidian_robot_clay_cost,
        geode_robot_ore_cost: geode_robot_ore_cost,
        geode_robot_obsidian_cost: geode_robot_obsidian_cost,
    }
}

fn geodes_opened(blueprint: Blueprint, remaining_time: usize) -> usize {
    let robots = Robots {
        ore_collecting: 1,
        clay_collecting: 0,
        obsidian_collecting: 0,
        geode_cracking: 0,
    };
    let inventory = Inventory {
        ore: 0,
        clay: 0,
        obsidian: 0,
        geodes: 0,
    };
    rec_geodes_opened(blueprint, remaining_time, robots, inventory, &mut 0)
}

fn rec_geodes_opened(
    blueprint: Blueprint,
    remaining_time: usize,
    robots: Robots,
    inventory: Inventory,
    max_overall_geodes: &mut usize,
) -> usize {
    if remaining_time > 26 {
        println!("{}", remaining_time);
    }
    let starting_inventory = inventory;

    let mut inventory = inventory;
    inventory.ore += robots.ore_collecting as u32;
    inventory.clay += robots.clay_collecting as u32;
    inventory.obsidian += robots.obsidian_collecting as u32;
    inventory.geodes += robots.geode_cracking as u32;

    let possible_geodes: usize = inventory.geodes as usize
        + (robots.geode_cracking as usize * remaining_time)
        + (remaining_time * (remaining_time - 1) / 2);

    let mut max_geodes = 0;

    let mut ore_costs = [
        blueprint.ore_robot_ore_cost,
        blueprint.clay_robot_ore_cost,
        blueprint.obsidian_robot_ore_cost,
        blueprint.geode_robot_ore_cost,
    ];
    ore_costs.sort();
    let max_ore_cost = ore_costs[ore_costs.len() - 1];

    if remaining_time > 1 && possible_geodes >= *max_overall_geodes {
        if blueprint.geode_robot_ore_cost <= starting_inventory.ore
            && blueprint.geode_robot_obsidian_cost <= starting_inventory.obsidian
        {
            let mut robots = robots.clone();
            let mut inventory = inventory.clone();
            inventory.ore -= blueprint.geode_robot_ore_cost;
            inventory.obsidian -= blueprint.geode_robot_obsidian_cost;
            robots.geode_cracking += 1;
            let geodes = rec_geodes_opened(
                blueprint,
                remaining_time - 1,
                robots,
                inventory,
                max_overall_geodes,
            );
            if geodes > max_geodes {
                max_geodes = geodes;
            }
        }
        if blueprint.obsidian_robot_ore_cost <= starting_inventory.ore
            && blueprint.obsidian_robot_clay_cost <= starting_inventory.clay
            && robots.obsidian_collecting < blueprint.geode_robot_obsidian_cost
        {
            let mut robots = robots.clone();
            let mut inventory = inventory.clone();
            inventory.ore -= blueprint.obsidian_robot_ore_cost;
            inventory.clay -= blueprint.obsidian_robot_clay_cost;
            robots.obsidian_collecting += 1;
            let geodes = rec_geodes_opened(
                blueprint,
                remaining_time - 1,
                robots,
                inventory,
                max_overall_geodes,
            );
            if geodes > max_geodes {
                max_geodes = geodes;
            }
        }
        if blueprint.clay_robot_ore_cost <= starting_inventory.ore
            && robots.clay_collecting < blueprint.obsidian_robot_clay_cost
        {
            let mut robots = robots.clone();
            let mut inventory = inventory.clone();
            inventory.ore -= blueprint.clay_robot_ore_cost;
            robots.clay_collecting += 1;
            let geodes = rec_geodes_opened(
                blueprint,
                remaining_time - 1,
                robots,
                inventory,
                max_overall_geodes,
            );
            if geodes > max_geodes {
                max_geodes = geodes;
            }
        }
        if blueprint.ore_robot_ore_cost <= starting_inventory.ore
            && robots.ore_collecting < max_ore_cost
        {
            let mut robots = robots.clone();
            let mut inventory = inventory.clone();
            inventory.ore -= blueprint.ore_robot_ore_cost;
            robots.ore_collecting += 1;
            let geodes = rec_geodes_opened(
                blueprint,
                remaining_time - 1,
                robots,
                inventory,
                max_overall_geodes,
            );
            if geodes > max_geodes {
                max_geodes = geodes;
            }
        }
        let geodes = rec_geodes_opened(
            blueprint,
            remaining_time - 1,
            robots,
            inventory,
            max_overall_geodes,
        );
        if geodes > max_geodes {
            max_geodes = geodes;
        }
    } else {
        return inventory.geodes as usize;
    }

    if max_geodes > *max_overall_geodes {
        *max_overall_geodes = max_geodes;
    }
    max_geodes
}

fn process(data: &str) -> usize {
    let blueprints: Vec<Blueprint> = data.split("\n").map(|s| parse_blueprint(s)).collect();

    blueprints
        .iter()
        .map(|b| b.id as usize * geodes_opened(*b, 24) as usize)
        .sum()
}

fn process2(data: &str) -> usize {
    let blueprints: Vec<Blueprint> = data
        .split("\n")
        .take(3)
        .map(|s| parse_blueprint(s))
        .collect();

    blueprints
        .iter()
        .map(|b| geodes_opened(*b, 32) as usize)
        .product()
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process2(data)); // process(data) for part 1
}

#[cfg(test)]
mod tests {
    use super::*;

    const DATA: &str = "Blueprint 1: Each ore robot costs 4 ore. Each clay robot costs 2 ore. Each obsidian robot costs 3 ore and 14 clay. Each geode robot costs 2 ore and 7 obsidian.
Blueprint 2: Each ore robot costs 2 ore. Each clay robot costs 3 ore. Each obsidian robot costs 3 ore and 8 clay. Each geode robot costs 3 ore and 12 obsidian.";

    #[test]
    fn test_parse() {
        let mut lines = DATA.split("\n");
        let line_1 = Blueprint {
            id: 1,
            ore_robot_ore_cost: 4,
            clay_robot_ore_cost: 2,
            obsidian_robot_ore_cost: 3,
            obsidian_robot_clay_cost: 14,
            geode_robot_ore_cost: 2,
            geode_robot_obsidian_cost: 7,
        };
        let line_2 = Blueprint {
            id: 2,
            ore_robot_ore_cost: 2,
            clay_robot_ore_cost: 3,
            obsidian_robot_ore_cost: 3,
            obsidian_robot_clay_cost: 8,
            geode_robot_ore_cost: 3,
            geode_robot_obsidian_cost: 12,
        };
        assert_eq!(parse_blueprint(lines.next().unwrap()), line_1);
        assert_eq!(parse_blueprint(lines.next().unwrap()), line_2);
    }

    #[test]
    fn test_find_largest_number_of_geodes_that_can_be_opened() {
        let mut lines = DATA.split("\n");
        assert_eq!(geodes_opened(parse_blueprint(lines.next().unwrap()), 24), 9);
        assert_eq!(
            geodes_opened(parse_blueprint(lines.next().unwrap()), 24),
            12
        );
    }

    #[test]
    fn test_part1() {
        assert_eq!(process(DATA), 33);
    }
}

Soundtrack

New Year, New Router!

Wed, 04 Jan 2023 00:00:00 +0000

Over Christmas break, I replaced my old router (a standard x86_64 PC running pfSense¹) with a new FriendlyElec NanoPi R2S, which has (for the first few days, anyway) been a fantastic bit of hardware. On the software side, I’m running OpenWRT, which is a stability upgrade from pfSense, and a major functionality bump from any other router OS I’ve tried. I’m now a few days in, and it’s been a really nice experience.

It’s worth clarifying that this isn’t the combination router/access point device that many people mean when they say “router”. An off-the-shelf wireless router/AP performs a whole bunch of functions, and this device performs many, but not of all, of those:

Packet routing/NAT
Firewall
DHCP server
DNS server/forwarder
VPN server (WIP, untested)
Wireless access point (I use a Unifi AC Lite, which may soon also be running OpenWRT)
Ethernet switch (I’m running a Netgear GS724T switch I pulled out of a dumpster; nothing fancy, but it gets the job done)
Modem, sometimes (I use a Motorola MB7220)

Hardware

Acquisition: Ordered from AliExpress

Shipping from the official FriendlyElec site was fairly expensive. The board, case, and USB cable totaled $41.99, and shipping was at least an additional $17. Plus, I’m too impatient to wait two months!

Instead, I found a reseller on AliExpress who included free shipping in their $3 markup ($45.81 total, free shipping, plus tax, at the time of purchase.)

https://www.aliexpress.us/item/3256802737557752.html

I placed my order December 1st, and the package arrived December 21st.

Performance: Adequate for 200mbps

I’m currently on a 200mbps down/10mbps up internet plan. I don’t have any trouble getting that on this hardware. However, gigabit or multi-gigabit might have trouble, based on a quick benchmark: When downloading at full throttle, just over 20% of my CPU time is used (mostly spent in softirqs). I’d expect this to scale with network traffic, so I’d plan to top out around a gigabit one way, or 500mbps full-duplex. If I ever wind up with gigabit fiber, I may want to look at upgrading; however, this is plenty of headroom for my current connection.

Thermals: Pretty good with the heatsink

The R2S has a quad-core processor, so I spun up four processes of cat /dev/urandom > /dev/null (a pretty effective single-core CPU stresser) and watched temperatures. They peaked around 65°C.

I haven’t done any kind of burn-in test, but my suspicion is that this chip should be able to run flat out indefinitely. I’d been considering opening this up and applying thermal paste (which, I suspect, is somewhere between unoptimally applied and nonexistent at the moment), but it looks like that won’t be necessary.

Note that I didn’t do anything fancy for testing here, so take these observations with a generously sized grain of salt. It’s plausible that the temperature sensor isn’t properly calibrated; it’s plausible that there’s throttling happening which I haven’t noticed; it’s plausible that this is a poor stress test, and that running something else would generate quite a bit more heat; it’s plausible that my house being 10°F warmer in the summer would cause some additional problems. All that said, I’m comfortable enough that I don’t feel the need to do more investigation unless I run into problems.

Further discussion on the Armbian forums

Power consumption: Typically 2-3 Watts

When idle, the router pulls as little as 2.0 to as much as 2.5 watts; it seems to average 2.1 or 2.2, measured with a Kill-A-Watt P4400. When running a speedtest (so about 200mbps down), I measured barely more power (2.3–2.6W), and when running a quad-core stress test, the highest observed peak was 4.2 watts.

Other considered hardware

I’d also been looking at some other models in the NanoPi line, and ooh boy do they ever make a lot of models. This feels like one of those choices that might be accidentally turing-complete! Here’s a comparison of all of NanoPi’s boards with at least two ethernet ports. Prices listed are all from FriendlyElec: I’d never have finished this post if I also collected the best AliExpress prices for each!

Product	Processor	Memory	Storage	Ethernet ports	Price with case
NanoPi R2C Plus	Rockchip RK3328: 4x Cortex-A53	1GB DDR4	8GB eMMC	1x native 1Gbps, 1x USB3 1Gbps	$39
NanoPi R2S	Rockchip RK3328: 4x Cortex-A53	1GB DDR4	—	1x native 1Gbps, 1x USB3 1Gbps	$39
NanoPi R4S	Rockchip RK3399: 4x Cortex-A72 + 4x Cortex-A53	4GB LPDDR4	—	1x native 1Gbps, 1x PCIe 1Gbps	$75
NanoPi R4SE	Rockchip RK3399: 4x Cortex-A72 + 4x Cortex-A53	4GB LPDDR4	32GB eMMC	1x native 1Gbps, 1x PCIe 1Gbps	$80
NanoPi R5C	Rockchip RK3568B2: 4x Cortex-A55	1GB LPDDR4X	8GB eMMC	2x PCIe 2.5Gbps²	$49
NanoPi R5C	Rockchip RK3568B2: 4x Cortex-A55	4GB LPDDR4X	32GB eMMC	2x PCIe 2.5Gbps²	$59
NanoPi R5S	Rockchip RK3568B2: 4x Cortex-A55	2GB LPDDR4X	8GB eMMC	1x native 1Gbps, 2x PCIe 2.5Gbps	$75
NanoPi R5S	Rockchip RK3568B2: 4x Cortex-A55	4GB LPDDR4X	16GB eMMC	1x native 1Gbps, 2x PCIe 2.5Gbps	$85
NanoPi R6S	Rockchip RK3588S: 4x Cortex-A76 + 4x Cortex-A55	8GB LPDDR4X	32GB eMMC	1x native 1Gbps, 2x PCIe 2.5Gbps	$139

Many of these have other features like various speed USB ports, GPUs and HDMI out, WiFi and Bluetooth adapters, and IR receivers. I didn’t care about any of those use cases, so those aren’t mentioned here. I also didn’t include the specifics on CPU and memory clock speeds, or on eMMC performance, though those may factor in to many use cases. My needs aren’t too demanding, which is why I ended up happy with the lowest-spec’ed device on the list!

All in all, the basic board I ordered was plenty for my use case: I don’t have fast enough internet (thanks, Comcast!) to saturate download bandwidth; I don’t have enough upload bandwidth to be CPU-limited when using a VPN from elsewhere; I don’t run vlans or anything that requires internal-only networking to pass through the router; and I don’t have >1Gbps networking set up anywhere in my house.

And, worth a mention: FriendlyElec’s documentation is really good. See for this board, for example: http://wiki.friendlyelec.com/wiki/index.php/NanoPi_R2S

Installing OpenWRT

OpenWRT’s documentation (despite being labeled “Under Construction”) for the R2S is fantastic.

https://openwrt.org/toh/friendlyarm/nanopi_r2s

Installing OpenWRT is pretty straightforward: Write the downloaded disk image to a microSD card, insert the card, and log into the router for first-time configuration. I’d guess the part that took me the longest was finding a blank microSD card!

Configuring OpenWRT

This is my first experience with OpenWRT, and I wanted easy, so I stuck to LuCI, the web-based administration portal LuCI is just a convenient wrapper around UCI, the Unified Configuration Interface, which amounts to editing text files in /etc/config.

Initial configuration amounted to setting up an administrator password, setting up some basics like time and time zone, and adding my SSH keys.

Basic routing: Plug and play

I just plugged in the router and the tubes started flowing. No configuration needed.

Static DHCP leases

In general, I try to use DHCP for everything I can; however, there are a few things on my network for which I need to have static addresses set up. (I set these up as static DHCP leases, so they can survive accidental networking configuration changes; I’ve done that more than once!) Those exceptions are, at the moment, the router, my desktop, and my HTTP server (since I can’t port forward to a dynamic IP address).

This was straightforward to setup: Network > DHCP and DNS > Static Leases.

Port forwarding

Network > Firewall > Port Forwards

At the point that I was trying to configure these, I hadn’t really gotten used to the UI yet, and I didn’t notice that below the menu and dropdown submenus at the top, there’s another tier of menus in a tabbed interface below, and port forwarding is hidden behind one of these.

I try to keep as few ports open as possible, but do forward :80, :443, :25565 (occasionally), and a few ports for custom internal services that I occasionally want friends to have access to without necessarily needing to be VPN’ed into my network.

DDNS

I use Cloudflare for my DNS.

I installed the luci-app-ddns, ddns-scripts, and ddns-scripts-cloudflare packages, and then configured DDNS through the web interface (Services > Dynamic DNS).

I tried to set the service up without reading through the documentation, which proved to be a mistake. I ran into a few issues that I had to fall back to reading the (excellent) documentation for:

For a domain like home.chandlerswift.com in the chandlerswift.com zone, I need to put home@chandlerswift.com in the Domain field. (Docs: “To use subdomains (CNAME or A records), use the format below when filling your credentials: domain {subdomain}@[zone]”)
Credentials aren’t my CloudFlare credentials, but they don’t use OAuth or similar either. Instead, I have to create a new API token through the Cloudflare dashboard, and use username Bearer and password {token} to authenticate. (Docs: “Create Custom Token by following the Creating API tokens guide. Make sure to add “Zone DNS Edit” Permission to your custom token. You can also “include Specific zone” under Zone Resources. … username Bearer, password [Your API token]”)

Docs: https://openwrt.org/docs/guide-user/services/ddns/client#cloudflarecom

Monitoring/pretty graphs with collectd

I’m monitoring an assortment of things with collectd, which I’d never used before and am now enamored with. It’s very plausible that I’m going to stand up a proper collectd collector somewhere on my network, and send a variety of stats there. I do like graphs!

I installed the luci-app-statistics package, plus collectd-mod-thermal for the thermal sensor images above, and now I have memory, networking, and CPU stats, plus potentially more in the future.

I haven’t yet figured out what the default LuCI install uses for its statistics (Status > Realtime Graphs), and if I can add stats to that instead of collectd. It does seem, though, that those stats are only graphed when I’m viewing the page, while I can get historic graphs at any time with collectd.

I’m currently not persisting my collectd logs anywhere. I may send them off to another server at some point: the NAS, possibly? But for now they’re just going into /tmp, and will be lost on reboot.

In addition to what I have set up, there are lots of other plugins for monitoring other data points with collectd:

List of available collectd packages

root@OpenWrt:~# opkg list | grep collectd-mod
collectd-mod-apache - 5.12.0-33 - apache status input plugin
collectd-mod-apcups - 5.12.0-33 - apcups status input plugin
collectd-mod-ascent - 5.12.0-33 - ascent status input plugin
collectd-mod-bind - 5.12.0-33 - BIND server/zone input plugin
collectd-mod-chrony - 5.12.0-33 - chrony status input plugin
collectd-mod-conntrack - 5.12.0-33 - connection tracking table size input plugin
collectd-mod-contextswitch - 5.12.0-33 - context switch input plugin
collectd-mod-cpu - 5.12.0-33 - CPU input plugin
collectd-mod-csv - 5.12.0-33 - CSV output plugin
collectd-mod-curl - 5.12.0-33 - cURL input plugin
collectd-mod-df - 5.12.0-33 - disk space input plugin
collectd-mod-dhcpleases - 5.12.0-33 - show dhcpleases plugin
collectd-mod-disk - 5.12.0-33 - disk usage/timing input plugin
collectd-mod-dns - 5.12.0-33 - DNS traffic input plugin
collectd-mod-email - 5.12.0-33 - email output plugin
collectd-mod-entropy - 5.12.0-33 - Entropy amount input plugin
collectd-mod-ethstat - 5.12.0-33 - Ethernet adapter statistics input plugin
collectd-mod-exec - 5.12.0-33 - process exec input plugin
collectd-mod-filecount - 5.12.0-33 - file count input plugin
collectd-mod-fscache - 5.12.0-33 - file-system based caching framework input plugin
collectd-mod-interface - 5.12.0-33 - network interfaces input plugin
collectd-mod-ipstatistics - 5.12.0-33 - ipstatistics input plugin
collectd-mod-iptables - 5.12.0-33 - iptables status input plugin
collectd-mod-irq - 5.12.0-33 - interrupt usage input plugin
collectd-mod-iwinfo - 5.12.0-33 - libiwinfo wireless statistics plugin
collectd-mod-load - 5.12.0-33 - system load input plugin
collectd-mod-logfile - 5.12.0-33 - log files output plugin
collectd-mod-lua - 5.12.0-33 - lua input/output plugin
collectd-mod-match-empty-counter - 5.12.0-33 - empty-counter match plugin
collectd-mod-match-hashed - 5.12.0-33 - hashed match plugin
collectd-mod-match-regex - 5.12.0-33 - regex match plugin
collectd-mod-match-timediff - 5.12.0-33 - timediff match plugin
collectd-mod-match-value - 5.12.0-33 - value match plugin
collectd-mod-memory - 5.12.0-33 - physical memory usage input plugin
collectd-mod-modbus - 5.12.0-33 - read variables through libmodbus plugin
collectd-mod-mqtt - 5.12.0-33 - transmit data with MQTT plugin
collectd-mod-mysql - 5.12.0-33 - MySQL status input plugin
collectd-mod-netlink - 5.12.0-33 - netlink input plugin
collectd-mod-network - 5.12.0-33 - network input/output plugin
collectd-mod-nginx - 5.12.0-33 - nginx status input plugin
collectd-mod-ntpd - 5.12.0-33 - NTP daemon status input plugin
collectd-mod-nut - 5.12.0-33 - UPS monitoring input plugin
collectd-mod-olsrd - 5.12.0-33 - OLSRd status input plugin
collectd-mod-openvpn - 5.12.0-33 - OpenVPN traffic/compression input plugin
collectd-mod-ping - 5.12.0-33 - ping status input plugin
collectd-mod-postgresql - 5.12.0-33 - PostgreSQL status input plugin
collectd-mod-powerdns - 5.12.0-33 - PowerDNS server status input plugin
collectd-mod-processes - 5.12.0-33 - process status input plugin
collectd-mod-protocols - 5.12.0-33 - network protocols input plugin
collectd-mod-routeros - 5.12.0-33 - MikroTik RouterOS input plugin
collectd-mod-rrdtool - 5.12.0-33 - RRDtool output plugin
collectd-mod-sensors - 5.12.0-33 - lm_sensors input plugin
collectd-mod-smart - 5.12.0-33 - smart input plugin
collectd-mod-snmp - 5.12.0-33 - SNMP input plugin
collectd-mod-snmp6 - 5.12.0-33 - snmp6 input plugin
collectd-mod-sqm - 5.12.0-33 - SQM/qdisc collection plugin
collectd-mod-swap - 5.12.0-33 - swap input plugin
collectd-mod-syslog - 5.12.0-33 - syslog output plugin
collectd-mod-table - 5.12.0-33 - table-like structured file input plugin
collectd-mod-tail - 5.12.0-33 - tail input plugin
collectd-mod-tail-csv - 5.12.0-33 - tail CSV input plugin
collectd-mod-tcpconns - 5.12.0-33 - TCP connection tracking input plugin
collectd-mod-teamspeak2 - 5.12.0-33 - TeamSpeak2 input plugin
collectd-mod-ted - 5.12.0-33 - The Energy Detective input plugin
collectd-mod-thermal - 5.12.0-33 - system temperatures input plugin
collectd-mod-threshold - 5.12.0-33 - Notifications and thresholds plugin
collectd-mod-ubi - 5.12.0-33 - Unsorted block images plugin
collectd-mod-unixsock - 5.12.0-33 - unix socket output plugin
collectd-mod-uptime - 5.12.0-33 - uptime status input plugin
collectd-mod-users - 5.12.0-33 - user logged in status input plugin
collectd-mod-vmem - 5.12.0-33 - virtual memory usage input plugin
collectd-mod-wireless - 5.12.0-33 - wireless status input plugin
collectd-mod-write-graphite - 5.12.0-33 - Carbon/Graphite output plugin
collectd-mod-write-http - 5.12.0-33 - HTTP POST output plugin

Disk space

The image that OpenWRT provides contains a partition just over 100MB in size. This, of course, wildly underutilizes the available space when extracted to a 16GB microSD card. The way it’s set up makes sense, as they need to build their images to the lowest common denominator of disk space, and most people don’t need that much space anyway. I don’t really care; I’m currently using 18.86 MiB / 102.33 MiB (18%) of my disk space, and I don’t anticipate installing many more packages. But for use cases like logging, it might be nice to have more disk space available.

It sounds like the best way to deal with this is to either manually extract each image onto the disk, or to build my own images with a custom amount of space. Either of these is a feasible workaround, but my current view is that I’m going to cross that bridge when I run out of space. Plus, I like having the microSD card as a mostly read-only filesystem, since they’re not known for their reliability.

Unimplemented/Unresolved issues

OpenWRT uses a self-signed certificate for LuCI over HTTPS. Can I use a proper cert? At least I’d like to use something from my internal CA.
VPN: I’ve explored both OpenVPN and Wireguard. These were painful and unimplemented-then-badly-implemented, respectively, on pfSense, and I haven’t finished setting them up here yet. https://openwrt.org/docs/guide-user/services/vpn/wireguard/basics, https://openwrt.org/docs/guide-user/services/vpn/openvpn/start
I’d like to do some kind of per-device bandwidth monitoring, which apparently OpenWRT has good support for. I’m rarely even remotely close to it, but Comcast does impose a data cap that I’m reasonably keen to avoid.
I use a small USB-enabled UPS (an Amazon Basics-branded white-label APC unit) to keep my network gear running through brief power outages. I’d love to use the router to monitor power draw on the UPS and its expected runtime. OpenWRT appears to have support for this: https://openwrt.org/docs/guide-user/services/ups/apcupsd_su700, https://networkupstools.org/ (Though at the moment, the battery appears to be on death’s door, as it dies an under-voltaged death on any significant load; I typically use <100w for all of my networking/server gear, and it’s a 600VA UPS so it should be able to handle the load without any issues!)
As long as I’m messing around with networking gear, now would be a great time to set up vlans for my IoT gear. My switch and access point both support this, so it might be nice to try. https://openwrt.org/docs/guide-user/network/vlan/start

Reboot time: 30 seconds

One thing I wanted to measure was how long the router takes to come back up after a reboot or power outage. My fairly unscientific test (“start pinging the router, SSH in and reboot the router, measure how many seconds’ worth of pings are lost”), repeated 3 times, yielded 29, 29, and 30 seconds. This isn’t fantastic (something this basic should probably be able to start and get network connected somewhat faster) but it’s definitely usable, and an improvement over what I had with pfSense.

UI samples

I’m pretty happy to be done with pfSense. Their software is definitely usable, but their business shenanigans plus the fact that their software isn’t really Open Source have left a pretty bad taste in my mouth, and I’m happy to be moving on. Plus, a major yikes at the whole pfSense/Wireguard fiasco! ↩︎
This board also includes an M.2 slot for a PCIe WiFi adapter on the bottom, which I can buy for an extra $18. I didn’t include that in my comparison, because I use a discrete access point, which is designed to be an access point and nothing else. ↩︎ ↩︎

Advent of Code 2022: Days 16 and 17

Tue, 03 Jan 2023 00:00:00 +0000

Well, it’s slow going. I was getting less than a day per day done, and now I’m back at work, so that’ll likely be even slower going forward. I’m retargeting mid-January for the end. But here are the next couple days’ worth!

Day 16

Not a particularly efficient implementation, but it works. We do a brute-force breadth first search here, and come up with an answer in just under two seconds, so I’m not too worried about optimization here. Both this and part 2 were run with -O for a fairly reasonable speedup.

$ time ./day16
2119
./day16  1.45s user 0.43s system 99% cpu 1.890 total

Part 1 source code

use std::collections::{HashMap, HashSet, VecDeque};
use std::fs;

#[derive(Debug)]
struct Valve {
    connections: Vec<String>,
    flow_rate: usize,
}

fn find_shortest_path<'a>(
    valves: &'a HashMap<&'a str, Valve>,
    origin: &'a str,
    destination: &str,
) -> Vec<&'a str> {
    // breadth-first search
    let mut nodes_to_check = VecDeque::new();
    nodes_to_check.push_back((origin, vec![]));
    while nodes_to_check.len() > 0 {
        let (current_node, path_to_current_node) = nodes_to_check.pop_front().unwrap();
        for connection in valves[&current_node].connections.iter() {
            let mut path_to_connection = path_to_current_node.clone();
            path_to_connection.push(connection.as_str());
            if connection == destination {
                return path_to_connection;
            }
            nodes_to_check.push_back((&connection, path_to_connection));
        }
    }
    panic!(
        "Could not find a route between {} and {}",
        origin, destination
    );
}

// Note that this only finds paths between nodes with non-zero flow rate
fn find_shortest_paths<'a>(
    valves: &'a HashMap<&'a str, Valve>,
) -> HashMap<(&'a str, &'a str), Vec<&str>> {
    let mut shortest_paths = HashMap::new();
    for (origin, _valve) in valves
        .iter()
        .filter(|(n, v)| v.flow_rate > 0 || **n == "AA")
    {
        for (destination, _valve) in valves.iter().filter(|(_, v)| v.flow_rate > 0) {
            if origin == destination {
                continue;
            }
            shortest_paths.insert(
                (*origin, *destination),
                find_shortest_path(valves, origin, destination),
            );
        }
    }
    shortest_paths
}

fn parse(data: &str) -> HashMap<&str, Valve> {
    let mut valves = HashMap::new();
    for line in data.split("\n") {
        let valve_name = &line[6..8];
        let (flow_rate, mut rest) = line[23..].split_once(';').unwrap();
        let flow_rate = flow_rate.parse().unwrap();
        // Curse you single path with single tunnel for breaking my parsing
        if rest.chars().nth(7).unwrap() == 's' {
            rest = &rest[24..]; // " tunnels lead to valves "
        } else {
            rest = &rest[23..]; // " tunnel leads to valve "
        }
        let connections = rest.split(", ").map(|s| s.to_owned()).collect();
        valves.insert(
            valve_name,
            Valve {
                connections: connections,
                flow_rate: flow_rate,
            },
        );
    }
    valves
}

fn process(data: &str) -> usize {
    let valves = parse(data);
    let mut starting_closed_flowing_valves: HashSet<&str> = valves
        .iter()
        .filter(|(_, v)| v.flow_rate > 0)
        .map(|(k, _)| *k)
        .collect();
    let shortest_paths = find_shortest_paths(&valves);

    let mut most_pressure_released = 0;

    // breadth-first search
    let mut nodes_to_check = VecDeque::new(); // (current_node, remaining_closed_valves, pressure_released, minutes_elapsed)
    if starting_closed_flowing_valves.contains("AA") {
        starting_closed_flowing_valves.remove("AA");
        nodes_to_check.push_back((
            "AA",
            starting_closed_flowing_valves,
            29 * valves["AA"].flow_rate,
            1,
        ));
    } else {
        nodes_to_check.push_back(("AA", starting_closed_flowing_valves, 0, 0));
    }
    while let Some((current_node, remaining_closed_valves, pressure_released, minutes_elapsed)) =
        nodes_to_check.pop_front()
    {
        for target_node in remaining_closed_valves.iter() {
            // can we reach it before 30 minutes is up?
            let opened_at =
                minutes_elapsed + shortest_paths[&(current_node, *target_node)].len() + 1;
            if opened_at < 30 {
                let new_pressure_released =
                    pressure_released + (30 - opened_at) * valves[target_node].flow_rate;
                let mut new_remaining_closed_valves = remaining_closed_valves.clone();
                new_remaining_closed_valves.remove(target_node);
                nodes_to_check.push_back((
                    target_node,
                    new_remaining_closed_valves,
                    new_pressure_released,
                    opened_at,
                ));
                if new_pressure_released > most_pressure_released {
                    most_pressure_released = new_pressure_released;
                }
            }
        }
    }
    most_pressure_released
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data));
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "Valve AA has flow rate=0; tunnels lead to valves DD, II, BB
Valve BB has flow rate=13; tunnels lead to valves CC, AA
Valve CC has flow rate=2; tunnels lead to valves DD, BB
Valve DD has flow rate=20; tunnels lead to valves CC, AA, EE
Valve EE has flow rate=3; tunnels lead to valves FF, DD
Valve FF has flow rate=0; tunnels lead to valves EE, GG
Valve GG has flow rate=0; tunnels lead to valves FF, HH
Valve HH has flow rate=22; tunnel leads to valve GG
Valve II has flow rate=0; tunnels lead to valves AA, JJ
Valve JJ has flow rate=21; tunnel leads to valve II";

    #[test]
    fn test_part1() {
        assert_eq!(process(DATA), 1651);
    }
}

For part 2, I wasn’t really sure what optimization I needed to do, if any. I ended up spending the better part of a day getting some very subtle bugs worked out of this one, so after that I was happy enough that it did terminate, eventually. (Notably, after running for about twelve hours!) I think there’s likely some fairly low-hanging fruit if I prune some of the possible paths that won’t work out, but…I give up!

$ /usr/bin/time -v ./day16b
[...]
-1583300000
-1583200000
-1583100000
2615
	Command being timed: "./day16b"
	User time (seconds): 40460.91
	System time (seconds): 0.66
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 11:15:26
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 739120
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 221124
	Voluntary context switches: 1
	Involuntary context switches: 125204
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Oh, and notice that I had an uncaught integer overflow here— I pretty quickly tried more than 2**31 routes, so that resulted in the negative numbers seen above.

Part 2 source code

use std::collections::{HashMap, HashSet, VecDeque};
use std::fs;

#[derive(Debug)]
struct Valve {
    connections: Vec<String>,
    flow_rate: usize,
}

fn find_shortest_path<'a>(
    valves: &'a HashMap<&'a str, Valve>,
    origin: &'a str,
    destination: &str,
) -> Vec<&'a str> {
    // breadth-first search
    let mut nodes_to_check = VecDeque::new();
    nodes_to_check.push_back((origin, vec![]));
    while nodes_to_check.len() > 0 {
        let (current_node, path_to_current_node) = nodes_to_check.pop_front().unwrap();
        for connection in valves[&current_node].connections.iter() {
            let mut path_to_connection = path_to_current_node.clone();
            path_to_connection.push(connection.as_str());
            if connection == destination {
                return path_to_connection;
            }
            nodes_to_check.push_back((&connection, path_to_connection));
        }
    }
    panic!(
        "Could not find a route between {} and {}",
        origin, destination
    );
}

// Note that this only finds paths between nodes with non-zero flow rate, plus the starting node
fn find_shortest_paths<'a>(
    valves: &'a HashMap<&'a str, Valve>,
) -> HashMap<(&'a str, &'a str), Vec<&str>> {
    let mut shortest_paths = HashMap::new();
    for (origin, _valve) in valves
        .iter()
        .filter(|(n, v)| v.flow_rate > 0 || **n == "AA")
    {
        for (destination, _valve) in valves.iter().filter(|(_, v)| v.flow_rate > 0) {
            if origin == destination {
                continue;
            }
            shortest_paths.insert(
                (*origin, *destination),
                find_shortest_path(valves, origin, destination),
            );
        }
    }
    shortest_paths
}

fn parse(data: &str) -> HashMap<&str, Valve> {
    let mut valves = HashMap::new();
    for line in data.split("\n") {
        let valve_name = &line[6..8];
        let (flow_rate, mut rest) = line[23..].split_once(';').unwrap();
        let flow_rate = flow_rate.parse().unwrap();
        // Curse you single path with single tunnel for breaking my parsing
        if rest.chars().nth(7).unwrap() == 's' {
            rest = &rest[24..]; // " tunnels lead to valves "
        } else {
            rest = &rest[23..]; // " tunnel leads to valve "
        }
        let connections = rest.split(", ").map(|s| s.to_owned()).collect();
        valves.insert(
            valve_name,
            Valve {
                connections: connections,
                flow_rate: flow_rate,
            },
        );
    }
    valves
}

fn process(data: &str) -> usize {
    let valves = parse(data);
    let starting_closed_flowing_valves: HashSet<&str> = valves
        .iter()
        .filter(|(_, v)| v.flow_rate > 0)
        .map(|(k, _)| *k)
        .collect();
    let shortest_paths = find_shortest_paths(&valves);

    let mut most_pressure_released = 0;

    // breadth-first search
    let mut nodes_to_check = VecDeque::new();
    nodes_to_check.push_back(("AA", "AA", starting_closed_flowing_valves, 0, 0, 0));
    let mut check_count = 0;
    while let Some((
        current_me_node,
        current_elephant_node,
        remaining_closed_valves,
        pressure_released,
        me_minutes_elapsed,
        elephant_minutes_elapsed,
    )) = nodes_to_check.pop_back()
    // I originally tried `pop_front`, but memory use explodes.
    {
        for target_me_node in remaining_closed_valves.iter() {
            for target_elephant_node in remaining_closed_valves.iter() {
                if target_me_node == target_elephant_node {
                    continue;
                }

                // can we reach them both before 30 minutes is up?
                let me_opened_at = me_minutes_elapsed
                    + shortest_paths[&(current_me_node, *target_me_node)].len()
                    + 1;
                let elephant_opened_at = elephant_minutes_elapsed
                    + shortest_paths[&(current_elephant_node, *target_elephant_node)].len()
                    + 1;
                if me_opened_at >= 26 && elephant_opened_at >= 26 {
                    continue;
                }

                let mut new_pressure_released = pressure_released;
                let mut new_remaining_closed_valves = remaining_closed_valves.clone();
                let me_final_node;
                let elephant_final_node;
                if me_opened_at < 26 {
                    new_remaining_closed_valves.remove(target_me_node);
                    me_final_node = target_me_node;
                    new_pressure_released += (26 - me_opened_at) * valves[target_me_node].flow_rate;
                } else {
                    me_final_node = &current_me_node;
                }

                if elephant_opened_at < 26 {
                    new_remaining_closed_valves.remove(target_elephant_node);
                    elephant_final_node = target_elephant_node;
                    new_pressure_released +=
                        (26 - elephant_opened_at) * valves[target_elephant_node].flow_rate;
                } else {
                    elephant_final_node = &current_elephant_node;
                }

                nodes_to_check.push_back((
                    me_final_node,
                    elephant_final_node,
                    new_remaining_closed_valves,
                    new_pressure_released,
                    me_opened_at,
                    elephant_opened_at,
                ));

                if new_pressure_released > most_pressure_released {
                    most_pressure_released = new_pressure_released;
                }
            }
        }
        check_count += 1;
        if check_count % 100_000 == 0 {
            println!("{}", check_count);
        }
    }
    most_pressure_released
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data));
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "Valve AA has flow rate=0; tunnels lead to valves DD, II, BB
Valve BB has flow rate=13; tunnels lead to valves CC, AA
Valve CC has flow rate=2; tunnels lead to valves DD, BB
Valve DD has flow rate=20; tunnels lead to valves CC, AA, EE
Valve EE has flow rate=3; tunnels lead to valves FF, DD
Valve FF has flow rate=0; tunnels lead to valves EE, GG
Valve GG has flow rate=0; tunnels lead to valves FF, HH
Valve HH has flow rate=22; tunnel leads to valve GG
Valve II has flow rate=0; tunnels lead to valves AA, JJ
Valve JJ has flow rate=21; tunnel leads to valve II";

    #[test]
    fn test_part2() {
        assert_eq!(process(DATA), 1707);
    }
}

Day 17

Part 1 was pretty straightforward. Part 2 would, as is typical, have needed until approximately the heat death of the universe to terminate, so we had to figure out a smarter approach. In this case, I just make note of every state, and if we see it twice, we know that we loop. Then, note that cycle, and keep applying that cycle until we’re near the end, and finish as normal.

One thing I struggled with for a while today was data types. Rust’s errors are excellent for the simplest cases, but can quickly spiral out of control for more complex errors. In my case, I had a lot of type errors, where I was trying to add, say, a usize to a u64 and put it in another u64. When stated like that, the fix is pretty straightforward: Cast the usize to u64 before doing the addition. However, when that type is buried in the middle of a program, and most of the types are inferred anyway, it’s not immediately obvious which variables’ types I need to change to make things work out the way I want.

For a more concrete example, imagine that height returned a u64 instead of a usize¹:

diff --git a/2022/day17/day17.rs b/2022/day17/day17.rs
index bb15c0d..0380e29 100644
--- a/2022/day17/day17.rs
+++ b/2022/day17/day17.rs
@@ -1,11 +1,11 @@
 use std::collections::HashMap;
 use std::fs;
 
-fn height(chamber: &Vec<[bool; 7]>) -> usize {
+fn height(chamber: &Vec<[bool; 7]>) -> u64 {
     for (i, row) in chamber.iter().enumerate().rev() {
         if row.iter().any(|x| *x) {
             // Is there a better function here?
-            return i + 1;
+            return i as u64 + 1;
         }
     }

The error here isn’t too terrible, but it’s not immediately obvious why a u64 was expected. (In this case it’s only a couple lines away, but during this problem I solved some cases where there was an extensive chain of “foo is u64 because it’s added to bar which is u64 because it’s cloned from spam which is u64 because that’s what the frobulate function returns”-esque shenanigans.)

Oh, and every time something went even slightly wrong, the compiler decided to start calling the offsets usizes instead of isizes. I can’t figure out what was going on here, since solving other issues made it go away, but every time any unrelated error popped up, this one would rear its ugly head too. I’m not sure what’s up with that! 🤷‍♂️

error[E0308]: mismatched types
   --> day17.rs:100:37
    |
100 |                 for _ in 0..h + 7 - chamber.len() {
    |                                     ^^^^^^^^^^^^^ expected `u64`, found `usize`

error[E0277]: cannot subtract `usize` from `u64`
   --> day17.rs:100:35
    |
100 |                 for _ in 0..h + 7 - chamber.len() {
    |                                   ^ no implementation for `u64 - usize`
    |
    = help: the trait `Sub<usize>` is not implemented for `u64`
    = help: the following other types implement trait `Sub<Rhs>`:
              <&'a f32 as Sub<f32>>
              <&'a f64 as Sub<f64>>
              <&'a i128 as Sub<i128>>
              <&'a i16 as Sub<i16>>
              <&'a i32 as Sub<i32>>
              <&'a i64 as Sub<i64>>
              <&'a i8 as Sub<i8>>
              <&'a isize as Sub<isize>>
            and 48 others

error[E0277]: the trait bound `usize: Neg` is not satisfied
  --> day17.rs:59:40
   |
59 |                 JetDirection::Left => (-1, 0),
   |                                        ^^ the trait `Neg` is not implemented for `usize`
   |
   = help: the following other types implement trait `Neg`:
             &f32
             &f64
             &i128
             &i16
             &i32
             &i64
             &i8
             &isize
           and 8 others

error: aborting due to 3 previous errors

Some errors have detailed explanations: E0277, E0308.
For more information about an error, try `rustc --explain E0277`.

Source code

use std::collections::HashMap;
use std::fs;

fn height(chamber: &Vec<[bool; 7]>) -> usize {
    for (i, row) in chamber.iter().enumerate().rev() {
        if row.iter().any(|x| *x) {
            // Is there a better function here?
            return i + 1;
        }
    }

    0
}

#[derive(Debug)]
enum JetDirection {
    Left,
    Right,
}

fn process(data: &str, rocks_to_fall: u64) -> u64 {
    // I'd have loved to put rock_shapes as a global constant, but
    // unfortunately since rocks are different sizes, rocks have to be
    // `Vec`s not arrays, and that can't be done statically,
    // apparently. Maybe there's a good workaround for this, but I
    // haven't found it. So, we just put it here instead!
    let rock_shapes = vec![
        vec![(0, 0), (1, 0), (2, 0), (3, 0)],
        vec![(1, 0), (0, 1), (1, 1), (2, 1), (1, 2)],
        vec![(0, 0), (1, 0), (2, 0), (2, 1), (2, 2)],
        vec![(0, 0), (0, 1), (0, 2), (0, 3)],
        vec![(0, 0), (1, 0), (0, 1), (1, 1)],
    ];

    let jets: Vec<JetDirection> = data
        .chars()
        .map(|c| match c {
            '<' => JetDirection::Left,
            '>' => JetDirection::Right,
            _ => panic!("Unknown characters"),
        })
        .collect();

    let mut falling_points = Vec::new();
    let mut chamber = vec![[false; 7]; 4];
    let mut current_falling_rock = 0;
    let mut is_moving_sideways = true;
    for point in &rock_shapes[0] {
        falling_points.push((point.0 + 2, point.1 + 3));
    }
    let mut seen_states: HashMap<([[bool; 7]; 10], usize), (u64, u64)> = HashMap::new();
    let mut jet_index = 0;
    let mut height_offset = 0;
    while current_falling_rock < rocks_to_fall {
        let target_offset;
        if is_moving_sideways {
            let jet_direction = &jets[jet_index];
            target_offset = match jet_direction {
                JetDirection::Left => (-1, 0),
                JetDirection::Right => (1, 0),
            };
            jet_index = (jet_index + 1) % jets.len();
        } else {
            target_offset = (0, -1);
        }
        let can_move = falling_points.iter().all(|(x, y)| {
            let new_x = x + target_offset.0;
            let new_y = y + target_offset.1;
            (0..7).contains(&new_x) && new_y >= 0 && !chamber[new_y as usize][new_x as usize]
        });
        if can_move {
            for point in falling_points.iter_mut() {
                *point = (point.0 + target_offset.0, point.1 + target_offset.1);
            }
        } else if !is_moving_sideways {
            if height_offset == 0 && height(&chamber) > 10 {
                let mut foo = [[false; 7]; 10];
                foo.clone_from_slice(&chamber[chamber.len() - 10..chamber.len()]);
                let state = (foo, jet_index);
                if seen_states.contains_key(&state) {
                    let remaining_falling_rocks = rocks_to_fall - current_falling_rock;
                    let rocks_fall_per_cycle = current_falling_rock - seen_states[&state].0;
                    let height_added_per_cycle = height(&chamber) as u64 - seen_states[&state].1;
                    let cycles = remaining_falling_rocks / rocks_fall_per_cycle;
                    height_offset = cycles * height_added_per_cycle;
                    current_falling_rock += rocks_fall_per_cycle * cycles;
                } else {
                    seen_states.insert(state, (current_falling_rock, height(&chamber) as u64));
                }
            }
            // Convert all falling points into stuck points
            while let Some((x, y)) = falling_points.pop() {
                chamber[y as usize][x as usize] = true;
            }

            current_falling_rock += 1;

            for point in &rock_shapes[current_falling_rock as usize % 5] {
                let h = height(&chamber);
                for _ in 0..h + 7 - chamber.len() {
                    chamber.push([false; 7]);
                }
                falling_points.push((point.0 + 2, point.1 + h as i64 + 3));
            }
        }
        is_moving_sideways = !is_moving_sideways;
    }

    height(&chamber) as u64 + height_offset
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data, 1_000_000_000_000)); // 2022 for part 1
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = ">>><<><>><<<>><>>><<<>>><<<><<<>><>><<>>";

    #[test]
    fn test_part1() {
        assert_eq!(process(DATA, 2022), 3068);
    }

    #[test]
    fn test_part2() {
        assert_eq!(process(DATA, 1_000_000_000_000), 1514285714288);
    }
}

Soundtrack

I did use u64s over usizes in a handful of places to try to write semi-portable code; many of the values in the second part were too large to fit in a 32-bit integer, so 32-bit platforms’ 32-bit usize would overflow, where a u64 is safe on all platforms. Turned out to be more hassle than it was worth, though! ↩︎

Advent of Code 2022: Days 12–15

Thu, 29 Dec 2022 00:00:00 +0000

Well, Advent is long over, but here I am! I was traveling and visiting family, which was delightful but really set back my AoC progress. But I’m hoping I’ll be caught up by the new year! New this time: eww, the Emacs Web Wowser.

Day 12

So this one basically requires a search for the optimal path. The given space is huge, so a brute-force search isn’t feasible. I attempted writing A* from memory (a bit of a challenge, since I’ve implemented it once in a college course, and once for a previous Advent of Code!). It went okay, in that I got something, but pretty poorly in my memory of A*! After some research, it turns out what I wrote was a slight modification of Dijkstra’s algorithm. Ah well, problem solved, anyway!

Today’s major learning was about eww, the Emacs Web Wowser, a basic web browser that runs within Emacs. It doesn’t have much for fancy features, but it’s great for reading AoC problem descriptions. Unfortunately, without functioning Javascript I can’t log into GitHub, so I’m still submitting my results via Firefox. However, displaying text is great, especially with eww-readable (R).

Part 1 source code

use std::collections::VecDeque;
use std::fs;

fn parse(data: &str) -> (Vec<Vec<i8>>, (usize, usize), (usize, usize)) {
    let mut start = None;
    let mut end = None;
    let mut elevations = Vec::new();
    for (i, line) in data.split("\n").enumerate() {
        let mut row = Vec::new();
        for (j, ch) in line.chars().enumerate() {
            if ch == 'S' {
                start = Some((i, j));
                row.push(1);
            } else if ch == 'E' {
                end = Some((i, j));
                row.push(26);
            } else {
                row.push(ch as i8 - 'a' as i8);
            }
        }
        elevations.push(row);
    }
    (elevations, start.unwrap(), end.unwrap())
}

fn process(data: &str) -> u32 {
    let (elevations, start, end) = parse(data);
    let map_height = elevations.len();
    let map_width = elevations[0].len();
    let mut costs = vec![vec![u32::MAX; map_width]; map_height];
    costs[start.0][start.1] = 0;

    let mut boundaries = VecDeque::new();
    boundaries.push_back(start);

    while !boundaries.is_empty() {
        let (x, y) = boundaries.pop_front().unwrap();
        let boundary_cost = costs[x][y];

        if x > 0 && elevations[x - 1][y] - elevations[x][y] >= 1 {
            let above_boundary_cost = &mut costs[x - 1][y];
            if *above_boundary_cost > boundary_cost + 1 {
                *above_boundary_cost = boundary_cost + 1;
                boundaries.push_back((x - 1, y));
            }
        }
        if x < map_height - 1 && elevations[x + 1][y] - elevations[x][y] <= 1 {
            let below_boundary_cost = &mut costs[x + 1][y];
            if *below_boundary_cost > boundary_cost + 1 {
                *below_boundary_cost = boundary_cost + 1;
                boundaries.push_back((x + 1, y));
            }
        }
        if y > 0 && elevations[x][y - 1] - elevations[x][y] <= 1 {
            let left_boundary_cost = &mut costs[x][y - 1];
            if *left_boundary_cost > boundary_cost + 1 {
                *left_boundary_cost = boundary_cost + 1;
                boundaries.push_back((x, y - 1));
            }
        }
        if y < map_width - 1 && elevations[x][y + 1] - elevations[x][y] <= 1 {
            let right_boundary_cost = &mut costs[x][y + 1];
            if *right_boundary_cost > boundary_cost + 1 {
                *right_boundary_cost = boundary_cost + 1;
                boundaries.push_back((x, y + 1));
            }
        }
    }

    costs[end.0][end.1]
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();

    println!("{}", process(data));
}

#[cfg(test)]
mod test {
    use super::*;

    #[test]
    fn test_small_input() {
        let data = "Sabqponm
abcryxxl
accszExk
acctuvwj
abdefghi";
        assert_eq!(process(data), 31);
    }
}

Part 2 source code

use std::collections::VecDeque;
use std::fs;

fn parse(data: &str) -> (Vec<Vec<i8>>, (usize, usize), (usize, usize)) {
    let mut start = None;
    let mut end = None;
    let mut elevations = Vec::new();
    for (i, line) in data.split("\n").enumerate() {
        let mut row = Vec::new();
        for (j, ch) in line.chars().enumerate() {
            if ch == 'S' {
                start = Some((i, j));
                row.push(1);
            } else if ch == 'E' {
                end = Some((i, j));
                row.push(26);
            } else {
                row.push(ch as i8 - 'a' as i8);
            }
        }
        elevations.push(row);
    }
    (elevations, start.unwrap(), end.unwrap())
}

fn process(data: &str) -> u32 {
    let (elevations, _, end) = parse(data);
    let map_height = elevations.len();
    let map_width = elevations[0].len();
    let mut costs = vec![vec![u32::MAX; map_width]; map_height];
    costs[end.0][end.1] = 0;

    let mut boundaries = VecDeque::new();
    boundaries.push_back(end);

    while !boundaries.is_empty() {
        let (x, y) = boundaries.pop_front().unwrap();
        let boundary_cost = costs[x][y];

        if x > 0 && elevations[x][y] - elevations[x - 1][y] <= 1 {
            let above_boundary_cost = &mut costs[x - 1][y];
            if *above_boundary_cost > boundary_cost + 1 {
                *above_boundary_cost = boundary_cost + 1;
                boundaries.push_back((x - 1, y));
            }
        }
        if x < map_height - 1 && elevations[x][y] - elevations[x + 1][y] <= 1 {
            let below_boundary_cost = &mut costs[x + 1][y];
            if *below_boundary_cost > boundary_cost + 1 {
                *below_boundary_cost = boundary_cost + 1;
                boundaries.push_back((x + 1, y));
            }
        }
        if y > 0 && elevations[x][y] - elevations[x][y - 1] <= 1 {
            let left_boundary_cost = &mut costs[x][y - 1];
            if *left_boundary_cost > boundary_cost + 1 {
                *left_boundary_cost = boundary_cost + 1;
                boundaries.push_back((x, y - 1));
            }
        }
        if y < map_width - 1 && elevations[x][y] - elevations[x][y + 1] <= 1 {
            let right_boundary_cost = &mut costs[x][y + 1];
            if *right_boundary_cost > boundary_cost + 1 {
                *right_boundary_cost = boundary_cost + 1;
                boundaries.push_back((x, y + 1));
            }
        }
    }

    let mut cheapest_coord_cost = u32::MAX;
    for (i, row) in elevations.iter().enumerate() {
        for (j, elevation) in row.iter().enumerate() {
            if *elevation == 0 && costs[i][j] < cheapest_coord_cost {
                cheapest_coord_cost = costs[i][j];
            }
        }
    }
    cheapest_coord_cost
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();

    println!("{}", process(data));
}

#[cfg(test)]
mod test {
    use super::*;

    #[test]
    fn test_small_input() {
        let data = "Sabqponm
abcryxxl
accszExk
acctuvwj
abdefghi";
        assert_eq!(process(data), 29);
    }
}

Day 13

Today’s puzzle took quite a bit of thought. Parsing the Lists required quite a bit of passing strings around, but I’m getting reasonably comfortable with the borrow checker, and this went okay. It’s certainly not pretty, but it does appear to get the job done! Most of the time/code spent was parsing the packets; once that was done, everything else was fairly straightforward.

One of the most unintuitive things I’ve run into with Rust this month is that indexing into strings sort-of works, but only when taking slices.

I found myself doing quite a bit of this type of thing:

let mut data = "foo bar";
data = &data[4..]; // data = "bar"

This works fine, taking the string starting at the fourth character. But what is the fourth character? Well, I found myself trying this pretty frequently, which would work fine in something like Python:

let data = "foo bar";
println!("{}", data[4]); // won't compile!

This doesn’t work, and for decent reason. In my particular case, I’m dealing with ASCII, and I know that I just want the fourth byte, so the naïve implementation of this that just returns the fourth byte would work for me. However, once you start dealing with multi-byte characters, this whole thing falls apart. Instead, I have to do the Right Thing, and account for the fact that multi-byte characters need to be planned for:

let data = "foo bar";
println!("{}", data.chars().nth(4).unwrap()); // "b"

This issue is definitely well-documented, but I kept tripping over it. To me, the major problem is the inconsistency. Why can I do this when I’m taking a slice, but not when I’m indexing into a particular character? Either I should be aware of the risks and always be able to index to a byte position (like I can with a slice), or I should be protected from the fact that this is almost certainly a thing that will come back to bite (byte? heheheh) me later, and so I shouldn’t be allowed to do it, like in the nth char case.

Anyway, rant aside, here’s today’s code:

Source code

use std::cmp::{min, Ordering};
use std::fs;

#[derive(Debug, PartialEq, Eq, Clone)]
enum List {
    Integer(u8),
    List(Vec<List>),
}

impl Ord for List {
    fn cmp(&self, other: &Self) -> Ordering {
        match (self, other) {
            // If both values are integers, the lower integer should
            // come first.
            (List::Integer(s), List::Integer(o)) => s.cmp(o),
            // If both values are lists, compare the first value of
            // each list, then the second value, and so on. If the
            // left list runs out of items first, the inputs are in
            // the right order.
            (List::List(s), List::List(o)) => {
                for i in 0..min(s.len(), o.len()) {
                    if s[i] != o[i] {
                        return s[i].cmp(&o[i]);
                    }
                }
                s.len().cmp(&o.len())
            }
            // If exactly one value is an integer, convert the integer
            // to a list which contains that integer as its only
            // value, then retry the comparison.
            (List::Integer(s), List::List(o)) => {
                List::List(vec![List::Integer(*s)]).cmp(&List::List(o.to_vec()))
            }
            (List::List(s), List::Integer(o)) => {
                List::List(s.to_vec()).cmp(&List::List(vec![List::Integer(*o)]))
            }
        }
    }
}

impl PartialOrd for List {
    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
        Some(self.cmp(other))
    }
}

fn parse(data: &str) -> List {
    let mut data = &data[1..data.len() - 1]; // Strip square brackets
    let mut packet = Vec::new();
    while data.len() > 0 {
        match data.chars().next().unwrap() {
            '[' => {
                // next element is a list
                let mut list_str = String::new();
                let mut depth = 0;
                loop {
                    list_str.push(data.chars().next().unwrap());
                    match data.chars().next().unwrap() {
                        '[' => depth += 1,
                        ']' => depth -= 1,
                        _ => (),
                    };
                    data = &data[1..];
                    if depth == 0 {
                        break;
                    }
                }
                packet.push(parse(&list_str));
            }
            ',' => data = &data[1..],
            _ => {
                // next element is an integer
                let mut num = String::new();
                while data.len() > 0 {
                    match data.chars().next().unwrap() {
                        ',' => break,
                        c => num.push(c), // TODO
                    };
                    data = &data[1..];
                }
                packet.push(List::Integer(num.parse().unwrap()));
            }
        }
    }
    List::List(packet)
}

fn process_part1(data: &str) -> usize {
    let mut sum = 0;
    for (index, packet_pair) in data.split("\n\n").enumerate() {
        let packets: Vec<&str> = packet_pair.split("\n").collect();
        let zeroth = parse(packets[0]);
        let first = parse(packets[1]);
        match zeroth.cmp(&first) {
            Ordering::Less => sum += index + 1,
            Ordering::Greater => (),
            Ordering::Equal => panic!("{:?} and {:?} are equal", zeroth, first),
        }
    }
    sum
}

fn process_part2(data: &str) -> usize {
    let mut packets: Vec<List> = data
        .split("\n")
        .filter(|p| p != &"")
        .map(|p| parse(p))
        .collect();
    let zeroth_divider = parse("[[2]]");
    let first_divider = parse("[[6]]");
    packets.push(zeroth_divider.clone());
    packets.push(first_divider.clone());
    packets.sort();
    let mut product = 1;
    for (index, packet) in packets.iter().enumerate() {
        if *packet == zeroth_divider || *packet == first_divider {
            product *= index + 1;
        }
    }
    product
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process_part1(data));
    println!("{}", process_part2(data));
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "[1,1,3,1,1]
[1,1,5,1,1]

[[1],[2,3,4]]
[[1],4]

[9]
[[8,7,6]]

[[4,4],4,4]
[[4,4],4,4,4]

[7,7,7,7]
[7,7,7]

[]
[3]

[[[]]]
[[]]

[1,[2,[3,[4,[5,6,7]]]],8,9]
[1,[2,[3,[4,[5,6,0]]]],8,9]";

    #[test]
    fn test_flat() {
        let data = "[1,1,3,1,1]\n[1,1,5,1,1]";
        assert_eq!(process_part1(data), 1);
    }

    #[test]
    fn test_part1() {
        assert_eq!(process_part1(DATA), 13);
    }

    #[test]
    fn test_part2() {
        assert_eq!(process_part2(DATA), 140);
    }
}

Day 14

Part 1 source code

use std::cmp::{max, min};
use std::fs;

fn format_map(map: &Vec<Vec<char>>, offset: usize) -> String {
    let mut ret = String::new();
    for row in map {
        ret.push_str(&row.iter().skip(offset).collect::<String>());
        ret.push('\n');
    }
    ret
}

fn parse_map(data: &str) -> Vec<Vec<char>> {
    // find max dimensions
    let mut max_x = 0;
    let mut max_y = 0;
    for line in data.split("\n") {
        for point in line.split(" -> ") {
            let mut coord_pair = point.split(",");
            let x = coord_pair.next().unwrap().parse().unwrap();
            let y = coord_pair.next().unwrap().parse().unwrap();
            if x > max_x {
                max_x = x;
            }
            if y > max_y {
                max_y = y;
            }
        }
    }

    let mut map = vec![vec!['.'; max_x + 1]; max_y + 1];

    for line in data.split("\n") {
        let points: Vec<(usize, usize)> = line
            .split(" -> ")
            .map(|c| {
                let mut coords = c.split(",").map(|p| p.parse().unwrap());
                (coords.next().unwrap(), coords.next().unwrap())
            })
            .collect();
        for i in 0..points.len() - 1 {
            let first = points[i];
            let second = points[i + 1];
            if first.0 != second.0 {
                // vertical wall
                assert_eq!(first.1, second.1); // No diagonal walls!
                let start = min(first.0, second.0);
                let finish = max(first.0, second.0);
                for j in start..=finish {
                    map[first.1][j] = '#';
                }
            } else {
                // horizontal wall
                assert_eq!(first.0, second.0);
                let start = min(first.1, second.1);
                let finish = max(first.1, second.1);
                for j in start..=finish {
                    map[j][first.0] = '#';
                }
            }
        }
    }

    map
}

// returns the active sand (if it hasn't fallen off the map)
fn tick(map: &mut Vec<Vec<char>>, active_sand: &(usize, usize)) -> Option<(usize, usize)> {
    if active_sand.1 == map.len() - 1 {
        return None; // We fell off the bottom of the map!
    }
    for x_offset in vec![0, -1, 1] {
        // first try down, then down-left, then down-right
        if map[active_sand.1 + 1][active_sand.0.checked_add_signed(x_offset).unwrap()] == '.' {
            return Some((
                active_sand.0.checked_add_signed(x_offset).unwrap(),
                active_sand.1 + 1,
            ));
        }
    }
    // If we got here, we can't move, so we stay put:
    Some(*active_sand)
}

fn process(map: &mut Vec<Vec<char>>, sand_source: (usize, usize)) -> usize {
    let mut sand_count = 0;
    let mut active_sand = sand_source;
    loop {
        match tick(map, &active_sand) {
            Some(new_active_sand) => {
                if new_active_sand == active_sand {
                    // This sand has come to rest; let's add more sand
                    map[active_sand.1][active_sand.0] = 'o';
                    active_sand = sand_source;
                    sand_count += 1;
                } else {
                    active_sand = new_active_sand;
                }
            }
            None => break, // Sand is falling off the map, so we're done
        }
    }
    sand_count
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(&mut parse_map(data), (500, 0)));
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "498,4 -> 498,6 -> 496,6
503,4 -> 502,4 -> 502,9 -> 494,9";

    #[test]
    fn test_parsing() {
        let goal = "......+...
..........
..........
..........
....#...##
....#...#.
..###...#.
........#.
........#.
#########.";
        let mut map = parse_map(DATA);
        map[0][500] = '+';
        assert_eq!(format_map(&map, 494).trim(), goal);
    }

    #[test]
    fn test_part1() {
        assert_eq!(process(&mut parse_map(DATA), (500, 0)), 24);
    }
}

Part 2 was the first challenge where the solution took a non-trivial amount of time to finish. (hyperfine puts it at about 1.4 seconds; the otherwise worst to date had been in the low tens of milliseconds range.) I briefly considered doing some optimization to make this a bit quicker, since this shouldn’t be quite that slow, but I realized I haven’t been compiling with any compiler optimizations on! Turning on the standard optimization level (-O is equivalent to -C opt-level=2, and didn’t fare any worse than -C opt-level=3) brought the time down to a level I found more than acceptable.

$ rustc day14b.rs
$ rustc -O -o day14b-optimized day14b.rs
$ hyperfine --style=basic --shell=none ./day14b ./day14b-optimized
Benchmark 1: ./day14b
  Time (mean ± σ):      1.445 s ±  0.035 s    [User: 1.437 s, System: 0.001 s]
  Range (min … max):    1.405 s …  1.504 s    10 runs

Benchmark 2: ./day14b-optimized
  Time (mean ± σ):      65.5 ms ±   8.0 ms    [User: 64.4 ms, System: 0.6 ms]
  Range (min … max):    61.0 ms …  92.0 ms    46 runs

Summary
  './day14b-optimized' ran
   22.07 ± 2.76 times faster than './day14b'

Part 2 source code

use std::cmp::{max, min};
use std::fs;

fn parse_map(data: &str) -> Vec<Vec<char>> {
    // find max dimensions
    let mut max_x = 0;
    let mut max_y = 0;
    for line in data.split("\n") {
        for point in line.split(" -> ") {
            let mut coord_pair = point.split(",");
            let x = coord_pair.next().unwrap().parse().unwrap();
            let y = coord_pair.next().unwrap().parse().unwrap();
            if x > max_x {
                max_x = x;
            }
            if y > max_y {
                max_y = y;
            }
        }
    }

    let mut map = vec![vec!['.'; max_x + max_y + 1]; max_y + 2];
    map.push(vec!['#'; max_x + max_y + 1]); // Add the infinite floor

    for line in data.split("\n") {
        let points: Vec<(usize, usize)> = line
            .split(" -> ")
            .map(|c| {
                let mut coords = c.split(",").map(|p| p.parse().unwrap());
                (coords.next().unwrap(), coords.next().unwrap())
            })
            .collect();
        for i in 0..points.len() - 1 {
            let first = points[i];
            let second = points[i + 1];
            if first.0 != second.0 {
                // vertical wall
                assert_eq!(first.1, second.1); // No diagonal walls!
                let start = min(first.0, second.0);
                let finish = max(first.0, second.0);
                for j in start..=finish {
                    map[first.1][j] = '#';
                }
            } else {
                // horizontal wall
                assert_eq!(first.0, second.0);
                let start = min(first.1, second.1);
                let finish = max(first.1, second.1);
                for j in start..=finish {
                    map[j][first.0] = '#';
                }
            }
        }
    }

    map
}

// returns the active sand (if it hasn't fallen off the map)
fn tick(map: &mut Vec<Vec<char>>, active_sand: &(usize, usize)) -> Option<(usize, usize)> {
    if active_sand.1 == map.len() - 1 {
        return None; // We fell off the bottom of the map!
    }
    for x_offset in vec![0, -1, 1] {
        // first try down, then down-left, then down-right
        if map[active_sand.1 + 1][active_sand.0.checked_add_signed(x_offset).unwrap()] == '.' {
            return Some((
                active_sand.0.checked_add_signed(x_offset).unwrap(),
                active_sand.1 + 1,
            ));
        }
    }
    // If we got here, we can't move, so we stay put:
    Some(*active_sand)
}

fn process(map: &mut Vec<Vec<char>>, sand_source: (usize, usize)) -> usize {
    let mut sand_count = 0;
    let mut active_sand = sand_source;
    loop {
        match tick(map, &active_sand) {
            Some(new_active_sand) => {
                if new_active_sand == active_sand {
                    // This sand has come to rest; let's add more sand
                    if map[active_sand.1][active_sand.0] != '.' {
                        break; // We're full
                    }
                    map[active_sand.1][active_sand.0] = 'o';
                    active_sand = sand_source;
                    sand_count += 1;
                } else {
                    active_sand = new_active_sand;
                }
            }
            None => break, // Sand is falling off the map, so we're done
        }
    }
    sand_count
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(&mut parse_map(data), (500, 0)));
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "498,4 -> 498,6 -> 496,6
503,4 -> 502,4 -> 502,9 -> 494,9";

    #[test]
    fn test_part2() {
        assert_eq!(process(&mut parse_map(DATA), (500, 0)), 93);
    }
}

Day 15

For part 1, I relied pretty heavily on the fact that we were only checking a single given line, and kept track of what regions on that line were seen by the beacon. The only slightly tricky thing I ran into was remembering that the beacon on the line that I’m counting non-beacon-spots on doesn’t count as a non-beacon spot! But thankfully, the automated test case reminded me of that, so that was fixed.

I’ve been trying to stick to the standard library for these challenges, but the Rust standard library does have some notable omissions. In this case, I’d have been interested in using the regex crate to parse out the input. However, the input is well-formatted enough that I can just count and index by character offsets. It’s a bit clunky, but works out fine, and is probably makes execution the tiniest bit quicker, too.

This is one thing that frustrates me a bit about the Rust ecosystem: There are some notable omissions from the standard library that it’s expected I pull in extra crates for, but that requires I jump over to using cargo rather than just a single .rs file; plus I have dependencies to vet. I’ve had blessed.rs recommended as a stopgap solution, with some widely-used crates and a bit of commentary listed for each use case.

Part 1 source code

use std::collections::HashSet;
use std::fs;

fn process(data: &str, target_row: i32) -> usize {
    let mut locations_without_beacon = HashSet::new();
    let mut beacons_on_target_row = HashSet::new();

    for sensor in data.split("\n") {
        let (sensor_x, rest) = &sensor[12..].split_once(',').unwrap();
        let sensor_x: i32 = sensor_x.parse().unwrap();
        let (sensor_y, rest) = &rest[3..].split_once(':').unwrap();
        let sensor_y: i32 = sensor_y.parse().unwrap();
        let (beacon_x, rest) = &rest[24..].split_once(',').unwrap();
        let beacon_x: i32 = beacon_x.parse().unwrap();
        let beacon_y: i32 = rest[3..].parse().unwrap();
        if beacon_y == target_row {
            beacons_on_target_row.insert(beacon_x);
        }

        let manhattan_distance: i32 = (beacon_x - sensor_x).abs() + (beacon_y - sensor_y).abs();

        let remaining_width = manhattan_distance - (sensor_y - target_row).abs();
        for c in (sensor_x - remaining_width)..=(sensor_x + remaining_width) {
            locations_without_beacon.insert(c);
        }
    }

    for beacon in beacons_on_target_row {
        locations_without_beacon.remove(&beacon);
    }

    locations_without_beacon.len()
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data, 2_000_000));
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "Sensor at x=2, y=18: closest beacon is at x=-2, y=15
Sensor at x=9, y=16: closest beacon is at x=10, y=16
Sensor at x=13, y=2: closest beacon is at x=15, y=3
Sensor at x=12, y=14: closest beacon is at x=10, y=16
Sensor at x=10, y=20: closest beacon is at x=10, y=16
Sensor at x=14, y=17: closest beacon is at x=10, y=16
Sensor at x=8, y=7: closest beacon is at x=2, y=10
Sensor at x=2, y=0: closest beacon is at x=2, y=10
Sensor at x=0, y=11: closest beacon is at x=2, y=10
Sensor at x=20, y=14: closest beacon is at x=25, y=17
Sensor at x=17, y=20: closest beacon is at x=21, y=22
Sensor at x=16, y=7: closest beacon is at x=15, y=3
Sensor at x=14, y=3: closest beacon is at x=15, y=3
Sensor at x=20, y=1: closest beacon is at x=15, y=3";

    #[test]
    fn test_part1() {
        assert_eq!(process(DATA, 10), 26);
    }
}

Part 2…went off the rails a bit! I was stumped for quite a while on how to make this finish in a reasonable amount of time; clearly the part 1 approach wasn’t going to work¹. So I ended up adding some optimization to check large chunks at a time, and discard large fractions of the search space at a time. Specifically, since the area we’re searching was four million units squared, I made sixteen million one-thousand-pixel-square regions, and eliminated the majority of those, before searching smaller areas.

I did check r/adventofcode after I finished, and saw two themes, one of which I’d considered and discarded, and another I hadn’t thought of at all (but definitely should have!).

The discarded possibility was that if there is a unique square that we can’t see, it must be along the perimeter of the area detected by a sensor. I could scan those perimeters and find the spot. However, after a bit of thought I wasn’t able to find a smart way to go about doing this, so I decided to go with the check-one-large-region-at-a-time approach.

The possibility that I hadn’t thought of, but is obvious in hindsight: recursion! Divide the area we want to check. Then divide it again! And again, until I have a single coordinate that isn’t detected by the sensors. This would have been faster, plus it would have meant I could have used the test cases.

Speaking of test cases, part two here was the first time I made it all the way to submitting an incorrect answer to the AoC website. This year was going so well, otherwise! Because of the scale difference of the test case and the actual solution, the test case didn’t exercise the region-checking code. And I made a pretty silly mistake:

- found_sensor = Some((x, y));
+ found_sensor = Some((region.0 + x, region.1 + y));

x and y are always in the range [0, 1000), and I forgot to add the region offset back in.

This part has to be run with the nightly Rust compiler at the moment, since I’m using #![feature(hash_drain_filter)], which hasn’t stabilized yet. It’s a nice feature, and I didn’t feel like messing around to figure out how to semi-efficiently avoid its use.

I probably spent close to eight hours on today’s problem, between figuring out a solution, writing the code, debugging the code, and doing a writeup. That may not bode well for finishing by the new year….

Oh, and while I was looking at other solution ideas on r/adventofcode, I found that there are great AoC memes, because of course there are!

Here’s the code; it took me about 40 seconds to run when compiled with -O.

Part 2 source code

// rustup run nightly rustc -O day15b.rs
#![feature(hash_drain_filter)]

use std::collections::HashSet;
use std::fs;

fn region_in_range(point: (i32, i32), manhattan_distance: i32, region: (i32, i32)) -> bool {
    let top_left_covered =
        (region.0 - point.0).abs() + (region.1 - point.1).abs() <= manhattan_distance;
    let top_right_covered =
        (region.0 + 1000 - point.0).abs() + (region.1 - point.1).abs() <= manhattan_distance;
    let bottom_left_covered =
        (region.0 - point.0).abs() + (region.1 + 1000 - point.1).abs() <= manhattan_distance;
    let bottom_right_covered =
        (region.0 + 1000 - point.0).abs() + (region.1 + 1000 - point.1).abs() <= manhattan_distance;
    return top_left_covered && top_right_covered && bottom_left_covered && bottom_right_covered;
}

fn process(data: &str) -> u64 {
    println!("Populating initial hash set...");
    let mut regions_without_beacon = HashSet::new();
    for x in 0..4000 {
        for y in 0..4000 {
            regions_without_beacon.insert((x * 1000, y * 1000));
        }
    }
    println!("hash set population complete!");

    let mut sensors = Vec::new();

    for sensor in data.split("\n") {
        let (sensor_x, rest) = &sensor[12..].split_once(',').unwrap();
        let sensor_x: i32 = sensor_x.parse().unwrap();
        let (sensor_y, rest) = &rest[3..].split_once(':').unwrap();
        let sensor_y: i32 = sensor_y.parse().unwrap();
        let (beacon_x, rest) = &rest[24..].split_once(',').unwrap();
        let beacon_x: i32 = beacon_x.parse().unwrap();
        let beacon_y: i32 = rest[3..].parse().unwrap();

        let manhattan_distance: i32 = (beacon_x - sensor_x).abs() + (beacon_y - sensor_y).abs();

        sensors.push(((sensor_x, sensor_y), manhattan_distance));

        let drained: HashSet<_> = regions_without_beacon
            .drain_filter(|r| region_in_range((sensor_x, sensor_y), manhattan_distance, *r))
            .collect();

        println!("Drained {}, {} remaining", drained.len(), regions_without_beacon.len());
    }

    let mut found_sensor = None;
    for (i, region) in regions_without_beacon.iter().enumerate() {
        if i % 100 == 0 {
            println!("Scanning regions {}-{}", i, i+99);
        }
        for x in 0..=1000 {
            'region: for y in 0..=1000 {
                for (sensor, distance) in sensors.iter() {
                    if (sensor.0 - region.0 - x).abs() + (sensor.1 - region.1 - y).abs()
                        <= *distance
                    {
                        continue 'region;
                    }
                }
                found_sensor = Some((region.0 + x, region.1 + y));
                println!("Found sensor at  {},{}", region.0 + x, region.1 + y);
            }
        }
    }
    let found_sensor = found_sensor.unwrap();
    found_sensor.0 as u64 * 4_000_000 + found_sensor.1 as u64
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data));
}

Soundtrack

Well, that’s apparently not true. If you’re absolutely insane (and I mean that as high praise), you can brute force it! https://reddit.com/r/adventofcode/comments/zmcn64/2022_day_15_solutions/j16tp3r/ ↩︎

Advent of Code 2022: Days 10 and 11

Tue, 13 Dec 2022 00:00:00 +0000

I’m getting caught back up, slowly!

Day 10

Day 10 was pretty quick. Both parts, but especially part 2, were sort of hacky, but that’s okay; this script only has to run once! In an ideal world, I might have made an iterator that generates per-cycle “bytecode” (this could be as simple as inserting an extra noop before each addx) so that each instruction takes exactly one cycle. But day 11 awaits, so move on we must.

Part 1 source code

use std::collections::HashMap;
use std::fs;

fn process(data: &str) -> i32 {
    let mut current_cycle = 1;
    let mut accumulator = 1;
    let mut value_at_cycle = HashMap::new();
    for line in data.split("\n") {
        let instruction: Vec<&str> = line.split(" ").collect();
        match instruction[0] {
            "noop" => {
                value_at_cycle.insert(current_cycle, accumulator);
                current_cycle += 1;
            }
            "addx" => {
                value_at_cycle.insert(current_cycle, accumulator);
                current_cycle += 1;
                value_at_cycle.insert(current_cycle, accumulator);
                current_cycle += 1;
                accumulator += instruction[1].parse::<i32>().unwrap();
            }
            _ => panic!("Unknown instruction {}", instruction[0]),
        }
    }
    let mut total_signal_strength = 0;
    for i in vec![20, 60, 100, 140, 180, 220] {
        total_signal_strength += i * value_at_cycle[&i];
    }
    total_signal_strength
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data));
}

#[cfg(test)]
mod test {
    use super::*;

    static DATA: &str = "addx 15
addx -11
addx 6
addx -3
addx 5
addx -1
addx -8
addx 13
addx 4
noop
addx -1
addx 5
addx -1
addx 5
addx -1
addx 5
addx -1
addx 5
addx -1
addx -35
addx 1
addx 24
addx -19
addx 1
addx 16
addx -11
noop
noop
addx 21
addx -15
noop
noop
addx -3
addx 9
addx 1
addx -3
addx 8
addx 1
addx 5
noop
noop
noop
noop
noop
addx -36
noop
addx 1
addx 7
noop
noop
noop
addx 2
addx 6
noop
noop
noop
noop
noop
addx 1
noop
noop
addx 7
addx 1
noop
addx -13
addx 13
addx 7
noop
addx 1
addx -33
noop
noop
noop
addx 2
noop
noop
noop
addx 8
noop
addx -1
addx 2
addx 1
noop
addx 17
addx -9
addx 1
addx 1
addx -3
addx 11
noop
noop
addx 1
noop
addx 1
noop
noop
addx -13
addx -19
addx 1
addx 3
addx 26
addx -30
addx 12
addx -1
addx 3
addx 1
noop
noop
noop
addx -9
addx 18
addx 1
addx 2
noop
noop
addx 9
noop
noop
noop
addx -1
addx 2
addx -37
addx 1
addx 3
noop
addx 15
addx -21
addx 22
addx -6
addx 1
noop
addx 2
addx 1
noop
addx -10
noop
noop
addx 20
addx 1
addx 2
addx 2
addx -6
addx -11
noop
noop
noop";

    #[test]
    fn test() {
        assert!(process(DATA) == 13140);
    }
}

Part 2 source code

use std::collections::HashMap;
use std::fs;

fn process(data: &str) -> i32 {
    let mut current_cycle = 1;
    let mut accumulator = 1;
    let mut value_at_cycle = HashMap::new();
    for line in data.split("\n") {
        let instruction: Vec<&str> = line.split(" ").collect();
        match instruction[0] {
            "noop" => {
                value_at_cycle.insert(current_cycle, accumulator);
                current_cycle += 1;
            }
            "addx" => {
                value_at_cycle.insert(current_cycle, accumulator);
                current_cycle += 1;
                value_at_cycle.insert(current_cycle, accumulator);
                current_cycle += 1;
                accumulator += instruction[1].parse::<i32>().unwrap();
            }
            _ => panic!("Unknown instruction {}", instruction[0]),
        }
    }
    let mut total_signal_strength = 0;
    for i in vec![20, 60, 100, 140, 180, 220] {
        total_signal_strength += i * value_at_cycle[&i];
    }
    total_signal_strength
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data));
}

#[cfg(test)]
mod test {
    use super::*;

    static DATA: &str = "addx 15
addx -11
addx 6
addx -3
addx 5
addx -1
addx -8
addx 13
addx 4
noop
addx -1
addx 5
addx -1
addx 5
addx -1
addx 5
addx -1
addx 5
addx -1
addx -35
addx 1
addx 24
addx -19
addx 1
addx 16
addx -11
noop
noop
addx 21
addx -15
noop
noop
addx -3
addx 9
addx 1
addx -3
addx 8
addx 1
addx 5
noop
noop
noop
noop
noop
addx -36
noop
addx 1
addx 7
noop
noop
noop
addx 2
addx 6
noop
noop
noop
noop
noop
addx 1
noop
noop
addx 7
addx 1
noop
addx -13
addx 13
addx 7
noop
addx 1
addx -33
noop
noop
noop
addx 2
noop
noop
noop
addx 8
noop
addx -1
addx 2
addx 1
noop
addx 17
addx -9
addx 1
addx 1
addx -3
addx 11
noop
noop
addx 1
noop
addx 1
noop
noop
addx -13
addx -19
addx 1
addx 3
addx 26
addx -30
addx 12
addx -1
addx 3
addx 1
noop
noop
noop
addx -9
addx 18
addx 1
addx 2
noop
noop
addx 9
noop
noop
noop
addx -1
addx 2
addx -37
addx 1
addx 3
noop
addx 15
addx -21
addx 22
addx -6
addx 1
noop
addx 2
addx 1
noop
addx -10
noop
noop
addx 20
addx 1
addx 2
addx 2
addx -6
addx -11
noop
noop
noop";

    #[test]
    fn test() {
        assert!(process(DATA) == 13140);
    }
}

Day 11

Today’s round 2 was a bit of a brainteaser. For round 1, I’d been doing what I imagine was intended: Make the number a sufficiently large type (I’m using usize, the pointer-sized uint, which is 64-bit on my platform), and rely on the division by three to keep things manageable. For a limited number of rounds and with repeated division by three, these numbers stay manageable; however, for this part of the puzzle, that’s not quite the case any more!

Since all of our tests are testing for divisibility, it’s modular arithmetic to the rescue! But suppose I have three divisibility conditions: divisible by 2, by 3, and by 5. If I use 2, or 3, or 5 as the modulus, that successfully shrinks my total, but at the expense of being completely wrong! So we can instead take the least common multiple of these; or, more simply, just multiply them all together! For my input, this results in a number less than a hundred thousand; very reasonable. This would fit fine in an u32, and certainly in a u64 with no trouble. (If this weren’t the case, repeated passes through the hands of the monkey that multiples the worry by itself would rapidly run me out of space; each squaring doubles the number of bits that my answer needs to use, so I have a maximum of 64 passes through that monkey’s hands before I’m out of space!)

I’ve been busy enough with the puzzles that I haven’t been taking much time to engross myself in new emacs features; instead, I’m focusing on refining my knowledge of the features I know, and retraining my muscle memory from Vim. That said, next up on the list is Version Control and/or Magit (or another of Emacs’ many git-related tools). I may also have to learn a bit about installing and managing packages, editing my emacs config file, and writing elisp. On to tomorrow!

Source code

use std::collections::VecDeque;
use std::fs;

#[derive(Eq, PartialEq, Debug)]
struct Monkey {
    starting_items: VecDeque<usize>,
    operation: Vec<String>,
    test_divisible_by_cond: usize,
    target_if_test: usize,
    target_if_not_test: usize,
    inspected_item_count: usize,
}

fn update_worry_level(operation: &Vec<String>, level: usize) -> usize {
    if operation[4] == "old" {
        // new = old * old
        level * level
    } else {
        match operation[3].as_str() {
            "*" => level * operation[4].parse::<usize>().unwrap(),
            "+" => level + operation[4].parse::<usize>().unwrap(),
            _ => panic!("Unknown operator {}", operation[3]),
        }
    }
}

fn parse(data: &str) -> Vec<Monkey> {
    let mut monkeys = Vec::new();
    for monkey_data in data.split("\n\n") {
        let lines: Vec<&str> = monkey_data.split("\n").collect();
        monkeys.push(Monkey {
            starting_items: lines[1][18..]
                .split(", ")
                .map(|i| i.parse().unwrap())
                .collect(),
            operation: lines[2][13..].split(" ").map(|s| s.to_string()).collect(),
            test_divisible_by_cond: lines[3][21..].parse().unwrap(),
            target_if_test: lines[4][29..].parse().unwrap(),
            target_if_not_test: lines[5][30..].parse().unwrap(),
            inspected_item_count: 0,
        });
    }
    monkeys
}

fn process(monkeys: &mut Vec<Monkey>, rounds: usize) -> usize {
    let test_product: usize = monkeys.iter().map(|m| m.test_divisible_by_cond).product();
    for _round in 0..rounds {
        for i in 0..monkeys.len() {
            while monkeys[i].starting_items.len() > 0 {
                monkeys[i].inspected_item_count += 1;
                let mut item = monkeys[i].starting_items.pop_front().unwrap();
                item = update_worry_level(&monkeys[i].operation, item);
                // item /= 3; // part 1 only
                item %= test_product; // part 2 only
                let target_monkey = if item % monkeys[i].test_divisible_by_cond == 0 {
                    monkeys[i].target_if_test
                } else {
                    monkeys[i].target_if_not_test
                };
                monkeys[target_monkey].starting_items.push_back(item);
            }
        }
    }
    let mut access_counts: Vec<usize> = monkeys.iter().map(|m| m.inspected_item_count).collect();
    access_counts.sort_unstable_by(|a, b| b.cmp(a));
    access_counts[0] * access_counts[1]
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    let mut monkeys = parse(data);
    println!("{}", process(&mut monkeys, 10000)); // 20 for part 1
}

#[cfg(test)]
mod test {
    use super::*;

    static DATA: &str = "Monkey 0:
  Starting items: 79, 98
  Operation: new = old * 19
  Test: divisible by 23
    If true: throw to monkey 2
    If false: throw to monkey 3

Monkey 1:
  Starting items: 54, 65, 75, 74
  Operation: new = old + 6
  Test: divisible by 19
    If true: throw to monkey 2
    If false: throw to monkey 0

Monkey 2:
  Starting items: 79, 60, 97
  Operation: new = old * old
  Test: divisible by 13
    If true: throw to monkey 1
    If false: throw to monkey 3

Monkey 3:
  Starting items: 74
  Operation: new = old + 3
  Test: divisible by 17
    If true: throw to monkey 0
    If false: throw to monkey 1";

    #[test]
    fn test_parse_one_monkey() {
        let data = "Monkey 0:
  Starting items: 75, 63
  Operation: new = old * 3
  Test: divisible by 11
    If true: throw to monkey 7
    If false: throw to monkey 2";
        let monkeys = parse(data);
        println!("{:?}", monkeys);
        let goal_monkeys = vec![Monkey {
            starting_items: VecDeque::from([75, 63]),
            operation: vec!["new", "=", "old", "*", "3"]
                .iter()
                .map(|s| s.to_string())
                .collect(),
            test_divisible_by_cond: 11,
            target_if_test: 7,
            target_if_not_test: 2,
            inspected_item_count: 0,
        }];
        println!("Goal:     {:?}\nActual: {:?}", goal_monkeys, monkeys);
        assert!(goal_monkeys == monkeys);
    }
    #[test]
    fn test() {
        let desired_results = vec![
            (1, 4 * 6),
            (20, 99 * 103),
            (1000, 5204 * 5192),
            (2000, 10419 * 10391),
            (3000, 15638 * 15593),
            (4000, 20858 * 20797),
            (5000, 26075 * 26000),
            (6000, 31294 * 31204),
            (7000, 36508 * 36400),
            (8000, 41728 * 41606),
            (9000, 46945 * 46807),
            (10000, 52166 * 52013),
        ];
        for (rounds, desired_result) in desired_results {
            let mut monkeys = parse(DATA);
            let result = process(&mut monkeys, rounds);
            println!("{} == {}", result, desired_result);
            assert!(result == desired_result);
        }
    }
}

Soundtrack

Advent of Code 2022: Days 8 and 9

Mon, 12 Dec 2022 00:00:00 +0000

Today I was out of the house most of the day. This was a good chance to check out TRAMP: emacs' Transparent Remote Access: Multiple Protocols, which satisfied my remote-editing needs pretty effectively.

For context, previously my remote editing generally consists of one of the following:

rsync down the project, edit the project, rsync up the project, though this doesn’t let me use the environment I have set up on my desktop (compilers, cached builds, configs, …)
A fresh git clone on the laptop, independent of the project files elsewhere, though this doesn’t give me any changes I haven’t committed and pushed
vim over ssh, though this isn’t terribly resilient to high-latency connections or packet loss, both of which are common in rural Minnesota, and isn’t perfect with things like clipboard integration
VSCode’s remote SSH extension, though this is a Microsoft proprietary extension and doesn’t work with Arch’s open source build of vscode

TRAMP does pretty well with all the concerns I’ve previously run into:

I can use the remote environment automatically with M-x shell
I’m editing my primary copy of the files, so I don’t have to worry about clobbering uncomitted work
Since the file is copied locally, it’s generally fairly tolerant of higher-latency connections; saving may just take a while
And, it’s open-source, and built into Emacs!

That’s not to say everything is perfect; I had some issues with interrupted sessions and autosaves when my VPN connection into my home network dropped, though I was able to figure all of these out eventually. (M-x recover-this-file goes a long way!)

Day 8

Part 1 source code

use std::fs;

fn is_visible(forest: &Vec<Vec<u8>>, row: usize, column: usize) -> bool {
    let height = forest[row][column];
    let mut hidden_from_north = false;
    for i in 0..row {
        if forest[i][column] >= height {
            hidden_from_north = true;
            break;
        }
    }
    let mut hidden_from_south = false;
    for i in row + 1..forest.len() {
        if forest[i][column] >= height {
            hidden_from_south = true;
            break;
        }
    }
    let mut hidden_from_west = false;
    for i in 0..column {
        if forest[row][i] >= height {
            hidden_from_west = true;
            break;
        }
    }
    let mut hidden_from_east = false;
    for i in column + 1..forest[0].len() {
        if forest[row][i] >= height {
            hidden_from_east = true;
            break;
        }
    }
    !hidden_from_north || !hidden_from_south || !hidden_from_west || !hidden_from_east
}

fn process(data: &str) -> u32 {
    let mut forest: Vec<Vec<u8>> = Vec::new();
    for line in data.split("\n") {
        let mut tree_row = Vec::new();
        for c in line.chars() {
            tree_row.push(c.to_digit(10).unwrap() as u8);
        }
        forest.push(tree_row);
    }
    let mut visible_count = 0;
    for row in 0..forest.len() {
        for column in 0..forest[0].len() {
            if is_visible(&forest, row, column) {
                visible_count += 1;
            }
        }
    }
    visible_count
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();

    println!("{}", process(data));
}

#[cfg(test)]
mod test {
    use super::*;

    static DATA: &str = "30373
25512
65332
33549
35390";

    #[test]
    fn test() {
        assert!(process(DATA) == 21);
    }
}

Part 2 source code

use std::fs;

fn scenic_score(forest: &Vec<Vec<u8>>, row: usize, column: usize) -> u32 {
    let height = forest[row][column];
    let mut visible_to_north = 0;
    for i in 1..=row {
        visible_to_north += 1;
        if forest[row - i][column] >= height {
            break;
        }
    }
    let mut visible_to_south = 0;
    for i in row + 1..forest.len() {
        visible_to_south += 1;
        if forest[i][column] >= height {
            break;
        }
    }
    let mut visible_to_west = 0;
    for i in 1..=column {
        visible_to_west += 1;
        if forest[row][column - i] >= height {
            break;
        }
    }
    let mut visible_to_east = 0;
    for i in column + 1..forest[0].len() {
        visible_to_east += 1;
        if forest[row][i] >= height {
            break;
        }
    }
    visible_to_north * visible_to_south * visible_to_west * visible_to_east
}

fn process(data: &str) -> u32 {
    let mut forest: Vec<Vec<u8>> = Vec::new();
    for line in data.split("\n") {
        let mut tree_row = Vec::new();
        for c in line.chars() {
            tree_row.push(c.to_digit(10).unwrap() as u8);
        }
        forest.push(tree_row);
    }
    let mut max = 0;
    for row in 0..forest.len() {
        for column in 0..forest[0].len() {
            if scenic_score(&forest, row, column) > max {
                max = scenic_score(&forest, row, column);
            }
        }
    }
    max
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();

    println!("{}", process(data));
}

#[cfg(test)]
mod test {
    use super::*;

    static DATA: &str = "30373
25512
65332
33549
35390";

    #[test]
    fn test() {
        assert!(process(DATA) == 8);
    }
}

Day 9

Source code

use std::cmp::Ordering;
use std::collections::HashMap;
use std::fs;

fn do_move(direction: &str, positions: &mut Vec<(i32, i32)>) {
    match direction {
        "U" => positions[0].1 += 1,
        "D" => positions[0].1 -= 1,
        "L" => positions[0].0 -= 1,
        "R" => positions[0].0 += 1,
        _ => panic!("Illegal direction {}", direction),
    }

    for i in 1..positions.len() {
        if (positions[i].0 - positions[i - 1].0).abs() > 1
            || (positions[i].1 - positions[i - 1].1).abs() > 1
        {
            positions[i].0 += match positions[i].0.cmp(&positions[i - 1].0) {
                Ordering::Less => 1, // TODO: There _must_ be a better way to do this
                Ordering::Equal => 0,
                Ordering::Greater => -1,
            };
            positions[i].1 += match positions[i].1.cmp(&positions[i - 1].1) {
                Ordering::Less => 1,
                Ordering::Equal => 0,
                Ordering::Greater => -1,
            };
        }
    }
}

fn process(data: &str, knot_count: usize) -> usize {
    let mut positions = Vec::new();
    for _ in 0..knot_count {
        positions.push((0, 0));
    }
    let mut visited_positions = HashMap::new();
    for row in data.split("\n") {
        let command: Vec<&str> = row.split(" ").collect();
        let direction = command[0];
        for _ in 0..command[1].parse().unwrap() {
            do_move(direction, &mut positions);
            visited_positions.insert(positions[positions.len() - 1], true);
        }
    }
    visited_positions.len()
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data, 10)); // 2 for part 1
}

#[cfg(test)]
mod test {
    use super::*;

    const DATA: &str = "R 4
U 4
L 3
D 1
R 4
D 1
L 5
R 2";

    #[test]
    fn test_move() {
        let cases = vec![
            // positions, dir, new_positions
            (vec![(1, 0), (0, 0)], "R", vec![(2, 0), (1, 0)]),
            (vec![(1, -2), (1, -1)], "D", vec![(1, -3), (1, -2)]),
            (vec![(2, -2), (1, -3)], "R", vec![(3, -2), (2, -2)]),
        ];
        for (original_positions, dir, new_positions) in cases {
            let mut positions = original_positions.clone();
            do_move(dir, &mut positions);
            println!("{:?} == {:?}", positions, original_positions);
            assert!(positions == new_positions);
        }
    }

    #[test]
    fn test_full() {
        assert!(process(DATA, 2) == 13);
        assert!(process(DATA, 10) == 1);
    }

    #[test]
    fn test_fuller() {
        let data = "R 5
U 8
L 8
D 3
R 17
D 10
L 25
U 20";
        assert!(process(data, 10) == 36);
    }
}

Soundtrack

Advent of Code 2022: The rest of Week 1

Fri, 09 Dec 2022 00:00:00 +0000

Day 2

I worked my way through the first half of the Emacs tutorial: document navigation, basic text insertion and deletion, cut and paste, and undo/redo. Still no syntax highlighting or anything fancy. I’d guess about two hours for the tutorial, and 20 minutes for writing code.

Part 1 source code

use std::fs;

fn main() {
    let data = fs::read_to_string("input.txt").expect("Could not read input.txt");
    let data = data.trim();

    let mut score = 0;

    for line in data.split("\n") {
        let choices: Vec<&str> = line.split(" ").collect();
        score += match (choices[0], choices[1]) {
            ("A", "X") => 1 + 3,
            ("A", "Y") => 2 + 6,
            ("A", "Z") => 3 + 0,
            ("B", "X") => 1 + 0,
            ("B", "Y") => 2 + 3,
            ("B", "Z") => 3 + 6,
            ("C", "X") => 1 + 6,
            ("C", "Y") => 2 + 0,
            ("C", "Z") => 3 + 3,
            _ => panic!("Unknown combination {:?}", choices),
        };
    }
    println!("{}", score);
}

Part 2 source code

use std::fs;

fn main() {
    let data = fs::read_to_string("input.txt").expect("Could not read input.txt");
    let data = data.trim();

    let mut score = 0;

    for line in data.split("\n") {
        let choices: Vec<&str> = line.split(" ").collect();
        score += match (choices[0], choices[1]) {
            ("A", "X") => 3 + 0,
            ("A", "Y") => 1 + 3,
            ("A", "Z") => 2 + 6,
            ("B", "X") => 1 + 0,
            ("B", "Y") => 2 + 3,
            ("B", "Z") => 3 + 6,
            ("C", "X") => 2 + 0,
            ("C", "Y") => 3 + 3,
            ("C", "Z") => 1 + 6,
            _ => panic!("Unknown combination {:?}", choices),
        };
    }
    println!("{}", score);
}

Day 3

I completed the rest of the Emacs tutorial: Buffers and buffer manipulation, modes, search, window/frame management, and the help system. I’m definitely feeling a bit more confident in my basic editor usage. Still leaving more advanced editor configuration for tomorrow. I spent about two more hours in the tutorial today, and a hair over 30 minutes writing code.

Part 1 source code

use std::fs;

fn find_duplicated_char(first: &str, second: &str) -> char {
    for char in first.chars() {
        if second.contains(char) {
            return char;
        }
    }
    panic!("no duplicate found");
}

fn priority(c: char) -> u32 {
    match c {
        'A'..='Z' => c as u32 - 64 + 26,
        'a'..='z' => c as u32 - 96,
        _ => panic!("This isn't a letter!"),
    }
}

fn process(data: &str) -> u32 {
    let mut score = 0;
    for line in data.split("\n") {
        let first_half = &line[0..line.len() / 2];
        let second_half = &line[line.len() / 2..line.len()];
        score += priority(find_duplicated_char(first_half, second_half));
    }
    score
}

fn main() {
    let data = fs::read_to_string("input.txt").expect("Could not read input.txt");
    let data = data.trim();

    println!("{}", process(data));
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn test() {
        let data = "vJrwpWtwJgWrhcsFMMfFFhFp
jqHRNqRjqzjGDLGLrsFMfFZSrLrFZsSL
PmmdzqPrVvPwwTWBwg
wMqvLMZHhHMvwLHjbvcjnnSBnvTQFn
ttgJtRGJQctTZtZT
CrZsJsPPZsGzwwsLwLmpwMDw";
        assert_eq!(process(data), 157);
    }
}

Part 2 source code

use std::fs;

fn find_duplicated_char(first: &str, second: &str, third: &str) -> char {
    for char in first.chars() {
        if second.contains(char) && third.contains(char) {
            return char;
        }
    }
    panic!("no duplicate found");
}

fn priority(c: char) -> u32 {
    match c {
        'A'..='Z' => c as u32 - 64 + 26,
        'a'..='z' => c as u32 - 96,
        _ => panic!("This isn't a letter!"),
    }
}

fn process(data: &str) -> u32 {
    let mut score = 0;
    let lines: Vec<&str> = data.split("\n").collect();
    let mut i = 0;
    while i < lines.len() {
        score += priority(find_duplicated_char(lines[i], lines[i + 1], lines[i + 2]));
        i += 3;
    }
    score
}

fn main() {
    let data = fs::read_to_string("input.txt").expect("Could not read input.txt");
    let data = data.trim();

    println!("{}", process(data));
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn test() {
        let data = "vJrwpWtwJgWrhcsFMMfFFhFp
jqHRNqRjqzjGDLGLrsFMfFZSrLrFZsSL
PmmdzqPrVvPwwTWBwg
wMqvLMZHhHMvwLHjbvcjnnSBnvTQFn
ttgJtRGJQctTZtZT
CrZsJsPPZsGzwwsLwLmpwMDw";
        assert_eq!(process(data), 70);
    }
}

Day 4

Today I read the info pages for Dired and figured out that M-! executes shell commands, so I did the entirety of my work in Rust – no external terminal needed! I’m still writing the blog post in VS Code, but I’m starting to C-x C-s to save there, which, of course doesn’t work!

Part 1 source code

use std::fs;

fn process(data: &str) -> u32 {
    let mut contains_count = 0;
    for line in data.split("\n") {
        let ranges: Vec<&str> = line.split(",").collect();
        let first: Vec<&str> = ranges[0].split("-").collect();
        let second: Vec<&str> = ranges[1].split("-").collect();
        let first_start: u32 = first[0].parse().unwrap();
        let first_end: u32 = first[1].parse().unwrap();
        let second_start: u32 = second[0].parse().unwrap();
        let second_end: u32 = second[1].parse().unwrap();
        if (first_start <= second_start && first_end >= second_end)
            || (second_start <= first_start && second_end >= first_end)
        {
            contains_count += 1;
        }
    }
    contains_count
}

fn main() {
    let data = fs::read_to_string("input.txt").expect("Could not read input.txt");
    let data = data.trim();

    println!("{}", process(data));
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn test() {
        let data = "2-4,6-8
2-3,4-5
5-7,7-9
2-8,3-7
6-6,4-6
2-6,4-8";
        assert_eq!(process(data), 2);
    }
}

Part 2 source code

use std::fs;

fn process(data: &str) -> u32 {
    let mut contains_count = 0;
    for line in data.split("\n") {
        let ranges: Vec<&str> = line.split(",").collect();
        let first: Vec<&str> = ranges[0].split("-").collect();
        let second: Vec<&str> = ranges[1].split("-").collect();
        let first_start: u32 = first[0].parse().unwrap();
        let first_end: u32 = first[1].parse().unwrap();
        let second_start: u32 = second[0].parse().unwrap();
        let second_end: u32 = second[1].parse().unwrap();
        if first_start <= second_end && second_start <= first_end {
            contains_count += 1;
        }
    }
    contains_count
}

fn main() {
    let data = fs::read_to_string("input.txt").expect("Could not read input.txt");
    let data = data.trim();

    println!("{}", process(data));
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn test() {
        let data = "2-4,6-8
2-3,4-5
5-7,7-9
2-8,3-7
6-6,4-6
2-6,4-8";
        assert_eq!(process(data), 4);
    }
}

Day 5

I didn’t make any meaningful Emacs progress today, other than reading the help as I needed for specific commands. I’m starting to think about the types of things I might want in an Emacs config. And I’m reliably C-x C-sing in VS Code now.

Part 1 source code

use std::collections::VecDeque;
use std::fs;

fn process(data: &str) -> String {
    let parts: Vec<&str> = data.split("\n\n").collect();

    let mut stack_lines: Vec<&str> = parts[0].split("\n").collect();
    stack_lines.truncate(stack_lines.len() - 1); // Remove indexing line

    let instructions: Vec<&str> = parts[1].trim().split("\n").collect();

    let mut stacks = Vec::new();
    for _ in 0..(stack_lines[0].len() + 1) / 4 {
        stacks.push(VecDeque::new());
    }

    for line in stack_lines {
        for i in 0..stacks.len() {
            let c = line.chars().nth(i * 4 + 1).unwrap();
            if c != ' ' {
                stacks[i].push_back(c);
            }
        }
    }

    for instruction in instructions {
        let instruction: Vec<&str> = instruction.split(" ").collect();
        let source: usize = instruction[3].parse().unwrap();
        let target: usize = instruction[5].parse().unwrap();
        for _ in 0..instruction[1].parse().unwrap() {
            let c = stacks[source-1].pop_front().unwrap();
            stacks[target-1].push_front(c);
        }
    }
    stacks.iter().map(|s| s.front().unwrap()).collect()
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();

    println!("{}", process(&data));
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn test() {
        let data = "    [D]    
[N] [C]    
[Z] [M] [P]
 1   2   3 

move 1 from 2 to 1
move 3 from 1 to 3
move 2 from 2 to 1
move 1 from 1 to 2";
        assert!(process(data) == "CMZ");
    }
}

Part 2 source code

use std::collections::VecDeque;
use std::fs;

fn process(data: &str) -> String {
    let parts: Vec<&str> = data.split("\n\n").collect();

    let mut stack_lines: Vec<&str> = parts[0].split("\n").collect();
    stack_lines.truncate(stack_lines.len() - 1); // Remove indexing line

    let instructions: Vec<&str> = parts[1].trim().split("\n").collect();

    let mut stacks = Vec::new();
    for _ in 0..(stack_lines[0].len() + 1) / 4 {
        stacks.push(VecDeque::new());
    }

    for line in stack_lines {
        for i in 0..stacks.len() {
            let c = line.chars().nth(i * 4 + 1).unwrap();
            if c != ' ' {
                stacks[i].push_back(c);
            }
        }
    }

    for instruction in instructions {
        let instruction: Vec<&str> = instruction.split(" ").collect();
        let source: usize = instruction[3].parse().unwrap();
        let target: usize = instruction[5].parse().unwrap();
        for i in 0..instruction[1].parse().unwrap() {
            let c = stacks[source-1].pop_front().unwrap();
            stacks[target-1].insert(i, c);
        }
    }
    stacks.iter().map(|s| s.front().unwrap()).collect()
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();

    println!("{}", process(&data));
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn test() {
        let data = "    [D]    
[N] [C]    
[Z] [M] [P]
 1   2   3 

move 1 from 2 to 1
move 3 from 1 to 3
move 2 from 2 to 1
move 1 from 1 to 2";
        assert!(process(data) == "MCD");
    }
}

Day 6

Today was the first day that I was a bit stumped writing code. The uniqueness check took a while to ponder through, but I got it working in just under an hour. Again, no major progress on anything Emacs-related.

Source code

use std::fs;

fn process(data: &str, len: usize) -> usize {
    let chars: Vec<char> = data.chars().collect();
    for i in len..chars.len() {
        let mut chars = chars[i-len..i].to_vec(); // TODO: this seems unidiomatic?
        chars.sort();
        chars.dedup();
        if chars.len() == len {
            // no duplicates
            return i;
        }
    }
    panic!("No valid marker found");
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data, 14)); // 4 for part A
}

#[cfg(test)]
mod test {
    use super::*;
    #[test]
    fn test_short() {
        let data = vec![
            ("mjqjpqmgbljsphdztnvjfqwrcgsmlb", 7),
            ("bvwbjplbgvbhsrlpgdmjqwftvncz", 5),
            ("nppdvjthqldpwncqszvftbrmjlhg", 6),
            ("nznrnfrfntjfmvfwmzdfjlvtqnbhcprsg", 10),
            ("zcfzfwzzqfrljwzlrfnpqdbhtmscgvjw", 11),
        ];
        for (datum, result) in data {
            assert!(process(datum, 4) == result);
        }
    }
    #[test]
    fn test_long() {
        let data = vec![
            ("mjqjpqmgbljsphdztnvjfqwrcgsmlb", 19),
            ("bvwbjplbgvbhsrlpgdmjqwftvncz", 23),
            ("nppdvjthqldpwncqszvftbrmjlhg", 23),
            ("nznrnfrfntjfmvfwmzdfjlvtqnbhcprsg", 29),
            ("zcfzfwzzqfrljwzlrfnpqdbhtmscgvjw", 26),
        ];
        for (datum, result) in data {
            assert!(process(datum, 14) == result);
        }
    }
}

Day 7

I finally installed rust-mode. I’m not sure why I waited so long—this was incredibly simple. M-x package-install rust-mode and then M-x rust-mode had me in syntax-highlighted, working-indentation paradise¹ in a grand total of maybe three minutes. (This only works in Emacs 28 and later, which includes the nongnu ELPA repos, and, therefore, rust-mode).

This problem took me quite a long time; I spent a lot of time figuring out how to best parse this data, and then a lot of additional time ~~fighting~~ negotiating with the borrow checker. A first iteration involved mutual references (e.g. for directories A and B, with B a subdirectory of A, A would have B in its list of children, and B would have A listed for its parent ("..")), which doesn’t work. I looked into Rc and RefCells, but ended up coming up with a moderately straightforward implementation that didn’t use either. I also had an epiphany about how borrowing works that finally (after an embarrassingly long time) made my mental model of the borrow checker click into place. So, all in all, definitely worth the struggle today!

I’d guess this puzzle took me three to four hours of intermittent work, over three days, which is why this post came out a bit later than targeted. Hopefully now that this one is done, I’ll be back on track going forward!

Source code

use std::collections::HashMap;
use std::fs;

struct Directory {
    children: HashMap<String, Directory>,
    files: HashMap<String, u32>,
}

impl Directory {
    fn size(&self) -> u32 {
        let mut size = 0;
        for (_name, child) in &self.children {
            size += child.size();
        }
        for (_name, file_size) in &self.files {
            size += file_size;
        }
        size
    }

    fn directories_at_most_size(&self, size: u32) -> u32 {
        let mut count = if self.size() <= size { self.size() } else { 0 };
        for (_name, child) in &self.children {
            count += child.directories_at_most_size(size);
        }
        count
    }

    fn smallest_directory_at_least(&self, target_size: u32) -> u32 {
        let mut size = u32::MAX;
        if self.size() < size && self.size() >= target_size {
            size = self.size();
        }
        for (_name, child) in &self.children {
            let child_size = child.smallest_directory_at_least(target_size);
            if child_size < size && child_size >= target_size {
                size = child_size;
            }
        }
        size
    }
}

fn process(data: &str) -> u32 {
    let lines: Vec<&str> = data.split("\n").collect();
    let mut root = Directory {
        children: HashMap::new(),
        files: HashMap::new(),
    };
    let mut path = Vec::new();
    for line in lines {
        let line: Vec<&str> = line.split(" ").collect();
        if line[0] == "$" {
            match line[1] {
                "cd" => match line[2] {
                    "/" => path = Vec::new(),
                    ".." => path.truncate(path.len() - 1),
                    dir => path.push(dir),
                },
                "ls" => {}
                _ => panic!("Unknown command"),
            }
        } else {
            // Part of `ls` output
            let mut pwd = &mut root;
            for dir in path.iter() {
                pwd = pwd.children.get_mut(&dir.to_string()).unwrap();
            }
            if line[0] == "dir" {
                if pwd.children.contains_key(line[1]) {
                    println!("We've seen {} before", line[1]);
                } else {
                    pwd.children.insert(
                        line[1].to_string(),
                        Directory {
                            children: HashMap::new(),
                            files: HashMap::new(),
                        },
                    );
                }
            } else {
                pwd.files
                    .insert(line[1].to_string(), line[0].parse().unwrap());
            }
        }
    }
    let size_available = 70_000_000 - root.size();
    let size_needed = 30_000_000 - size_available;
    root.smallest_directory_at_least(size_needed) // directories_at_most_size() for part 1
}

fn main() {
    let data = fs::read_to_string("input.txt").unwrap();
    let data = data.trim();
    println!("{}", process(data));
}

#[cfg(test)]
mod test {
    use super::*;

    static DATA: &str = "$ cd /
$ ls
dir a
14848514 b.txt
8504156 c.dat
dir d
$ cd a
$ ls
dir e
29116 f
2557 g
62596 h.lst
$ cd e
$ ls
584 i
$ cd ..
$ cd ..
$ cd d
$ ls
4060174 j
8033020 d.log
5626152 d.ext
7214296 k";

    #[test]
    fn test() {
        assert!(process(DATA) == 24933642); // 95437 for part 1
    }
}

Notes

At this point, I’m maintaining a bit of a checklist of things I want to figure out how to do in Emacs, and I’m making a bit of progress:

Use Dired instead of a terminal or file manager
Rust syntax highlighting¹
Rust display errors inline
Fix wonky indentation (partially—spaces/tabs are still mixed)
- Run rustfmt automatically?
Single key combination to compile and run binary
Git integration within Emacs, without falling back to M-x shell
Modify config file (aka learn elisp)
- Set auto-revert-mode by default (useful for e.g. rustfmt)

I’ve really grown to like a lot of things about Emacs:

Emacs’ help system is incredible. The tutorial is great, and the documentation available for anything I could want to do is somewhere in C-h. Apropos is nice; having explainers for each key combo with C-h k; it’s all fantastic.
Emacs’ built-in shell works really nicely. (Vim gained a terminal in 8.1, but I never got along too well with it.) There are lots of integrations that work really nicely. I was particularly impressed when git asked for an SSH key passphrase, and Emacs prompted me in the minibuffer for the passphrase so that it wouldn’t be echoed in the terminal.
In addition to being a very capable terminal editor, it’s also nicely integrated with the desktop. Copy/paste works fine with Wayland, and the mouse works nicely, and having a list of menus and icons, while I don’t use them heavily, is rather convenient.

I’ll note, to be fair to Vim, that I was never quite this intentional about learning its features. Most everything I’m able to do this far, I was also able to do in Vim, or would have been able to figure out how to, had I wanted.

Soundtrack

Maybe I’m just becoming an old grouch, but having not had syntax highlighting for much of the week, I honestly don’t miss it as much as I thought I would. I often find it useful for finding trivial syntax errors (“I didn’t close a quote” being a common one), and I suppose to some extent as an additional visual cue for navigation, but I didn’t really find either of those to be significantly impeding here. ↩︎ ↩︎
Cory Wong is on Bandcamp! Here’s The Golden Hour. I 10/10 recommend both Cory Wong and Bandcamp: wicked awesome guitar, and DRM-free no-nonsense FLAC music downloads, respectively. ↩︎

Advent of Code 2022: Day 1

Thu, 01 Dec 2022 00:00:00 +0000

This is it: I’m finally going to finish Advent of Code this year! (A strong claim, given that this is my third attempt, but we’ll see!) This year, I’m planning to use this as an excuse to learn/relearn Emacs and Rust.

I also started AoC in 2020 (19 stars; Python, mostly) and 2021 (36 stars; Rust). Last year, I went in with the specific intention of learning Rust, which honestly went pretty well; the challenges got more difficult at a pretty similar rate to which the language grew more familiar, so I ended up spending a pretty consistent amount of time on each day’s challenge.

This year, I’m going to continue with Rust, but with a different learning component: Emacs! I’ve been using Vim and VS Code as my primary editors for quite a while now, but Emacs came back across my radar yesterday as I saw the news of the looming Emacs 29 release. Lots of neat new features (Eglot for LSP integration, and native Wayland support) make this a pretty exciting release.

I used Emacs a bit in school (it was the required text editor for our second-semester CS class), so I’m not going in totally blind, but I’ve…basically forgotten everything I ever knew. C-X C-S to save; C-X C-C to exit? Arrow keys for navigation, type to insert text. That’s literally it, off the top of my head. That should get me through today, anyway; maybe tomorrow I’ll do the tutorial.

VS Code makes Rust easy; built-in syntax highlighting and good completion with the rust-analyzer extension really minimize the amount of thinking I have to do. With Emacs, I won’t be able to take the same tab-completion mental shortcuts. That means I’ll be relying a lot more heavily on my knowledge of the language to carry me through, as well as exploring some new editor features, which I’m excited about.

Setting up a dev environment

I already had Rust installed from last year’s attempt, so that’s easy: rustup update stable and I have the latest version ready to go. (I installed rustup through my system’s package manager, rather than the curl | bash official install.)

Emacs 29 isn’t stable, so the Arch package is currently still on v28. The various versions in the AUR all appear to be tracking the master branch, so I’m just going to go ahead and compile it myself. Apparently I had all the dependencies installed, so that was as simple as:

git clone git://git.savannah.gnu.org/emacs.git
cd emacs
./configure --with-pgtk
make -j16
./src/emacs

Some notes: https://lars.ingebrigtsen.no/2014/11/13/welcome-new-emacs-developers/

The ./configure step could be left out, but then Emacs fails looking for an X display. (I’m not entirely sure why it didn’t use Xwayland, which I do have set up.)

Code

use std::fs;

fn main() {
    let data = fs::read_to_string("input.txt")
        .expect("Could not read input.txt");
    let data = data.trim();

    let mut highest_totals = vec![0, 0, 0];
    let mut current_total = 0;
    for line in data.split("\n") {
        if line == "" {
            // we've reached the end of an elf's food
            highest_totals.push(current_total);
            highest_totals.sort_by(|a, b| b.cmp(a));
            highest_totals.truncate(3);
            current_total = 0;
        } else {
            current_total += line.parse::<u32>().unwrap();
        }
    }
    highest_totals.push(current_total);
    highest_totals.sort_by(|a, b| b.cmp(a));
    highest_totals.truncate(3);
    println!("{:?}, sum: {}", highest_totals, highest_totals.iter().sum::<u32>());
}

Notes

I’m using Emacs in GUI mode. I’ll probably also explore the -nw option to run it from a terminal.

Emacs didn’t have any syntax highlighting for Rust in its default configuration. I hear it has tree-sitter support as of v29, but I assume I need an extension to make it support any particular language? Not sure; a project for tomorrow!

Indentation by default is pretty wonky; Emacs appears to default to a tab inserting…three spaces? But then it replaces eight spaces with a tab? But eight isn’t a multiple of three? I fixed some indentation manually; tomorrow I’ll have to look a bit more into what’s going on here. For today, I just fixed things up manually with rustfmt.

I’m not entirely sure I understand what the pgtk stuff in Emacs means. I’ll have to check that out. More info here? https://batsov.com/articles/2021/12/19/building-emacs-from-source-with-pgtk/

Here’s where I wound up. Note that I don’t have the faintest idea how to kill the default intro buffer at the moment, and that I’m missing any kind of window decorations (though sway adds its own for me; not shown here).

Time spent

Download/Compile Emacs: 20m

Write/compile/submit code: 20m (~roughly evenly split between fighting Emacs and Rust)

Blog post: 1h

Soundtrack

Twos

Thu, 10 Nov 2022 00:00:00 +0000

Twos is one of my favorite card games. (I’ve never been able to find reference to the game outside my friend circle, so if this is a game that’s known by other names, please let me know!)

Full rules...

Here are the rules, as I was originally taught.

Twos is played with three players. A standard deck of playing cards (no jokers) is shuffled, and each player is dealt thirteen cards. (One hand of thirteen cards is left out of play, which helps reduce the effectiveness of card counting.)

The object of the game is to be the first to run out of cards.

Each card has a distinct value. Aces and twos are high (twos being the highest card in the game, hence the name). Within each rank, hearts are worth more than diamonds, which are worth more than clubs, which are worth more than spades.

The player with the lowest valued card (generally the three of spades, unless it was in the undealt hand this round) starts the first trick.

A trick begins with the player either playing 1, 2, 3, 4 of the same rank, or a run of 3 or more (optionally of the same suit). Subsequent players need to play a higher value (using the highest-value cards to compare) of the same pattern; for example, if the first player played a three of spades and a three of diamonds, the second player could play any other pair: the remaining threes include a three of hearts, which outvalues the three of diamonds, and any other pair outvalues the threes. Or, if the first player plays a run of three ending in a six of diamonds, subsequent players would continue the trick with a run of three, ending in a six of hearts or a seven or higher. This continues until no player can (or chooses – a player can pass if they are able to but do not wish to continue) to outdo the previous player. The player who put down the highest-valued card(s) then sets up the next trick.

The one exception to following the pattern is the “bomb”, which can either consist of a multiple run (e.g. two threes, two fours, and two fives), or a four of a kind. A bomb beats any other set of cards played, except a higher-valued bomb. For example, if the first player leads with a single three of spades, and the second player immediately escalates to a two of hearts, the third player can only continue play using a bomb, since no single card can beat a two of hearts, the highest-valued card. A bomb consisting of a double run of three can be beaten by a double run of three with a higher top-valued card than in the other double run, or by a four-of-a-kind bomb. (For a discussion of higher-valued bombs, see below.)

Play continues until the first player runs out of cards; that player wins.

Käthe and Isaac and I played a few rounds of Twos last night. After Käthe called it for the night, Isaac and I tried adjusting the rules for two players. We decided to try out much larger hands, with two players each being dealt twenty cards.

This ended up being totally insane! The first round Isaac led with a full run of 13 cards: 3-4-5-6-7-8-9-10-J-Q-K-A-2, with the two of spades. I immediately countered with an equivalent run with the two of clubs. This was definitely anomalous, but not wildly so when playing with hands that large.

The largest issue we ran into was that we were seeing multiple bombs each round, and of much higher magnitudes than usual. A double run of three occurred roughly once per round; a four-of-a-kind every second or third round; and it wasn’t uncommon to see a triple run of three, or a double run of four, five, or (once) six! This led to some questions: Does a four of a kind beat a double run of three? of four? of five? Where do triple runs fall in?

Based on our intuition after playing probably twenty or so rounds with the 20-card hands, Isaac and I settled on this ordering for bomb strength, ordered from weakest to strongest:

Double run of three
Four-of-a-kind
Double run of four
Double run of five
Triple run of three
Double run of six

If pressed, we’d probably continue that list with

Triple run of four
Double run of seven
Quadruple run of three
Triple run of five
Triple run of six
Quadruple run of four

although we never ran into any of those, and we can’t say with any certainty which are most/least likely to occur.

But those were just guesses, and we were more or less making it up as we went along. To solidify these answers a bit, I wrote a short simulator in Rust that deals hands of varying sizes from a 52-card deck¹, and records the frequency that a hand contains each type of bomb.

Source code (Rust)

Running through ten million hands takes about a minute on my computer. I highly suggest cargo run --release; --debug is substantially slower.

use std::collections::HashMap;

use rand::{seq::SliceRandom, thread_rng};

const HANDS_TO_TRY: u32 = 10_000_000;
const HAND_SIZE: u8 = 13; // See also: 20

fn contains_run_of(card_count_by_rank: &HashMap<u8, u8>, width: u8, length: u8) -> bool {
    for starting_rank in 1..=(13 - length + 1) {
        let mut is_run = true;
        for i in 0..length {
            if card_count_by_rank[&(starting_rank + i)] < width {
                is_run = false;
                break;
            }
        }
        if is_run {
            return true;
        }
    }
    false
}

fn check_for_bombs(hand: &[u8], hands_with_bombs: &mut HashMap<(u8, u8), u64>) {
    let mut card_count_by_rank = HashMap::new();
    for rank in 1..=13 {
        card_count_by_rank.insert(rank, 0);
    }
    for card in hand {
        card_count_by_rank
            .entry(*card)
            .and_modify(|count| *count += 1);
    }

    for width in 1..=4 {
        for length in 1..=13 {
            if contains_run_of(&card_count_by_rank, width, length) {
                hands_with_bombs
                    .entry((width, length))
                    .and_modify(|count| *count += 1);
            }
        }
    }
}

fn main() {
    // Construct a 52-card deck
    let mut deck = Vec::new();
    for _suit in 0..4 {
        for rank in 1..=13 {
            deck.push(rank);
        }
    }

    let mut hands_with_bombs = HashMap::new();
    for width in 1..=4 {
        for length in 1..=13 {
            hands_with_bombs.insert((width, length), 0);
        }
    }

    let mut rng = thread_rng();

    let hands_per_round = 52 / HAND_SIZE;

    for i in 0..(HANDS_TO_TRY / hands_per_round as u32) {
        if i * hands_per_round as u32 % 10000 == 0 {
            println!("Checking {}", i * hands_per_round as u32);
        }

        deck.shuffle(&mut rng);

        // Deal cards
        let mut hands = Vec::new();
        for i in 0..hands_per_round {
            hands.push(&deck[(HAND_SIZE * i) as usize..(HAND_SIZE * (i + 1)) as usize]);
        }

        for hand in hands {
            check_for_bombs(hand, &mut hands_with_bombs);
        }
    }

    let mut results = Vec::from_iter(hands_with_bombs.iter());
    results.sort_by(|a, b| a.1.cmp(b.1));

    for ((width, length), result) in results {
        if *result > 0 {
            println!("{}x{}: {}", width, length, result);
        }
    }
}

Given that an n×m run is a run n cards wide and m cards long (e.g. 4 4 5 5 6 6 7 7 would be a 2x4 run), in ten million hands you would expect to see this many of each of the following runs:

3x4: 3
2x6: 172
4x2: 219
1x13: 979
3x3: 1124
2x5: 7250
1x12: 10579
1x11: 46448
2x4: 106649
3x2: 126110
1x10: 145179
4x1: 342976
1x9: 367730
1x8: 813649
2x3: 917211
1x7: 1632653
1x6: 3033189
3x1: 4906188
2x2: 5164667
1x5: 5181797
1x4: 7797492
1x3: 9652373
1x2: 9998564
2x1: 9999021
1x1: 10000000

But, of course, not all of those are valid bombs. Here’s just the ones that are valid (that is, two or more wide and three or more long, plus the 4-of-a-kind bomb).

3x4: 3
2x6: 172
3x3: 1124
2x5: 7250
2x4: 106649
4x1: 342976
2x3: 917211

For 20-card hands (again, the number of hands that contain at least one of this type of bomb, out of 10 million hands):

2x10: 24
4x3: 62
3x5: 169
2x9: 931
3x4: 7829
2x8: 10502
4x2: 20085
2x7: 63317
3x3: 159321
2x6: 274643
1x13: 897647
2x5: 950825
1x12: 1405553
3x2: 1965803
1x11: 2059459
4x1: 2198049
2x4: 2767478
1x10: 2882221
1x9: 3899799
1x8: 5141079
2x3: 6482887
1x7: 6634212
1x6: 8161395
1x5: 9281048
3x1: 9567731
2x2: 9783220
1x4: 9874388
1x3: 9996418
1x2: 10000000
1x1: 10000000
2x1: 10000000

Filtered to just valid bombs:

2x10: 24
4x3: 62
3x5: 169
2x9: 931
3x4: 7829
2x8: 10502
2x7: 63317
3x3: 159321
2x6: 274643
2x5: 950825
4x1: 2198049
2x4: 2767478
2x3: 6482887

With this in mind, our original proposal stands pretty well with the numbers we found. For hands of 13 our original list was entirely correct; from most to least likely:

Double run of three
Four-of-a-kind
Double run of four
Double run of five
Triple run of three
Double run of six

For twenty-card hands, though, this doesn’t hold. I hadn’t considered this, but in hindsight it seems obvious: bigger hands will increase the expected length of runs (e.g. getting a 1x13 run is horribly improbable in a thirteen-card hand, but fairly likely in a twenty-card hand) more than the width of them (e.g. I’ll never see a five-of-a-kind dealt out of a standard deck, no matter how large the hands are!). So for larger hands the order changes up a bit, with a double run of four becoming more likely than a four of a kind.

Another interesting problem would be drawing hands with replacement. Not that you would actually draw hands with replacement, but often we play card games like Twos with a deck made of the remains of lots of old, partially-complete decks. With sufficiently many decks, this ends up (1) alleviating a lot of the shuffling, and (2) effectively being an infinite deck to draw from. (And, of course making card counting effectively impossible!) I didn’t do the math on this, though, but if you do, please let me know what you find! ↩︎

Ages of Minnesota Wild Players

Sun, 24 Apr 2022 00:00:00 +0000

Tonight the Minnesota Wild won their game against the Nashville Predators, in a nailbiting finish—we (specifically Dmitry Kulikov, age 31) scored our final goal to end 5-4 with 1.3 seconds left of overtime.

It’s always amazing to me as I get older that a large component of professional sports players are younger than I am. (Though hockey players are somewhat older than I’d imagined: Both goalies are in their mid to late thirties, and the median player is still a few years older than I am.)

The NHL has pretty good information available about their games, so I poked around a bit. (Here’s the (extensive!) data for tonight’s game.)

import requests, datetime

def age(born):
    today = datetime.date.today()
    return today.year - born.year - ((today.month, today.day) < (born.month, born.day))

roster = requests.get("https://statsapi.web.nhl.com/api/v1/teams/30/roster").json()['roster']

# Retrieve full data on each player
players = []
for player in roster:
    players.append(requests.get(f"https://statsapi.web.nhl.com{player['person']['link']}").json()['people'][0])
players.sort(key=lambda p: p['birthDate'], reverse=True)

# Display table by age
print("| Name | Position | DOB | Age |\n|-|-|-|-|") 
for p in players:
    born = datetime.date.fromisoformat(p['birthDate'])
    print(f"| [{p['fullName']}](https://www.nhl.com/player/{p['id']}) | {p['primaryPosition']['name']} | {p['birthDate']} | {age(born)} |")

# Find average player age
average_dob = datetime.date.fromtimestamp(sum([datetime.datetime.combine(datetime.date.fromisoformat(p['birthDate']), datetime.time()).timestamp() for p in players])/len(players))
print(f"Average age: {age(average_dob)}")

# Count players by age
min_age = age(datetime.date.fromisoformat(players[0]['birthDate']))
max_age = age(datetime.date.fromisoformat(players[-1]['birthDate']))
players_by_age = {}
for a in range(min_age, max_age+1):
    players_by_age[a] = []
for p in players:
    players_by_age[age(datetime.date.fromisoformat(p['birthDate']))].append(p)
for age, players in players_by_age.items():
    print(f"{len(players)} players {age} years old")

2021-2022 Minnesota Wild players by age:

Boldy

Dewar

Jost, Duhaime, Kaprizov

Greenway, Eriksson Ek, Fiala

Zuccarello, Benn, Talbot

Goligoski

Fleury

The average player is 29 years and one month old; the median player is 28.

Name	Position	DOB	Age¹
Matt Boldy	Left Wing	2001-04-05	21
Connor Dewar	Center	1999-06-26	22
Tyson Jost	Center	1998-03-14	24
Brandon Duhaime	Right Wing	1997-05-22	24
Kirill Kaprizov	Left Wing	1997-04-26	24
Jordan Greenway	Left Wing	1997-02-16	25
Joel Eriksson Ek	Center	1997-01-29	25
Kevin Fiala	Left Wing	1996-07-22	25
Jacob Middleton	Defenseman	1996-01-02	26
Ryan Hartman	Right Wing	1994-09-20	27
Matt Dumba	Defenseman	1994-07-25	27
Jonas Brodin	Defenseman	1993-07-12	28
Frederick Gaudreau	Center	1993-05-01	28
Joseph Cramarossa	Center	1992-10-26	29
Nick Bjugstad	Center	1992-07-17	29
Jon Merrill	Defenseman	1992-02-03	30
Marcus Foligno	Left Wing	1991-08-10	30
Nicolas Deslauriers	Left Wing	1991-02-22	31
Dmitry Kulikov	Defenseman	1990-10-29	31
Jared Spurgeon	Defenseman	1989-11-29	32
Mats Zuccarello	Right Wing	1987-09-01	34
Jordie Benn	Defenseman	1987-07-26	34
Cam Talbot	Goalie	1987-07-05	34
Alex Goligoski	Defenseman	1985-07-30	36
Marc-Andre Fleury	Goalie	1984-11-28	37

As of today, April 24, 2022. ↩︎

2021 Taxes: Open Source Edition

Mon, 18 Apr 2022 00:00:00 +0000

Happy Tax Day! The tax industry (by which I primarily mean its largest player, Intuit) sucks.¹ As a whole, the industry is an excellent example of rent-seeking behavior, and not one I wish to encourage with my patronage. So I tried out a few options when filing my 2021 taxes, and found out a few interesting things along the way².

Introduction

In the past, I’ve fluctuated between filling out paper forms from the IRS and, more recently, having TurboTax do my taxes for me. This year, I’m tired of TurboTax’s shenanigans, and they no longer (appear to?) participate in the Free File program, so I’d be paying through the nose for it. In the interest of finding something better, I tried a few other providers. Also, it’s possible that this will be my last year that I’ll fall in the income range where I’ll be eligible for the Free File program, so I figured I’d try out some of the free-as-in-beer software while it’s available to me.

I <3 open source, so I tried out the two major open-source contenders:

USTaxes.org is a modern-feeling GPL3-licensed web/desktop app written with React. It’s very usable, though certainly won’t cover every use case for everyone, as its scope is currently fairly limited. I did my taxes with their web version, but there’s a desktop version built with Tauri, if you want to make extra sure your data never leaves your computer.

OpenTaxSolver is a desktop application, GPL2 licensed, written in C. It exposes a very neat plaintext file format for managing tax forms, and then uses that data to generate the final fillable PDFs. However, its interface feels slightly archaic, as does its ecosystem: Sourceforge, Subversion³, and this excellent logo. However, that’s not a bad thing in my book—the implementation also tends towards Boring (C, Makefiles, limited dependencies), which is definitely a plus⁴!

More about OpenTaxSolver

Here’s a summary from their website:

OpenTaxSolver (OTS) is a set of programs and templates for helping you fill out your income tax forms. It performs the tedious arithmetic. OTS is intended to assist those who normally prepare their tax forms themselves, and who generally know on which lines to enter their numbers. It is meant to be used in combination with the instruction booklet corresponding to a given form.

This package contains programs and templates for:

US-1040 - which also does the Schedules A, B, D, and forms 8949.

Schedule C for US-1040.

State Income Taxes for Ohio, New Jersey, Virginia, Pennsylvania, Massachusetts, North Carolina, and California taxes updated for the 2021 Tax-Year.

Also contains an Automatic PDF Form-Fillout function:

Supports all Federal Forms and State Forms. Saves time by filling out many of the numbers. You may still need to enter some information or check boxes that are not handled by OTS. Tested to work properly with many PDF viewers.

You can edit your forms with Libre-Office.

Here’s an architecture diagram from their website:

Their file format is really interesting. Here’s a snippet from my file from this year:

Title:  US Federal 1040 Tax Form - 2021

 { --- Your Filing Status & Exemptions --- }
Status	Single	 { Single, Married/Joint, Head_of_House, Married/Sep, Widow(er) }
You_65+Over?	N	 { Were you born before January 2, 1957 ? (answer: Yes, No) }
You_Blind?	N	 { Are you blind ? (answer: Yes, No) }
Spouse_65+Over?	N	 { Was Spouse born before January 2, 1957 ? (answer: Yes, No) }
Spouse_Blind?	N	 { Is Spouse blind ? (answer: Yes, No) }
Dependents	1	 { Number of Dependents, (answer: 1, 2, 3, 4, 5, 6, 7, 8, 9, ...)
				self=1, spouse, etc. }
VirtCurr?	N	 { During 2021, did you buy/sell/exchange/trade any Virtual Currency ? (answer: Yes, No) }
 { ---- Income ---- }
 { -- Wages, salaries, tips (W-2's Box-1). -- }
L1	12345.67	
		;
 { --- Interest --- }
 { -- Tax-Exempt Interest. (Only used for SocialSecurity calculations). --
      (Any private activity bond interest exempt from regular tax, is entered under Schedule 2 below.) }
L2a			;
 { -- Taxable Interest -- 1099-INT(s) box 1 }
L2b	20.13	
		
		;
 { --- Dividends --- }
 { -- Qualified Dividends 1099-DIV box 1b -- }
L3a	72.97	
		;
 { -- Ordinary Dividends 1099-DIV box 1a. -- }
L3b	120.95	
		;
 { --- Other Income & Credits --- }
L4a			; { IRA distributions. }
L4b			; { Taxable IRA distributions. }
L5a			; { Pensions, Annuites. }
L5b			; { Taxable Pensions, Annuites. }
L6a			; { Social Security benefits.  Forms SSA-1099 box-5. }
CharityCC	200		; { Charity contributions by Cash or Check. }
CharityOT			; { Charity contributions Other Than cash or check. }
CharityCO			; { Charity contributions CarryOver from prior year. }
L13			; { Qualified business income deduction. }
L19			; { Child tax credit/credit for other dependents. }
L25a	1234.56		; { Federal income tax withheld, from W-2's, box-2. }
L25b			; { Federal income tax withheld, from 1099's. }
L25c			; { Federal income tax withheld, from other forms. }
L26			; { Estimated tax payments made for the year. }

As a control group, I compared the results with several other providers:

TurboTax may be Satan incarnate, but I wanted to include it for completeness’ sake.

FreeTaxUSA.com is a frequently recommended competitor to TurboTax.

OLT.com was the only Free File provider that would also file my state returns for free.

Not under consideration this time around:

Filling out forms myself: While doing my taxes five consecutive times gave me a pretty good overview of what I’d need to do, I wasn’t prepared to attempt doing my taxes unassisted this year. Perhaps next year, since I should be able to mostly copy/paste from my 2021 tax return.

Free File Fillable Forms https://www.freefilefillableforms.com/home/default.php

A paid tax preparer like H&R Block (which I’ve heard is the TurboTax of in-person tax preparers) or your local CPA. This option tends to be prohibitively expensive, and won’t likely end up saving me much time.

Getting started

UsTaxes.org

Perhaps unsurprisingly, the open-source version was the easiest to sign up for, in that no signup whatsoever is required. (This does mean that they may lose your data if you start a return and don’t complete it. It wasn’t entirely clear to me how the data was stored—localStorage mostly, I think?—but I avoided refreshing the page just in case.) Just visit their website and go!

Otherwise, you can download their desktop app and run locally:

https://github.com/ustaxes/UsTaxes/releases

Selecting and downloading an appropriate executable for one’s platform is perhaps the least user-friendly part of the process. The download takes you straight to GitHub releases, with no indication of whether you want the .msi (Windows), the .dmg (MacOS), the .deb (Debian/Ubuntu/Mint/…), the .AppImage, …. While I might feel comfortable recommending UsTaxes.org to my parents from a general usability standpoint, I wouldn’t expect them to necessarily figure out GitHub Releases just to install the software.

OpenTaxSolver

Download OpenTaxSolver from http://opentaxsolver.sourceforge.net/download2021.html, extract and run:

tar xf OpenTaxSolver2021_19.07_linux64.tgz
cd OpenTaxSolver2021_19.07_linux64
./Run_taxsolve_GUI

Documentation is available online: http://opentaxsolver.sourceforge.net/usage.html

There are prebuilt versions, and some distros have it packaged. Still, not ready for non-nerds.

Paid providers

All three required me to sign up for an account, but there was nothing noteworthy about the process for any of them. Except FreeTaxUSA, which is convinced that usernames are secrets? (Shh, don’t tell anyone!)

Each suggests, with varying degrees of intensity, that I add a phone number to “secure my account”—presumably with SMS-based 2fa. I didn’t dig around in any menus to figure out if they supported TOTP or not. I get enough SMS spam as it is, so I declined giving any of the three my phone number. Based on the number of promotional emails I’ve gotten this week from Intuit, this may have been wise.

Oh, and TurboTax wants to get its hands on all the information it can:

Pricing

The open-source options are, unsurprisingly, free of cost.

TurboTax

TurboTax provides a questionnaire that helpfully slots you into one of their product ranges. (“Expert help as you go” (“Get unlimited help from tax experts and a review before you file.”) and “We do your taxes for you” (“A tax expert will prepare, sign, and file your taxes for you.”) are a substantial additional cost.)

I was immediately put off to discover that their pricing is different whether or not you’ve already created an account. (I have notes that they also had a $69⁵ Deluxe price listed somewhere, but was unable to find that. Might be an error on my part, or that I wasn’t able to reproduce the set of conditions for which they quote that particular price point.) Note the difference in the Premier and Self-Employed price tiers:

Without an account:

These prices, too, are pretty deceptive; they fairly clearly advertise the cost as the federal cost—and oh, by the way, it’s $50 extra to file your state taxes too. (I suppose this is reasonable in states that don’t have a state income tax, like Washington, Florida, and Texas; those states will just pay the quoted base fee.) After seeing this, I’d expect to pay $59, but the real cost is $108 once you add the $49 state filing fee on:

They do have a free offering, but it’s pretty limited. From Intuit’s website:

A simple tax return is Form 1040 only.

Situations covered in TurboTax Free Edition include:

W-2 income

Limited interest and dividend income reported on a 1099-INT or 1099-DIV

Claiming the standard deduction

Earned Income Tax Credit (EIC)

Child tax credits

Student Loan Interest deduction

Situations not covered in TurboTax Free Edition include:

Itemized deductions

Unemployment income reported on a 1099-G

Business or 1099-NEC income

Stock sales

Rental property income

Credits, deductions and income reported on schedules 1-3

Since I participated in Pearson’s Employee Stock Purchase Program, I would have needed to pay for the $89 Federal/$49 State/maybe additional hidden fees? for Premier:

FreeTaxUSA

FreeTaxUSA, as the name implies, allows you to fill out your federal returns for free, and charges $15 for your state return.

They participate in the Free File program, but only up to an AGI of $41,000, and they do make that page rather difficult to find—it’s not linked to from their home page at all.

OLT.com

OLT.com offers free federal and state tax returns up to a $73k AGI cap. Above that, it’s $9.95. They also have a $7.95 premium edition that offers phone/chat support and audit support.

Pricing Summary

I really don’t understand why Intuit draws a 5-10x premium over the other providers. Their service is somewhere between somewhat better and slightly worse depending on which metric you go by; certainly not 5-10x better!

Service	Cost (federal)	Cost (state)	Cost (if eligible for Free File)	Free File AGI Cap
UsTaxes.org	$0	$0	$0	N/A
OpenTaxSolver	$0	$0	$0	N/A
TurboTax	$0-$300+	$49.99+	N/A	No Free File Offer
FreeTaxUSA	$0	$14.99	$0	$41k
OLT.com	$0	$9.95	$0	$73k

Results Summary

Turns out, especially for fairly straightforward returns like mine, most services do a pretty comparable job. Each paid service got me the same return on each form to the dollar. OpenTaxSolver got me the same federal return after a few attempts, and UsTaxes.org offered me a $47 lower refund thanks to not supporting deductions for line 12a.

Service	Federal return	State Return	M1PR
UsTaxes.org	$(X-47)	N/A	N/A
OpenTaxSolver	$X	N/A	N/A
TurboTax	$X	$Y	$Z
FreeTaxUSA	$X	$Y	$Z
OLT.com	$X	$Y	$Z

(Dollar values omitted for privacy reasons.)

User Experience

UsTaxes.org

Definitely the best UI, narrowly edging out TurboTax. They use Material UI, which feels very familiar to me, and have clearly put thought into things like tab indexing, so navigating around feels very natural. I didn’t try filling it out on a mobile device (I’m not that much of a masochist), but their mobile UI seems very passable as well.

There’s very little friction to getting started: No account to create, no terms of service or privacy policy to agree to, no application to install; just visit the website and get started.

Also, I noticed it was a bit quicker to run through my forms too. Many of the other services had me copying over data from every box of every form I received, but UsTaxes.org seemed to only have me extract the relevant bits.

There’s an open issue tracking progress on allowing federal return submission through FreeFileFillableForms, but otherwise UsTaxes.org expects you to print out and mail in your forms, which isn’t nearly as convenient as e-filing.

OpenTaxSolver

OpenTaxSolver was the only piece of software with which I made mistakes – and I did it last, so I knew what I was looking for. I had to go back twice and correct values I’d omitted or misplaced. Unfortunately, this does just speak to the confusing interface of OpenTaxSolver. As its website says, it’s definitely targeted at people who have a high degree of familiarity with the forms it’s filling out for you, though at that point I’m not sure why I wouldn’t want to just fill out forms myself.

Here’s one place I messed up: I received a 1099-B from Pearson’s employee stock purchase plan. The 1099-B, and every other software, refers to these fields, among others:

Description
Date acquired
Date sold
Proceeds
Cost or other basis
Type of gain/loss (short or long term)

Here’s OpenTaxSolver’s interpretation of those fields:

I have no doubt that if I had to file my taxes weekly or even monthly, I’d be using OpenTaxSolver. However, I’m not prepared to learn a domain-specific language (even if it’s super cool!) for a set of forms I have to fill out once annually.

OpenTaxSolver expects you to print and mail your tax returns.

OLT.com

Form fields are a little bit odd. Certain elements edit hyphens and such in with JavaScript. The phone number fields are all formatted as two fields; one for the area code, and another for the exchange and subscriber number (so a number like (320) 555-1212 would be represented in two fields: [ 320 ][ 5551212 ]), which led to me reliably either typing all ten digits of phone numbers into the area code box, or hitting tab again after the exchange code (555 in this example). The tab order for addresses is “Address Line”, “ZIP Code”, “City”, “State”, which often leads me to begin typing the city in the ZIP field. (This is sort of reasonable, though, since they do attempt to auto-fill the city and state from the ZIP Code.)

I mistakenly clicked to add a form which I didn’t have. The site wouldn’t allow me to move on from that page until I’d filled out the title and the TIN, which, of course, I didn’t have! It did let me log out and log back in, which apparently reset the session sufficiently to allow me to continue.

FreeTaxUSA

Pleasant to use. Not over-the-top fancy like TurboTax, nor as barebones as OLT. I got a single attempt to upsell during the process, and likely would have had one or two more if I attempted to file with them. (However, I find a $7 Deluxe package a lot less offensive than the several-hundred-dollar upgrades that Intuit tries to push!)

TurboTax

Prompts me to enter a phone number (skippable, but not obvious) every time I log in.

Spams me with email after signing up. It’s possible I’m a dummy, and somehow clicked something to sign me up for Mint, but I never intentionally navigated to Mint, so these emails are just all unsolicited Intuit spam:

Auto-import is…almost nice? Except I have to enter employer TIN, Box 1 of my W-2, and the box d (“Control number”) of my W-2, and then proofread—at this point, I might as well just fill in all the information myself! I don’t consider this much of a benefit at all; I don’t really think it saved me any time, though it was able to import the majority of the information I needed.

The number of animations, interstitials, and aggressive upgrade prompts I had to sit/click through really tested my patience. It’s incredible they’re leading the market with such a user-hostile product.

How much legal gunk I had to read to use each service

Service	`cat privacy_policy terms_of_use \| wc` (lines, words, characters)
UsTaxes.org	0
OpenTaxSolver	0
TurboTax	`1006 24328 155006`
FreeTaxUSA	`183 6901 43255`
OLT.com	`323 8250 52270`
Franz Kafka’s Metamorphosis (for scale)	`2266 25094 142017`

So TurboTax’s privacy policy and terms of use is both longer and more boring than Kafka’s Metamorphosis. Hmm.

So, is open-source tax software ready?

Honestly, as much as I’d like to say otherwise, no, it’s not. If your tax situation is exceptionally simple, then you can use UsTaxes.org—but TurboTax’s free product should cover most of those cases too, and will allow you to e-file. If you’re eligible for Free File, either OLT.com or FreeTaxUSA is perfectly fine.

If you have a lot of patience and interest in understanding what you’re doing, you can use OpenTaxSolver—but you can also fill in the forms yourself, and again, FreeFileFillableForms will allow you to e-submit here.

The great thing about open source software is that I can contribute! I’ve been working on some PRs to UsTaxes.org (which unfortunately didn’t make it in before tax day, but may help late filers), and I’ve been playing around with getting Minnesota’s state returns supported. Hopefully, in another year or two, it will be in a state where I’ll feel comfortable recommending it to everyone. And, of course, the sooner TurboTax dies a flaming death, the happier I’ll be!

Unless you work in the tax industry, you probably don’t need citations on this one. But there’s a ton of excellent writing out there!

From NPR:
From ProPublica’s Series “The TurboTax Trap”:
Elsewhere:
↩︎
One interesting thing I picked up this time around is that the IRS will send you alternate forms of communication. Besides the usual standard print letters, there’s large print and braille, both of which I’d expect, and the option to send MP3 files and text/braille files on a USB drive, which I did not expect!

From https://www.irs.gov/forms-pubs/about-form-9000:
I elect to receive written communications from the IRS in the following accessible format. Check only one. Forms with more than one box checked will not be processed.
- 00 Standard Print (Cancels prior election)
- 01 Large Print
- 02 Braille
- 03 Audio (MP3)
- 04 Plain Text File (TXT)
- 05 Braille Ready File (BRF)
Note: You will also receive a standard print copy.
↩︎
Compiling the source for OpenTaxSolver did give me my first introduction into git-svn, which is pretty great!
```
git svn clone https://svn.code.sf.net/p/opentaxsolver/SrcCodeRepo/ opentaxsolver
```
↩︎
```
% du -hs opentaxsolver/trunk/*
1016K	opentaxsolver/trunk/OTS_2017
7.2M	opentaxsolver/trunk/OTS_2018
6.9M	opentaxsolver/trunk/OTS_2019
7.0M	opentaxsolver/trunk/OTS_2020
9.0M	opentaxsolver/trunk/OTS_2021
```
Compare this to USTaxes.org’s software, written in Javascript:
```
% du -hs node_modules
656M	node_modules
% find node_modules | wc -l
93728
```
Plus Rust stuff, for Tauri, not known for its compile speed.

Building OpenTaxSolver didn’t make me install or download anything except the gcc/make toolchain, and gtk+-2.0, all of which I already had installed, and occupies more than an order of magnitude less disk space. ↩︎
Nice. ↩︎

The Holy (Midwestern) Trinity: Culvers, Kwik Trip, and Menards

Sat, 16 Apr 2022 00:00:00 +0000

If you live in the Upper Midwest, you know what I mean: There’s no place like Culvers. And Kwik Trip. And Menards. (All Wisconsin-based, for that lovely hometown charm.) But the worst part of visiting all three is the period of time between stepping out of one and arriving at another. Where is the best location we can minimize this pain?

First, obligatory memes:

I grew up near Hutchinson, MN. Hutchinson gained a Menards in 2001¹, a Culvers in 2005², and a Kwik Trip in 2013³. They’re all within a few blocks of each other—a good start for minimizing the total trip time of a Culver’s–Kwik Trip–Menards lap.

But is there a location that has a closer clustering? Here we brute force every Culver’s/Kwik Trip/Menards combination, checking the sum of their distances and finding the top ten results. (We use the sum of distances as a rough estimate of the stores’ total distance apart; one could also find the “center of mass”—the point where the sum of distances to each of the other points is minimal—and find the sum of distances to the other points, or any number of other scoring methods.)

import json
import math

# Should take ~20 seconds to run

# http://edwilliams.org/avform147.htm#Dist
def dist(store1, store2):
    lat1 = store1['lat'] * math.pi/180
    lon1 = store1['long'] * math.pi/180
    lat2 = store2['lat'] * math.pi/180
    lon2 = store2['long'] * math.pi/180
    distance_in_radians = 2*math.asin(math.sqrt((math.sin((lat1-lat2)/2))**2 + 
                math.cos(lat1)*math.cos(lat2)*(math.sin((lon1-lon2)/2))**2))
    earth_radius = 3959 # freedom units (miles)
    return distance_in_radians * earth_radius

def total_dist(store1, store2, store3):
    return dist(store1, store2) + dist(store2, store3) + dist(store3, store1)

# Format an address
def a(x):
    return  f"[{x['address']}, {x['city']}, {x['state']} {x['zip']}]({x['website']})"

if __name__ == '__main__':
    # A naïve approach might be to check for each city how close the Culver's,
    # Kwik Trip, and Menards are (if a city has all three). However, this would
    # break if there happen to be close clusterings across city lines. Instead,
    # we make a list of the locations of each chain, and process that. (While
    # that does make a difference for some results, none of them make it into
    # the top 10.)
    with open("stores.json", 'r') as f:
        stores = json.loads(f.read())

    culverses = list(filter(lambda s: s['chain'] == "Culver's", stores))
    kwiktrips = list(filter(lambda s: s['chain'] == "Kwik Trip", stores))
    menardses = list(filter(lambda s: s['chain'] == "Menards", stores))

    top_matches = []

    for culvers in culverses:
        for kwiktrip in kwiktrips:
            # This is a hack to speed the program up; without this the program
            # takes hours to run. With this optimization, it takes <1 minute.
            if abs(kwiktrip['lat'] - culvers['lat']) + abs(kwiktrip['long'] - culvers['long']) > 2: # More than ~50 miles apart
                continue
            for menards in menardses:
                if abs(kwiktrip['lat'] - menards['lat']) + abs(kwiktrip['long'] - menards['long']) > 2:
                    continue
                top_matches.append({
                    "culvers": culvers,
                    "kwiktrip": kwiktrip,
                    "menards": menards,
                    "dist": total_dist(culvers, kwiktrip, menards),
                })
                top_matches.sort(key=lambda s: s['dist'])
                top_matches = top_matches[:10]

    print("| Rank | Culver's | Kwik Trip | Menards | Total distance |")
    print("|-|-|-|-|-|")
    for rank, top_match in enumerate(top_matches, start=1):
        print(f"| {rank} | {a(top_match['culvers'])} | {a(top_match['kwiktrip'])} | {a(top_match['menards'])} | {top_match['dist']:.2f} miles |")

Rank	Culver’s	Kwik Trip	Menards	Total distance
1	1101 2nd St S, Waite Park, MN 56387	106 10Th Ave S, Waite Park, MN 56387	251 10Th Ave S, Waite Park, MN 56387	0.39 miles
2	3604 University Dr, Muscatine, IA 52761	3605 University Dr, Muscatine, IA 52761	3408 N Highway 61, Muscatine, IA 52761	0.43 miles
3	525 Village Walk Ln, Johnson Creek, WI 53038	465 Village Walk Ln, Johnson Creek, WI 53038	440 Wright Rd, Johnson Creek, WI 53038	0.53 miles
4	W6606 State Rd 23, Fond du Lac, WI 54937	1123 W Johnson St, Fond Du Lac, WI 54937	1200 Rickmeyer Dr, Fond Du Lac, WI 54937	0.58 miles
5	2270 Westowne Ave, Oshkosh, WI 54904	1090 N Washburn St, Oshkosh, WI 54904	2351 Westowne Ave, Oshkosh, WI 54904	0.74 miles
6	1510 Montreal St, Hutchinson, MN 55350	10 Denver Ave Se, Hutchinson, MN 55350	1525 Montreal St Se, Hutchinson, MN 55350	0.74 miles
7	1499 Lawrence Dr, De Pere, WI 54115	1620 Lawrence Dr, De Pere, WI 54115	1313 Lawrence Dr, De Pere, WI 54115	0.86 miles
8	1601 E 1st St, Grimes, IA 50111	2351 E 1St St, Grimes, IA 50111	300 Ne Destination Dr, Grimes, IA 50111	0.89 miles
9	3048 5th Ave S, Fort Dodge, IA 50501	3121 5Th Ave S, Fort Dodge, IA 50501	3319 5Th Ave S, Fort Dodge, IA 50501	0.93 miles
10	2520 Folsom St, Eau Claire, WI 54703	2327 N Clairemont Ave, Eau Claire, WI 54703	3210 N Clairemont Ave, Eau Claire, WI 54703	1.18 miles

Waite Park, MN comes out on top:

with Muscatine, IA a close second:

And Hutchinson does make the list! The 0.74 miles for a Culvers-Menards-Kwik Trip lap puts it in 6th place.

While we have all this fun data, let’s check out a few more things! I’ve noticed that Menardses and Culver’ses seem to have a habit of being located right next to each other though I have yet to see a Culver’s in a Menards, the way you might see a Subway in a Walmart, or something). This was true in Hutchinson, Eden Prairie⁴, Burnsville/Savage, Richfield/Bloomington – every Menards I’ve ever been to, I believe, except Apple Valley.

It’s very possible that this is just confirmation bias, though – it’s not true in Apple Valley, at least. When else isn’t this the case? That is, what cities have the biggest distances between their Culver’s and Menards?

import json
import math
from find_closest_trio import dist, a

if __name__ == '__main__':
    most_widely_spaced_pairs = []
    with open("stores.json", 'r') as f:
        stores = json.loads(f.read())

    culverses = list(filter(lambda s: s['chain'] == "Culver's", stores))
    menardses = list(filter(lambda s: s['chain'] == "Menards", stores))

    for culvers in culverses:
        for menards in menardses:
            if culvers['city'] == menards['city'] and culvers['state'] == menards['state']:
                most_widely_spaced_pairs.append((culvers, menards))
                most_widely_spaced_pairs.sort(key=lambda x: dist(x[0], x[1]), reverse=True)
                most_widely_spaced_pairs = most_widely_spaced_pairs[:10]

    print("| Rank | Culver's | Menards | Distance |")
    print("|-|-|-|-|")
    for rank, pair in enumerate(most_widely_spaced_pairs):
        print(f"| {rank+1} | {a(pair[0])} | {a(pair[1])} | {dist(pair[0], pair[1]):.2f} miles |")

Rank	Culver’s	Menards	Distance
1	4701 Kentucky Ave, Indianapolis 46221	7145 E 96Th St, Indianapolis 46250	19.41 miles
2	7953 State Line Rd, Kansas City 64114	3701 Nw 90Th St, Kansas City 64154	18.84 miles
3	7105 E 96th St, Indianapolis 46250	7140 S Emerson Ave, Indianapolis 46237	18.28 miles
4	8232 Country Village Dr, Indianapolis 46214	7145 E 96Th St, Indianapolis 46250	17.85 miles
5	5020 W 71st St, Indianapolis 46268	7140 S Emerson Ave, Indianapolis 46237	17.36 miles
6	1444 Rentra Dr, Columbus 43228	6800 E Broad St, Columbus 43213	17.14 miles
7	11050 S. Doty Avenue W, Chicago 60628	2601 N Clybourn Ave, Chicago 60614	16.82 miles
8	11050 S. Doty Avenue W, Chicago 60628	4501 W North Ave, Chicago 60639	16.38 miles
9	575 W Layton Ave, Milwaukee 53207	8110 W Brown Deer Rd, Milwaukee 53223	16.07 miles
10	5115 Shear Avenue, Indianapolis 46203	7145 E 96Th St, Indianapolis 46250	15.77 miles

Unsurprisingly, it’s just a bunch of really big cities with multiple stores across their respective metro areas. (With some cities with widely spread locations highly overrepresented, too!) Here’s Indianapolis:

Finally, since I have all this neat data, here’s a map of every Culvers, Kwik Trip, and Menards.

Here's how I retrieved the data I used:

#!/usr/bin/python3

import csv
import requests
import json # TODO: remove

stores = []

# Culvers
print("Searching for Culver's locations")
response = requests.get('https://hosted.where2getit.com/lite?action=getstategeojsonmap&appkey=EC84928C-9C0B-11E4-B999-D8F4B845EC6E').json()

state_names = {}
for f in response['states']['features']:
    state_names[f['properties']['name']] = f['properties']['regiondesc']

for s in response['labels']['features']:
    if s['properties']['num_stores'] > 0:
        state = s['properties']['name']
        print(f"Searching {state}...", end="")
        json_data = {
            'request': {
                'appkey': '1099682E-D719-11E6-A0C4-347BDEB8F1E5',
                'formdata': {
                    'geolocs': {
                        'geoloc': [
                            {
                                'addressline': state,
                                'state': state_names[state],
                            },
                        ],
                    },
                    'stateonly': 1,
                },
            },
        }

        response = requests.post('https://hosted.where2getit.com/culvers/rest/locatorsearch', json=json_data)
        count = 0
        for store in response.json()['response']['collection']:
            # I tried a bunch of other things and this is the only one that matched :facepalm:
            if "coming soon" in store['name'].lower(): # store['comingsoondate']: # not store['dine_in'] and not store['takeout']: # not store['opendate']: # store['comingsoondate']:
                # print(f"{store['name']} not yet open")
                continue
            stores.append({
                'chain': "Culver's",
                'lat':  float(store['latitude']),
                'long': float(store['longitude']),
                'address': store['address1'],
                'city': store['city'],
                'state': store['state'],
                'zip': store['postalcode'],
                'website': store['url'],
            })
            count += 1
        print(count)
        if not count == s['properties']['num_stores']:
            print(f"Inequal for {state}: {count} != {s['properties']['num_stores']}")
print(f"""{len(stores)} locations found""")


# Kwik Trip
# Export to CSV from https://www.kwiktrip.com/Maps-Downloads/Store-List
print("Searching for Kwik Trip locations")
kwiktrip_count = 0
with open('stores.csv') as f:
    reader = csv.DictReader(f)
    for row in reader:
        kwiktrip_count += 1
        stores.append({
            'chain': "Kwik Trip",
            'lat':  float(row['Latitude']),
            'long': float(row['Longitude']),
            'address': row['Address'].title(),
            'city': row['City'].title(),
            'state': row['State'],
            'zip': row['Zip'],
            'website': f"https://www.kwiktrip.com/locator/store?id={row['Store Number']}",
        })
print(f"{kwiktrip_count} locations found")


# # Menards
print("Searching for Menards locations")

# Visit https://www.menards.com/store-details/locator.html; view source; find a
# value for `data-initial-stores`; copy its value into data here. Incapsula, the
# DDoS mitigation platform that Menards uses, makes this reeeeaaaally hard to do
# with `requests`. Those lines should look something like this:
#
# 125 | <meta
# 126 |     id="initialStores"
# 127 |     data-initial-stores="[{&quot;number&quot;:3132,..."
# 128 |   >
# 129 | </head>
#
# (line 127 is the one you want here)
data="[]"

menardses = json.loads(data.replace("&quot;", '"'))
for menards in menardses:
    stores.append({
        'chain': "Menards",
        'lat':  float(menards['latitude']),
        'long': float(menards['longitude']),
        'address': menards['street'].title(),
        'city': menards['city'].title(),
        'state': menards['state'],
        'zip': menards['zip'],
        'website': f"https://www.menards.com/store-details/store.html?store={menards['number']}",
    })
print(f"{len(menardses)} locations found")

with open('stores.json', 'w') as f:
    f.write(json.dumps(stores))

Source: Eric says so. (The Menards API used to expose this information, but has since stopped.) ↩︎
https://weblink.hutchinsonmn.gov/WebLink/DocView.aspx?id=16157&dbid=0&repo=Hutchinson&cr=1 p.36-38 ↩︎
https://mblsportal.sos.state.mn.us/Business/SearchDetails?filingGuid=3adf3dfd-6c16-e311-8e3a-001ec94ffe7f ↩︎
The Culver’s here moved across the street a few years ago; it’s now on the north side of 212. However, the Culver’s restaurant, while updated to the correct address, still has the wrong coordinates: https://www.culvers.com/locator?key=220 ↩︎

Lessons learned from implementing the BitTorrent protocol

Thu, 10 Mar 2022 00:00:00 +0000

Last week, Jeff and I had some questions about some specifics about how the BitTorrent protocol works. Over the course of a conversation, I realized that, despite years of using torrents to download Linux ISOs¹, I had at best a surface-level understanding of the protocol. That’s a shame, because peer to peer file sharing involves some really neat tech! One of the best ways to understand how something works is to build it, so I wrote a toy BitTorrent client that implements the bare minimum necessary to download a file.

Protocol overview

At a high level, here’s how the BitTorrent protocol works, in the simplest case:

I browse to Debian’s CD torrent download page and download a .torrent file, which contains a tracker URL and a list of checksums for the list of pieces I’ll be downloading—all of the information needed to retrieve the actual file.
I import this .torrent file into a torrent client.
The client connects to the tracker’s url, provides the SHA1 hash associated with the torrent file. The tracker adds me to its list of peers associated with that hash, and returns to me a list of other peers that may be sharing that file.
The client connects to those peers, and identify with the hash what file we want to download. When a peer is ready, it “unchokes” us, and begins sending chunks of the file as we request them, typically in a random-ish order.
Data comes in from as many sources as we can pull from until we have the entirety of the file.
As a good internet citizen, I continue to seed the file I’ve just received to other peers in the swarm so others are able to download from me.

`femtotorrent` project scope

If this isn’t the world’s worst BitTorrent client, it must certainly be in the running! That’s okay though; it was only intended to be a learning tool, and not a production-ready software suite.

Features it lacks (in no particular order):

Support for multi-file torrents
Support for >4GB torrents
Support for multiple peers
Support for peers that don’t have the full file
Support for peers disconnecting in the middle of the download
An option to choose where to save the file
Seeding back to the swarm
Support for magnet links
A UI
Support for torrents that aren’t Debian 11 (hardcoded in)
Protocol extensions
- Distributed Hash Table – this allows for trackerless torrents
- Peer Exchange
- Any kind of UDP
- Compact peer lists
A smart choking algorithm for congestion management

Features it has (alphabetized for convenience):

Can download a file. Usually.

Notes on the protocol

The BitTorrent protocol is brilliant. It’s a great example of software that does what it was designed for incredibly well. It also does it in such a way that, once it exists, makes one say, “Why didn’t I think of that?”. It’s a simple and effective solution that fits its problem domain very well.

The original release of the BitTorrent application was in July 2001. I, naturally, was closely following every development in the space, having recently turned four years old at the time. While my commentary is limited by a lack of context in many ways, I do find it interesting to understand many of the decisions in context. “Why bencoding? Why not just JSON-encode everything?” Well, Chandler, you see, JSON didn’t exist yet! (Or, at least, JSON.org and the JSON spec wouldn’t come to be until 2002. But apparently YAML was already around²?)

Resources available

Unfortunately, the thing that drew me to implement the BitTorrent spec also ended up being a thorn in my side the whole time I implemented it: The spec, printed, would fit on about five and a half pages. Unfortunately, it’s written in a way that leaves a lot of details ambiguous and hard to understand. For example, one thing that stood out to me as particularly confusing was the specifics on the length prefix on messages. As far as I can tell, quoted here is the entirety of what the protocol doc has to say about message length prefixes:

peer protocol

[…]

The peer wire protocol consists of a handshake followed by a never-ending stream of length-prefixed messages.

[…]

All later integers sent in the protocol are encoded as four bytes big-endian.

[…]

That’s it for handshaking, next comes an alternating stream of length prefixes and messages. Messages of length zero are keepalives, and ignored. Keepalives are generally sent once every two minutes, but note that timeouts can be done much more quickly when data is expected.

peer messages

All non-keepalive messages start with a single byte which gives their type.

I eventually got this figured out, but it took a lot of looking back and forth between my code and the spec to figure out why I wasn’t getting the responses I expected. Some specifics that weren’t obvious:

To start out, I entirely missed the length prefix note altogether; it’s not grouped with other information that I need to send a message. While the messages are described in some detail ("‘request’ messages contain an index, begin, and length. …"), there’s no mention of the length prefix here.
How is the length prefix structured?
Does the length prefix include the length of its own bytes?
A keepalive isn’t actually of length zero; it includes the length…right?
If “all non-keepalive messages start with a single byte”, but also all messages start with a length…which comes first?

With the benefit of hindsight, most of this ends up being reasonably clear. I’d been conflating what I was thinking of as a message (“the thing that gets sent across the wire”, possibly Layer 5; includes a length prefix) with what the spec calls a “message” (“the thing after the length prefix that starts with a byte indicating its type”). The length is the length of the message that follows. All perfectly clear now, but far from it on the first many read-throughs!

Also, I was tripped up by two specific points on bencoding that despite repeated careful reading I can’t find in the spec. First, dicts’ keys must be strings, which I assumed (because I didn’t want to deal with map[interface]interface{}s everywhere!) but didn’t see verified. Second, dict keys are ordered alphabetically.

D’oh! I need to get my vision checked. As I was proofreading this post, I went back to check one more time, and found it, right there, right where I should have been looking, both points, clearly spelled out:

Dictionaries are encoded as a ’d’ followed by a list of alternating keys and their corresponding values followed by an ’e’. For example, d3:cow3:moo4:spam4:eggse corresponds to {‘cow’: ‘moo’, ‘spam’: ’eggs’} and d4:spaml1:a1:bee corresponds to {‘spam’: [‘a’, ‘b’]}. Keys must be strings and appear in sorted order (sorted as raw strings, not alphanumerics).

Credit to the author(s) on this one, I guess!

Other helpful resources

I ended up writing a full first pass at the entirety of the implementation using just the original spec. I deliberately didn’t “cheat” by looking at code from other implementations; this was partially a challenge to myself to see if I could do it from the spec alone.

However, once I had prototyped out the code and started debugging the flaws in my implementation, I ended up being confused on a handful of points, and I did reach out for some additional resources. (Specifically, length prefixes, as I implemented them, included the four bytes of the length prefix as part of the length, causing Transmission, the client on the other end of the connection, to expect more data and not respond.)

Third party documentation turned out to be much more enlightening (and verbose) than the official spec. (https://wiki.theory.org/BitTorrentSpecification in particular is a fantastic resource, and one I’d highly recommend to anyone else looking to implement the protocol.) They provide essentially the same information that the spec does, but more clearly spelled out, and in a way that (to me) was much more obvious on a first read. Particularly, I found their descriptions of choking/unchoking and declaring interest much more enlightening than the original spec’s descriptions.

Wireshark, as recommended by the BitTorrent spec, ended up being very helpful for showing what packets I was sending and receiving, comparing them to a known-good implementation of the spec, and dissecting the parts of the response to validate that my parser was parsing properly.

Interestingly, I would have done a poor job of predicting in advance what resources I would make the most use of. Ol’ reliable, the debugger, barely came into play; my issues for the most part weren’t in understanding what my code was doing, but rather what it was supposed to be doing.

Instead, one of the most (unexpectedly) helpful tools ended up being Transmission, the torrent client I was testing against. The peer list that you get from a client is surprisingly detailed, while still helpful, and (beyond the basics) lists information like whether or not we’re choked or interested, what the state of the connection is, and how much we’ve uploaded/downloaded.

Takeaways

Even though I didn’t come up with a complete implementation, I was able to answer the question I’d had at the beginning: Why don’t BitTorrent trackers get hit by DMCA takedown requests? Turns out, it’s awfully hard to enforce copyright on a map of a 20-byte SHA1 hash to a list of IP addresses (and not even the hash of the tracked file; rather, the hash of some data including hashes of the file’s pieces!). Trackers don’t know the contents of the torrent, nor even the name. (Plus, a torrent can contain multiple trackers, pull peer information from the swarm via DHT, and exchange peer lists with other peers, which in total severely limit the ramifications of a tracker being taken offline, even if it’s legally possible.)

Specifics aside, I wish I had done this years ago! Writing a program from a (simple) spec and getting it to interoperate with other software is a really fun and enlightening project. It was a great way to dive into a problem area I didn’t know much about and learn a bunch about how it works (plus a chance to write lower level code than I often otherwise might!). The protocol wasn’t, perhaps, as nicely specified as I’d have hoped, but it wasn’t bad, either; it’s relatively simple, especially if I’m not trying to implement every feature perfectly.

One of the neat things about having done a few of this type of project is that it really acts as a great confidence booster when I’m considering implementing new protocols. I’d been reading about CardboardWM, and a month ago I never would have considered what it would take to write my own window manager. But now? …maybe that’s next!

Normally I’d expect this to be a joke, where “Linux ISOs” is a euphemism for “things I’ve pirated”, but strangely enough in my case it happens to be the truth! I don’t watch much for movies or TV; I use nearly exclusively Free and Open Source Software (including not playing much for games, and I’ll gladly pay for those I do!); and streaming music services have solved the “Piracy is almost always a service problem and not a pricing problem” problem to such a degree that Spotify is worth its cost as a library management tool alone, to say nothing about compensating artists! ↩︎
The first draft I could find of the YAML spec was captured by the Internet Archive on March 30, 2001. It bears limited resemblance to modern YAML. Some time later, the YAML editors decided that draft wasn’t worth mentioning, and later editions (linked: the first edition with an example which, upon replacing tabs with spaces, parses as modern YAML) don’t bother mentioning that anything from before May 15 ever existed. ↩︎

How not to send an email asking for personal information

Thu, 10 Feb 2022 00:00:00 +0000

May-July 2017: Equifax is hacked. September 7, 2017: Equifax discloses the breach. July 24?, 2019: I make a claim against the settlement fund, since my information was involved in the breach. January 27, 2022: I receive an email from the Equifax Breach Settlement Administrator telling me how I can claim the credit monitoring I selected. January 30, 2022: I spend several hours trying unsuccessfully to decide if the email is legitimate or not.

As a professional in a computer-related field, I spend a lot of time fielding computer-related questions, and trying to help friends and family understand how to safely use technology. It’s a constant struggle—there are a lot of scams out there, and new ones popping up all the time. I have a few pieces of fairly general advice I tend to hand out for people looking to see if an email they’ve received is legitimate or not, summarized here:

Check if the domain is known to be trusted¹. For example, a password reset email from noreply@google.com may be safe, but an email from noreply@google.account-service.info is almost certainly out to get you.
Check if the links lead where you expect. If I have an email with the FROM header² as noreply@accounts.google.com, but all the links lead to http://555.867.530.900/… (a personal favorite “IP address”—big scare quotes here—that Pearson gave as part of an internal phishing awareness training; spot the Tommy Tutone reference!), then it’s a malicious link.
If you see something you’re not suspecting, or your gut tells you something is suspicious, it probably is. Make sure to validate anything you’re being sent, usually by looking it up on the website that the email claims to be from. (“Call the number on the back of your card” is a good example of this.) Or check in with an expert! I encourage my friends and family to forward me anything they’re unsure about, if they want a second opinion on if it’s legit or not.
And, of course, have caution proportionate to the risk. I’m less concerned about losing my Spotify password than I am my bank’s password, and it would be a red flag if a link claiming to be from Spotify started asking me about my bank info.

The email I received telling me how to claim my credit monitoring is…terrible. It violates every single principle in the list above, to such a comical degree that I felt compelled to write it up. If you need to send out an email, just do the opposite of what they did, and you’ll probably be good! Let’s see how they violate every single point I’ve laid out above:

Check if the domain is known to be trusted.

An email from Equifax should come from equifax.com or a subdomain. Well, I guess this isn’t from Equifax, it’s from some law firm or something. Okay, a quick Google search says equifaxbreachsettlement is right. (And, as a person who loves using the view-source button, I can also verify that SPF, DMARC, and DKIM all pass, but this took me a while to remember how to do (it’s just in the headers), and wouldn’t be something I’d expect most people to check.)

Authentication-Results: aspmx1.migadu.com;
	dkim=pass header.d=equifaxbreachsettlement.com header.s=smtp header.b=SqYR676H;
	dmarc=pass (policy=quarantine) header.from=equifaxbreachsettlement.com;
	spf=pass (aspmx1.migadu.com: domain of "bounce+17882d.ae1aa8-chandler=chandlerswift.com@equifaxbreachsettlement.com" designates 69.72.40.1 as permitted sender) smtp.mailfrom="bounce+17882d.ae1aa8-chandler=chandlerswift.com@equifaxbreachsettlement.com"

Check if the links lead where you expect.

equifaxidworks.com is the type of domain I’d register if I were trying to phish Equifax customers. I can’t get equifax.com, of course, but I can tack some other words onto the end, and call that my site. My current leading contender is experianidentityprotection.com, which is available for <$9 at the time of writing, but any number of others would be equally convincing.

  <td>Visit the Experian IdentityWorks Website:=20
  <a href=3D"https://email.equifaxbreachsettlement.com/c/eJxFjk2PgjAURX8NLE=
n7-CgsuiCKExbiRF3Mjjza19CI6JRO6s8fJmomuYub3OScqyUSRyxjK4EBMA6CCyagSrggbVLNO=
ZKpMj1EGaPvH2vwMThCNS7k_URXmn2ibtd4lIpXxVAwQMQsL_M8LZVJixIyMMSFwniSo_f3JUrr=
CHZrQggJPe7kLM5Wh5u7LH-odXmJ_hWxk2rEWU_k1h_vugRrnnYvm91Xvzk22_a8P3Tt-XDsN_X=
-s24_uv7UdNu--gX9ZUxy">www.experianidworks.com/equifaxsettlement</a></td>

Oh, and that’s just the text of the link—the actual link doesn’t lead there. They’ve obscured the actual link behind a tracking redirect, so even the location I already thought was suspicious isn’t what I see. Nice.

If something is suspicious…

Well, it kinda ticks all the boxes: “claim your free ______”, “time limited”, “click here”, “enter all your personal information, including DOB and SSN”…

Validate information on the website the email claims to be from.

In this case, the email is from equifaxbreachsettlement.com, so their website should have some note that they’re currently sending out these emails, and that you’ll be able to claim your credit monitoring at experianidworks.com in the email they’ll send. But https://www.equifaxbreachsettlement.com mentions experianidworks.com…exactly zero times. Not once! The closest they come is in the FAQ, where they say

I received an email regarding free, three-bureau (Equifax, Experian, and TransUnion) credit monitoring from Experian for four years. Is this email legitimate?

If you made a valid claim for Credit Monitoring Services, you will receive an email from the Equifax Data Breach Settlement Administrator (from the email address info@equifaxbreachsettlement.com) by February 25, 2022 providing you with information on how to activate your credit monitoring. The Settlement Administrator will provide you with an activation code and link to the Experian website where you can enroll and activate your Credit Monitoring Services.

But none of that would prevent an enterprising scammer from realizing that these are emails going out, and sending out their own emails using the same template, that link to another service. (As mentioned above, experianidprotection.com is available for $9, and looks no less trustworthy than experianidworks.com!)

Well, if equifaxbreachsettlement.com doesn’t give me any information, let’s see if Experian links there? Not that I can find. They do have a product called “Experian IdentityWorks™” but that entire registration flow happens on the experian.com domain, as I’d have expected.

How about a quick web search? Well, all the results for “experian id works” and a few other related search queries either just return links to the experianidworks.com page, or to similarly named pages on the experian.com domain that experianidworks.com seems to be a clone of, or reviews/articles talking about Experian’s credit monitoring program (again, named IdentityWorks, but all on the experian.com domain, bolstering my suspicion that this could just be a domain impersonating Experian).

Hm. Let’s try the command line, and make sure that Experian is actually the one to have registered the domain:

% whois experian.com
[...]
Domain Name: EXPERIAN.COM
Registry Domain ID: 5503875_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.networksolutions.com
Registrar URL: http://networksolutions.com
Updated Date: 2021-08-30T16:21:45Z
Creation Date: 1996-10-28T05:00:00Z
Registrar Registration Expiration Date: 2022-10-27T04:00:00Z
Registrar: Network Solutions, LLC
Registrar IANA ID: 2
Reseller: 
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Registry Registrant ID: 
Registrant Name: Information Solutions, Experian
Registrant Organization: Experian Information Solutions, Inc.
Registrant Street: 475 ANTON BLVD
Registrant City: COSTA MESA
Registrant State/Province: CA
Registrant Postal Code: 92626-7037
Registrant Country: US
Registrant Phone: +1.7148307000
Registrant Phone Ext: 
Registrant Fax: 
Registrant Fax Ext: 
Registrant Email: domainadmin@experian.com
[...]
% whois experianidworks.com
[...]
Domain Name: experianidworks.com
Registry Domain ID: 1978500106_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.corporatedomains.com
Registrar URL: www.cscprotectsbrands.com
Updated Date: 2021-11-07T01:04:51Z
Creation Date: 2015-11-11T13:43:29Z
Registrar Registration Expiration Date: 2022-11-11T18:43:29Z
Registrar: CSC CORPORATE DOMAINS, INC.
Sponsoring Registrar IANA ID: 299
Registrar Abuse Contact Email: domainabuse@cscglobal.com
Registrar Abuse Contact Phone: +1.8887802723
Domain Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
Registry Registrant ID: 
Registrant Name: Domain Management
Registrant Organization: Consumerinfo.com, Inc.
Registrant Street: 535 Anton Blvd, Suite 100
Registrant City: Costa Mesa
Registrant State/Province: CA
Registrant Postal Code: 92626
Registrant Country: US
Registrant Phone: +1.8003570821
Registrant Phone Ext: 
Registrant Fax: +1.8003570821
Registrant Fax Ext: 
Registrant Email: DNSAdmin@consumerinfo.com
[...]

So they’re not registered by the same organization…or even the same registrar! There’s no link on equifaxbreachsettlement.com or experian.com to this site; it’s registered by and through a different organization than the base Experian domain; it’s on a domain that looks like it’s intended to be deceptive…there are a lot of reasons not to trust this domain at this point.

Have caution proportionate to the risk.

Well, it may be sketchy, but at least they ask for all my personal information, including my full name, email address, mailing address, date of birth, and some common security questions and answers!

Oh, and news organizations are somewhere between useless and actively harmful.

Perhaps unsurprisingly, I’m not the only one to be asking whether this is legitimate, so there are a decent number of answers out there.

Snopes has a decent answer, but they sort of hide any nuance behind an “is this email legit” question and a big green “Yes!” at the top, which doesn’t really encourage people to appreciate that it’s not necessarily a yes/no question; but that there’s a bit more subtlety behind it.

Another of the top links on a web search for “experianidworks”, at the time of writing, was https://www.10news.com/news/fact-or-fiction/fact-or-fiction-equifax-settlement-email-is-legit, which has the following advice:

We’re clearing up some confusion over an email you may have received.

The email appears to concern a settlement over the 2017 Equifax data breach.

It offers you 4 free years of the credit monitoring service Experian Identity Works.

The body of the email contains a link to Experian’s site in which you have to provide a lot of personal information.

But don’t worry, it’s legit.

This is, in my opinion, exceptionally bad advice, given without qualification. (Remember, only a Sith deals in absolutes!) A more nuanced take might look something like

That link should lead to experianidworks.com. If it does, then it’s legit.

As usual, Reddit does a much better job. Here’s u/lunarchuck:

I received the email yesterday at my gmail address. Whois lookup on experianidworks.com produces a registration address (535 Anton Blvd, Suite 100, Costa Mesa, CA, 92626, US) that appears to be associated with Experian per BBB. Googling the domain brings up a bunch of legitimate sources that link to it (universities, state governments).

Gmail headers show that the sending domain equifaxbreachsettlement.com passes dkim and dmarc and the sending ip address passes SPF for that domain. And that domain is linked from the FTC site.

I think it’s legit. It would have to be one of the all-time greatest phishing/hacking scams to not be, especially since it’s been out there for at least 3 days.

That is, “Given this domain, here’s what I checked. It all checks out; here’s how to reproduce my results; and despite checking, I’m still not going to claim it’s absolutely safe, so you should still use appropriate caution”—exactly what I’d have wanted from any other analysis.

So it’s legit?

Yeah, I think so. Despite the total lack of effort of anyone involved to help reassure me that it’s legit, I’ve filled in my information and feel relatively comfortable having done so. Plus, in the time between when I received this email and did the original checking, and when I wrote the post, there’s a lot of new, better information out there.

The clincher, for me, which should have been plastered all over the place, is a link to the FTC’s Equifax settlement page, which mentions both info@equifaxbreachsettlement.com as the email source and https://www.experianidworks.com/equifaxsettlement as the link’s target.

That said, for a settlement regarding poor security practice by a company that definitely should have known better, this is embarrassing! They fail every single heuristic I’d use to verify that this is legit. And, perhaps more importantly, they’re working against the community of security professionals who try to educate people on good security practice. Do better!

One issue I’m glossing over here is that to a person who hasn’t spent a lot of time around computers, it’s certainly non-obvious which bits of a URL are important and which aren’t. I trust google.com/*, and account-info.google.com/* (or I would, if I didn’t happen to know that Google uses accounts.google.com for account-related pages), but I definitely wouldn’t trust google.account-info.com. And watch out for the particularly devious ones like accounts.google.com-helpandsupport.tk/…! The thing that makes this especially difficult is that I can’t simply say to look at the beginning of the URL, because that’s just a subdomain. I can’t say to look at the end, because that’s not the host. I have to explain to look for the first slash after https://, and then look at whatever’s one (or two, in cases like *.gov.uk) dot-separated chunk before that. Or suggest to read MDN’s “What is a URL?”³—or, perhaps, read any of the relevant RFCs (or better yet, the Living Spec on URLs)? Riiiiiight, I’m trying to save people the hassle of needing a degree in networking to not leak their bank password!

This is one reason I’m generally not opposed to browsers drastically reducing the amount of information they display in the URL bar: It’s a pain for people who do want all that information, but for most people, the query string isn’t information they care about. ↩︎
As a reminder, the FROM line in an email is just text, and I can set it to whatever I want (though this is getting better! SPF, DKIM, and DMARC are all strides in the right direction). Eric and I had a fun time in high school spoofing emails from school administration to our friends, which somehow didn’t turn out poorly for anyone involved. ↩︎
Despite my tone in the rest of that paragraph, I’m a huge fan of that document. MDN does a surprisingly excellent job of explaining these concepts in an approachable way. Despite the technical nature of the topic, I think I would feel comfortable sending my mother this document if I thought she really needed to know a bit more about the nitty-gritty of URL structure than she currently does. ↩︎

Azure Active Directory SAML sample code/configuration

Fri, 14 Jan 2022 00:00:00 +0000

At work, I occasionally have to integrate things I’ve written with Azure Active Directory. Unfortunately, their Go documentation is somewhere between light and nonexistent, and there are a few quirks that make it not immediately obvious how this works, so here’s an example.

This example is based off the example code in crewjam/saml, but tweaked to work with Microsoft’s Azure Active Directory SAML authentication.

The full code is available at https://github.com/ChandlerSwift/azure-saml-example.

Code

package main

import (
	"context"
	"crypto/rsa"
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"net/http"
	"net/url"

	"github.com/crewjam/saml/samlsp"
)

func hello(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello, %s!", samlsp.AttributeFromContext(r.Context(), "http://schemas.microsoft.com/identity/claims/displayname"))
}

func main() {
	keyPair, err := tls.LoadX509KeyPair("myservice.crt", "myservice.key")
	if err != nil {
		panic(err) // TODO handle error
	}
	keyPair.Leaf, err = x509.ParseCertificate(keyPair.Certificate[0])
	if err != nil {
		panic(err) // TODO handle error
	}

	idpMetadataURL, err := url.Parse("https://login.microsoftonline.com/<your tenant UUID>/federationmetadata/2007-06/federationmetadata.xml")
	if err != nil {
		panic(err) // TODO handle error
	}
	idpMetadata, err := samlsp.FetchMetadata(context.Background(), http.DefaultClient,
		*idpMetadataURL)
	if err != nil {
		panic(err) // TODO handle error
	}

	rootURL, err := url.Parse("http://localhost:8000")
	if err != nil {
		panic(err) // TODO handle error
	}

	samlSP, _ := samlsp.New(samlsp.Options{
		EntityID:    "spn:<your application's Client UUID>",
		URL:         *rootURL,
		Key:         keyPair.PrivateKey.(*rsa.PrivateKey),
		Certificate: keyPair.Leaf,
		IDPMetadata: idpMetadata,
	})
	app := http.HandlerFunc(hello)
	http.Handle("/hello", samlSP.RequireAccount(app))
	http.Handle("/saml/", samlSP)
	http.ListenAndServe(":8000", nil)
}

Generating a Certificate Pair

We use roughly the same command as in the upstream example, but use a .crt extension rather than .cert, since Azure’s certificate upload process doesn’t recognize .cert files. The common name is not used as part of the authentication process, so can be anything.

openssl req -x509 \
    -newkey rsa:2048 \
    -keyout myservice.key \
    -out myservice.crt \
    -days 365 \
    -nodes \
    -subj "/CN=myservice.example.com"

Setting up Auth with an Azure Active Directory Application

Unlike SAMLtest, Azure Active Directory doesn’t accept a metadata upload, so we need to take a few extra manual steps in the Portal.

Generate a certificate pair as above. Ensure your application is set to use the generated files.
In the Azure Portal, go to Azure Active Directory, and select “App registrations” from the menu on the left. Click “New Registration”. Choose a name. This example is written as a single tenant application. Under Redirect URI, select “Web” and enter “http://localhost:8000/saml/acs”. Click “Register”.
Copy the “Application (client) ID” from the main page of the app registration you’ve created. Put this where the code sample says <your application's Client ID>.
From “Endpoints”, copy the “Federation metadata document” URL. It should be https://login.microsoftonline.com/<your tenant UUID>/federationmetadata/2007-06/federationmetadata.xml. Put this on line 29 for the idpMetadataURL’s value. (This is what your application will use to discover the endpoints for other functions.)
On the “Certificates & secrets” page, Select the “Certificates” tab and upload the certificate from step 1.
Compile and run!

Notable changes

Without the EntityID (which defaults to the empty string), the following error is seen:

AADSTS700016: Application with identifier 'http://localhost:8000/saml/metadata' was not found in the directory 'mydirectory'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.

If the EntityID is simply set to the UUID (without the spn: prefix), the request succeeds, but the SAML library rejects the response as invalid, because the response, which is compared to the EntityID, does contain the prefix.

assertion invalid: assertion Conditions AudienceRestriction does not contain "<your application's UUID>"

(This comparison is made in service_provider.go.)

The response that Microsoft returns doesn’t have a cn attribute; instead, we use the http://schemas.microsoft.com/identity/claims/displayname attribute.

Caveats

We notably didn’t upload our certificate. Given this use case (this application doesn’t make actual requests once we have the user’s name and email address), this isn’t a security concern, but I’m not sure it’s safe in all cases. (Though I have only an afternoon’s understanding of SAML, so don’t take my word for it!)

Hats

Tue, 19 Oct 2021 00:00:00 +0000

As of about 11pm yesterday, I’m the proud owner of 0hats.com.

On Friday, I received a link to a document hosted on 17hats.com, which as far as I can tell is some kind of CRM. 17 is a fun number¹, and the concept of wearing 17 is a silly one and got a chuckle out of me. On a whim, I checked if 19hats.com (the next largest prime quantity of hats) was taken. It was. 23hats? Taken. 29hats? Also taken. Well then, what’s the smallest numbered hat quantity that I could register as my own domain?

A first attempt went something like this:

% for i in {1..50}; do dig ${i}hats.com | grep -i nxdomain >/dev/null && echo ${i}hats.com is available; done
5hats.com is available
16hats.com is available
26hats.com is available
28hats.com is available
30hats.com is available
35hats.com is available
36hats.com is available
37hats.com is available
41hats.com is available
43hats.com is available
44hats.com is available
46hats.com is available
48hats.com is available
49hats.com is available
%

(That’s not exactly right, as 5hats.com is registered, but doesn’t have a DNS entry associated with it.)

Anyway, I ended up getting well and truly nerd sniped by the problem—how many of these are registered? What are they used for? What’s the smallest number of hats whose domain name I could own?

So a few hours of Go code later, 0hats.com ~~graced~~ belabored the world with its presence.

In short, for each domain from 0hats.com to some limit (I’ve chosen 50, rather arbitrarily):

Checks if the domain is registered with a whois lookup, and if it is, notes the owner
Does a DNS lookup to see if there may be anything hosted at the site
Opens a geckodriver instances, browses to the page, grabs the title, and takes a screenshot.
and then renders the whole thing to one big list.

I’ve already snagged the smallest possible non-negative quantity of hats, but if anyone is interested, 16hats.com is available at the time of writing!

I also did a very basic scan of {51..1000} hats; it seems that registrations drop off pretty quickly after 50.

package main

import (
	"fmt"

	"github.com/likexian/whois"
	whoisparser "github.com/likexian/whois-parser"
)

func main() {
	for i := 0; i < 1000; i++ {
		domainName := fmt.Sprintf("%vhats.com", i)
		// Check if domain is registered
		query_result, err := whois.Whois(domainName)
		if err != nil {
			panic(err)
		}
		_, err = whoisparser.Parse(query_result)
		if err == whoisparser.ErrNotFoundDomain {
			fmt.Println(domainName, " is available")
		} else if err != nil {
			panic(err)
		} else {
			fmt.Println(domainName, " is taken")
		}

	}

}

% time go run main.go >/dev/null
go run main.go  2.58s user 1.96s system 1% cpu 5:03.92 total
% go run main.go | grep "taken" | wc -l  # How many in total?
77
% go run main.go | tail +52 | grep "taken" | wc -l  # How many above 50?
39
% go run main.go | tail +52 | grep "taken"
51hats.com  is taken
52hats.com  is taken
58hats.com  is taken
66hats.com  is taken
68hats.com  is taken
69hats.com  is taken
73hats.com  is taken
80hats.com  is taken
83hats.com  is taken
86hats.com  is taken
88hats.com  is taken
89hats.com  is taken
97hats.com  is taken
99hats.com  is taken
100hats.com  is taken
101hats.com  is taken
112hats.com  is taken
123hats.com  is taken
208hats.com  is taken
210hats.com  is taken
234hats.com  is taken
241hats.com  is taken
247hats.com  is taken
252hats.com  is taken
300hats.com  is taken
310hats.com  is taken
360hats.com  is taken
365hats.com  is taken
386hats.com  is taken
420hats.com  is taken
444hats.com  is taken
500hats.com  is taken
501hats.com  is taken
586hats.com  is taken
710hats.com  is taken
777hats.com  is taken
802hats.com  is taken
906hats.com  is taken
911hats.com  is taken

The answer to this is very simple. It was a joke. It had to be a number, an ordinary, smallish number, and I chose that one. Binary representations, base thirteen, Tibetan monks are all complete nonsense. I sat at my desk, stared into the garden and thought ‘42 will do’ I typed it out. End of story.

— Douglas Adams, on the choice of 42

↩︎

Hacking together a workaround for Audio issues

Sat, 02 Oct 2021 00:00:00 +0000

I’ve had a longstanding issue somewhere in my system’s config where when I adjust my system volume using the hardware wheel, the left and right channels get out of sync. It’s certainly something I can work around, but it’s frustrating to have to either have to turn the volume all the way down, which resynchronizes the channels, or to open alsamixer or similar and manually adjust the right and left channels to match.

Here’s me spamming the volume wheel, and watching the issue in alsamixer:

It’s an inconvenient reality that sometimes fixing the root cause¹ of the issue just isn’t nearly as easy as patching together a service that addresses the symptom! (This was also a good excuse to gradually introduce myself to ALSA’s libraries; I’m not sure I would have figured out the root cause of the issue if I hadn’t been playing with this project.) This program periodically checks the volume of the default output’s left channel, and adjusts the right channel’s volume to match:

// sudo pacman -Sy alsa-lib
#include <alsa/asoundlib.h>
#include <time.h>

// Based on the code at
// https://stackoverflow.com/questions/6787318/set-alsa-master-volume-from-c-code
int main()
{
	// Set up the mixer for card "default"
	snd_mixer_t *handle;
	snd_mixer_open(&handle, 0);
	snd_mixer_attach(handle, "default");
	snd_mixer_selem_register(handle, NULL, NULL);
	snd_mixer_load(handle);

	// Obtain the mixer element "Master" from the mixer
	snd_mixer_selem_id_t *sid;
	snd_mixer_selem_id_alloca(&sid);
	snd_mixer_selem_id_set_index(sid, 0);
	snd_mixer_selem_id_set_name(sid, "Master");
	snd_mixer_elem_t *elem = snd_mixer_find_selem(handle, sid);

	for (;;) {
		// If any events (say, volume changes) happened since the last
		// iteration, update our handle to reflect that. Without this, no volume
		// changes will be picked up:
		// https://www.raspberrypi.org/forums/viewtopic.php?t=175511
		snd_mixer_handle_events(handle);

		// Find the left channel's volume, and sync it with the right channel
		long left_ch_vol;
		snd_mixer_selem_get_playback_volume(elem, SND_MIXER_SCHN_FRONT_LEFT, &left_ch_vol);
		snd_mixer_selem_set_playback_volume(elem, SND_MIXER_SCHN_FRONT_RIGHT, left_ch_vol);

		// Don't monopolize the CPU; it's okay if the levels desync for 100ms
		nanosleep(&(struct timespec){0, 100 * 1000 * 1000}, NULL); // 100ms
	}

	// Clean up
	snd_mixer_close(handle);
	handle = NULL;
	snd_config_update_free_global();
}

For the time being, I’m running this as a systemd user service. (Since this exists to deal with an issue in my sway config, it could also be run with an exec line in my sway config.)

chandler@xenon ~ % systemctl status --user volume-balancer.service
● volume-balancer.service - Keep L/R channel volumes in sync
     Loaded: loaded (/home/chandler/.config/systemd/user/volume-balancer.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-10-02 20:00:56 CDT; 42s ago
   Main PID: 218353 (volume-balancer)
      Tasks: 2 (limit: 19096)
     Memory: 1.1M
        CPU: 217ms
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/volume-balancer.service
             └─218353 /home/chandler/bin/volume-balancer

Oct 02 20:00:56 xenon systemd[2288]: Started Keep L/R channel volumes in sync.
chandler@xenon ~ % cat .config/systemd/user/volume-balancer.service
[Unit]
Description=Keep L/R channel volumes in sync

[Service]
ExecStart=/home/chandler/bin/volume-balancer
Type=simple
Restart=always

[Install]
WantedBy=default.target
chandler@xenon ~ %

Update 2024-04-18: Hajo Noerenberg informs me that this can be made substantially more efficient by waiting for a mixer event rather than polling inputs:

- // Don't monopolize the CPU; it's okay if the levels desync for 100ms
- nanosleep(&(struct timespec){0, 100 * 1000 * 1000}, NULL); // 100ms
+ // Wait for a mixer to become ready (i.e. at least one event pending)
+ snd_mixer_wait(handle, 1000);

Hajo also provides a much cleaner solution for a similar problem:

In the meantime I’ve released a slightly more complex daemon to watch for audio activity and volume changes:

https://github.com/hn/linkplay-a31/blob/main/openwrt-linkplay-a31/linkplay-emu/src/linkplay-emu.c

Might be helpful for some people perhaps.

Thanks, Hajo!

It seems that amixer runs into a race condition when run multiple times concurrently. This happens as a result of my sway config; amixer is called on each “press” of the XF86AudioRaiseVolume key, which may happen begin the previously called amixer instance terminates. ↩︎

Retrieving Kwik Trip Gas Prices

Mon, 16 Aug 2021 00:00:00 +0000

Kwik Trip (a popular gas station in the MN-WI-IA area, if you’re not from the Midwest) has an API to display their fuel prices, among other things. Here’s a short Python program to retrieve pricing of nearby stores.

import requests

# Burnsville, MN
lat=44.767778
long=-93.2775

params = {
    'latitude': lat,
    'longitude': long,
    'limit': 20 # 1-255
}
stores = requests.get("https://api.kwiktrip.com/api/stores/nearby", params=params).json()["stores"]

for store in stores:
    has_gas = [p for p in store['properties'] if p['name'] == 'GAS'][0]
    if not has_gas:
        continue

    storedata = requests.get(f"https://api.kwiktrip.com/api/location/store/information/{store['id']}").json()

    unleaded_price = [p for p in storedata["fuel"] if p["type"] == "UNLEADED"][0]
    location_str = f'{store["address"]["city"].title()}, {store["address"]["state"]}'
    print(f'{store["id"]} ({location_str}): {unleaded_price["currentPrice"]}')

Sample output (as of date of publishing):

585 (Burnsville, MN): 3.099
695 (Apple Valley, MN): 2.999
309 (Burnsville, MN): 3.099
179 (Eagan, MN): 3.099
406 (Apple Valley, MN): 3.059
694 (Lakeville, MN): 3.049
663 (Savage, MN): 3.099
421 (Apple Valley, MN): 3.059
343 (Lakeville, MN): 3.059
397 (Apple Valley, MN): 3.059
447 (Shakopee, MN): 3.099
443 (Farmington, MN): 3.059
281 (Prior Lake, MN): 3.099
692 (Lakeville, MN): 3.099
697 (Rosemount, MN): 3.099
662 (Eagan, MN): 3.099
693 (Farmington, MN): 3.099
441 (Shakopee, MN): 3.049
492 (Chanhassen, MN): 3.099
178 (South St Paul, MN): 3.099

If you want pricing information on all Kwik Trip stores, it’s a bit more difficult, as the limit parameter on the /stores/nearby request is capped at a maximum of 255. In this case, a CSV of all Kwik Trip stores can be found at https://www.kwiktrip.com/Maps-Downloads/Store-List (click Export to CSV above the table), and that can be used as a basis for the requests:

import csv
import requests

with open('stores.csv') as f:
    reader = csv.DictReader(f)
    for row in reader:
        if row["Sells Gas"] == "No":
            continue
        if row["State"] != "MN":
            continue
        r = requests.get(f"https://api.kwiktrip.com/api/location/store/information/{row['Store Number']}").json()
        unleaded_price = [p for p in r["fuel"] if p["type"] == "UNLEADED"][0]
        print(f'{row["Store Number"]} ({row["City"]}, {row["State"]}): {unleaded_price["currentPrice"]}')

Output

804 (ALBERT LEA, MN): 3.049
805 (ALBERT LEA, MN): 3.049
875 (ALBERT LEA, MN): 3.049
1020 (ALBERTVILLE, MN): 3.049
812 (ANDOVER, MN): 3.099
1017 (ANOKA, MN): 3.099
474 (ANOKA, MN): 3.039
406 (APPLE VALLEY, MN): 3.059
695 (APPLE VALLEY, MN): 2.819
397 (APPLE VALLEY, MN): 3.099
421 (APPLE VALLEY, MN): 3.059
250 (AUSTIN, MN): 3.099
689 (AUSTIN, MN): 3.099
246 (AUSTIN, MN): 3.099
445 (AUSTIN, MN): 3.099
330 (BELLE PLAINE, MN): 3.049
168 (BIG LAKE, MN): 3.149
925 (BLAINE, MN): 3.049
1022 (BLAINE, MN): 2.999
896 (BLAINE, MN): 3.049
1014 (BLAINE, MN): 2.999
646 (BLUE EARTH, MN): 3.089
880 (BROOKLYN PARK, MN): 3.099
448 (BROOKLYN PARK, MN): 3.099
458 (BUFFALO, MN): 3.039
309 (BURNSVILLE, MN): 3.099
585 (BURNSVILLE, MN): 3.099
1045 (BYRON, MN): 3.149
622 (BYRON, MN): 3.149
code 733 (CALEDONIA, MN): 3.099
571 (CARLTON, MN): 3.149
1095 (CARVER, MN): 2.989
492 (CHANHASSEN, MN): 3.099
402 (CHANHASSEN, MN): 3.099
886 (CHASKA, MN): 3.099
608 (CHATFIELD, MN): 3.079
874 (CHISAGO CITY, MN): 3.059
206 (CIRCLE PINES, MN): 3.099
104 (CLEARWATER, MN): 3.099
247 (CLOQUET, MN): 3.149
234 (CLOQUET, MN): 3.149
144 (CLOQUET, MN): 3.149
1120 (COLD SPRING, MN): 2.999
465 (COON RAPIDS, MN): 3.059
497 (DAKOTA, MN): 3.049
1031 (DELANO, MN): 3.099
789 (DODGE CENTER, MN): 3.149
273 (DULUTH, MN): 3.099
274 (DULUTH, MN): 3.099
941 (DULUTH, MN): 3.099
218 (DULUTH, MN): 3.099
117 (DULUTH, MN): 3.099
224 (DULUTH, MN): 3.059
489 (DUNDAS, MN): 3.099
179 (EAGAN, MN): 3.099
662 (EAGAN, MN): 3.099
753 (EYOTA, MN): 3.099
424 (FAIRMONT, MN): 3.049
744 (FARIBAULT, MN): 3.099
745 (FARIBAULT, MN): 3.099
793 (FARIBAULT, MN): 3.099
772 (FARIBAULT, MN): 3.099
693 (FARMINGTON, MN): 3.099
443 (FARMINGTON, MN): 3.059
429 (FOREST LAKE, MN): 3.089
1049 (GLENCOE, MN): 3.149
848 (HARMONY, MN): 3.079
249 (HASTINGS, MN): 3.039
220 (HERMANTOWN, MN): 3.059
572 (HERMANTOWN, MN): 3.059
216 (HERMANTOWN, MN): 3.059
186 (HINCKLEY, MN): 3.199
825 (HOKAH, MN): 3.099
454 (HUGO, MN): 3.099
316 (HUTCHINSON, MN): 2.919
1019 (ISANTI, MN): 2.979
619 (KASSON, MN): 3.149
814 (KELLOGG, MN): 2.999
614 (LA CRESCENT, MN): 3.099
437 (LA CRESCENT, MN): 3.099
844 (LAKE CITY, MN): 3.049
600 (LAKE CITY, MN): 3.049
1078 (LAKE ELMO, MN): 3.099
248 (LAKE ELMO, MN): 3.099
343 (LAKEVILLE, MN): 3.059
692 (LAKEVILLE, MN): 3.099
694 (LAKEVILLE, MN): 3.049
737 (LEWISTON, MN): 3.059
449 (MANKATO, MN): 3.049
1011 (MANKATO, MN): 3.049
344 (MANKATO, MN): 3.049
334 (MANKATO, MN): 3.049
379 (MANKATO, MN): 3.049
275 (MANKATO, MN): 3.049
431 (MANKATO, MN): 3.049
1026 (MILACA, MN): 3.099
623 (MINNESOTA CITY, MN): 3.099
345 (MONTICELLO, MN): 3.149
177 (MONTICELLO, MN): 3.149
166 (MOOSE LAKE, MN): 3.199
1037 (MORA, MN): 3.079
926 (NEW PRAGUE, MN): 2.959
1090 (NEW PRAGUE, MN): 2.959
432 (NEW ULM, MN): 3.049
931 (NORTH BRANCH, MN): 3.099
930 (NORTH BRANCH, MN): 3.099
615 (NORTH MANKATO, MN): 3.049
385 (NORTHFIELD, MN): 3.099
854 (NORWOOD YOUNG AMERICA, MN): 3.099
111 (OAK PARK HEIGHTS, MN): 3.099
869 (OAKDALE, MN): 3.099
162 (OTSEGO, MN): 3.099
1088 (OWATONNA, MN): 2.929
403 (OWATONNA, MN): 2.959
641 (OWATONNA, MN): 2.929
806 (OWATONNA, MN): 2.929
237 (OWATONNA, MN): 2.929
435 (OWATONNA, MN): 2.929
598 (PAYNESVILLE, MN): 3.099
640 (PINE ISLAND, MN): 3.099
245 (PLAINVIEW, MN): 3.099
411 (PLYMOUTH, MN): 3.099
928 (PRINCETON, MN): 2.959
281 (PRIOR LAKE, MN): 3.099
481 (RED WING, MN): 2.999
305 (RED WING, MN): 2.999
376 (RED WING, MN): 2.999
590 (ROCHESTER, MN): 3.099
672 (ROCHESTER, MN): 3.099
279 (ROCHESTER, MN): 3.099
418 (ROCHESTER, MN): 3.099
659 (ROCHESTER, MN): 3.099
341 (ROCHESTER, MN): 3.099
382 (ROCHESTER, MN): 3.099
388 (ROCHESTER, MN): 3.099
321 (ROCHESTER, MN): 3.099
438 (ROCHESTER, MN): 3.099
335 (ROCHESTER, MN): 3.099
357 (ROCHESTER, MN): 3.099
364 (ROCHESTER, MN): 3.099
433 (ROCHESTER, MN): 3.099
464 (ROCHESTER, MN): 2.999
570 (ROCKFORD, MN): 3.099
697 (ROSEMOUNT, MN): 3.099
592 (RUSH CITY, MN): 3.149
609 (RUSHFORD, MN): 3.099
477 (SAINT BONIFACIUS, MN): 3.099
754 (SAINT CHARLES, MN): 3.029
120 (SAINT CLOUD, MN): 2.999
158 (SAINT CLOUD, MN): 2.999
146 (SAINT CLOUD, MN): 2.999
151 (SAINT CLOUD, MN): 2.999
150 (SAINT CLOUD, MN): 2.999
149 (SAINT CLOUD, MN): 2.999
943 (SAINT FRANCIS, MN): 2.939
147 (SAINT JOSEPH, MN): 2.999
575 (SAINT JOSEPH, MN): 2.999
681 (SAINT MICHAEL, MN): 3.099
466 (SAINT PETER, MN): 3.099
153 (SARTELL, MN): 2.999
154 (SAUK RAPIDS, MN): 2.999
663 (SAVAGE, MN): 3.099
441 (SHAKOPEE, MN): 3.049
447 (SHAKOPEE, MN): 3.099
178 (SOUTH ST PAUL, MN): 3.099
736 (SPRING GROVE, MN): 3.099
831 (SPRING VALLEY, MN): 3.149
213 (STACY, MN): 3.149
803 (STEWARTVILLE, MN): 3.149
414 (STEWARTVILLE, MN): 3.149
415 (STILLWATER, MN): 3.099
141 (TWO HARBORS, MN): 3.049
152 (VADNAIS HEIGHTS, MN): 3.099
843 (WABASHA, MN): 2.999
460 (WACONIA, MN): 3.099
137 (WAITE PARK, MN): 2.999
160 (WAITE PARK, MN): 2.999
1034 (WASECA, MN): 3.049
442 (WASECA, MN): 3.049
937 (WILLMAR, MN): 3.099
944 (WILLMAR, MN): 3.099
945 (WILLMAR, MN): 3.099
746 (WINONA, MN): 3.099
810 (WINONA, MN): 3.099
654 (WINONA, MN): 3.099
778 (WINONA, MN): 3.099
824 (WINONA, MN): 3.099
811 (WINONA, MN): 3.099
407 (WOODBURY, MN): 3.099
408 (WOODBURY, MN): 3.099
1028 (WORTHINGTON, MN): 3.099
463 (ZUMBROTA, MN): 2.989

Although, of course:

Setting up an ODROID-HC4 as a NAS

Fri, 07 May 2021 00:00:00 +0000

I bought a new NAS, a Hardkernel ODROID-HC4! It’s a real upgrade. Before: 6C/12T Xeon processor; After: 4C/4T ARM processor. Before: 12GB ECC RAM, After: 4GB RAM. Before: 6 HDD slots/SATA ports; After: 2. But most importantly:

Before: After:

I keep the Kill-A-Watt measuring the whole homelab, so those numbers are for the while rack. However, without any NAS at all the total sits just over 100W. That is, I cut my power consumption by something like forty percent by replacing one of my four machines—and not even the fastest, by a good margin!

With electricity prices just below the national average of $0.13/kWh, I generally figure 1W=$1/year. So if I cut my power consumption by a bit more than 70W, this should pay for itself in just over a year.

Note that this is steady-state power; I suspect numbers would go up under heavy access. However, my NAS spends 90+% of its time idle, so I’m most concerned about its performance there. I did see nearly 40W on startup, with both disks spinning up plus higher CPU usage, so that seems like a good upper bound. Even if it was at this state all the time, I’d still be using less than half of my previous power consumption.

Purchase process

There are several North American resellers of Hardkernel products, but their markup is big enough that I opted to buy directly from Hardkernel¹. I did have some issues the first time through the purchase process (the largest of which was that when I was filling out my info in the purchase form, many of the labels weren’t in English…until some reloads later, when they suddenly were!), but ended up making it through. The shipping was surprisingly fast for crossing an ocean; from clicking “ORDER” to opening the box took just under seven days.

Hardware impressions

Wow. Super slick. This thing is pretty great! The case is sturdy, and seems to support the drives well. The “toaster” form factor is kind of cute, and definitely serves its job well. I suspect it may not protect the drives as well as fully enclosed models, but on the upside it does do a nice job keeping good airflow for cooling.

Speaking of cooling, there’s a built in fan that I never once saw spin. I did a bit of compiling, and the temperature indicator budged from 50°C where it usually sat to nearly 60°C, but it never did actually break 60°. Perhaps if I stressed the CPU and GPU (not sure why, with a headless device) and put a good load on the disks so the power supply and PCIe controller are working, I might be able to get the fans to kick in, but under normal use it’s cool as a cucumber.

The CPU is reasonably speedy, and 4GB of memory isn’t anything to scoff at. I did some development of the OLED display driver software on the device itself, and though I eventually moved to cross compiling, this was more for toolchain reasons than for speed. The hardware is certainly more than sufficient for serving files over the network, as long as I’m not doing particularly heavy-duty encryption or compression.

Single-core performance for encryption was the one issue I did run into; scping files within my network did bottleneck on my single-core speeds, limiting me to about 60MB/s instead of the full 100-110MB/s that’s typical with a gigabit network. Of course, Comcast’s anemic upload speeds ensure I’ll never have to worry too much about that!

OLED Display

I opted to pay the extra $10 for the integrated OLED display. In hindsight, totally worth it! (It also adds a battery backed realtime clock, so that adds extra value too, but the display is easily worth it on its own.) While I was running some initial backups (in case I had problems switching over to the new NAS) I took the time to try to figure out how I wanted to use the display.

The wiki page for the ODROID-HC4’s OLED display links to some sample software, written in Python, that can be used for running the display. However, it’s fairly dependency heavy, and requires maintaining a Python installation. This seems like a fairly good candidate for a zero-dependency statically-compiled drop-in binary, so I set about writing a small app in Go to do what I wanted.

The result is on GitHub; I’m pretty happy with it!

Setting up software

I have in the past used FreeNAS (now TrueNAS CORE), but I was hoping to migrate away in favor of something I could manage more directly. Even if I had wanted to stay with FreeNAS, though, my hand would have been relatively forced since this board doesn’t have good support for FreeBSD. Linux support for ZFS is supposedly quite good at this point, and since Jeff set up his NAS in about 15 minutes on Void Linux, I figured I could make it work.

Hardkernel offers official images, but they seem to be stuck on Linux 4.9 (an LTS release from 2016) with no plans to move to more recent images. Also, while they do provide the kernel source code, I wasn’t able to find packaging info or build scripts for the images they produce. I prefer more transparency than that in the build process, so I went searching for something else.

Ideally, I’d have tried for a Debian install, but Debian’s ARM support seems patchy. My next attempt was Arch Linux ARM, but they don’t support the HC4. After some searching, it turns out Armbian seems to be the best supported distro for the HC4, so that’s what I chose.

After running through Armbian’s setup process, here’s the config I had to change to get the NAS set up the way I wanted:

sudo hostname nas

# Set up SSH
mkdir .ssh
curl https://github.com/chandlerswift.keys | head -n1 > .ssh/authorized_keys
sudo vim /etc/ssh/sshd_config # set PasswordAuthentication=no, PermitRootLogin=no
sudo service sshd restart # Apply new config

# Update packages
sudo apt update
sudo apt upgrade

# Set up ZFS
sudo armbian-config # Software > Headers_Install
sudo apt install zfs-dkms zfsutils-linux
sudo reboot # Could possibly `modprobe` instead?

Importing data

Now we’re ready for some disks! I had two 8TB disks set up in Freenas as a mirrored pool. Some historical baggage remained – the pool was still called Duluth from two moves ago². Plug the disks in, and let’s re-import the pool as nas:

# Still some remnants of prior living locations here!
sudo zpool import Duluth nas

Future steps

This isn’t terribly useful as a NAS yet. I can mount filesystems from Linux with SSHFS quite easily and with decent performance (assuming I remember to turn compression off, as single-core compression limits my throughput); but for MacOS or Windows clients I’m out of luck. With this in mind, I intend to get NFS or SMB (or both!) shares set up. This will also let me easily mount shares to VMs when I want to offload data from the VM hosts, and keep them backed up.

I also intend to set up an automated backup solution. I’ve had fairly good luck with pointing Duplicity at B2. I’ve also heard good things about Borg Backup, but I don’t believe that it directly supports targeting B2, and I don’t have a remote server to stash backups on.

Finally, I’m still not thrilled with Armbian; it’s done reasonably well, but I do prefer to trust the most important piece of server hardware in my house to a big-name distribution like Debian. I’m considering trying to get support for the ODROID-HC4 merged into Debian and/or Arch Linux ARM so I have a distribution I’m more directly familiar with to use. Also, after getting settled in with Armbian, I did see that Alpine is supposed to have decent aarch64 support, so that may be worth exploring as well.

I paid $79 for the device, $9.40 for the power supply, and $25.13 for shipping, totaling $113.53. AmeriDroid, who was the on-the-same-continent-with-a-website-entirely-in-a-language-I-can-read option, would have charged $79.95 for the base (no OLED) model, $12.95 for the power supply, and $15.29 for shipping, for a total of $108.19. I did want the display/RTC, and the prices were close, so I chose to buy straight from Hardkernel. ↩︎

root@nas:~# zfs get creation nas
NAME  PROPERTY  VALUE                  SOURCE
nas   creation  Fri Aug  9  8:59 2019  -

↩︎

Data mining my Spotify history

Thu, 29 Apr 2021 00:00:00 +0000

I have a few songs I listen to very frequently. Enough that I wonder, “Does the amount I stream this song have a noticeable effect on the song’s popularity?” It turns out, yes, there are at least a handful of songs for which that is the case.

First, the main findings:

Song	Listens/Total	Percentage of total
Nun komm, der Heiden Heiland, BWV 659	67/<1000¹	6.700%
Come To Mama	73/1118	6.530%
10 Pièces - Organ: Scherzo	68/1156	5.882%
Dandaya	83/1753	4.735%
How Long Does It Take?	44/<1000¹	4.400%
The Eye of the Hurricane	42/<1000¹	4.200%

Neat! I have four (and possibly more¹ ²) songs for which I account for at least 4% of the Spotify listens. Interestingly, none of these are among my top played tracks, some of which have hundreds of plays.

That said, there are two ways to get a high listen percentage: have my personal total be very high, or have the global total be very low. Given this, it’s not particularly surprising that a large number of the songs for which I have a high listen percentage have a very low total number of listens.

Methodology

The full source code I wrote and used is available on Github.

I have my listening history mostly recorded on last.fm, and Spotify provides total listen counts for songs, at least through their web client. Here’s an abbreviated runthrough of the code I used to find the data, ignoring most of the error handling and boilerplate.

#!/usr/bin/env python3

import pylast
import spotify.sync as spotify

Python has nice packages to do the heavy lifting here: pylast for last.fm and spotify.py for Spotify.

lastfm_top_tracks = lastfm_client.get_user("chandlerswift").get_top_tracks(stream=True, limit=None)

First, we get a generator that returns (lazily – no need to pull all 11000ish of my listens in one go!) the tracks I’ve played the most frequently, in descending order. (If you want to try this, note that limit=None does require pylast/pylast#367 to be merged.)

track_data=[]
for i, lastfm_track in enumerate(lastfm_top_tracks):
    spotify_track = spotify_client.search(
        f"{lastfm_track.item.artist.name} {lastfm_track.item.title}", types=["track"], limit=1
    ).tracks[0]

Then, for each track we’ve retrieved from last.fm, we find its corresponding track on Spotify. This can be issue-prone; tracks don’t necessarily have identical names between services, and often the same artist will have the same track across many albums. Apparently last.fm considers these to be the same, while Spotify counts them differently. Despite these problems, this generally seemed to work well.

    res = requests.get(f"http://localhost:8080/albumPlayCount?albumid={spotify_track.album.id}").json()

Here’s some magic! It turns out Spotify doesn’t provide a way to retrieve play count information from their API, so we have to use a third party tool. (I could likely have figured out what calls this tool makes, exactly, and integrated it, but that seemed more complex than integrating an extra tool into a one-off workflow.) I downloaded a .jar file from sp-playcount-librespot’s latest release, ran it, and directed my API requests there.

    found_track = None
    for spotify_disc in res['data']['discs']:
        for spotify_track_info in spotify_disc['tracks']:
            if spotify_track_info['name'].lower() == lastfm_track.item.title.lower():
                found_track = spotify_track_info

We end up having to extract the Spotify track from that album, since it doesn’t seem to be possible to get data for an individual track from sp-playcount-librespot. This naïve comparison did have some issues, but for the most part it worked fairly well.

    if found_track:
        found_track['my_playcount'] = lastfm_track.weight
        track_data.append(found_track)
        print(f"{i}. {found_track['name']}: {found_track['my_playcount']}/{found_track['playcount']} ({100*found_track['my_playcount']/found_track['playcount']:.3f}%)")
    else:
        find_manually.append(lastfm_track)
        print(f"No track {lastfm_track.item.title} found on album {spotify_track.album.name} (will find later)")

To each track, we tack on its “weight” (play count) from last.fm, and save it for later. We do a note of songs that we couldn’t find on the album—I’ll clean those up later. In the end, I wound up effectively running through this whole process again on the initially failed tracks, with a manual track comparison instead of an automatic one.

Data/Analysis

At this point, we’re done gathering the data; let’s see what we have from it. I ran the script with ipython -i main.py, so when I was done, it just dropped me into an ipython shell to manipulate the data as I wanted. I sorted it by percentages, and printed them out in the format used to generate the table at the beginning of the article:

track_data.sort(key=lambda track: track['my_playcount']/max(track['playcount'], 1000), reverse=True)
for i, track in enumerate(track_data[:50]):
    print(f"[{track['name']}](https://open.spotify.com/track/{track['uri'].split(':')[2]}) | {track['my_playcount']}/{track['playcount'] if track['playcount'] > 0 else '<1000'} | {track['my_playcount']/max(track['playcount'], 1000)*100:.3f}%")

[Nun komm, der Heiden Heiland, BWV 659](https://open.spotify.com/track/3iVaSPY04q6lm6yk1JZfw8) | 67/<1000 | 6.700%
[Come To Mama](https://open.spotify.com/track/4X4f9s89oqRzA6cbG3Qqpm) | 73/1118 | 6.530%
[10 Pièces - Organ: Scherzo](https://open.spotify.com/track/2QJDUqE12goajiRvX2ZGA9) | 68/1156 | 5.882%
[Dandaya](https://open.spotify.com/track/2OD2twxgZDvzzZk0LBFcG0) | 83/1753 | 4.735%
[How Long Does It Take?](https://open.spotify.com/track/5nJdSukxjQT2GMRroy2T4t) | 44/<1000 | 4.400%
[The Eye of the Hurricane](https://open.spotify.com/track/6HNfSCcXvcfzm8P0atsoYB) | 42/<1000 | 4.200%
[It's A Shame, It's A Mystery](https://open.spotify.com/track/3b92Si6WeUY3AtgIdIauIC) | 63/1757 | 3.586%
[Behind My Back](https://open.spotify.com/track/2HajmdZaKcSGjVVwkbDD7n) | 36/1095 | 3.288%
[Lucky Southern (Live)](https://open.spotify.com/track/35bMR4tsBzexE4sbfQ7iHW) | 27/<1000 | 2.700%
[Pussy Cat Moan](https://open.spotify.com/track/1trxN9IV3zuKKm8DwLq6MS) | 80/3103 | 2.578%
[You're Nobody 'Til Somebody Loves You](https://open.spotify.com/track/579seIobCvjKFMA5iWVzIZ) | 24/<1000 | 2.400%
[Imagine](https://open.spotify.com/track/4llK1G7gFgs9Of6TeblOU6) | 29/1351 | 2.147%
[Come Along and Join Me](https://open.spotify.com/track/3AlEiVaGtg9juGSdHhe20T) | 20/<1000 | 2.000%
[Sweet Inspirations](https://open.spotify.com/track/3ev9yqA7eEQlj5YX6Dkxj4) | 23/1175 | 1.957%
[Soul Shine](https://open.spotify.com/track/3ps0pyaq8qIS6S01cWhZ0l) | 57/2954 | 1.930%
[How High the Moon](https://open.spotify.com/track/5Lwk2DGagiaJ91HzKPcxjQ) | 18/<1000 | 1.800%
[I'm Happy With Me](https://open.spotify.com/track/3ZJ21R1Hj5CgqpWdEVpCpq) | 18/<1000 | 1.800%
[I Want To Be Happy](https://open.spotify.com/track/5c6OtW2SbbQQAAmIX9tZhk) | 18/<1000 | 1.800%
[I Don't Want To Hurt You Baby](https://open.spotify.com/track/31EyidBsi5nOeIKDXnUpaF) | 52/3042 | 1.709%
[You Gotta Move](https://open.spotify.com/track/1FRly0ajiFsnhDfI55hqu9) | 54/3374 | 1.600%
[To Dream The Impossible Dream](https://open.spotify.com/track/5U0NztxEG065s4p2eKJl6K) | 15/<1000 | 1.500%
[Mannenberg - Pts. 1 & II (Feat. Sons of Table Mountain)](https://open.spotify.com/track/7afjyhiW4mtBJnoLuXYO9I) | 37/2471 | 1.497%
[Jump Blues Jam Track in D_160 bpm](https://open.spotify.com/track/6VmbBxl8RIZQebBbM1NT7C) | 38/2670 | 1.423%
[You'll Never Walk Alone](https://open.spotify.com/track/05CePy9cYchqsf8HVmzrt7) | 18/1267 | 1.421%
[All Things Are Possible](https://open.spotify.com/track/2zpnKFy41KROv5AEAAtoAX) | 14/<1000 | 1.400%
[Gospel Beat](https://open.spotify.com/track/3xjdt4l1W7L8iVc1PkjMI8) | 14/<1000 | 1.400%
[Business is Tough (in Db)](https://open.spotify.com/track/0Cx1kdVuhZzD0zxrhpVwoc) | 183/13218 | 1.384%
[Schefel](https://open.spotify.com/track/72Wn6cgsupR0shGI70bCZ3) | 14/1047 | 1.337%
[If I Only Had a Brain](https://open.spotify.com/track/2ABlZfY0YhluHszFZzwzat) | 37/2792 | 1.325%
[Milestones](https://open.spotify.com/track/6qRngToVR7MQKjO4zh1ZV0) | 13/<1000 | 1.300%
[Let It Be](https://open.spotify.com/track/4LecnEWdIg6ZgQBUZjVBFJ) | 17/1338 | 1.271%
[Centerpiece](https://open.spotify.com/track/19baWZcpp71fvy6zJlzCZi) | 20/1590 | 1.258%
[Hero](https://open.spotify.com/track/4RRMcdtPxLJyUFGCRbGi3l) | 11/<1000 | 1.100%
[Why Did You Leave My Child?](https://open.spotify.com/track/7nHXsKpgWz7zSwG2cb8toD) | 11/<1000 | 1.100%
[A Chance To Breathe](https://open.spotify.com/track/6rEzSUOZiRr9AOiDXy2kRM) | 45/4325 | 1.040%
[Live In The Spirit](https://open.spotify.com/track/4f7rI4Su9pYkmFO94kq43d) | 10/<1000 | 1.000%
[Jesus, Oh What a Wonderful Child (In the Style of Mariah Carey) [Karaoke Version]](https://open.spotify.com/track/5RT29wO0uGJzUYm7tTmFrX) | 23/2407 | 0.956%
[Songs of Praise Toccata for Organ](https://open.spotify.com/track/4gx4oXLSZJ0qpCzZmzprIf) | 26/2829 | 0.919%
[Bye Bye Blackbird](https://open.spotify.com/track/7xCnMzJzNuPlU0ynd6u4An) | 9/<1000 | 0.900%
[Leave the Door Open](https://open.spotify.com/track/2FG2bGSmK2al4ZbMzxUw8m) | 28/3226 | 0.868%
[I Wish](https://open.spotify.com/track/5YtgljxatdsJNbVkus464N) | 17/2041 | 0.833%
[The Walking Wounded](https://open.spotify.com/track/43uFShzMWIOnJzqq9MoYLH) | 26/3175 | 0.819%
[Suite brève: IV. Dialogue sur les mixtures](https://open.spotify.com/track/0Y4cwqqIcvzcKSQsNhjkqf) | 8/<1000 | 0.800%
[It Had To Be You](https://open.spotify.com/track/6cSIcX545GS9kjxUrF9eW2) | 8/<1000 | 0.800%
[Atlanta Blue](https://open.spotify.com/track/3VZCyBekMReptNB5cwPLmp) | 8/<1000 | 0.800%
[I'm A Woman](https://open.spotify.com/track/2knT9LjDV2M8wgp2zAhHeM) | 34/4500 | 0.756%
[Honey It's Your Fault](https://open.spotify.com/track/3vfyT1JduiIdezBcG30ypL) | 28/3862 | 0.725%
[Bathtub Blues](https://open.spotify.com/track/2X0W1IZBy92uAYb91j510D) | 7/<1000 | 0.700%
[Cookin' At The Colonels](https://open.spotify.com/track/13wUXxTzGk939tcE80Al2w) | 7/<1000 | 0.700%
[Let's Have a Natural Ball](https://open.spotify.com/track/5om3jWnIwPyXtHygKVsUsI) | 7/<1000 | 0.700%

A few more questions: Of the songs I’ve listened to at least 5 times, how many have <1000 listens?

[f"{i}. {t['name']} by {', '.join([a['name'] for a in t['artists']])}" for i, t in enumerate(list(filter(lambda track: track['playcount'] == 0, track_data)))]

['0. Nun komm, der Heiden Heiland, BWV 659 by Johann Sebastian Bach, Matti Hannula',
 '1. How Long Does It Take? by Sista Monica Parker',
 '2. The Eye of the Hurricane by D Squared',
 '3. Lucky Southern (Live) by Thirteen Degrees',
 "4. You're Nobody 'Til Somebody Loves You by Swingin' Fireballs",
 '5. Come Along and Join Me by The Chancellors Quartet',
 '6. How High the Moon by Les DeMerle, Bonnie Eisele',
 "7. I'm Happy With Me by Sista Monica Parker",
 '8. I Want To Be Happy by The Carl Fontana - Arno Marsh Quintet',
 '9. To Dream The Impossible Dream by Sista Monica Parker',
 '10. All Things Are Possible by Sista Monica Parker',
 '11. Gospel Beat by Sista Monica Parker',
 '12. Milestones by The Carl Fontana - Arno Marsh Quintet',
 '13. Hero by Sista Monica Parker',
 '14. Why Did You Leave My Child? by Sista Monica Parker',
 '15. Live In The Spirit by Sista Monica Parker',
 '16. Bye Bye Blackbird by The Carl Fontana - Arno Marsh Quintet',
 '17. Suite brève: IV. Dialogue sur les mixtures by Jean Langlais, John Balka',
 '18. It Had To Be You by The Carl Fontana - Arno Marsh Quintet',
 '19. Atlanta Blue by Bill Walker, The Bill Walker Orchestra',
 '20. Bathtub Blues by Joe Scruggs',
 "21. Cookin' At The Colonels by Steve Einerson",
 "22. Let's Have a Natural Ball by The Blue In Blues",
 '23. Now the Green Blade Rises (arr. P. Manz for pipe organ) by J. M. C. Crum, Paul Manz',
 '24. Too Many Drivers at the Wheel by AJ Crawdaddy',
 '25. Singet frisch und wohlgemut op. 12,4 - II. by Hugo Distler, MonteverdiChor Muenchen, Konrad von Abel',
 '26. Peas Porridge Hot by Joe Scruggs',
 '27. Organ Symphony No. 1 in D Major, Op. 14: VI. Final by Louis Vierne, Fabien Chavrot',
 '28. Old Devil Moon by Michael Gott',
 "29. It's a Beautiful Day in the Neighborhood by Rich Szabo, Curtis McKonly, Mark Vinci, Bill Kirschner",
 '30. Dr. Mlk & Obama Impossible Dream Tribute by Sista Monica Parker',
 '31. Old Devil Moon by Michael Gott']

Only 32 of the 1989 tracks we’re inspecting; sounds like my tastes aren’t too obscure!

Spotify gives me the length of each song. What songs have I listened for the longest total time?

import datetime

track_data.sort(key=lambda track: track['my_playcount'] * track['duration'], reverse=True)
for i, track in enumerate(track_data[:10]):
    print(f"{i}. {track['name']}: {track['my_playcount']} listens at {datetime.timedelta(seconds=round(track['duration']/1000))}: {datetime.timedelta(seconds=round(track['my_playcount'] * track['duration'] / 1000))}")

0. You Look Good To Me: 690 listens at 0:04:52: 2 days, 7:58:46
1. Lucky Southern: 649 listens at 0:03:46: 1 day, 16:45:00
2. Change the World: 569 listens at 0:03:55 seco: 1 day, 13:07:19
3. Strasbourg / St. Denis: 473 listens at 0:04:39: 1 day, 12:36:30
4. Rock of Ages: 306 listens at 0:05:28: 1 day, 3:52:58
5. At Long Last Love - Live: 282 listens at 0:04:56: 23:12:39
6. Sultans of Swing: 229 listens at 0:05:50: 22:17:22
7. Mo' Better Blues (feat. Terence Blanchard): 281 listens at 0:03:39: 17:05:31
8. Here We Go Again: 231 listens at 0:03:58: 15:16:18
9. Love Me or Leave Me - 2013 Remastered Version: 270 listens at 0:03:21: 15:06:15

It looks like this list is still roughly in order of listen count, as one might expect. Most songs tend to be around the same length, so it makes sense that it wouldn’t be wildly different from the most-listened-to song list.

I did limit the script run to songs with at least 5 plays; however, a song with only 4 plays would need to be almost 4 hours long to make this list, so I think it’s safe to ignore those.

Potential improvements

A solution to many of the issues I’ve been having would be to use Spotify’s data on my listening history rather than last.fm’s.

Because I’d be correlating Spotify’s data with Spotify’s data, I wouldn’t have to worry about track title mismatches (the most notable being “¿Quién Será?” vs “Quien Sera?”; the most common being “[track title]” vs “[track title] - Remastered 20xx”). In this case, I could simply match the internal Spotify IDs (if that’s something their data dumps do actually provide).
Spotify should have my complete, precise listening history, compared to last.fm which only has the last few years’ worth.
Since I’m using Spotify’s search, rather than selecting a track by ID, occasionally I receive an obscure track on a weird album that isn’t what I’m looking for.

I did want to base this analysis on Spotify’s data, but I wasn’t particularly patient, and Spotify seems to try to barely scrape under GPDR’s 30-day deadline for delivering data exports.

And finally, this isn’t an improvement, but worth mentioning: While reading through Spotify’s API docs, I found that they expose an Audio Analysis for a Track (if the link doesn’t take you there, reload or search the page for “Get Audio Analysis for a Track” – anchors don’t always seem to work on the first page load), which includes information like time and key signature! This is probably enough data that I’d be able to write the music player feature of which I’ve long been dreaming: Create playlists with no key changes between songs!

Oh man, let me tell you all about it!

Any song has a starting key signature and an ending key signature. For many songs, these are the same. Those are the easy ones. Start with a song in the key of D, say, and play more songs in the key of D. Here's an example from the last time I seriously thought about this:

The hard part comes in when songs have one or more modulations (changes in key signature, essentially) in the middle. As a relatively quick example, you could run through the following songs:

Track	Artist	Album	Start Key	End Key
This Will Be (An Everlasting Love)	Natalie Cole	Inseparable	B♭	D♭
I Just Called To Say I Love You - Live/1995	Stevie Wonder	The Complete Stevie Wonder	D♭	E♭
You’re Nobody ‘Til Somebody Loves You	Swingin’ Fireballs	Live in Bremen	E♭	G
Beyond the Sea (La Mer)	George Benson	20/20	G	A♭
My Buick, My Love and I - Bonus Track	Seth MacFarlane, Elizabeth Gillies	In Full Swing	A♭	F
The Liberty Bell March	John Philip Sousa	Last Night of the Proms	F	B♭

Each one begins in the key the other left off in, creating what is (for me) a relatively seamless listening experience! I had abandoned this project years ago when I figured I’d have to manually classify every song I wanted to listen to, but if Spotify will do do it for me…problem succesfully avoided!

My ideal listening client, then, would be an improvement on the standard “shuffle” functionality. Instead of picking songs completely at random, it would pick song n+1 from the set of songs that begin in the same key that the song n ended in.

Spotify appears to not show listen counts for songs with fewer than 1000 plays. It’s not impossible that some of the more obscure things I listen to (artists I know in person, or obscure organ recordings, likely) have only tens to hundreds of plays. However, unless I can find out otherwise, I’m considering them conservatively, as if each track without a count had exactly one thousand listens. ↩︎ ↩︎ ↩︎ ↩︎
I only signed up for last.fm in 2017, so this won’t account for any listening I did before then. Also, for a while, I’m not convinced the integration was very good, so I think a fair number of my listens got skipped for the first year. ↩︎

Notes on Building AsteroidOS

Tue, 06 Apr 2021 00:00:00 +0000

Building AsteroidOS isn’t terribly difficult, and reasonably well documented, but there are a handful of gotchas I ran into during the process.

Build with one thread at a time

The OpenEmbedded build system, which AsteroidOS is based on, seems to have problems running concurrent builds. I tried to do some debugging to isolate the problem, but gave up after a few tries; it’s hard to stay interested with a several hour long build-run-debug cycle. It also didn’t seem to be the same problem every time; perhaps there’s some kind of race condition somewhere.

The solution is pretty simple:

export BB_NUMBER_THREADS=1

Experimentally, after a successful initial run, I haven’t had problems doing subsequent parallel builds. YMMV!

Building on Arch etc. with Docker

Arch Linux ships with newer versions of packages than AsteroidOS allows by default, so I’m running the build inside of Docker. Here’s what I use:

#!/bin/sh
docker build -t asteroidos-toolchain .

docker rm -f asteroidos-toolchain
docker run --name asteroidos-toolchain -it \
	-u `id -u`:`id -g` \
	-v "$HOME/.gitconfig:/$HOME/.gitconfig" \
	-v "$(pwd):/asteroid" \
	asteroidos-toolchain \
	bash -c "source ./prepare-build.sh update && \
		source ./prepare-build.sh harmony && \
		BB_NUMBER_THREADS=1 nice bitbake asteroid-image || bash"

This provides a handful of niceties I’ve come to appreciate. Line by line:

docker build -t asteroidos-toolchain .
# This ensures the image is up to date. Thanks to Docker's caching, though,
# it won't take more than a second or so if the image is already built.

docker rm -f asteroidos-toolchain
# This cleans up after the last image. I could just add --rm to the `docker run`
# command, but this lets it stay around afterwards if I want to do any post hoc
# troubleshooting

docker run --name asteroidos-toolchain -it \
	-u `id -u`:`id -g` \
	# Use the same user and group inside the container as outside. If I didn't
	# do this, I'd have permissions issues on files created in the volume mounts

	-v "$HOME/.gitconfig:/$HOME/.gitconfig" \
	# It seems that the prepare-build.sh script occasionally attempts to make
	# commits, so I need my user.name and user.email config values.

	-v "$(pwd):/asteroid" \
	# I want the files present after the container exits, so I do all my work
	# in a volume mount here. This also means I can keep caches between runs.
	asteroidos-toolchain \ # The container we built earlier

	bash -c "source ./prepare-build.sh update && \
		source ./prepare-build.sh harmony && \
		BB_NUMBER_THREADS=1 nice bitbake asteroid-image || bash"
		# The prepare-build script invocations are pretty standard. I have a
		# KW88 Pro, so I use the `harmony` image. The BB_NUMBER_THREADS=1 is
		# documented above, to work around that concurrency bug. `nice`, by
		# default, sets a niceness (roughly the reverse of priority; processes
		# that are more "nice" will run with lower priority, with the scheduler
		# giving up their timeslices if there's contention for CPU time) of 10,
		# so this will soak up all my available processing power in the
		# background without much of an impact on my foreground work. The
		# `|| bash` at the end simply drops me into a bash shell if the previous
		# command fails so that I can troubleshoot in the container if I want.

One more note: Since I’m running ./prepare-build.sh update in the container, I need to be able to git pull all of the repos. I’m not passing in my ssh private keys, so I can’t pull over ssh. That’s okay, since I’m not pushing upstream anyway; I’ll just switch to https:

git remote set-url origin https://github.com/AsteroidOS/asteroid.git

Running a build server

I try to automatically build up-to-date packages for supported watches. This lets me keep my own watch updated, and verify that the changes I make don’t break on other builds as well.

Those builds are available at home.chandlerswift.com/asteroid/ and run periodically – let me know if you use these, or if you want to use them and you need help getting opkg set up to use them!

I built this on my local machine originally, and didn’t want to start from square one on the build VM, so I copied over my build files. One particular gotcha here is that apparently the OpenEmbedded build system makes extensive use of hard links to save space. I tried rsyncing my asteroid directory to the VM, and the 60GB directory filled up the 250GB of disk space the VM had available! That’s not right. However, rsync will honor hardlinks if you ask it politely:

rsync --archive \
      --verbose \
      --info=progress2 \
      --keep-dirlinks \
      --hard-links \
      --delete \
      ./ asteroidos-build:asteroid/

That did the trick for me.

On the build side, I have a short script that builds everything. (Note: this might take days!)

#!/bin/bash

cd ~/asteroid/

WATCHES="anthias bass dory harmony inharmony lenok mooneye smelt sparrow sprat sturgeon sawfish swift tetra wren"
ADDITIONAL_PACKAGES="vim gdb alsa-utils "
#ADDITIONAL_PACKAGES+="quake "

for watch in $WATCHES; do
	. prepare-build.sh $watch
	BB_NUMBER_THREADS=1 bitbake asteroid-image $ADDITIONAL_PACKAGES
	cd ~/asteroid
done

I can add any extra packages that I want built, and they’ll just show up in the repos for each watch. Pretty slick!

Keeping Tabs on my Tabs – a quick Python hack

Thu, 04 Mar 2021 00:00:00 +0000

I’m a bit of a tab hoarder. Firefox, thankfully, is pretty good about not using excessive resources for infrequently-accessed tabs, but when my Tab Counter extension told me I’d topped 300 tabs, I figured it was time to start a more serious diet plan. And what’s a diet without a scale? Here’s a script that I use to keep track of how many tabs I have open at a time, so I can measure my progress in the fight against my hoarding instincts.¹

Firefox² saves a sessionstore file that contains information about your windows and tabs. Unfortunately, that seems to be only saved when the browser is closed. However, Firefox also saves³ a $FIREFOX_PROFILE/sessionstore-backups/recovery.jsonlz4 with current data. This is the file I used. file tells us that .jsonlz4 is “Mozilla lz4 compressed data” – apparently they’re using a vaguely custom compression scheme to store this data. I didn’t have any luck stripping the header and decompressing it with the standalone lz4 tool, but Python proved workable. Once I have the data, parsing it is fairly straightforward. I wound up running the script once a day on a cronjob, frequently enough to give me that little extra prod to close my tabs, and one day in…I’ve closed almost 30!

#!/usr/bin/env python
import lz4.block as lz4
import json
from datetime import date, timedelta
import urllib.parse
import shutil

RECOVERY_FILE="/home/chandler/.mozilla/firefox/cu3gxbtx.default-release/sessionstore-backups/recovery.jsonlz4"

today=date.today().isoformat()
# Take a snapshot of today's file
shutil.copy(RECOVERY_FILE, f"recovery-{today}.jsonlz4")

with open(f"recovery-{today}.jsonlz4", 'rb') as f:
    assert f.read(8) == b'mozLz40\0'
    today_data=json.loads(lz4.decompress(f.read()))

yesterday=(date.today() - timedelta(days=1)).isoformat()
with open(f"recovery-{yesterday}.jsonlz4", 'rb') as f:
    assert f.read(8) == b'mozLz40\0'
    yesterday_data=json.loads(lz4.decompress(f.read()))

def count_total_tabs(data):
    return sum([len(window['tabs']) for window in data['windows']])

# This is a unique tab identifier. However, it's not consistent, it seems to
# change when a tab is unloaded and reloaded (suspended?)
IDENTIFIER="docshellUUID"

def find_matching_tab(needle, haystack, remove_from_haystack=False):
    # If they have the same docshellUUID, they're definitely the same
    for tab in haystack:
        if tab['entries'][-1][IDENTIFIER] == needle['entries'][-1][IDENTIFIER]:
            if remove_from_haystack:
                haystack.remove(tab)
            return tab
    # If they have the same URL they're probably the same
    for tab in haystack:
        if tab['entries'][-1]['url'] == needle['entries'][-1]['url']:
            if remove_from_haystack:
                haystack.remove(tab)
            return tab
    # TODO: if they share history, they may be the same


def find_matching_window(needle, haystack, remove_from_haystack=False):
    leading_contender = None
    leading_contender_matches = 0
    leading_contender_percent = 0
    for potential_match in haystack:
        matches = 0
        for tab in needle['tabs']:
            if find_matching_tab(tab, potential_match['tabs']) is not None:
                matches += 1

        match_percent = matches / len(needle['tabs'])
        if (matches > leading_contender_matches or match_percent > leading_contender_percent) and match_percent > .5:
            leading_contender = potential_match
            leading_contender_matches = matches
            leading_contender_percent = match_percent

    if remove_from_haystack:
        haystack.remove(leading_contender)
    return leading_contender

def format_tab(tab, leading_spaces=0):
    entry = tab['entries'][-1]
    title = entry['title']
    if len(title) > 50:
        title = title[:50] + "…"
    url = urllib.parse.urlsplit(entry['url'])
    return f"{' ' * leading_spaces}{title} ({url.netloc})"


# TODO: persist these data
# TODO: graph these data
print(f"Yesterday: {len(yesterday_data['windows'])} windows, {count_total_tabs(yesterday_data)} tabs.")
print(f"Today:     {len(today_data['windows'])} windows, {count_total_tabs(today_data)} tabs.")

net_closed_tabs = count_total_tabs(yesterday_data) - count_total_tabs(today_data)
if net_closed_tabs > 0:
    print(f"Congrats! You closed {net_closed_tabs} tabs!")
elif net_closed_tabs == 0:
    print(f"Good work. You didn't open any more tabs today.")
else:
    print(f"You opened {-net_closed_tabs} tabs. Dishonor on you! Dishonor on your cow!")

# TODO: tab entries have a lastAccessed field. Do some counting on that? (ms since epoch)

yesterday_windows = yesterday_data['windows'][:]
for i, window in enumerate(today_data['windows']):
    yesterday_window = find_matching_window(window, yesterday_windows, remove_from_haystack=True)
    if yesterday_window:
        print(f"Window {i} ({len(window['tabs'])} tabs)")
    else:
        print(f"Window {i} (NEW, {len(window['tabs'])} tabs)")

    tabs = window['tabs'][:]
    if yesterday_window:
        yesterday_tabs = yesterday_window['tabs'][:]
    else:
        yesterday_tabs = []

    opened_tabs = []
    changed_tabs = []

    for tab in tabs:
        yesterday_tab = find_matching_tab(tab, yesterday_tabs, remove_from_haystack=True)
        if yesterday_tab is None:
            opened_tabs.append(tab)
        elif yesterday_tab['entries'][-1]['url'] != tab['entries'][-1]['url']:
            changed_tabs.append(tab)

    if opened_tabs:
        print(f"  Opened ({len(opened_tabs)}): ")
        for tab in opened_tabs:
            print(format_tab(tab, 4))

    if yesterday_tabs:
        print(f"  Closed ({len(yesterday_tabs)}): ")
        for tab in yesterday_tabs:
            print(format_tab(tab, 4))

    if changed_tabs:
        print(f"  Changed ({len(changed_tabs)}): ")
        for tab in changed_tabs:
            print(format_tab(tab, 4))
    
    if not (opened_tabs or yesterday_tabs or changed_tabs):
        print("  (unchanged)")

for window in yesterday_windows:
    print(f"CLOSED: Window {i} ({len(window['tabs'])} tabs)")
    print(f"  Closed ({len(window['tabs'])}): ")
    for tab in window['tabs']:
        print(format_tab(tab, 4))

Potential enhancements that’ll probably never come since this was just a quick evening project to scratch an itch I had:

Clean up the code, Chandler!
Rather than snapshotting the (reasonably large) file each time, save the relevant bits to a database
Make the tab matching more robust. Currently if I’m visiting example.com and click a link to example.org, the script thinks I closed example.com and opened example.org. However, since I do have access to each tab’s history, I could reasonably easily modify the script to process this correctly. I actually got something mostly working here, but stripped it out since it made the output logic a lot more complex.
Add a trendline for the n-day trend (alternatively, a streak–“You’ve closed as many tabs as you’ve opened for N days in a row!”)

Generally, my usage consists of opening a bunch of tabs for a project, and closing them when I finish the project. This wouldn’t be that big of a deal, considering that a good project is generally in the 20-30 active tab range (generally consisting of documentation I’ll want to refer back to, obscure forum threads where DenverCoder9 mentioned my issue, API references, and of course some Stack Overflow tabs). However, this becomes a lot more of a problem when I’m swapping back and forth between several large projects, where I have a window for each. Currently, that list is:
- DTANM - adding automated testing
- Debugging screensharing on Wayland using Pipewire
- Writing bare-metal Risc-V assembly
- Fixing up an old super-tic-tac-toe server for better multiplayer
- Finishing the Gallery app for AsteroidOS
- Interfacing with MIDI instruments in Rust
as well as a handful of smaller projects. One hope here is that this will be a nice kick in the pants to finish up some of those projects that have been sitting on the back burner for months. ↩︎
I’m not looking forward to the day when this changes and it breaks everything that relies on this partially-undocumented feature. ↩︎
It seems to save this file at least once a minute. I didn’t really care if it was once a minute or once an hour, since my script runs daily; I just didn’t want it to be “whenever you close your browser”! ↩︎

A random accompaniment on EVENTIDE

Mon, 01 Feb 2021 00:00:00 +0000

How good can we make a random accompaniment sound? I was playing with EVENTIDE, William Henry Monk’s tune written for Abide with Me, and was pondering this. I was trying to play randomly on the piano from octaves and notes in the scale, but suspected my “randomess” of playing wasn’t particularly good. But what does have good randomness? Python! (If you don’t want to read the code, just check out the results at the bottom.)

#!/usr/bin/python

# pip install pymusicxml
# https://pypi.org/project/pymusicxml/
# https://git.sr.ht/~marcevanstein/pymusicxml
from pymusicxml import *
from random import choice

score = Score(title="EVENTIDE", composer="Chandler Swift")
accompaniment = Part("Piano")

### GENERATE ACCOMPANIMENT:

# First, we do some boilerplate. We generate the note names and octaves to be
# used later in the program. This section generates the 88 keys on a piano,
# with a result like ["a0", "bb0", "b0", "c1", "db1", ... "bb7", "b7", "c8"]
note_names = ["c", "db", "d", "eb", "e", "f", "gb", "g", "ab", "a", "bb", "b"]
notes_with_octaves = ["a0", "bb0", "b0"]
for octave in range(1,8):
    for note in note_names:
        notes_with_octaves.append(note + str(octave))
notes_with_octaves.append("c8")


# to save some writing later, scales are major, minor, or dominant sevenths.
# I could have just written out "major" or "minor" everywhere, but simply being
# able to write `M` saves a fair number of quotes typed!
#
# Abide with me does have some slightly more complex chords than these three,
# but we can get nearly there with these. The biggest thing that I miss having
# is probably the nice Csus2/E that we have in the second bar and elsewhere.
M="major"
m="minor"
dom7="dominant seventh"

# Now let's write out the notes. A C major scale is C D E F G A B; offsets from
# the starting C of 0, 2, 4, 5, 7, 9, and 11 notes. The others are similar.
#
# After some playing around, I think I like the melodic minor best, but I'm
# still not sure. If we wanted, we could simplify any of these; perhaps omitting
# a 7 altogether from the major scale would be nice; for example, in the second
# chord in the first measure (assuming key of C, for simplicity), we really
# don't want an F# (the major seventh of the V chord, G). We could switch all
# of our V chords to V7 chords, and then at least we have F naturals instead of
# F sharps, but I was happy enough with the result here to let it be.
scales={
    M: [0,2,4,5,7,9], # M2, M3, P4, P5, M6
    m: [0,2,3,5,7,10], # M2, m3, P4, P5, m7
    dom7: [0,2,4,5,7,9,10]
}

# Here are the changes for the tune. Each tuple has the chord root (c, d, f, g, 
# and a, here), the chord type (major, minor, or dominant seventh), and the
# (whole) number of beats for which that chord is played. Hey, it's the world's
# lousiest lead sheet!
changes=[
    # simplified, and transposed to C so I don't have to deal with accidentals
    # see also e.g. https://hymnary.org/page/fetch/LUYH2013/504/low
    ("c", M, 2),
    ("g", dom7, 2),
    ("a", m, 4),
    # ^ 2, Csus2/E 2
    ("f", M, 2),
    ("g", dom7, 2),
    ("c", M, 4),
    ("c", M, 4),
    ("f", M, 2),
    ("c", M, 2),
    ("d", m, 2),
    ("d", dom7, 2),
    ("g", M, 2),
    ("g", dom7, 2),
    ("c", M, 2),
    ("g", dom7, 2),
    ("a", m, 4),
    # ^ 2, Eb/G 2
    ("f", M, 2),
    ("a", dom7, 2),
    ("d", m, 4),
    ("g", dom7, 4),
    ("c", M, 1),
    ("g", M, 1),
    ("a", m, 1),
    ("f", M, 1),
    ("g", M, 2),
    ("g", dom7, 2),
    ("c", M, 4),
    # TODO: intro/outro
]

# We randomly generate an array of notes to be added to the score. So that we
# don't have to keep track of the measures, and since the notes are all the same
# duration, we just generate them all at once, and then slot 4*NOTES_PER_BEAT of
# them into each measure at the end.
notes = []

BEATS_PER_MINUTE=80
NOTES_PER_BEAT=4
for chord in changes:
    chord_root, chord_type, beats = chord

    # Play the root of the chord
    tonic = chord_root + "3" # start in the third octave
    notes.append(Note(tonic, 1/NOTES_PER_BEAT, directions=StartPedal()))
    notes_left = beats * NOTES_PER_BEAT - 1

    # If the chord is held for longer than a beat, add a fifth in the bass. We
    # do this check specifically for the second measure of the last phrase,
    # which has four chords each held for one beat. If we added both the tonic
    # and fifth to each of those, we wouldn't have room for any random notes.
    if beats > 1:
        # add a major fifth above the tonic
        fifth = notes_with_octaves[notes_with_octaves.index(tonic) + 7]
        notes.append(Note(fifth, 1/NOTES_PER_BEAT))
        notes_left -= 1

    # fill the rest with random notes from the top end of the piano.
    while notes_left:
        interval = choice(scales[chord_type])
        octave = choice([4,5,6,7,8]) # Notes will be in the range of C4 to C8
        note = note_names[(note_names.index(chord_root) + interval) % len(note_names)]
        note_with_octave = note + str(octave)
        if note_with_octave in notes_with_octaves:
            #print(f"from {chord_root} {chord_type}, picked interval {interval}"
            #       "octave {octave} so note {note_with_octave}")
            directions=StopPedal() if notes_left == 1 else () # unpedal at the end of the chord
            notes.append(Note(note_with_octave, 1/NOTES_PER_BEAT, directions=directions))
            notes_left -= 1

# Now take those notes and break them up into measures
measures = []
first_measure = True
while len(notes) > 0:
    if first_measure:
        measure = Measure(
            time_signature=(4,4),
            directions_with_displacements=[(MetronomeMark(1.0, BEATS_PER_MINUTE), 0)]
        )
        first_measure = False
    else:
        measure = Measure()
    measure.extend(notes[:NOTES_PER_BEAT*4])
    notes = notes[NOTES_PER_BEAT*4:]
    measures.append(measure)

# and tack those measures onto the score.
accompaniment.extend(measures)


### ADD VOICE PART, FOR REFERENCE

# This bit adds the melody line to EVENTIDE. Mostly busy work, I debated not
# even including it.

voice = Part("Voice")

# I took a long time to convince myself that it wasn't worth it to reimplement
# lilypond to make this section work. We're like 80% of the way there, but the
# pareto principle suggests that it's still not worth it to get that last 20%.
voice_notes="""
e4 2 e4 1 d4 1 c4 2 g4 2 a4 1 g4 1 g4 1 f4 1 e4 4
e4 2 f4 1 g4 1 a4 2 g4 2 f4 1 d4 1 e4 1 f#4 1 g4 4
e4 2 e4 1 d4 1 c4 2 g4 2 g4 1 f4 1 f4 1 e4 1 d4 4
d4 2 e4 1 f4 1 e4 1 d4 1 c4 1 f4 1 e4 2 d4 2 c4 4
"""
voice_notes = voice_notes.split()
# https://stackoverflow.com/a/15480540/3814663
voice_notes = list(zip(voice_notes[::2], voice_notes[1::2]))
# [('g4', '2'), ('g4', '1'), ('f4', '1'), ('eb4', '2'), ...]

current_measure = Measure(time_signature=(4,4))
beats_through_current_measure = 0
measures = []
for note, duration in voice_notes: # this only works because no notes are held through bar lines
    duration = int(duration)
    current_measure.append(Note(note, duration))
    beats_through_current_measure += duration
    if beats_through_current_measure == 4: # should never be greater than four, see above.
        # new measure
        measures.append(current_measure)
        current_measure = Measure()
        beats_through_current_measure = 0
voice.extend(measures)

score.append(PartGroup([voice, accompaniment]))
score.export_to_file("Eventide.xml")

The code above is built off the pymusicxml python library, and is therefore licensed under the GPLv3.

If you want to run it yourself, here’s my Makefile:

Makefile

Eventide.xml:
	python eventide.py

Eventide.pdf: Eventide.xml
	musescore Eventide.xml -o Eventide.pdf

Eventide.mp3: Eventide.xml
	musescore Eventide.xml -o Eventide.mp3

Eventide.mscz: Eventide.xml
	musescore Eventide.xml -o Eventide.mscz

.PHONY: clean
clean:
	rm -f Eventide.xml Eventide.mp3 Eventide.pdf Eventide.mscz

all: Eventide.pdf Eventide.mp3 Eventide.mscz

Results

Here’s one sample of the output. (This isn’t hand-picked from lots of results; it’s the result of the first complete run of the program.) You can view it on Musescore’s community site (embedded below), or download the MusicXML file, Musescore file, PDF, or MP3.

Bonus: While looking up information about EVENTIDE, I found that the Thelonious Monk Septet has a recording of Abide with Me!

Edited 2021-02-09 to add – Bonus bonus: What’s almost as random as random.choice()? Chickens! (Thanks Jeff!)

Exploring paths for Sudoku variants with Python

Sat, 09 Jan 2021 00:00:00 +0000

One of my college friends has an interest in Sudoku variants. Generally, that manifests in the two of us solving puzzles curated by Cracking the Cryptic, but occasionally it ends up with me trying to solve a puzzle he’s created. A while back, he and I were chatting about one of his ideas for the basis of a puzzle. It’s based around a snake in the grid (for an example of another puzzle using a snake, check out this puzzle by CtC). His puzzle requires that its snake alternates successive even digits and odd digits, is unambiguous, and visits at least one cell in every box (as well as, of course, being a valid sudoku puzzle!).

Let’s talk about those rules in a bit more depth:

Snake: A snake is a one-wide orthogonally connected path. Much like in the classic Snake game, it can’t touch itself anywhere along its length (although diagonal adjacency is fine). The one in this puzzle has alternating even and odd digits along its length, represented in this post by red and blue respectively:
visits a cell in every box: The above snake fills this requirement as well. The following (shorter) snake does, too:
and the big constraint, is unambiguous: Given the pattern of even and odd digits in the grid, there is exactly one direction the snake could continue at any point and still satisfy the parity constraint. To see this happen, we also have to have some information about the parity of the cells surrounding the snake. In general, each cell of the snake is touching either one (if it’s the head/tail) or two (if it’s part of the body) other cells that are snake cells. These must be of the opposite parity. The other cells must be of the same parity.

For example, suppose that the head of the snake is even, and continues into the (odd) cell to its right:

To ensure there’s no ambiguity that the snake continues to the right, all of the other cells must be invalid for the snake to enter; that is, they must all have the same parity as the head:

Now, each box must contain a 1 through 9; that is, four even and five odd digits. This one already contains all four even digits, so the unfilled cells must be odd:

The only place the snake can proceed, then, is directly into the second box:

If we continue to the right, we would need at least five red cells in the top middle box (try it out, if you like!) so now we have to move down…

Repeating this logic eventually gets us here. This scenario, however, is already broken—there simply isn’t a way to get the snake back into the top right box, if we can’t put a red cell in that middle row.

This type of logic is fairly time-consuming and error prone, but it’s not really that complex¹, so we put together a Python program to try out paths for us that satisfy the even-odd and uniqueness constraints. Given that it was written hastily to solve a one-time problem, it’s certainly neither very neat nor readable, but it seems to do what we want!

Open if you dare!

#!/usr/bin/python3
import copy
from time import sleep

# Originally these were just letters (X and O for even and odd snake, and x and
# o for even and odd empty cell), but as long as they're distinct, they do what
# we need. So these include ANSI color codes, which colors the snake red and
# blue.
EMPTY = "_"
ODD = "\033[34m_\033[0m"
EVEN = "\033[31m_\033[0m"
ODD_SNAKE = "\033[34mX\033[0m"
EVEN_SNAKE = "\033[31mX\033[0m"

# Here we define a few helper functions to make our lives easier later.
attempts = 0
def print_grid(grid):
    global attempts
    print(f"Attempt {attempts}:")
    attempts += 1
    for row in range(9):
        print(" ".join([r[row] for r in grid]))
    print()

def opposite_parity(x):
    if x == EVEN:
        return ODD
    else:
        return EVEN

def snake(x):
    if x == EVEN:
        return EVEN_SNAKE
    else:
        return ODD_SNAKE

def has_two_adjacent_snakes(grid, cell):
    """
    has_two_adjacent_snakes checks to see if two or more of the adjacent
    cells are filled with snake. If one is filled, we must be at the head of a
    snake; if two are filled we're along the body. No more than two can be
    filled, because then we'd have broken the snake; it would fork.
    """
    adjacent_snakes = 0
    if cell[1] > 0 and grid[cell[0]][cell[1] - 1] in [ODD_SNAKE, EVEN_SNAKE]: # north
        adjacent_snakes += 1

    if cell[0] < 8 and grid[cell[0] + 1][cell[1]] in [ODD_SNAKE, EVEN_SNAKE]: # east
        adjacent_snakes += 1

    if cell[1] < 8 and grid[cell[0]][cell[1] + 1] in [ODD_SNAKE, EVEN_SNAKE]: # south
        adjacent_snakes += 1
    
    if cell[0] > 0 and grid[cell[0] - 1][cell[1]] in [ODD_SNAKE, EVEN_SNAKE]: # west
        adjacent_snakes += 1
    return adjacent_snakes >= 2

def check_grid_parity(grid):
    """
    check_grid_parity employs a multi-pass approach. For each pass, it counts
    the even and odd cells in each row, column, and box. Then, if it has
    either four even or five odd cells, it knows that all of the others must be
    of the opposite parity, so it fills them in. This, however, may complete
    other rows/columns/boxes, so we'll do another pass after this one is done.
    We continue to do passes until either something breaks (returning False),
    or nothing is left to change.

    This is a lot of repetitive but just-slightly-different-each-time logic, so
    it can't easily be condensed. However, it's not particularly complicated,
    just rather verbose.
    """
    do_another_pass=False
    # check each row
    for row in grid:
        even_count = odd_count = 0
        for cell in row:
            if cell == EVEN or cell == EVEN_SNAKE:
                even_count += 1
            elif cell == ODD or cell == ODD_SNAKE:
                odd_count += 1
        if even_count > 4 or odd_count > 5:
            #print("broken on row")
            return False
        elif even_count == 4:
            for cell_index in range(len(row)):
                if row[cell_index] == EMPTY:
                    #print("filling row with odd")
                    row[cell_index] = ODD
                    do_another_pass=True
        elif odd_count == 5:
            for cell_index in range(len(row)):
                if row[cell_index] == EMPTY:
                    #print("filling row with even")
                    row[cell_index] = EVEN
                    do_another_pass=True

    # check each column
    for col_index in range(9):
        even_count = odd_count = 0
        for row in grid:
            if row[col_index] == EVEN or row[col_index] == EVEN_SNAKE:
                even_count += 1
            elif row[col_index] == ODD or row[col_index] == ODD_SNAKE:
                odd_count += 1
        #print(f"row {col_index}: even: {even_count}, odd: {odd_count}")
        if even_count > 4 or odd_count > 5:
            #print("broken on column")
            return False
        elif even_count == 4:
            for row in grid:
                if row[col_index] == EMPTY:
                    row[col_index] = ODD
                    #print("filling col with odd")
                    do_another_pass=True
        elif odd_count == 5:
            for row in grid:
                if row[col_index] == EMPTY:
                    row[col_index] = EVEN
                    #print("filling col with even")
                    do_another_pass=True

    # check each box
    for box_x in range(3):
        for box_y in range(3):
            even_count = odd_count = 0
            for x in range(3*box_x, 3*box_x+3):
                for y in range(3*box_y, 3*box_y + 3):
                    if grid[x][y] == EVEN or grid[x][y] == EVEN_SNAKE:
                        even_count += 1
                    elif grid[x][y] == ODD or grid[x][y] == ODD_SNAKE:
                        odd_count += 1
            if even_count > 4 or odd_count > 5:
                #print(f"bad parity on box {box_x},{box_y}")
                return False
            elif even_count == 4:
                for x in range(3*box_x, 3*box_x+3):
                    for y in range(3*box_y, 3*box_y + 3):
                        if grid[x][y] == EMPTY:
                            grid[x][y] = ODD
                            #print("filling box with odd")
                            do_another_pass=True
            elif odd_count == 5:
                for x in range(3*box_x, 3*box_x+3):
                    for y in range(3*box_y, 3*box_y + 3):
                        if grid[x][y] == EMPTY:
                            grid[x][y] = EVEN
                            #print("filling box with even")
                            do_another_pass=True

    if do_another_pass:
        return check_grid_parity(grid)
    else:
        return True

def has_adjacent_cells_of_parity(grid, cell, parity):
    adjacent_cells_of_parity = 0
    if cell[1] > 0 and grid[cell[0]][cell[1] - 1] == parity: # north
        adjacent_cells_of_parity += 1

    if cell[0] < 8 and grid[cell[0] + 1][cell[1]] == parity: # east
        adjacent_cells_of_parity += 1

    if cell[1] < 8 and grid[cell[0]][cell[1] + 1] == parity: # south
        adjacent_cells_of_parity += 1
    
    if cell[0] > 0 and grid[cell[0] - 1][cell[1]] == parity: # west
        adjacent_cells_of_parity += 1
    return adjacent_cells_of_parity > 0


def fill_empty_cells_around(grid, cell, value):
    """
    fill_empty_cells_around is applied to the "neck" (that is, the cell next to
    the head) of the snake. Supposing we have a snake like this, where E and O
    represent even and odd snake cells, and e and o non-snake cells:
        e o e _
        E O E O E
        e o e _
    then fill_empty_cells_around will be used with the above as `grid`, the
    last O in the row as `cell`, and "o" as `value`.
    """

    if cell[1] > 0 and grid[cell[0]][cell[1] - 1] == opposite_parity(value): # north
        raise ValueError
    elif cell[1] > 0 and grid[cell[0]][cell[1] - 1] == EMPTY: # north
        grid[cell[0]][cell[1] - 1] = value

    if cell[0] < 8 and grid[cell[0] + 1][cell[1]] == opposite_parity(value): # east
        raise ValueError
    elif cell[0] < 8 and grid[cell[0] + 1][cell[1]] == EMPTY: # east
        grid[cell[0] + 1][cell[1]] = value

    if cell[1] < 8 and grid[cell[0]][cell[1] + 1] == opposite_parity(value): # south
        raise ValueError
    elif cell[1] < 8 and grid[cell[0]][cell[1] + 1] == EMPTY: # south
        grid[cell[0]][cell[1] + 1] = value
    
    if cell[0] > 0 and grid[cell[0] - 1][cell[1]] == opposite_parity(value): # west
        raise ValueError
    elif cell[0] > 0 and grid[cell[0] - 1][cell[1]] == EMPTY: # west
        grid[cell[0] - 1][cell[1]] = value


def recursive_dfs(grid, head, neck, last_parity):

    # disable this if you don't want to see the solving visualization. Then you
    # can also disable the sleep(10) about 20 lines below, as we'll only be
    # printing the solutions.
    print_grid(grid)

    # check if we broke the grid:
    if not check_grid_parity(grid):
        return # try the next case

    # give the terminal a chance to keep up. On a 60hz display, this should
    # leave the grid on display for 1-2 frames.
    sleep(0.03)

    # base case: have we solved it (reached every box at least once)
    snakes_everywhere = True
    for box_x in range(3):
        for box_y in range(3):
            snake_in_grid = False
            for x in range(3*box_x, 3*box_x+3):
                for y in range(3*box_y, 3*box_y + 3):
                    if grid[x][y] in [ODD_SNAKE, EVEN_SNAKE]:
                        snake_in_grid = True
            if not snake_in_grid:
                snakes_everywhere = False
    if (snakes_everywhere and
        not has_adjacent_cells_of_parity(grid, head, opposite_parity(last_parity)) and
        not has_adjacent_cells_of_parity(grid, neck, last_parity)):
        print("Solved!")
        print_grid(grid)
        sleep(10) # give us a chance to look at what we found
        # note that we don't return here; we can try to see if there's a longer
        # snake that's also a solution.

    # if the cell is empty, then it doesn't have a parity, and therefore a snake
    # does not pass by in an adjacent cell, so it's a valid cell to try to enter
    #
    # For each of the directions (ordered north, east, south, and west), we make
    # sure we're not along that wall (if we're along the north wall, of course
    # we can't go north); we verify that either cell to the $direction of the
    # current head is empty (no constraints, so we can enter), or that it's the
    # same color as the one we're entering _and_ doesn't have two adjacent
    # snakes (that is, another besides the one we're entering from).
    #
    # If we have a neck cell (which we always do except on the very first cell
    # of the snake), we fill the cells around the neck with the appropriate
    # color. Note that we're filling the cells around the neck and not the
    # head. Although it does take one step longer to fail than would otherwise
    # be necessary, this saves us a lot of headache later: Say our head is even.
    # Then all of the cells around it except the neck cell (if applicable) and
    # the cell that will be the head next iteration (if the snake is not
    # complete) will also be even, to make the snake unambiguous. However, then
    # next iteration we will need to continue the snake into one of those even
    # cells---which one? Conversely, if we simply apply it to the neck after the
    # location of the new head has been decided, we avoid all this difficulty.
    #
    # We repeat this process for each possible direction.
    if head[1] > 0 and (grid[head[0]][head[1] - 1] == EMPTY or
        (grid[head[0]][head[1] - 1] == opposite_parity(last_parity) and
            not has_two_adjacent_snakes(grid, (head[0], head[1] - 1)))): # north
        new_grid = copy.deepcopy(grid)
        new_grid[head[0]][head[1] - 1] = snake(opposite_parity(last_parity))
        if neck is not None:
            try:
                fill_empty_cells_around(new_grid, neck, opposite_parity(last_parity))
            except ValueError:  # Cell is required to be both even and odd---abort!
                return # the neck is broken so this _can't_ be the head
        recursive_dfs(new_grid, (head[0], head[1] - 1), head, opposite_parity(last_parity))

    if head[0] < 8 and (grid[head[0] + 1][head[1]] == EMPTY or
        (grid[head[0] + 1][head[1]] == opposite_parity(last_parity) and
            not has_two_adjacent_snakes(grid, (head[0] + 1, head[1])))): # east
        new_grid = copy.deepcopy(grid)
        new_grid[head[0] + 1][head[1]] = snake(opposite_parity(last_parity))
        if neck is not None:
            try:
                fill_empty_cells_around(new_grid, neck, opposite_parity(last_parity))
            except ValueError:  # Cell is required to be both even and odd---abort!
                return
        recursive_dfs(new_grid, (head[0] + 1, head[1]), head, opposite_parity(last_parity))

    if head[0] > 0 and (grid[head[0] - 1][head[1]] == EMPTY or
        (grid[head[0] - 1][head[1]] == opposite_parity(last_parity) and
            not has_two_adjacent_snakes(grid, (head[0] - 1, head[1])))): # west
        new_grid = copy.deepcopy(grid)
        new_grid[head[0] - 1][head[1]] = snake(opposite_parity(last_parity))
        if neck is not None:
            try:
                fill_empty_cells_around(new_grid, neck, opposite_parity(last_parity))
            except ValueError:  # Cell is required to be both even and odd---abort!
                return
        recursive_dfs(new_grid, (head[0] - 1, head[1]), head, opposite_parity(last_parity))

    if head[1] < 8 and (grid[head[0]][head[1] + 1] == EMPTY or
        (grid[head[0]][head[1] + 1] == opposite_parity(last_parity) and
            not has_two_adjacent_snakes(grid, (head[0], head[1] + 1)))): # south
        new_grid = copy.deepcopy(grid)
        new_grid[head[0]][head[1] + 1] = snake(opposite_parity(last_parity))
        if neck is not None:
            try:
                fill_empty_cells_around(new_grid, neck, opposite_parity(last_parity))
            except ValueError:  # Cell is required to be both even and odd---abort!
                return
        recursive_dfs(new_grid, (head[0], head[1] + 1), head, opposite_parity(last_parity))

# We attempt to start in each cell in the grid, though to cut our search space
# and time in half, we constrain start_y <= start_x. Any snakes that would be
# found with start_y > start_x will be a mirror of one that will be found here.
# This still leaves quite a few duplicates by reflection and rotation, but those
# will be harder to weed out by simply adjusting the ranges here.
for start_x in range(9):
    for start_y in range(start_x + 1):
        empty_grid = []
        for i in range(9):
            empty_grid.append([EMPTY] * 9)
        empty_grid[start_x][start_y] = ODD_SNAKE
        recursive_dfs(empty_grid, (start_x, start_y), None, ODD)

While the rate my terminal emulator can display the text seems to currently be the bottleneck, it’s still certainly not slow, so I left the output displayed at each step the algorithm takes. This was essential as a debugging tool, but now it’s just nice to watch it run. It’s really a rather pretty visual; I forgot how much time I could spend staring at visualizations of searches²! If I’m ever bored for an evening, I may consider porting it to Javascript so it can run in the browser, but for now if you want to see more, you’ll just have to download and run it yourself. All you need is Python.

A version of this program with all the print statements removed took about five minutes to find 3241 possible configurations that satisfied the constraints.

Since we’ve found all the possible combinations (recognized by this program, at least), we can also figure out the shortest and longest possible paths that satisfy the constraints—21 and 37, respectively. And both are pretty neat shapes!

Images from this post were created using penpa-edit.

Over-optimistic as always, I think I had said we’d probably have a solution together in an hour. I think by the time we’d worked out all the bugs and edge cases, we were somewhere between three and four hours in, but it was a fun challenge to write! ↩︎
If you also enjoy watching this sort of thing, Jamis Buck’s page on mazes has a bunch of really neat animations! ↩︎

PinePhone software from the ground up, part 2

Wed, 09 Dec 2020 00:00:00 +0000

A few weeks ago, I compiled and installed a Linux kernel and a bit of other software to run on the PinePhone. Today, let’s look at some next steps.

First, let’s talk about the init process. When the kernel finishes its initialization process, it launches a single userspace program. Typically, the kernel will attempt to run /init (if launched from a ramdisk), followed by /sbin/init, /etc/init, /bin/init, and, as a final fallback, /bin/sh. If you’re using a recent mainstream Linux distribution, your init process is likely systemd. Ubuntu tried creating a system called Upstart, which didn’t really catch on; before that, they used some variant of SysVinit¹.

The init process is responsible for setting up a computer and spawning all the subsequent processes that are run². Commonly, this might mean mounting disks, setting up your path, setting up your network, and starting any services that you run. When you’re ready to turn off or restart your computer, you can send your init process a signal, and it will (hopefully) stop other running processes gracefully, and then tell the kernel to shut down or reboot, as the case may be³.

Last time, we installed Busybox and ran a shell script that just looped. A shell script (and, therefore, a shell) is pretty overkill for printing some strings. Instead, below is a C reimplementation of what we did using Busybox last time. I’m pretty much just resuming where I left off—we have a functioning bootloader and kernel, hopefully, so I just need to adjust the initramfs.

cd ~/mobile/pinephone
mkdir my_init
cat > my_init/init.c <<EOF
#include <stdio.h>
#include <unistd.h>

int main() {
	unsigned int counter = 0;
	while (1) {
		printf("Hello world %i\n", counter++);
		sleep(1);
	}
}
EOF
# Because the compiled init is the only thing in the initramfs, we of course
# can't dynamically link to libraries like (in this case) glibc, which is what
# gcc tries to do by default. Luckily, we can just add -static and gcc will give
# us a nice statically linked binary.
aarch64-linux-gnu-gcc my_init/init.c -o initramfs/init -static

# rebuild initramfs
cd initramfs
find . | cpio -H newc -o > ../bootloader/initramfs.cpio
cd ..

# deploy to SD card
sudo p-boot/dist/p-boot-conf-native bootloader /dev/sda1

It still seems like the disk usage is a bit high, though:

du -hs initramfs/init
# 584K	initramfs/init

There’s no way a program that’s just incrementing a counter and printing 13 bytes of text needs that much space! How does it compare without the statically linked glibc parts embedded in the binary?

aarch64-linux-gnu-gcc my_init/init.c -o init-dynamic

du -hs init-dynamic
# 12K	init-dynamic

Yeah, that seems better. How can we cut out some of that bloat? Well, glibc has drawn complaints for being pretty heavyweight. How does musl stack up?

# I don't really want to figure out how to get my system's toolchain to link to
# musl, so I'll just do it in a prebuilt container.
docker run --volume `pwd`/my_init:/src \
    --user 1000 \
    muslcc/i686:aarch64-linux-musl \
    sh -c "gcc /src/init.c -o /src/init-musl -static"

du -h my_init/init-musl
# 136K	my_init/init-musl

That’s a bit better. And with musl’s easier to understand libraries besides? I’ll call it a victory. In theory, these two approaches should be interchangeable, so you can compile with whichever libc you prefer.

Of course, at this point, we haven’t managed to create a system that does anything particularly useful; we’ve just removed some dependencies to make our initramfs a bit smaller. Instead, to demonstrate a bit of I/O, here’s a (relatively) brief C program that draws a sine wave on the screen⁴, responding to touch input.

#include <stdio.h>
#include <sys/mount.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <math.h>
#include <unistd.h>
#include <errno.h>
#include <linux/input.h>
#include <string.h>

#define SCREEN_WIDTH 720
#define SCREEN_HEIGHT 1440

// For more information on writing to framebuffers, check out the slides at
// https://archive.fosdem.org/2020/schedule/event/fbdev/attachments/slides/3595/export/events/attachments/fbdev/slides/3595/fosdem_2020_nicolas_caramelli_linux_framebuffer.pdf

void reset(unsigned char* fbmem) {
	memset(fbmem, 0, SCREEN_WIDTH * SCREEN_HEIGHT * 4);
	// The naïve approach, here, is an order of magnitude slower than above
	//for (int p = 0; p < SCREEN_WIDTH * SCREEN_HEIGHT * 4; p++) {
	//	fbmem[p] = 0;
	//}
}

void draw_sine(unsigned char* fbmem, int x_offset, int h) {
	for (int x = 0; x < SCREEN_WIDTH; x++) {
		int y = SCREEN_HEIGHT/2 + (h/2)*sin((x - x_offset)/100.0);

		// Ensure we don't write out of bounds
		y = fmin(fmax(y, 0), SCREEN_HEIGHT - 1);

		fbmem[(SCREEN_WIDTH * y + x) * 4] = 255;
		fbmem[(SCREEN_WIDTH * y + x) * 4 + 1] = 255;
		fbmem[(SCREEN_WIDTH * y + x) * 4 + 2] = 255;
	}
}

int main() {
	// Since Linux 2.6.32, devtmpfs (https://lwn.net/Articles/330985/) has
	// provided an easy way to automatically mount special device files.
	// (Earlier versions of this program were much longer, and relied on the
	// `mknod` system call to create these as they were used. This turned out
	// to be a much less wieldy approach.) The `mount` system call mounts a
	// `devtmpfs` on `/dev`, so that we can use it to read touch inputs from a
	// file in `/dev/input`, and write output to the framebuffer `/dev/fb0`.
	mount("devtmpfs", "/dev", "devtmpfs", 0, "");

	// Open the framebuffer
	int fb = open("/dev/fb0", O_RDWR);
	if (fb < 0) {
		printf("Error opening /dev/fb0: %i\n", errno);
		return 1;
	}
	// To give us convenient random-write access to the framebuffer, we map the
	// file into memory. Technically, we could likely use fseek() and fwrite(),
	// but being able to easily index and write makes this much more pleasant.
	unsigned char* fbmem = mmap(NULL, SCREEN_WIDTH * SCREEN_HEIGHT * 4,
					PROT_WRITE, MAP_SHARED, fb, 0);

	// Open the touch input device
	//
	// In my case, the touchscreen was at `/dev/input/event3`, where `event0`,
	// `event1`, and `event2` belong to other input devices---power and volume
	// buttons among them. This ordering may not be always the same, and may
	// vary between PinePhone hardware revisions. I checked using a simple init
	// script based on the BusyBox install from last time, included in the
	// footnotes.
	int touchscreen = open("/dev/input/event3", O_RDONLY);
	if (touchscreen < 0) {
		printf("Error opening /dev/input/event3: %i\n", errno);
		return 1;
	}

	// Print and update a sine wave on the framebuffer
	int h = 100; // height of the sine wave, in px
	int x = 0; // horizontal offset

	struct input_event ev;
	int pressed_slot = -1; // no slot selected
	int reporting_slot = 0;
	int previous_x_pos = -1; // no previous position
	int previous_y_pos = -1; // no previous position
	while (1) {
		if (read(touchscreen, &ev, sizeof(struct input_event)) < 0) {
			printf("Error reading touchscreen input\n");
			return 1;
		}

		// Handle touchscreen input
		//
		// Because handling touch input isn't the focus of this post, I'm not
		// going to go into great detail here. If you want, check out docs at
		// https://www.kernel.org/doc/html/latest/input/multi-touch-protocol.html
		if (ev.type != EV_ABS) { // Only listen for absolute position events
			continue;
		}
		switch(ev.code) {
		case ABS_MT_SLOT:
			reporting_slot = ev.value;
			break;
		case ABS_MT_TRACKING_ID:
			if (ev.value > 0 && pressed_slot < 0) {
				pressed_slot = reporting_slot;
			} else if (ev.value < 0 && reporting_slot == pressed_slot) {
				pressed_slot = -1;
				previous_x_pos = -1;
				previous_y_pos = -1;
			}
			break;
		case ABS_MT_POSITION_X: // move left-to-right
			if (reporting_slot == pressed_slot) {
				if (previous_x_pos >= 0) {
					x += ev.value - previous_x_pos;
				}
				previous_x_pos = ev.value;
			}
			break;
		case ABS_MT_POSITION_Y: // adjust vertical scale
			if (reporting_slot == pressed_slot) {
				if (previous_y_pos >= 0) {
					h += ev.value - previous_y_pos;
				}
				previous_y_pos = ev.value;
			}
			break;
		}
		reset(fbmem);
		draw_sine(fbmem, x, h);
	}
}

Compile and write to the SD card, and here’s what we have:

At this point, we have our program running; no other userspace is needed! That’s pretty neat. But for now, we’re still running entirely from a ramdisk. We can’t persist any changes, and the initramfs model will get unwieldy quickly as more assets are added to the image. Next time, we’ll take a quick look at what’s needed to mount a root filesystem and persist changes to disk.

I didn’t start using Linux enough to be aware of things like init systems until Ubuntu was well into the Upstart→systemd migration, if you want to know anything about an init system older than systemd, ask somebody who lived through a different init system! ↩︎

While doing some research for this post, I found out about pstree, which does pretty much what it says on the tin: prints a pretty listing of processes that you have running. It’s part of the psmisc package, which seems to be required by the base system on Arch, at least. This is a listing of a pretty bare-bones system before a desktop environment is started up. It would be a fair bit lighter without Docker.

chandler@xenon ~ % pstree
systemd─┬─agetty
        ├─containerd───11*[{containerd}]
        ├─dbus-daemon
        ├─dhcpcd─┬─dhcpcd───dhcpcd
        │        └─2*[dhcpcd]
        ├─dockerd───9*[{dockerd}]
        ├─login───zsh───pstree
        ├─polkitd───7*[{polkitd}]
        ├─ssh-agent
        ├─sshd
        ├─systemd─┬─(sd-pam)
        │         └─dbus-daemon
        ├─systemd-journal
        ├─systemd-logind
        ├─systemd-timesyn───{systemd-timesyn}
        └─systemd-udevd
chandler@xenon ~ %

↩︎

One thing that threw me off initially was that the init process shouldn’t ever exit, and the kernel will complain if it does. Instead, the process ensures everything is ready for shutdown, and then sends a reboot or shutdown system call to the kernel. ↩︎

Here’s the script to check which /dev/input device your touchscreen is on:

#!/bin/sh

Create all the symlinks to /bin/busybox
/bin/busybox --install -s

mount -n -t devtmpfs -o mode=0755,size=4M devtmpfs /dev
echo dev:
ls /dev
echo dev input:
ls /dev/input
# we see event0, event1, event2, event3, so we check which one is correct:
for i in 0 1 2 3; do
	echo i: $i
	timeout 5 cat /dev/input/event$i
done

↩︎

Phone a Month 4?: Setting up PinePhone software from the ground up

Tue, 10 Nov 2020 00:00:00 +0000

I never actually ran anything new on my phone this month. Instead, I’ve been tinkering with the PinePhone. Instead of the usual update post, here’s some info on how to install and boot the most basic possible Linux installation for the PinePhone.

In my defense, I did make quite a few attempts to get something (anything!) working on the PinePhone, but after trying up-to-date versions of Ubuntu Touch, PostmarketOS, and Mobian, I found major dealbreakers in each. I can’t remember which had which problem(s), but I struggled with terrible call quality, missed incoming calls and SMS messages, lack of functioning camera apps, and probably other issues as well.

In any case, I’m currently running the marvelous p-boot bootloader with Megi’s latest PinePhone kernel (optionally prebuilt here) and a manually configured initramfs. I don’t expect this to ever be a fully-functional system, but I wanted to build this system to make sure that I understood all the parts I’d have to be troubleshooting later on in the process, and it’s proven quite useful for that.

The end result of this process is, more or less, the smallest system that can be said to be running Linux¹. It boots to a shell, and thanks to Busybox, I do have a fairly complete environment. I can write (with cat and sed as my editors) and execute shell scripts, and…that’s about it!

The notes below are given with the caveat that I’m running Arch Linux (and so x86_64 architecture), and that I have the base-devel package installed. It assumes I’m installing to an SD card located at /dev/sda, and building for the PinePhone 1.2 (where 1.0 was the dev kit, 1.1 was Braveheart, and 1.2 was the UBports, PostmarketOS, and Manjaro CE’s). Adjust as needed; your mileage may vary.

mkdir -p ~/mobile/pinephone
cd ~/mobile/pinephone

mkdir -p bootloader/files # This is where the setup for p-boot will go

# Step 1: set up p-boot bootloader
# I don't compile this myself; I use the pre-compiled portions provided.
# Documentation and more info at https://megous.com/git/p-boot/tree/README
git clone https://megous.com/git/p-boot
# the p-boot-conf that comes pre-compiled is compiled for ARM, but we're running
# x86_64. We'll compile our own. If we were using this directly on the PinePhone
# we would be able to skip this step.
gcc -o p-boot/dist/p-boot-conf-native p-boot/src/conf.c

# Step 2: partition the SD card. 
# BE CAREFUL here; you will lose all data on /dev/sda, which could be your hard
# drive! If you're not sure, running `sudo fdisk -l` will tell you the size and
# model of your disks.
#
# Press <m> for help. My configuration looks like this:
## Disk /dev/sda: 28.89 GiB, 31016878080 bytes, 60579840 sectors
## Disk model: Card-Reader
## Units: sectors of 1 * 512 = 512 bytes
## Sector size (logical/physical): 512 bytes / 512 bytes
## I/O size (minimum/optimal): 512 bytes / 512 bytes
## Disklabel type: dos
## Disk identifier: 0x3a6e5394
## 
## Device     Boot  Start      End  Sectors  Size Id Type
## /dev/sda1  *     65536   589823   524288  256M 83 Linux
## /dev/sda2       589824 60579839 59990016 28.6G 83 Linux
#
# I did start the first partition quite a ways into the disk. This may not have
# been necessary, but I didn't want to run into issues with the p-boot binary
# itself (which gets written 8K into the disk) and the first partition, so I
# decided to offset it by 64K sectors (32MiB) into the disk.
sudo fdisk /dev/sda

# Step 3: Write the p-boot binary to the SD card
sudo dd if=p-boot/dist/p-boot.bin of=/dev/sda bs=1024 seek=8

# Step 4: Compile the Linux kernel
# More info at https://github.com/megous/linux/blob/orange-pi-5.10/README.md
# We need a cross-compiler. On Debian derivatives, this would probably be the
# build-essential package.
sudo pacman -Sy aarch64-linux-gnu-gcc
# I'm using megi's kernel from https://megous.com/git/linux/, but for bandwidth
# reasons cloning from the GitHub mirror. (This is a 4GBish repo!)
git clone https://github.com/megous/linux
cd linux
# I was initially confused that the PinePhone branch (currently pp-5.10) wasn't
# the correct branch to build. Instead, there are defconfigs for the PinePhone
# in the orange-pi-* branches.
git checkout orange-pi-5.10
# We'll be using the default configs here:
ARCH=arm64 make pinephone_defconfig
# Time to build! I have an 16-thread processor; scale this for your machine. You
# may have to answer a few questions that the default config did not cover; I'm
# not sure exactly why they don't get populated in the previous step.
ARCH=arm64 \
    CROSS_COMPILE=aarch64-linux-gnu- \
    make -j16 Image dtbs
mv arch/arm64/boot/dts/allwinner/sun50i-a64-pinephone-1.2.dtb ../bootloader/
mv arch/arm64/boot/Image ../bootloader/
cd ..

# Step 5: Build the initramfs
# Initramfs uses a cpio archive to store the filesystem, so we need the `cpio`
# tool. https://en.wikipedia.org/wiki/Initial_ramdisk#Implementation
sudo pacman -Sy cpio
# https://www.jootamam.net/howto-initramfs-image.htm gives a decent walkthrough
# of this process, as well as some background behind why this works.
# First, we create the filesystem:
mkdir -p initramfs/{bin,sbin,etc,proc,sys,usr/bin,usr/sbin}
# Install busybox for a reasonable range of utilities:
curl -O https://www.busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-armv8l
mv busybox-armv8l initramfs/bin/busybox
chmod +x initramfs/bin/busybox
ln -s busybox initramfs/bin/sh
# We create something for the kernel to run as the init process:
cat > initramfs/init <<EOF
#!/bin/sh

# Create all the symlinks to /bin/busybox
/bin/busybox --install -s

i=1
# Loop forever, displaying an incrementing value
while true; do
	echo Hello world \$i
	i=\$((i+1))
	sleep 1
done
EOF
# The init process should never exit; in theory we should be calling shutdown()
# somewhere in here, but we don't have a shutdown program written yet! However,
# since we're in the initramfs, we don't really have much to worry about with
# an improper shutdown here.
chmod +x initramfs/init
# And create our image!
cd initramfs
find . | cpio -H newc -o > ../bootloader/initramfs.cpio
cd ..

# Step 6: Install all this to the SD card
cat > bootloader/boot.conf <<EOF
no=1
name=Chandler's custom Linux
dtb=sun50i-a64-pinephone-1.2.dtb
atf=fw.bin
linux=Image
initramfs=initramfs.cpio
EOF
cp p-boot/dist/fw.bin bootloader # ATF (Arm Trusted Firmware)
sudo p-boot/dist/p-boot-conf-native bootloader /dev/sda1

# Boot and run!

Next steps

With this done, we should be able to insert your SD card, and boot to the init script that we added to the initramfs. Unfortunately, we’re missing quite a few things necessary for a functional system. Specifically¸ we don’t have util-linux (Wikipedia), which would normally provide us with tools like mount that are fairly essential for a functioning system (particularly so since we need to mount things like /dev to talk to hardware!). We don’t do anything yet with the second partition on the kernel; while we could use that instead of an initramfs, that would add some time to the boot process while providing little benefit at this stage.

Piano Heatmap Analysis (Part 3)

Sat, 07 Nov 2020 00:00:00 +0000

A third post in a series (part 1, part 2) on exploring what keys of a piano I use the most.

I’m a sucker for data. Doesn’t really matter about what—lately, election polling has been interesting to watch—but I really like seeing numbers about things I do. Music is one interesting thing, and I like that I can see my stats on last.fm, or the annual email from Spotify with my year in review. Whether it’s using time-tracking software or using a fitness app to track duration/pace/heart rate/etc when running or using GPS data and OBD sensors to figure out what part of my commute burns the most gas¹, I really enjoy being able to see changes even in fairly useless data. With that in mind, here’s an update on my attempt to keep track of the keys I play.

We left off with a page that would keep track of keys played, but every time the page was reloaded (which wasn’t infrequent, given that Edge² crashed more than occasionally when MIDI devices were added and removed) the data would be reset, so it was difficult to keep a running total.

To mitigate this, I added support for saving data per-session. If Edge decides to stop accepting MIDI data, I can simply save the session and restart the browser. This uses Dexie as an easy wrapper for IndexedDB, and stores that data in the browser’s storage³. I added some options to save the current play session, and a few extra page links:

I can then see my per-day/per-session stats:

This probably isn’t the endgame for this project, but it’s a nice move in the direction of collecting more data. Like I said, I like data! And it’s much easier to do any kind of analysis later if I’ve already been collecting the data than it would be if I wasn’t already keeping track! Here’s the aggregated total of the first few days’ collected data:

As usual, the source and the application are available to view or modify!

Unfortunately, this one was starting to bubble up to the top of my project list just as full COVID-19 stay-at-home measures started, so it hasn’t borne any fruit yet. Someday! ↩︎
Since the laptop I’m using occasionally also needs to run Hauptwerk, I’m kind of stuck running Windows. Now that Edge is Chromium-based, I don’t have a strong need to install Firefox as a default browser, if all I’m doing is occasional browsing. ↩︎
This does mean that you can’t (easily?) sync your sessions across devices, nor share your data with anyone else. That said, it wouldn’t be too difficult to automatically upload and share this data publicly, à la WhatPulse. I just haven’t done that. ↩︎

Smartwatches

Fri, 16 Oct 2020 00:00:00 +0000

I’ve owned two smartwatches, across 3 OS’s. They’re all terrible. Here’s why.

Samsung Galaxy Watch Active

In June 2019 I bought a Samsung Galaxy Watch Active. It’s a very slick device, and checked off the few checkboxes I had specifically selected it for:

Decent battery life: Unlike many ~~Android Wear~~ WearOS devices, I typically get two, sometimes three days of battery life with low-moderate use.
Offline Spotify playback: This was purchased specifically as a device to take running when I didn’t want a phone, that I could listen to music through. Spotify playback when offline, however, is pretty buggy, especially with newer updates. Spotify on WearOS doesn’t have this capability. As far as I know, offline playback on WearOS is limited to Google Play Music, which has, I believe, been deprecated in favor of some YouTube offering.
Compact: At the time of purchase, I think this may have been the smallest non-Apple full smartwatch¹ on the market. I don’t want to strap a huge device to my wrist; I’m trying to get away from my huge devices!

With all the features, though, I’ve found that I want my watch to do three things, and this does none of them exceptionally well:

I want it to be able to play music. Local music is flawless, but Spotify stutters and often outright crashes when disconnected from the phone. Recently, it started indicating that songs are saved offline, but then saying “Song not available” when I try to play it. Connecting to Bluetooth earbuds is not difficult, but doesn’t always work the first time either.
Fitness tracking. This may be its best point; it’s pretty decent. However, it requires Samsung’s “Samsung Health” app, and doesn’t offer good integration with other platforms. If you’re on a Samsung phone, it’s convenient; if not, it’s a bit more of a hassle. (Supposedly Strava support is also okay, but I haven’t tested this.)
I just want to know the dang time! Ironically, this device is quite poor at that, compared to the ~~dumb~~ supremely cool Timex Expedition digital watches that lasted me years as a kid². The battery life is pretty bad if I use the always-on display, but the raise-to-wake feature doesn’t always work, especially if I’m laying down. And “I’m sorry, I don’t know what time it is because I forgot to charge my watch”…just seems a bit silly!

A fourth, less tangible point is the ability to do what I want with it. It’s running Samsung’s locked down OS; I can’t easily develop or install Android apps for it since it’s running some Tizen-based OS, and of course I don’t have root access on it. Samsung is reputable enough that they’re probably not shipping off all my biometrics to China, but that’s not a sure bet either.

One thing that it does do well is notifications. My phone is now nearly always on silent mode, and important notifications (select texts and calls, mostly) buzz my watch. Bluetooth range is good throughout the house if I leave my phone on my centrally located desk, so it’s not uncommon for me to go a full day or two without knowing where my phone is. And if I do lose my phone, the find-my-phone feature is nice as well—no more “lost my phone, can you call it please?”!

KingWear KW88 Pro

I’ve been peripherally aware of the AsteroidOS Project for a few years now, but hadn’t ever taken the plunge to try it out. I purchased a KingWear KW88 Pro from GearBest in November of 2019, when it was on sale for $79.99—reasonable enough, I figure, that even if it never works for me, it’s not a $200+ experiment like many other watches. Also, it had some unique features that drew my attention:

A speaker that, on testing, is as loud as most smartphones.
1GB RAM, 16GB ROM, when many WearOS smartwatches come with 512M/4GB.
Camera! It’s a bit gimmicky and not particularly high quality, but it’s a camera.
It does have a heart rate monitor, which many low-end watches seem to lack.
While I don’t plan to use this, there’s a 3G modem and SIM slot, so I could use it for placing calls and texts without a phone.
Decent display: full circle (no flat tire); quite bright, decent size, 400x400px.

I wouldn’t even consider using the version of Android that came with this watch, however. I don’t have words to express how bad it is. As far as I can tell, they took the full smartphone Android and made as many tweaks as was necessary to get it to compile on the watch (and no more), and shipped it. The menus are mostly English, but the text that isn’t included in stock Android looks like it hasn’t been proofread by a native English-speaking proofreader³—lots of phrases like “Please waiting for update…” or “No notification” rather than “No notifications”. It’s running an ancient version of Android, last patched in 2017, and with a 3.18.x Linux kernel. The apps’ support for hardware is good (camera, accelerometer, etc. all work), with the exception of the display: apps made for a square display haven’t been modified, so important buttons are offscreen.

Check out this quick screen capture for an idea of what I mean. Nothing is unusable, or even that remarkable by itself, but after using it for a while it’s definitely a death-by-a-thousand-cuts UI disaster. The black overlay around the outside shows where the screen ends. Play Store and Maps seem to be the worst offenders, with both displaying important controls and information outside of the visible area. Other apps are (mostly) better, but suffer from their own localization, discoverability, and functionality issues. (The Contacts app, for example, seemed prone to random crashes.)

There’s an app that goes along with it. They don’t provide a Play Store link, just a link to a download page. The app won’t start without location, camera/microphone, and file access permissions, and suffers from similar UI issues as the watch, though at least all of the important buttons are on screen. And then there’s permissions: The app won’t even let you connect to the watch unless it can:

Take pictures and record video
Access this device’s location
Access photos, media, and files on your device

It does seemingly allow you to bypass call log, contacts, phone, display over other apps, and notification permissions, but I had to put in some effort to make that happen.

Now, these all make some sense as to why they might be necessary. That said, for an app specifically downloaded outside of the Play Store from a manufacturer I’ve never heard of, I’m a bit hesitant to grant pretty much every major permission on my device.

KingWear KW88 Pro, AsteroidOS

AsteroidOS is lovely. I can ssh in to a root shell, and run lots of neat things⁴ on it! But it’s nowhere near mature enough where I’d consider it ready for daily use, particularly not on this watch. I’m attempting to contribute to get it to a more stable and feature-complete state, but the going is slow.

I’m still missing, at a minimum:

Some kind of fitness tracking—I like to know time, pace, and distance when I’m running. Heart rate, GPS tracks, cadence, step count are all nice, but not strictly necessary. I also really like having music support over Bluetooth headphones.
A stable Bluetooth connection—Right now I can connect things to the watch, but generally only for a few seconds at a time, and it’ll disconnect, which I haven’t fixed yet. This applies to both the BLE connection to my phone and full-fat Bluetooth connections to game controllers, headsets, and the like.
Wifi connection—I currently have to plug the watch into my computer to connect to a network. Wifi seems to be in worse shape than Bluetooth; scanning for networks works, but wpa_supplicant keeps DEAUTHing itself.
Better battery life. It seems intermittent; sometimes I get nearly two days, and sometimes it doesn’t last a day. May have something to do with whether Wifi and Bluetooth are active and scanning?

Conclusions

In addition to the watches I’ve used, I’ve spent some time with plain old digital watches (which I generally quite like), a Huawei Watch 1, which was impressive in its day but lackluster compared to newer offerings, and a Ticwatch S, which was good for its price point, and offered a nice Android Wear experience. I have not tried either a Fitbit or an Apple Watch, though I hear encouraging things about both. I definitely see a lot of flaws in the smartwatch ecosystem, but I also see a lot of progress! I’m genuinely excited to see what watches will look like in a few years, as technology continues to improve.

For all the watches: Notifications on my wrist are really, really nice. It feels to me like I’ve finally moved from polling to interrupts, and anything that can reduce the amount of time I spend looking at my phone is good in my book!

Currently, AsteroidOS seems most strongly on track to provide the experience I want, and unlike the other options, I can actually fix what I don’t like about AsteroidOS. Both proprietary options are okay, but have gripes that simply can’t be fixed, and I can’t use the tools I’m used to to develop for or use them. (I was stunned to find out that rsync Just Worked™ to copy my media over; now all I have to do is write a music player!) Perhaps a sub-conclusion—it’s amazing how much more comfortable I feel working on these projects when I have a familiar POSIX-ish environment everywhere!

Besides the practicalities, AsteroidOS provides some nice intangibles too. I like the community, and I’ve felt welcome and appreciated as I’ve tried to step up and contribute to the project. I like the design philosophy; Linux and Wayland and DBus and BlueZ and all the other things I use on my desktop work on my watch too, just on a smaller scale. I may not be running Firefox and Sway… but I could if I wanted to! (I have contemplated a terminal, if I can get a bluetooth keyboard working!)

With all this in mind, I’m sticking with AsteroidOS for the near future, and I’m hoping to be able to deliver some progress for the ecosystem myself. Ah, the joys of open source!

Thanks to Käthe for reviewing a draft of this post.

For this comparison, I’m not counting Fitbit watches and other similar watches, since they won’t do music playback, although they do excel at the time-telling and fitness-tracking portions of my use case. Basically, “can I write and install an app” is probably the defining question for this. ↩︎
As I recall, with only occasional backlight use, I tended to get between two and three years of battery life out of each iteration of this or a similar watch. At the end, the battery would die, I’d replace it, but once the waterproof seal on the back cover was broken, it would last maybe three to six months before failing in a light rainstorm. This sequence repeated three or four times between third grade and high school. ↩︎
Though, credit where it’s due, I can nearly always tell what they’re trying to say, and their localization (like MM-DD-YYYY for USA, and options for imperial units in addition to metric) is pretty good. ↩︎
So far, vim and a lightweight web server, a standard POSIX environment, Quake 1, a Minecraft server (only kinda-sorta-working), and probably others I’m not recalling. No k8s cluster yet! ↩︎

A phone a month for a year: September: Custom LineageOS 16.0/MicroG

Thu, 08 Oct 2020 00:00:00 +0000

Here’s another post about the open-source phone exploration I started in July. Last month, I ran an unofficial LineageOS/microG build (containing no Google apps) to pretty good results.

The fun thing about being with you is that when I call and you don’t answer, there are so many reasons it could be.

–Käthe, yesterday, after my PinePhone looked like it might not receive calls

Not too many interesting points this month, which, in my opinion, is a good thing. Overall, it’s an OS I’m quite happy with, and one that strikes a good balance of reliability and privacy.

Apps

LineageOS takes its apps pretty much straight from AOSP, with a few tweaks that I generally appreciate. It’s a nice default set of apps—browser, email, camera, file manager, dialer; all functional enough that I don’t feel compelled to immediately add more. I did make the following shifts:

Play Store → F-Droid and Aurora Store (for apps like Spotify that I can’t get from F-Droid)
Google’s sync → DAVx⁵, with which I can sync my calendar and contacts.
Google Keep → OpenTasks
YouTube → NewPipe

I haven’t found a Maps app I particularly like, but Google Maps works fine with microG, so that’s my one remaining Google app. I’d also like an Android Auto app, but support for that is required and not currently implemented in microG, so that’ll have to wait for the time being.

Beyond the Lineage apps and Google replacements, I have a handful of apps that cover the rest of what I want to use my device for:

AsteroidOS Sync (F-Droid)
Bitwarden, via a custom F-Droid repo
Menards (Play Store, works without Google libraries)
Spotify (Play Store, works without Google libraries)
Pancheros (Play Store, sort of works without Google libraries)
RevolutionIRC (F-Droid)

Miscellany

I was pleasantly surprised that the Moto Actions I had set up persisted after I reset my phone, so I still have my chop-chop for a flashlight. Presumably the Moto Actions app writes something to the gesture detection chip that survives an internal storage wipe?

I’ve only run into one noticeable bug: Display colors are sometimes wonky after turning the phone on. I’ve found that this can be fixed by changing the LiveDisplay mode in Settings, or toggling sRGB in Developer Options…perhaps this resets something in the graphics chip that isn’t set properly on boot? It seems like it doesn’t really matter what setting I change nor what I change it to; any combination works, as long as it’s not the combination the phone was booted with. Here’s what I see, compared to what the phone thinks I should be seeing:

LineageOS quietly added official LOS 17.1 builds for ocean early in the month, so perhaps that’s in the future! MicroG has them too!

Next Month

[chandler@xenon ~]$ ssh chandlers-pinephone
Welcome to postmarketOS!

This distribution is based on Alpine Linux.
Read both our wikis to find a large amount of how-to guides and
general information about administrating and development.
See <https://wiki.postmarketos.org> and <https://wiki.alpinelinux.org>.

You may change this message by editing /etc/motd.

chandlers-pinephone:~$

(meta-)*tic-tac-toe

Sun, 04 Oct 2020 00:00:00 +0000

After a particularly interesting talk at UMD’s math club, to fill some left-over time, the speaker of the day introduced to us a game that was new to me: Meta Tic-Tac-Toe¹. After a few games on the chalkboard, I was really interested, and spent a few weeks asking anyone who would listen to play a game or three. I can’t say I was ever good at it, but it was certainly fun!

If you haven’t seen the rules, here’s an overview. The gist is that you’re playing on a tic-tac-toe grid full of tic-tac-toe grids, and you must win three in a row. There’s a twist to make it interesting, though: The location of the previous play determines which grid the opponent plays on. For example, an X in the top left corner of a grid means the opponent will next have to play in the top left grid, and that move determines where you’ll play next.

     │ │  ┃  │ │  ┃  │ │      ?│?│? ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃  │ │ 
    ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─
     │ │  ┃  │ │  ┃  │ │      ?│?│? ┃  │ │  ┃  │ │       │ │o ┃  │ │  ┃  │ │ 
    ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─
     │ │  ┃  │ │  ┃  │ │      ?│?│? ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃  │ │ 
    ━━━━━━╋━━━━━━━╋━━━━━━━    ━━━━━━╋━━━━━━━╋━━━━━━━    ━━━━━━╋━━━━━━━╋━━━━━━━
     │ │  ┃ x│ │  ┃  │ │       │ │  ┃ x│ │  ┃  │ │       │ │  ┃ x│ │  ┃ ?│?│?
    ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─
     │ │  ┃  │ │  ┃  │ │   ->  │ │  ┃  │ │  ┃  │ │   ->  │ │  ┃  │ │  ┃ ?│?│?
    ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─
     │ │  ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃ ?│?│?
    ━━━━━━╋━━━━━━━╋━━━━━━━    ━━━━━━╋━━━━━━━╋━━━━━━━    ━━━━━━╋━━━━━━━╋━━━━━━━
     │ │  ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃  │ │ 
    ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─
     │ │  ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃  │ │ 
    ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─     ─┼─┼─ ┃ ─┼─┼─ ┃ ─┼─┼─
     │ │  ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃  │ │       │ │  ┃  │ │  ┃  │ │

If you haven’t played, it’s worth noting that moves near the beginning often feel pretty random, and for me that’s usually the case. I haven’t worked out much strategy for moves at the beginning, where they don’t have much (obvious) impact on later game state. Moves toward the end tend to have much more immediate impact, and are usually where the bulk of my gameplay time is spent.

To make it easier to get other people (say, those not in the same room as me) to play, I wrote a server that supports playing games online. It was a nice weekend hack, though for a variety of reasons not exactly production ready!

But here’s the thing about nesting tic-tac-toe boards: Why only once? What if we went deeper? Well, here’s what we came up with for meta-meta-tic-tac-toe² ³. It’s played pretty much the same as meta-tic-tac-toe, with a few rules clarifications. Like 2-deep tic-tac-toe, previous moves determine where you’re allowed to play. The conclusion we came to for 3-deep is that your top-level board is that of the second-level board from last time, and the second-level board is determined by which square was chosen last turn. More generally, for m-nested boards⁴, your nth turn is played at top-level board n-m, second level board n-m-1, …, and the smallest board at n-1. For example, here are the first few moves of a possible game:

I find this confusing to reason about logically, but intuitively it makes fairly decent sense; each turn you’re progressively less affected by older moves. For me, the lightbulb moment was when I realized that each move the area a move is constrained to by an old move expands one board level. With that in mind, I visualize an expanding board when I’m playing. Suppose I’m playing in the yellow highlighted area below. Expanding the second-level board to cover the top-level board gives an idea of where the next move will direct the players. In this case, a move in the orange square will send one’s opponent to the bottom left sub-sub-grid of the middle right sub-grid.

This scales, too: The move after the next can be visualized by zooming out once more. In this example, the next move will be in the bottom-left sub-sub-grid of the center sub-grid, and the following move will be in the bottom-left sub-grid, with a sub-sub-grid dependent on the location of the next move.

Here are some more examples⁵:

And, because why not, here’s a four-deep example. I don’t know how long exactly this would take to play, but assuming you have to fill about half the spaces, two players placing collectively one token per second would take almost an hour to play. And I generally take >10 seconds a move! This can slow down a browser a bit, so click the image, which is a link to the animated version.

The other note, as compared to meta tic-tac-toe, is that when you’re directed to play on a bottom-level grid, you recursively escalate boards until you reach a playable board. So if your opponent made a move sending you to a board that’s full, you can play anywhere on a second level board; if that’s full, you can play anywhere on a next-level board. These rules seem (mostly?) unavoidable, but I haven’t played enough games to work out if there are odd implications of this.

If anybody else has been playing by these (or different!) rulesets, let me know! I’d enjoy hearing about your games, and whether or not you’ve liked the ruleset!

Thanks to Jeff and Isaac for reviewing a draft of this post.

It was introduced as Super Tic-Tac-Toe, a name I tend to like better, but Meta* Tic-Tac-Toe seems to capture the recursive nature of what we’re trying to do here more effectively. ↩︎
or super-duper-tic-tac-toe, if you’re so inclined. ↩︎
I haven’t been able to find any rules for how exactly the game is played, nor analysis on whether or not it’s fair. But if there’s any mention on the ’net that I missed, I’d be grateful to hear about it! ↩︎
This does not counting the top level board; 0-nested is normal tic-tac-toe; 1-nested is 9 boards inside one larger board, etc. ↩︎
These examples were all generated using a short Python script available here. The script should support arbitrary nesting with maybe minor tweaks, but the SVG files get large pretty quickly!

I had initially tried using Inkscape to generate these, but I had to do some manual editing of the SVG files, and Inkscape doesn’t do a very nice job of outputting human-readable SVG. Between a surprisingly readable SVG spec and the wonder of modern technology that is MDN’s reference on the subject, it wound up being a pretty quick project. And arbitrary recursion is fun!

Much of the logic was borrowed from an client I started trying to develop, which hopefully should support arbitrary nesting. When that’s done, it’ll probably show up on GitHub.

To generate the images I’ve been using, you can simply
```
./generate.py -l 1 1,0 2,1 0,2 > anim-1.svg
./generate.py 1,0 2,1 0,2 > anim-2.svg
./generate.py 1,1 1,1 0,1 > anim-3.svg
./generate.py 2,0 0,2 2,0 > anim-4.svg
./generate.py -n 0,1 1,2 2,1 1,0 > too-meta-static.svg
./generate.py 0,1 1,2 2,1 1,0 > too-meta.svg
```
↩︎

Chandler Swift's Blog

Notes on installing LineageOS on OnePlus 9 ("lemonade")

Contents

adb/fastboot on NixOS

Correct Firmware Version

Root

Advent of Code 2022: Day 22

Soundtrack

Advent of Code 2022: Days 20 and 21

Day 20

Day 21

Soundtrack

Advent of Code 2022: Days 18 and 19

Day 18

Day 19

Soundtrack

New Year, New Router!

Contents

Hardware

Acquisition: Ordered from AliExpress

Performance: Adequate for 200mbps

Thermals: Pretty good with the heatsink

Power consumption: Typically 2-3 Watts

Other considered hardware

Installing OpenWRT

Configuring OpenWRT

Basic routing: Plug and play

Static DHCP leases

Port forwarding

DDNS

Monitoring/pretty graphs with collectd

Disk space

Unimplemented/Unresolved issues

Reboot time: 30 seconds

UI samples

Advent of Code 2022: Days 16 and 17

Day 16

Day 17

Soundtrack

Advent of Code 2022: Days 12–15

Day 12

Day 13

Day 14

Day 15

Soundtrack

Advent of Code 2022: Days 10 and 11

Day 10

Day 11

Soundtrack

Advent of Code 2022: Days 8 and 9

Day 8

Day 9

Soundtrack

Advent of Code 2022: The rest of Week 1

Day 2

Day 3

Day 4

Day 5

Day 6

Day 7

Notes

Soundtrack

Advent of Code 2022: Day 1

Setting up a dev environment

Code

Notes

Time spent

Soundtrack

Twos

Ages of Minnesota Wild Players

2021 Taxes: Open Source Edition

Contents

Introduction

Getting started

UsTaxes.org

OpenTaxSolver

Paid providers

Pricing

TurboTax

FreeTaxUSA

`adb`/`fastboot` on NixOS

`femtotorrent` project scope