Incredible dive into something I’ve only dreamed of doing, this post is definitely one of my favorites. If the author is reading this, would love to know where you got those chairs!
Gentoo is the best! Once you get the hang of creating a bootable system and feel comfortable painting outside the lines, it feels like Linux from Scratch just without needing to manually build everything. I automated building system images with just podman (to build the rootfs) and qemu (test boot & write the rootfs, foreign arch emulation) and basically just build new system images once a week w/ CI for all my hardware + rsync to update. Probably one of the coolest things I’ve ever built, at this point I’m effectively building my own Linux distro from source and it’s all defined in Containerfiles! I have such affection for the Gentoo team for enabling this project, shocking to discover how little they operate on I’m definitely setting up a recurring donation.
I think it is a great learning opportunity, but after using Gentoo for a decade or so, I prefer Arch these days. So if you want to learn more about Linux and its ecosystems, go for it, do it for a few months or years.
That said, I haven't tried Gentoo with binaries from official repositories yet. Maybe that makes it less time-consuming to keep your system up to date.
Been happily and very successfully using the official binpkgs, it works really well, sometimes there's a slight delay for the binary versions of the source packages to appear in the repositories, but that's about it. I guess it's kind of running Arch, but with portage <3! And the occasional compilation because your use flags didn't really match the binaries
I would encourage you to write about it as well. It seems interesting and unconventional.
I used to tinker a lot with my systems but as I gotten older and my time became more limited, I've abandoned a lot of it and now favor "getting things done". Though I still tinker a lot with my systems and have my workflow and system setup, it is no longer at the level of re-compiling the kernel with my specific optimization sort of thing, if that makes sense. I am now paid to "tinker" with my clients' systems but I stay away from the unconventional there, if I can.
I did reach a point where describing systems is useful at least as a way of documenting them. I keep on circling around nixos but haven't taken the plunge yet. It feels like containerfiles are an easier approach but they(at least docker does) sort of feel designed around describing application environments as opposed to full system environments. So your approach is intriguing.
> It feels like containerfiles are an easier approach but they(at least docker does) sort of feel designed around describing application environments as opposed to full system environments.
They absolutely are! I actually originally just wanted a base container image for running services on my hosts that a.) I could produce a full source code listing for and b.) have full visibility over the BoM, and realized I could just ‘FROM scratch’ & pull in gentoo’s stage3 to basically achieve that. That also happens to be the first thing you do in a new gentoo chroot, and I realized that pretty much every step in the gentoo install media that you run after (installing software, building the kernel, setting up users, etc) could also be run in the container. What are containers if not “portable executable chroots” after all? My first version of this build system was literally to then copy / on the container to a mounted disk I manually formatted. Writing to disk is actually the most unnatural part of this whole setup since no one really has a good solution for doing this without using the kernel; I used to format and mount devices directly in a privileged container but now I just boot a qemu VM in an unprivileged container and do it in an initramfs since I was already building those manually too. I found while iterating on this that all of the advantages you get from Containerfiles (portability, repeatability, caching, minimal host runtime, etc) naturally translated over to the OS builder project, and since I like deploying services as containers anyways there’s a high degree of reuse going on vs needing separate tools and paradigms everywhere.
I’ll definitely write it up and post it to HN at some point, trying to compact the whole project in just that blurb felt painful.
Not what was mentioned by parent but I've been working on an embedded Linux build system that uses rootfs from container images: https://makrocosm.github.io/makrocosm/
The example project uses Alpine base container images, but I'm using a Debian base container for something else I'm working on.
Honestly this is just sorta a Tuesday for an advanced Gentoo user? There are lots of ways to do this documented on the Gentoo wiki. Ask in IRC or on the Forum if you can't find it. "Catalyst" is the method used by the internal build systems to produce images, for instance https://wiki.gentoo.org/wiki/Catalyst.
Gentoo is LFS but with the interdependence between packages mapped out for you (all hail the USE flags!) Or, alternatively, Arch with even more customization knobs to twiddle.
I have had Gentoo in at least one nearby system (physical and/or VM) since about 15 years ago. It's always a blast interacting with it.
It's still 100% pure Gentoo (and actually these days even vanilla Gentoo itself offers precompiled binaries) so you still can compile things in rare cases that binary isn't already compiled with use/config that you want.
That’s mostly why I build system images in CI; my slowest builds (qemu user mode emulation of aarch64 for e.g. raspberry pi boards) can take multiple days so I just declared myself a 1 week window between updates and then just pull in the changes via rsync. I even boot the images with qemu as part of the testing cycle. At some point I might try building and hosting prebuilt bins like gentoo does now, I don’t use those though because I explicitly want to build everything from source.
We burned thru pretty much all of our public /8, RFC1918, and have begun digging into RFC6589 (a /10 I didn’t even know existed prior to job). Still shocks me. Hardly an expert in the space, but I think the issue comes from subnetting to distribute ranges to teams that need a consistent IP address space for some project or another. Lots of inefficiency & hoarding over time. We’ve had legitimate outages and impending platform death staved off by last minute horse-trading & spooky technical work due to such things. IPV6 has always been a distant aspiration.
I’ve been toying around with the idea of using chaos engineering as a method of training new on-call folks. My first ever on-call shift was during a major product launch for a FAANG and I more or less just hoped that’d I’d be able to handle whatever broke. I got lucky and it turned out that I can usually fix things when they break, but have also found that jumping people in like that isn’t exactly consistent. I wonder if controlled, limited outages (maybe even as a surprise) would be a less hellish way of doing it. could be a good way to build instinct under pressure without risking too much.
Currently we do shadow shifts for a month or two first, but still eventually drop people into the deep end with whatever experience production gifts them in that time. That experience is almost certainly going to be a subset of the types of issues we see in a year, and the quantity isn’t predictable. Even if the shadowee drives the recovery, the shadow is still available for support & assurance. I don’t otherwise have a good solution for getting folks familiar with actually solving real-world problems with our systems, by themselves, under severe time pressure, and I was thinking controlled chaos could help bridge the gap.
You are making things harder for newer hires than the environment you came into. It is a sink over swim strategy that introduces stress without any apparent compensation in training. It creates new bases for evaluation you were not subject to.
Hazing us a cycle of abuse that expresses in a magnification of the abuse inflicted in the hazing than was suffered in the previous cycle.
Thanks for this perspective, I think I’ll reconsider this plan (to be clear, haven’t done it) and try to think up some alternative training strategy that doesn’t involve live issues.
Yep, have been on constant "pager duty" for 2+ years, although I have more help now and I get paged 1-3 times a week instead of per night. Still, carry my lappy everywhere I go. Bought an ARM Windows laptop to get that 20hr battery life so I could worry less during my travels. You know, fancy things like going get food or going grocery shopping.
Rough shift, my worst was every other week and my boss prior to hiring me was 24/7 just like you. I just carry a backpack with a few batteries + my work laptop, fortunately only a few really bad stories but hooooo boy me and that backpack have seen some fun times.
There's a company in England called "Cascode" who make firefighter alerters. These are really basic "beeper" pagers, which you can program to have a bunch of different tones and LED patterns based on the RIC and Subcode.
I look after several thousand of these across several hundred paging sites.
They're relatively inexpensive (70 quid or so in quantity) and they last about six weeks on a commonly-available AA battery. The batteries go flat enough to trigger the "low battery" beep at about 3am, for some reason. I don't know why.
There's no messaging involved, although the encoders are capable of sending a text string. The message is "get up and get down to the fire station right now", which generally needs no further explanation. POCSAG is unencrypted, so there would be privacy concerns with sending actual incident information in the clear with it.
While we're on the subject of old tech, until BT finally cut the last of them off, we use dialup modems to control the encoders (not dialup internet, just a hundreds-of-miles serial cable) as a backup, and dot-matrix printers to print out a hardcopy message for the crews to pick up.
All very low-tech. All very fixable. All stays working if you don't mess with it.
It's doable but it would be custom firmware and it's not really necessary. Two way paging isn't really worth doing because then you need a massive device with a massive battery, or something that uses uncontrolled mobile phone networks (and generally still has a massive battery, that lasts about a day).
You wouldn't even need particularly good encryption, you'd just need something adequate to stop casual eavesdropping really - "keep them busy for half an hour" would stop people from sniffing the POCSAG traffic and tweeting it, so that people show up at incidents and hang around filming it on their phones.
This incidentally is what a guy in England got arrested for a few years ago, exactly that. It's perfectly legal to listen to and decode pager messages (or any other radio messages), you're just not allowed to pass them on to people or act upon them, and posting them on twitter and then going round to rubberneck at the ongoing incident very much ticks those boxes. As with so many things in the UK, to paraphrase Aleister Crowley, "Don't Be A Dick shall be the whole of the law".
Oh no, I just always hear it termed that way and it captures the “feeling” for me since it feels like a dedicated device. I just just carry a work phone w/ PagerDuty during my shift.
We need to resist this stuff or else there will be Flock, stalker cars, and some other new nightmare they excuse by saying “well we’re already watching…”. Can’t let ourselves accept this is normal!
Why would you want different kernels for different device types?
Genuine question! I maintain my own Linux distro (upstream Linux + portage) for all my devices and haven’t found much reason to go beyond kernel per arch. I’m curious if there’s something I could be missing.
vm.swappiness defaults to 60, which is default from when everyone was still running spinning rust with a swap partition. Servers these days usually have very specific storage+memory configurations, whereas the usual desktop or laptop has an SSD and 16GB+ of RAM with RAM compression expanding it.
Lazy RCU loading is good on a laptop because you only lose about 10% performance and only with specific workloads, but your idle and light load energy consumption improves. Most laptops spend like 95%+ in light or idle load scenarios. Conversely, on a desktop you don't care (much) about idle and light load energy consumption, you only care about keeping max load consumption low enough so that your fans stay quiet. And on a workstation you don't care about a system being whisper quiet so you can go nuts with the energy consumption.
> vm.swappiness defaults to 60, which is default from when everyone was still running spinning rust with a swap partition. Servers these days usually have very specific storage+memory configurations, whereas the usual desktop or laptop has an SSD and 16GB+ of RAM with RAM compression expanding it.
You don't need to compile a specific kernel for that, this is setup via sysctl.
Do you mean RCU_LAZY? Most distros will already enable that: it doesn't do anything without rcu_nocbs, so there's no negative impact on server workloads.
[calvin@debian-trixie ~] grep RCU_LAZY /boot/config-6.12.57+deb13-amd64
CONFIG_RCU_LAZY=y
# CONFIG_RCU_LAZY_DEFAULT_OFF is not set
[calvin@debian-trixie ~] grep RCU_NOCB_CPU /boot/config-6.12.57+deb13-amd64
CONFIG_RCU_NOCB_CPU=y
# CONFIG_RCU_NOCB_CPU_DEFAULT_ALL is not set
Swappiness and many others can be changed by some sort of system preset rather built that way. I know not ALL options can be done that way, but I'd want to see changes start there where feasible.
I totally missed that part of your comment, my bad. Thanks for elaborating on those, I feel inspired to experiment!
So far my kernel journey has been about making my hardware work + enabling features, and that’s mostly how I’ve been discovering config options. Do you have any suggestions on where one aught to read further on this sort of kernel tuning?
EDIT: doing some further research, couldn’t you just set those options via sysctl w/o needing to build a separate kernel?
Yes you can adjust them via sysctl or directly as kernel parameter arguments. That isn't my point. My point is that Linux has some horrible defaults :+)
I generally have three types of Linux devices I typically use. My desktop, servers locally/remotely, and "mobile" devices (more like tablets I guess).
For the first, I want the lowest latency for everything I do, together with the highest burstable speed whenever possible, for pretty much all the components.
For the servers, I basically have two types, one which does storage, they just need large disks that can be slow, and one which users actually connect to, that one needs focus on throughput, latency and performance isn't as important as "can serve all requests in a reasonable timeframe, even under load".
Finally, many of the portable devices run on batteries, so on those the focus is power-saving, even if it compromises on performance.
I'm sure others out there have more device types, like ultraweight watches, security devices, monitors, radios and much more. Each one of these have different tradeoffs, and tuning the kernel and OS for each use case makes it a lot better usually. Personally I use NixOS for everything except my desktop (CachyOS right now!), and it makes it really trivial to create profiles based on the same configuration, deployed to all devices, and today they're are tuned for exactly their purpose, as Linus intended :)
