A few weeks ago, Apple released new versions of Xcode and Command Line Tools. Not thinking too hard about how my pkgsrc developer environment often gets broken by OS or tool updates, I updated promptly. For one thing, I’m kinda used to it. For another, it doesn’t usually break. For a third thing, managing dependencies — anything not my code that can break my code — means responsibility for dealing with the inevitable trouble, and therefore the sooner I find it the better. (More on my approach to life with dependencies.)

A vendor-provided toolchain is a significant dependency. So I accepted the Command Line Tools update, and it commandeered my spare time for two weeks as I hurried carefully to repair one of the world’s biggest continuous-integration cauldrons on one of its most popular platforms. When I ran my usual pkg_rolling-replace -suv to rebuild anything outdated, it did not go well at all.

Did I mention that a few weeks ago we were aiming to stabilize for yesterday’s quarterly release? Suddenly, if we didn’t scramble to straighten things out for macOS users, we’d have to manage a complicated situation for a while. But if we created a mess on other platforms by moving rashly, that’d be even worse.

The usual feedback mechanism for proposed infrastructure changes is to compare full bulk builds before and after. There was no time for that.

Happily, the conclusion of the story is boring: as always, the pkgsrc 2024Q1 stable branch supports macOS and its developer tools, including the latest releases of each. (So does -current pkgsrc, of course, if that’s your thing.)

Curious what we had to do to keep it boring? Read on.

Stricter clang defaults

Upstream Clang 16 and GCC 14 have promoted several warnings to errors by default, and Apple’s Clang 15 has followed suit. (Gentoo has very helpfully documented this for packagers.) These changes are intentional and well-intentioned, pushing maintainers to ship more reliable code. But pkgsrc’s job is to build nearly 30,000 codebases we don’t maintain. And stricter compiler defaults break a lot of builds.

As you might hope, we can make the breakage go away in one place.

In pkgsrc, packages declare which programming languages are required for their build. The compiler framework then selects package-and-platform-appropriate compilers, places them preferentially in the package’s build environment, and — crucially — intercepts compiler invocations and rewrites them for a variety of purposes.

When we look into pkgsrc’s clang logic, we find prior art for this specific class of problem. In September 2020, Xcode 12 (and its associated Command Line Tools) arrived even later in our quarterly schedule and promoted -Wimplicit-function-declaration to an error. The surgical fix: on macOS only, if invoking clang reveals the new stricter default, we pass -Wno-error=implicit-function-declaration to demote the error back to a warning.

Apple Clang 15’s new strictures aren’t observable in the same way, so we adjust our workaround: if clang doesn’t complain when we try demoting the new errors back to warnings, we pass those arguments too, via the same compiler-framework mechanism.

Missing m4 and yacc

This messy regression found only in the Command Line Tools 15.3.0.0.1.1708646388 update — not in the corresponding full Xcode 15.3 (build 15E204a) update — must have been unintended.

On macOS, some of the familiar Unix tools in /usr/bin are in fact stubs. When invoked, they either execute into the corresponding installed program (living somewhere under /Library/Developer) or prompt the user to install the Command Line Tools.

This Command Line Tools update uninstalls m4 and yacc from /Library/Developer. But since the OS-provided /usr/bin/m4 and /usr/bin/yacc stubs still exist, running m4 or yacc still does something: it pops up a window prompting you to reinstall the CLT. Unfortunately, as the latest available version doesn’t provide those tools, reinstalling is a waste of time.

As you might once again hope, we can hide the problem without personally visiting 29,000+ packages.

In pkgsrc, we also have a framework to control which non-compiler tools are invoked during builds. Packages declare which tools are required for their build. The tools framework then selects package-and-platform-appropriate incarnations of the declared tools and places them preferentially in the package’s build environment.

We just got handed a few new twists to handle in the framework, is all.

First, because this clever new CLT failure mode outfoxes our usual tool-detection mechanism, we special-case m4 and yacc detection on macOS, performing an existence check for the stubs’ targets. Then the selection mechanism’s usual fallback logic can provide them some other way. This prevents the primary source of needless CLT install popups. For non-macOS platforms, no change.

Second, because some packages might not yet be declaring all their tool dependencies, we special-case m4 and yacc handling on macOS: when they’re not declared, we place them in the build environment anyway, as no-ops. If the package build happens to invoke them, nothing happens. This prevents the secondary source of needless CLT install popups, at the risk of breaking macOS builds for packages that are missing these tool declarations and have heretofore gotten lucky; in such cases, the breakage will be obvious and the fix easy. For non-macOS platforms, no change. (At leisure, we might like to broaden this approach to help find and fix all undeclared tools on all platforms.)

Third, because the flex tool expects to invoke a GNU-compatible m4, we adjust the tools framework to infer gm4 from a flex declaration so that the framework controls which m4 gets found. This more correctly expresses our intent on all platforms, and in the macOS package build environment it restores /usr/bin/flex to a working state.

Broader xcrun search

We were already relying on xcrun for a couple of things, so when our new tool-detection special cases were sometimes getting surprising results from it, that was concerning. Turns out xcrun no longer looks solely in Apple-controlled locations, but also consults the environment’s $PATH. By invoking xcrun with an empty PATH and --no-cache, we obtain controlled, predictable tool detection.

Conclusion

Under the constraints, we changed as little as possible, as safely as possible, as similarly as possible to previous proven changes, avoiding novel constructs or any whiff of unforeseen consequences. We could not have done nearly as safe or thorough a job without good abstractions already in place. Total lines of pkgsrc infrastructure code changed: less than 100. Now that 2024Q1 is out, we have room to refactor.

These 15.3 updates also include a brand new linker. So far it hasn’t given us any trouble. If that changes, wanna guess whether we have one place to take care of it?