the arc of software bends towards understanding
Those that espouse the value of backwards compatibility often claim that backwards compatibility is simply a matter of never removing things. But anyone who has published APIs that involve data structures know that the story is not so simple. I’d like to describe my thought process on a recent BC problem I’m grappling with on the Cabal file format. As usual, I’m always interested in any insights and comments you might have.
In the PVP, you increment the minor version number if you add functions to a module, and the major version number if you remove function to a module. Intuitively, this is because adding functions is a backwards compatible change, while removing functions is a breaking change; to put it more formally, if the new interface is a subtype of the older interface, then only a minor version number bump is necessary.
Suppose that you want to parse a list separated by newlines, but you want to automatically ignore extra newlines (just in the same way that import declarations in a Haskell file can be separated by one or more newlines.) Historically, GHC has used a curious grammar to perform this parse (here, semicolons represent newlines):
decls : decls ';' decl
| decls ';'
| decl
| {- empty -}
It takes a bit of squinting, but what this grammar does is accept a list of decls, interspersed with one or more semicolons, with zero or more leading/trailing semicolons. For example, ;decl;;decl; parses as:
It’s not too hard to convince people that version bounds are poor approximation for a particular API that we depend on. What do we mean when we say >= 1.0 && < 1.1? A version bound is a proxy some set of modules and functions with some particular semantics that a library needs to be built. Version bounds are imprecise; what does a change from 1.0 to 1.1 mean? Clearly, we should instead write down the actual specification (either types or contracts) of what we need.
Rich Hickey recently gave a keynote at Clojure/conj 2016, meditating on the problems of versioning, specification and backwards compatibility in language ecosystems. In it, Rich considers the “extremist” view, what if we built a language ecosystem, where you never, ever broke backwards compatibility.
A large portion of the talk is spent grappling with the ramifications of this perspective. For example:
In general, Rich wants us to avoid breakage by turning all changes into accretion, where the old and new can coexist. “We need to bring functional programming [immutability] to the library ecosystem,” he says, “dependency hell is just mutability hell.” And to do this, there need to be tools for you to make a commitment to what it is that a library provides and requires, and not accidentally breaking this commitment when you release new versions of your software.
Backpack, a new system for mix-in packages in Haskell, has been released with GHC 8.2. Although Backpack is closely integrated with the Cabal package system, it’s still possible to play around with toy examples using a new command ghc --backpack. Before you get started, make sure you have a recent enough version of GHC:
ezyang@sabre:~$ ghc-8.2 --version
The Glorious Glasgow Haskell Compilation System, version 8.2.1
By the way, if you want to jump straight into Backpack for real (with Cabal packages and everything), skip this tutorial and jump to Try Backpack: Cabal packages.
There’s a shitty API and it’s ruining your day. What do you do?
Without making a moral judgment, I want to remark that there is something very different about these two approaches. In Dropbox’s case, Dropbox has no (direct) influence on what APIs Apple provides for its operating system. So it has no choice but to work within the boundaries of the existing API. (When Apple says jump, you say, “How high?”) But in Adam’s case, POSIX is implemented by an open source project Linux, and with some good ideas, Adam could implement his new interface on top of Linux (avoiding the necessity of writing an operating system from scratch.)
One of the early posts from this blog, from 2010, was on the subject of how to pick your string library in Haskell. Half a decade later, the Haskell ecosystem is still largely in the same situation as it was half a decade ago, where most of the boot libraries shipped with GHC (e.g., base) still use the String type, despite the existence of superior string types. The problem is twofold:
A common claim I keep seeing repeated is that there are too many language-specific package managers, and that we should use a distribution’s package manager instead. As an example, I opened the most recent HN discussion related to package managers, and sure enough the third comment was on this (very) dead horse. (But wait! There’s more.) But it rarely feels like there is any forward progress on these threads. Why?
Here is my hypothesis: these two camps of people are talking past each other, because the term “package manager” has been overloaded to mean two things:
When building a module system which supports parametrizing code over multiple implementations (i.e., functors), you run into an important implementation question: how do you compile said parametric code? In existing language implementations are three major schools of thought:
The repository perspective sounds nice, until you realize that it requires major architectural changes to the way your compiler works: most compilers don’t try to write intermediate results into some shared cache, and adding support for this can be quite complex and error-prone.