I have a built-in, knee-jerk sense of annoyance whenever I see a new compiled language. I can’t help it. It’s probably because I grew up on interactive BASIC. My first instinctive response is: “Great! A new language! We absolute need new and better approaches to programming! But, why on Earth did you sabotage your ideas right out of the box by making it compiled? Now we’ll have to start over from scratch with the next language.”
It is hard to explain this intuitive dislike for “compilation” to someone who didn’t grow up with interactive, interpreted languages. After all, compilation is currently considered the Best Practice of Best Practices in language design. It’s just what we do - it’s almost (perhaps not quite but almost) even what Computer Science itself is defined as being, the science of compiler design. If you’re not designing a compiler, are you even doing Programming Language Design? How could you have a language without compilation? What kind of strange person would you have to be, to be interested in programming languages yet NOT interested in compilation?
This essay is an attempt to sketch out a rationale for my intuitive sense of distress at compilation, perhaps just to try to simplify any future conversations I might have with bright-eyed language designers with yet another slightly modified take on C.
(Or in this case, and the language which prompted this essay: “Fennel”, which appears to be Clojure For Lua, and which sadly trips my sense of “How very close to being a good idea, and yet how very far. Please take the compiler out of this thing first, and then we can begin to talk.” Fennel: https://fennel-lang.org)
Okay. So what annoys me about compilation is, it breaks what I think is a major quality of good programming language design, but one that isn’t often identified. I will give this quality a name: “Smooth Gradient of Composability”.
So what is a Smooth Gradient of Composability?
What it means is that, in my aesthetic opinion, programming is about building Systems by means of Composition. A System is a thing made out of smaller parts, that are glued together to make a bigger thing with multiple parts but which is itself, in some important sense, still a single thing. That’s a System. One thing, multiple parts, made by putting parts together. So far, so good, right? We all agree on this.
Next, Composability is the ability to put parts together. Any tool or language intended for building Systems must have some amount of Composability in it. Otherwise it’s just going to be one big opaque lump - not a system-creating thing but a black box, an Application perhaps. It’s probably impossible for any piece of software to actually be incapable of Composability at some level: just by being able to send messages to a thing, means you can build it into a wider System. But not all tools or languages are necessarily equally good at this task.
A Smooth Gradient of Composability means specifically that you can compose parts together to make Systems at every scale, from small to big. As you put parts together, each set of parts then must have the same composability properties that an individual part has, and the same systemic properties that any system has. The tool or language must not impose arbitrary restrictions on how small or how large your parts or your system can be. If it does impose such restrictions, that’s where your Gradient of Composability is no longer Smooth: it has big jumps in it.
I want tools that don’t have big jumps in their Gradient of Composability. I want everything from a function call to a function or type declaration to an Object to a Program to a Process to a Virtual Machine to an Operating System to a Network… I want all of these kinds of Systems, and all things at levels in between them, to be equally able to be plugged together or taken apart. A smooth transition from small to large, at all scales.
Compiled languages, at least in the form we currently see them, and which bright-eyed and happy young language designers keep churning out, do not exhibit a Smooth Gradient of Composability. Instead they have a very lumpy gradient. This is how:
Compiled languages are based on Compilation Units. A Compilation Unit is either a file, or a set of files. (Sometimes just naming and selecting the set of files to be compiled is an extremely non-trivial problem and possibly an undecidable one: that’s why Package Managers exist, why all of them are awful, and why they all fight). However, a Compilation Unit is not really a thing that exists inside a compiled program, and it is certainly not the smallest thing. The smallest thing is usually an Expression or a Statement; the next smallest might be a Declaration or a Statement Block or a Function; there might be something like an Object. None of these things, however, can generally exist on their own in a compiled language. The compiler gets to ingest one huge chunk of Program, at once, and it emits one huge opaque chunk of Compiled Artifact (object code, machine code, bytecode, intermediate representation, or whatever).
Further, both source compilation units and binary compiled artifacts are pretty much obscure to the running program. They have structure inside them, yes, but this structure is greyed out and hidden.
This is very much Not a Smooth Gradient of Composibility. It is a large chunky gradient. Worse, it’s two entirely different sets of gradients, two completely separate worlds. One one side of the compiler you have a language built out of Types, Objects, Functions, Expressions, Statements etc. Then it goes through the distressingly loud and very clever machine full of parsers and optimisers and rotating blades and mulchers, and out the other side comes extruded sealed blocks: Libraries, Executables, Files.
So we are maintaining entirely separate worlds of our programming stuff, and the fine-grained world of the source language gets chunked into opaque grey mush once the compiler has had at it.
Then the pain of chunkiness doesn’t stop at just two separate worlds: A compiled program will then usually further go through the operating linker/loader/exec system and come out as Processes, Memory Pages, Network Sockets, etc.
Finally, in the Cloud world, we are still moving in an even chunkier direction: our equivalent of Compilation is Deployment, and that tends to work at the level of Containers and VMs. We’ve made an entire computer - or network of computers - our smallest unit of Composability.
This massive chunkiness all up in my composability gradients hurts my aesthetic sense immensely. I want something like a Cloud (without the privacy issues, but let’s assume I have my own Cloud that I own), but where functions and types and even expressions can exist as first-class entities, just one big soup of them.
In the worlds of Forth, Smalltalk and Lisp, we get closer to my ideal of smooth gradients. Not really there yet, but closer. For one, we don’t have such a separation between Language Runtime and Operating System: the language often controls the bare machine. Compilation in all three languags, where it occurs, happens at the level of individual functions. There is a strong sense that a system is a living, running entity, not a dead stopped one, and that changes happen in small increments.
I believe there is plenty of room in design space for more programming languages which put an emphasis on composability at small levels, and where the mechanisms of composability continue to scale smoothly up to the level of networks. I can see no reason why we need many of the arbitrary 1970s-era limitations that we have: the idea of process and machine, for example, shouldn’t really even be a thing in 2023.
To get there, though, we need this quality and this vision to be articulated in clear terms. We need an approach to compilation, where it’s necessary (and I’m not convinced it’s always necessary), that doesn’t throw away the fine structure inside a program when it turns it into object code. And we need an approach to maintaining and interacting with software systems that puts a priority on the liveness and always-on nature of the system (while, yes, also preserving the “can rebuild from scratch” quality that we have in files of source code).
The “gradient of composabilty” is I think one possible way of articulating this quality and this vision.