If you need quick hands-on experience with accidental complexity, just try reproducing a computational experiment paper, which I unfortunately did recently, and very likely you will find these extremely helpful features:
- The main program finds the dataset by name and parses it, so you don’t need to worry about paths and file types.
- The main program puts the results in a pre-configured well-known path, so their evaluation code can find it.
- The main program finishes the experiment and calls evaluation code, so I don’t need to know where the evaluation code is.
Of course, all of these features were buggy. When I ran into bugs, I needed to look at all the implementation details the author imagined I could skip. In the end, we could have been better off writing documentation for the core functions, instead of writing a “friendly” main program.
But why do people keep writing “friendly” main programs? A great amount of accidental complexity is repeated in every program not because of programmer incompetence, but systemic reasons. The following important problems are not adequately solved by the UNIX operating system/programming environment.
Persistence
Persistence is already a widely agreed requirement of substrates.
If the main program does not include unnecessary I/O code, the convention is to use standard input and output. Standard output is not persisted and also dangerous (control characters). Standard output is by default unreliable: it scrolls too far, breaks the terminal, and is not saved.
But we also have shell redirection, and it has none of these adverse effects. Why are people not using it? I can only guess: maybe the shell is not “proactive” enough; users want the shell to save the output first, then let them consider other options. This way they have one fewer thing to worry about. (Feel free to share your theory.)
In conclusion, if there is only one output stream, a mandatory --output-file argument is just unnecessary complexity. We should simply modify the shell to persist what the user wants. Although uncommon in practice, this can still be done with more output streams.
Of course, my previously proposed nonlinear operating system should be a systematic solution to this problem.
File Types
If the operating system is a programming language, each variable is a “file” which already has a type. This is not true for the UNIX file, which is always a byte array. So, the main program needs to write file type abstraction code.
On one hand, this “programming language as OS” stops making sense if you have byte arrays everywhere like a regular filesystem. It really is necessary to have typed variables everywhere. If the abstraction layer is present, the serialized presentation is not important at all. Are we ready to demote serialized representation? Let the runtime take over all serialization?
On the other hand, it is conceptually sound to have binary formats, as long as the user knows the format. You can always build abstractions on top. Untyped filesystems are successful because they care no more about bytes. Additionally, the byte array file can be used with any abstraction implementation, while a variable of a certain type can only be backed by a single abstraction implementation. Are we ready to lose this flexibility?
My suggestion: Have a type-agnostic kernel, and allow multiple independent abstraction views of one underlying data.
Core Function Discovery
The main program can only be as helpful as its help message, which shows the core (business logic) functions I can call, and what I need to do to be routed to each core function. Without a main program, how will I discover the core functions?
Let’s consider static analysis. We can get a dependency graph of the functions. The unused functions (no other code calls them) are definitely entry points. But some used functions can also be entry points.
The ordering of the functions can also be exploited. The author has usually decided to order them in a certain meaningful way. Many IDEs reorder functions lexically in a context, which is really detrimental to function discovery.
In general, since this is quite subjective, I think no one but the author must indicate the core functions. For example, they can annotate core functions to be displayed with more importance. They can also write a literate example program that calls all core functions.
Note: I did not mention “exports” because I still want to be able to call the helpers. They just need to be very unimportant when I initially explore the code.
