Feedback on a malleable operating system

Hi, I’m Chase :wave:

I’m part of a small team that’s working on a new kind of operating system which embodies many of the properties discussed in this forum. Here’s how we describe it:

Pallas is an event sourced, purely functional operating system, called an operating function.

Pallas provides an entirely unique set of features, but is inspired by a long history of systems and language research, including MIT’s exokernel. Pallas collapses the distinction between database, virtual machine, and language platform, creating a stable, simple, and extensible programming environment.

The cost to this design is a clean break with legacy systems, including Unix and most popular programming languages. In return, Pallas offers a set of features that are found nowhere else:

  • All application data is automatically persisted, without the need for imports or boilerplate. To create a database, you write a pure function. Open system calls are included in persisted state, and so are resumed on reboot.
  • Partially applied functions can be serialized and stored on-disk, or sent over the wire. Programs in mid-execution can be paused, moved to a new machine, and resumed with no impact.
  • Program execution can be parallelized via a process model. A single machine can spawn and manage multiple processes.
  • The system’s default language blends features from both Lisp and Haskell, with a syntax that is more readable and flexible than S-expressions without sacrificing regularity or homoiconicity. Metaprogramming capabilities include hot reload, zero-overhead virtualization, macro-based type systems, all the way up to custom compilers.
  • Data and code is deduplicated, merkleized, and stored in content-addressable memory pages. This creates a global referentially-transparent content store, which is naturally complemented by protocols like BitTorrent.

The foundation of Pallas is untyped, but conceptually we can say that a database is a function of the type

type DB = Input -> (Output, DB)

If a user supplies such a function, the Pallas runtime will create a database using a snapshot-and-event-log system. The user can write their programs as if they were keeping their data “in memory”, without any need for manual persistence or other forms of cache management.

The recursive part of the type above might seem strange. You can think of it almost as a normal stateful function:

type OtherDB = (State, Input) -> (State, Output)

The difference is that instead of changing the state value, the recursive version would change itself. The current version of the function is the state. In other words: programs can upgrade themselves dynamically. Code can construct code. Because of this, we can put an entire compiler toolchain inside the system and the programs it generates have zero dependencies on the outside world.

Pallas is open-source, but before we start sharing it publicly, we’ve been trying to find communities of engineers and researchers that can provide private feedback. The model is sufficiently different from the mainstream that we’re concerned about confusing first impressions, so we’re holding ourselves to a high bar by running “UX” tests on the repo and documentation site.

We would really love the opportunity to conduct a live onboarding session with anyone that’s interested. They generally last for about 90 minutes and do require screensharing. I have an open calendar available here.

If you’re interested in this idea, but not in a call, we’re aiming for a finalized first version of the repo and docs within a couple weeks. I’ll update this thread with more info when it’s available.


Edit 8/26/2024

5 Likes

Your system certainly sounds interesting in theory! Could you tell us a bit more about it? In particular:

  • What kind of physical or virtual computer does it run on?
  • What are the application domains you are envisaging?
  • How far does your “clean break with legacy systems” go? Is there any way to interface to today’s computing world, e.g. by accessing file systems, Web APIs, etc.?

This sounds interesting.

At first blush it seems like an operating system based on Unison. Or the same principles, at least.

Sure! Thanks for the questions.

  • What kind of physical or virtual computer does it run on?

Pallas runs as a VM right now. There’s a more complete prototype Haskell runtime and a less complete—but minimal dependency—C runtime. We have prebuilt binaries for Mac and Linux and you can build a dev environment with Nix. Although we currently assume an underlying OS, the system has been designed to eventually run on something like a hypervisor.

  • What are the application domains you are envisaging?

We’re targeting cloud environments right now. If I had to give a pithy description of the project, it would be something like, “a personal cloud computer that your grandma could use.” That doesn’t quite tell the entire story (we won’t be trapped in the cloud forever), but it’s currently the most complete part of the system.

Pallas is not well-suited for most high performance use cases today. For example, we wouldn’t recommend using it as a backend for a multiplayer game (though we have prototyped a small MUD). The places where we think it will be most immediately useful:

  • CRUD apps
  • Messaging applications, including running relays in federated architectures
  • Personal data storage (photos, images, documents, etc.)
  • Hosting personal websites or blogs
  • As a syncing layer for local-first apps
  • Content aggregation (say from RSS feeds or social media)
  • Content streaming

The Haskell runtime pretty happily handles 50+ GBs of storage and we have a clear path for increasing that to many terabytes.

  • How far does your “clean break with legacy systems” go? Is there any way to interface to today’s computing world, e.g. by accessing file systems, Web APIs, etc.?

Yes, there’s an HTTP server, as well as other “hardware devices.” We don’t have plans to expose direct access to file systems.

The system standardizes two things:

  1. An evaluation model called PLAN (see below)
  2. A set of abstract hardware interfaces

By hardware I mean things like time, network, http, etc. The specifics of the implementation are left up to the runtime though. From inside the system, the programmer just has a single uniform interface.

PLAN
The entire system is bootstrapped from a small combinator interpreter called PLAN. It’s basically just lambda calculus, restricted to a type of lambda term with no free variables (called a supercombinator). It is concrete, concise, and relatively readable, considering it’s essentially a compiler binary.

It’s also fast to compile to and easy to map back and forth between memory and disk - which is how you get a single-level store that essentially makes no distinction between in-memory and on-disk. You can unplug it while it’s running, move it to another physical machine, turn it back on and it picks up right where it left off.

Formally, it looks like this:

PLAN ::= <PLAN>           # Pin
       | {Nat Nat PLAN}   # Law
       | (PLAN PLAN)      # App
       | Nat              # Nat

Where a Nat is a natural number and a Law is a user-defined function. () / App denotes function application, {} / Law is a list of values and <> / Pin is a sort of runtime hint that has to do with optimizing memory layout.

This is both the entire data model of our system and anything that users write using this system.

I mention this because one of the ways in which we do intend to be compatible with legacy systems is as a compiler target for other functional languages. For example, a language like Elm could compile to PLAN.

3 Likes

Yea, Unison is awesome!

There are definitely similar ideas shared by both projects, though we’re starting from the goal of designing an operating system, and Unison seems to have started more from an insight about code representation.

But “like an operating system based on Unison” isn’t a bad mental model to hold.

2 Likes

I wanted to follow up on my first post with a link to our repo and documentation:

We received a bunch of useful feedback over the last couple weeks and are working on updates. The repo is the best place to start for now. If you venture into the docs, be aware that our reference material is incomplete and there will be inconsistencies in architecture descriptions. For example, we found that “operating system” is confusing and are now using “application platform”, but the docs still use the old term.

Feedback of any kind is deeply appreciated and I’ll do my best to respond within 24 hours.

2 Likes