Exploration of live systems - Hooliganism or Play

Let us assume that you find a specific server with an interface and you try to interact with it. You create your program and you start experimenting.

As you change your program, it immediately interacts with a production system, you have the ability to see the responses of the server. If you dont like the outcome, you continue to change your software, till you finally end up with something that works.

All of this exploration sends many erroneous msgs to the server, that might have crashed it or worse. At the same time, we want to have play and exploration of new possibilities and it is very useful to get live responses.

To allow for exploration, we thus have to put the responsibility of failure to the server side.
In general, this might be called localized responsibility. Any code that is reachable from the internet has the sole responsibility of being correct, of specifying exactly the behaviors that it allows, and to handle them correctly.

If you push for live exploration of production systems, they might break, and it is not our fault.

1 Like

What software design principles should thus permit for exploration?

This is the question. For me, the answer is that we need to find methods that guarantee that the server will not fail, or that it fails gracefully, without any damage.

For me, the answer is that we need to find methods that guarantee that the server will not fail, or that it fails gracefully, without any damage.

My thinking is that there are several approaches we should be trying:

1 Least-privilege and capability-based referencing which allow systems, servers, or objects to connect with only the minimal rights they need, to limit the damage a broken server processing a broken or malicious request can do.

If you’re doing playful exploration of a live production corporate database (or even your personal hard disk) that’s fine and all, but, it would be software malpractice to give a flaky new component and/or a potential attacker write access to more than a minimal set of records that it absolutely needs. We need APIs which allow us to preemptively restrict the scope of exposure and damage at all levels of data storage. The Unix filesystem APIs, sadly, is not one of these good APIs, and that means we need to stop using it. The Windows filesystem API isn’t any better.

For data, I think we need something general-purpose like a filesystem (I don’t think strictly-typed SQL tables are really sufficient for semi-structured data of the kind we put in filesystems), but where any program can create a subset of the view of the data store that it has, and pass that subset to other programs.

For a network, it might be that something like “DNS name” or “IP address” is really not sufficient as an identifier, because it’s too easily forgeable. We should be thinking about how to define networks in the same ways that we write programs, and ideally in the same (parallel) languages. After all, large chunks of our networks are converging into software (software-defined networks inside clouds); we should be thinking about what that implies on the level of system construction. Ie we should ask ourselves what languages might be useful for defining all of: – 1) an event-based desktop GUI, 2) a set of Unix processes talking over IPC or a filesystem, 3) an emulated system motherboard and set of components, and 4) a bunch of Linux containers all running microservices over TCP/IP – if all four of those are just going to end up running as emulations on the very same chunk of silicon.

2 On the level of servers and programs, we should be looking for ways to define the protocol/API/contract in a way that can be used to guarantee that a program really and truly fulfils it. Object-oriented classes and interfaces aren’t enough to give us this strong a guarantee. Even pure-functional types don’t have enough information to describe not only what kind of things a function inputs and outputs, but what relationship those inputs and outputs must have with each other. This gets especially hard to define (yet very important to define) as the state of an object or a chunk of data changes over time and across multiple versions of functions/classes, or the boundary of multiple systems.

I think we should be looking at something like logic programming, or at least something in the “relational” rather than “functional” sector of mathematics, because functions are about irreversible one-way transformations and while those are nice and simple to think about in terms of “processing pipelines”, what we face in the problem of specifying and validating contracts and invariants seems to be more than just opaque one-way transformations but a parseable, iteratable network of connections between those transformations.

That is: while it’s not just okay but active a good thing that a function does something mysterious when you run it and you can’t find out, at runtime, what that thing is that it does - and in that opacity lies security and privacy guarantees - we need the opposite property for the specification and validation of contracts. We need clearly defined webs of relationships which can be peeled apart by other components at runtime, in order to look inside and check that yes, that relationship does in fact hold, and yes, it does in fact logically follow and can be proven. But I don’t actually know whether these two properties (opacity vs transparency of definitions) really belong to a division between the “functional” vs “relational” worlds of mathematics - or whether they belong to two separate social contexts that the “functional programming” vs “logic programming” subcultures just purely accidentally happened to grew up in.

Regardless of the functional/relational split, though, my feeling is that a suitably simple and usable version of a “dependent typing” theory (or even “calculus of constuctions”) will probably turn into something very like a logic programming language. But, I don’t think the big theorem prover languages are anywhere near the gold standard of simplicity that, say, Lisp or Prolog have. They need to become smaller before they can be useful for actual programming.

2 Likes

A lot of this seems like rocket science, and I don’t really do research anymore. But I just wanted to respond to one statement:

For me, the answer is that we need to find methods that guarantee that the server will not fail, or that it fails gracefully, without any damage.

I think a guarantee is too much to expect when it’s not your server. Clients don’t get to specify SLAs. It’s enough for the server to not blame me for sending it requests that follow its rules.

1 Like

1 Least-privilege and capability-based referencing which allow systems, servers, or objects to connect with only the minimal rights they need, to limit the damage a broken server processing a broken or malicious request can do.

Maybe web assembly.

If you’re doing playful exploration of a live production corporate database (or even your personal hard disk) that’s fine and all, but, it would be software malpractice to give a flaky new component and/or a potential attacker write access to more than a minimal set of records that it absolutely needs. We need APIs which allow us to preemptively restrict the scope of exposure and damage at all levels of data storage. The Unix filesystem APIs, sadly, is not one of these good APIs, and that means we need to stop using it. The Windows filesystem API isn’t any better.

Here I think that having multiple actors with their own vm and their own data might be the answer.
Compartmentalizing code and data into very small parts.

but where any program can create a subset of the view of the data store that it has, and pass that subset to other programs.

Personally I like the actor model as a method of doing this. It is similar to how smalltalk performs encapsulation of data. Each objects has methods that act upon it and returns other objects.
In the actor model, it is msgs and actors.

We should be thinking about how to define networks in the same ways that we write programs

I don’t know a lot about this. I only have experience on the application level, not the lower ones.
In the abstract sense, two actors can communicate when an appropriate token is shared between them. This is also my method of defining scope of communication. In lambda calculus, the scope is defined based on the structure of the term. When we have parallel processes, each process could communicate with any other if it wanted, thus scope is absent. Tokens are the answer to that.

the problem of specifying and validating contracts and invariants seems to be more than just opaque one-way transformations but a parseable, iteratable network of connections between those transformations.

I would like to know more about this. I need to learn more about logic programming.

opacity vs transparency of definitions

I think opacity is inherent in the distributed setting. You do not know how one implements a specific type of function. In the dependent type world, it might say that it implements a specific contract, but in fact it doesn’t and you have no way of inspecting the runtime of an actor at another computer. I call this the Byzantine Type Problem. All type systems assume that the programs respect the type system at runtime. This is not the case here.

Thus, the type system is useful strictly for social purposes, it allows the coordination of developers when creating new software.

The type system can provide local guarantees only, meaning that it can specify the behavior of your own system and the external behaviors with which it can interact.

We could also have a group of actors in different computers that can implement a “behavior” with respect to an external environment. The only way to create such a protocol is to treat all such actors inside the group as hostile, ie that they will disrespect the specification.

the big theorem prover languages are anywhere near the gold standard of simplicity that, say, Lisp or Prolog have

The sentiment is also shared among the practitioners of theorem provers.

By the way, the purpose of this post is to extend the programming on live objects that smalltalk has to the distributed setting. Is there a way to extend smalltalk’s liveness to the distributed setting?

@akkartik I think here we are both the client and the server. We don’t want SLAs because we want exploration on both sides, on the client and server or in the p2p setting.

Oh I see. So it is your server? I mean, are the server and client in a single “zone” of ownership.

No different ownership, but similar technology and culture.

My proposal is to change culture , enabled by technology, so that we program with live objects, like in smalltalk.

There are technical problems to this. So it is speculative, up to discussion. Is there technology that could mitigate any damage, so that this change of culture will be accepted?

Maybe this is a more accurate question.

For me, culture changes in accordance to technological capabilities. Noone wants to lose data.

I wish I had more experience with pharo and smalltalk. I would be interested to see how they handle change. Last time, I remember that I was scared to store anything in the live image.

Smalltalk has always had recovery techniques for the code stored in an image. Basically, it logs all changes and allows you to replay the edits, individually or in chunks. So if you crash your image seriously, you don’t lose code though you do lose the results of computations.

Today’s Smalltalk systems, such as Pharo, live as subsystems inside standard OS, and the image is just a file in the host OS’ file system. Users tend to treat those images as persistent workspaces of little value. If you damage an image, you just throw it away and start from a fresh one. Code is stored in a textual format, usually managed in Git repositories.

BTW, if you want to crash a Pharo system in a really classy way, try

true become: false

This exchanges the objects true and false in the image. Nothing much happens after that.

2 Likes

Maybe we need something like croquet. I think they store human interactions in a logical clock. Then we can replay computations, due to changes in code or data.

Then we could work on live objects without fear.

Well, part of this is my personal axe to grind, having occasionally played with Prolog and Prolog derivatives and really liking part of what I saw, but only part of it. And logic programming isn’t necessarily the best or only way forward. But, the part of LP that I like is that it seems to unify three otherwise separate things- “functions”, “types”, and “data structures” - into a single concept, the “predicate/relation”. Given that we seem to be facing a crisis of accidental complexity in programming, a major foundation-level simplification like that seems like a fairly important breakthrough to me.

The current trending logic programming “language” - MiniKanren or MicroKanren - isn’t even a language, but rather an algorithm that is usually implemented in a functional/procedural/OOP programming language. It’s a very nice thing to exist, and the Kanren family of algorithms I think has made some important steps forward, but I still would like a Kanren derivative to be its own fully self-hosting language.

Anyway, what I mean by “parseable, iterable network” is this:

In Prolog you define a predicate as a series of assertions, which can be either literals or function-like rules, and can intermingle them freely, eg:

city(seattle,washington).

city(losangeles,california).

city(nyc,newyork).

region(westcoast,california).
region(westcoast, washington).
region(eastcoast, newyork).

region(R,City) :- region(R,State),city(City,State).

So now you can treat “region” as a mixture of a data structure/table, a function testing membership, and also a computed query / simulated table that calculates and returns (or constructs) data on the fly, as it’s queried.

region(westcoast, nyc) ← if that’s an assertion, then it’s just a dumb data literal. But if it’s a query then it’s a query that’s currently not true (the “region” table/predicate does not contain these values, or a rule that allows you to infer those values), so this query “fails”, which is either like exiting your whole function or like returning false, depending on how you handle it

region(westcoast, Place) ← if that’s a query, then variable Place becomes bound to a keyword which can be found defined in the region table/predicate, but it stores state somehow on the stack/heap and will backtrack if a later expression using those variable bindings fails. So now you have something like an iterator or a generator function / coroutine. Every function can return multiple (and potentially infinite) values, that’s automatically baked in.

region(R,Place) ← now you have an iterator/generator/coroutine with two free variables.

So where in functional programming or even OO programming you’d probably implement “region” as three or four very different literal data structures / abstract data types / objects / functions, here we just have one thing, a predicate, that can sort of shapeshift into being a definition, an evaluator, a query, or just a dumb set of data, depending on the circumstances.

It’s a very powerful abstraction in my opinion, and would be really great for situations like personal databases, configuration files, build systems, a lot of things that we use a lot of incompatible ad-hoc config/code languages for at the moment. But there’s a cost. The logic programming mindset is tricky to connect up to more ordinary code because of all the backtracking, and the variables having to be carried around everywhere. You never just have a function that returns a simple value, you always have a bit of a “returning a gorilla holding the banana and the whole jungle” problem in that you’re returning “a set of variable mappings that somewhere hold your value, plus a call stack to generate a new set of mappings in case those don’t work for you”.

If this could be simplified and streamlined a bit, I feel like we might have something really nice that could meet our config, data and programming needs for the next 100 years. And could potentially work to define systems both in the small and in the large. But it’s not quite there yet.

For example (to bring this back to the purpose of this thread), defining how a “predicate call” would/should work between two distributed systems: that’s a tricky part. We could probably think of it like an OOP message send, but since every predicate call creates backtracking state that can later be resumed, it’s probably like every message send would have to create a state object on the machine it’s sent to, and return not a value but a reference to that state object, so that the call could be resumed at any point. That’s a lot of object creation and persistence. And how does garbage collection work? Does a server have to hold on to state objects forever, or can it silently let them go at some point? Does the whole thing become an instant DDOS machine?

tldr: In logic programming, the equivalent of a “function call” doesn’t return simple values but rather a metadata structure containing statements about those values (ie, variable mappings, and a backtracking stack). And that metadata structure is the thing that can be “parsed and iterated” in a way that a simple function return value can’t be. That metadata-ness is what gives 1970s-era Prolog its startling ability to do very complex things with very simple definitions. But perhaps it’s just shoved the complexity under the carpet and the total amount of complexity remains the same. Still, shoving complexity around is what we do in programming, so I think it’s worthwhile to look at different ways of doing it.

Edit: Ok, on reflection, a distributed predicate call wouldn’t need to create a state object on the server that could be resumed later. It could instead return something like source or object code for a query that would return the remaining values. But that then hits another question of privacy/security: we aren’t really in the habit of understanding functions that routinely return, across the wire, parts of their own code and/or call stack. We like our functions to be black boxes that conceal whatever is within, and exposing the runtime internals to the world makes us nervous. Doing this probably could still be done in a secure manner, but it would need serious thinking about, because it violates a lot of our standard intuitions.

2 Likes