Self-hostable visual systems

Is anyone aware of any visual languages / systems that are powerful enough to construct the same system within itself (self-hosting)?

In the visual systems I can think of, you can typically recreate the logic / behaviour features of the system in itself, but IO access is restricted in various ways such that you can’t create the UI. For example, a spreadsheet does not have the means to draw a spreadsheet UI itself or receive user input events. (Here I am thinking of only the “cell” formula languages… Perhaps there are sufficiently powerful things that can be done in the vendor-specific spreadsheet scripting languages, but I suppose I often ignore those given they aren’t portable…)

Perhaps one way around such IO restrictions might be to just output e.g. lists of drawing commands that can then be sent to another (less restricted) process via some kind of RPC channel to then act on…

Anyway, I find self-hostable systems, reflective towers, etc. all quite fascinating, so I was curious if any such things exist for visual systems. :nerd_face:

2 Likes

I am stumped trying to come up with examples of that! That’s an amazing line of inquiry, this could be an interesting community challenge.

1 Like

Sadly a lot of links to Viewpoint seem to have rotted in the last couple of years.

Ah yeah, I’ve watched the Viewpoint demo video a few times, but even so I don’t think I’ve fully internalised the ideas from it… I should read through the thesis perhaps. It’s certainly quite a unique system…!

Is there one one that list that is self-hosted? I couldn’t find any.

I am not sure I understand your exact criteria for self-hosting. Would a Smalltalk system qualify? If not, why not?

Well, good question, I am not sure I’ve pinned it down precisely yet…! :innocent:

I am hoping for visual systems that can build the same system inside itself, so it would seem like such a system can’t depend special primitives in its runtime that are kept hidden from user programs. Or said in a positive way: If the primitives used to construct the system itself are also user accessible, then surely the system can be built in itself. (Hmm, do even text-based self-hosting systems meet this test, or am I being too strict…?)

I suppose the boundary is fuzzy, e.g. do you need to expose literally OS-level drawing commands, or is it enough to have an in-language canvas… Not entirely sure, so either is okay for now. Just “enough” to “make the thing in the thing”. :smile:

I suppose (as a starting point) I am curious who even gets close to this fuzzy goal.

Perhaps Smalltalk does achieve it, as I know it exposes quite a lot to the user, as its effectively like its own internal OS / desktop in a way… I’ll have to refresh my memory of whether you’d need anything that’s part of the Smalltalk runtime internals.

I’ve been scratching my head on this problem all morning, I have some ideas how this can be achieved but I’ll need some time to put together my notes.

I think the requirement should be that: the drawing environment can evaluate or compile the drawing environment and run the interpreter/compiler. For example, a visual neural nets runtime written in a neural net backend, color computers written in color computers, orca running in orca, game of life in game of life(shared above), etc..

I think the trick is that the visual representation has to be encoded in machine readable code somewhere down the line.

Thought Experiment

An extremely weak form(aka, it’s unusable) of this would be like…

Here’s a drawing program that is modifying its own binary(in the case of a compiler), otherwise that same drawing program is modifying the code in memory(in the case of an interpreter).

Potential Future Work

Another possible candidate, that’s not quite there, but might be close, Tromp’s Lambda Diagrams, if there was an editor for them, closing that loop by changing the representation(like in Scratch) that might also work. The reason I mention this one is that the visual representation is rather simple, and the resulting abstraction for writing them, rather high level. Similarly to interaction nets I guess.

I think something like FSA(wireworld, neural nets, etc..) could never self host?

I didn’t realize that Piet had no IO, that would have been a brilliant candidate.

2 Likes

I definitely agree that you have to have some relation between the machine code and the visual system. I have had some very (extremely) loose thoughts here. Mostly in the mental fantasy I’ve been calling “my code is a garden” which takes some of my work with Fungus to the extreme. I don’t have anything concrete enough to start working on tho. I’ve been fixated on the idea of a 2D world you can walk around, build things in, and then also script those things all from the same foundation.

As for the raw drawing approach, it could be more interesting to design 8x8 sigils that are easy for the eye to read. (I also know there was old talk in the catlang discord between us about how Tote could bootstrap itself). I am a sucker for a programming system I can use completely from my drawing tablet.

The issue I have with the glyph building block approach, like this program that runs foward and back to print the letters TENET

Is that it’s not really a visual system, it’s an alphabetic system, not unlike using conlang words or assembly opcodes for programming. I’m not sure if that would really qualify tho..

binary_keyboard

Tote does use symbols, that could be thought of as alphabetic, but somehow it seems to break out of this representation a bit since the symbols themselves have no semantic meaning, like a brush stroke, a node in a AND graph? Although self hosting tote would be a massive endavour.

What about wang tiles? That seems like a potential candidate to write an editor in wang tiles and visual representation.

1 Like

it’s an alphabetic system

That’s a good point. And, I do think that could be a fault with “editing a sequence of pixels” (such as the example with Noodle). The raw binary opcodes are still textual programming. A sequence of symbols, placed in a linear order, with no way to see how different parts connect together (i.e, you don’t get some kind of visualization of where a jump instruction goes beyond the raw address). The actual way your program connects together has to be held in your head.

(This is a thing that has done my head in when a program has too many links for me to keep in my head all at once. Also a possible reason I get lost-in-the-sauce with Lisp and its deeply nested operations. I have to keep the whole operation “loaded” until I can finally resolve the whole tree all at once.)

I think that’s why Tote seems to escape this. The symbols of your program aren’t tacked together one after the other to describe logic. Additionally, rules don’t contain jump operations so you can think of each in isolation. And, most importantly, the state of your program is exposed to you.

For me, a big thing with visual programming is the ability to step-back and see the overall shape of your program. It’s something you get with visual art: you can lean in (or zoom if it’s digital) to work on fine details, then lean back to see how things are going overall. This is why box-and-wire style systems have always had some appeal to me. They let you see a physical link from operation to operation. Zooming out gives you some kind of overview to your program.

What about wang tiles? That seems like a potential candidate to write an editor in wang tiles and visual representation.

I think Wang tiles would be a similar challenge to Tote. It seems like you’d need a massive buffer and rule set to implement a program with Wang tiles. However, wang tiles would have great inspect ability. It seems like you get a history of your program for free.

Maybe a visual Fungeoid could be fruitful? One thing that motivated Fungus was a surprising lack of “immersive” editors for Befunge. They were all basically Normal Text Editors with better 2D input. I was honestly a little surprised how novel the idea was. The only other Fungeoid I know with an immersive editor is ORCA. I guess you could count Baba Is You and maybe some other video games.

P.S. Also, game editors in general can have some interesting tools such as Trials Evolution’s editor.

1 Like

I think as far as I could find, Game Of Life might be the only system that qualifies.

yeah, that’s accurate. It’s piss poor UX as well. It’d have to be something though from the ground up to be visual, ideally it would have a single node type, I was looking at the Trial’s editor’s nodes, and they’re unscruptable similarly to opcodes in a way.

So alright, lemme see what could be potential routes

Going Down

NOR Nodes(1 node type, 1 wire type)

I’ve toyed with single-node type systems a little. Here’s a system where each node is a NOR, I think this might be capable to build something kind of rough UI were there device I/O in the spec.

The idea here is taken from Sejo’s coloring computers

Non-electronic computers that work when you color them according to a simple set of rules. The booklet contains three series of computers: computers that compare, computers that count, and computers that play. From a technical standpoint they are all NOR-based logic circuits designed by using truth tables, karnaugh maps, and maxterm expansions.

Threashold Neurons(1 node types, two wire types)

Alternatively, maybe something like threshold neurons might work too. Single rule, but two wire types.

SICs(3 node types, 4 rules)

Somewhere inbetween Tote and Neural Nets, there’s symmetric interaction nets, 3 node types, 4 rules.

Going Up

I think Self? was written largely in Self, it’s not strictly a visual programming language but something worth mentioning. Although, the Morphic interface is all prefabs that live outside of the observable Self code.

Boxer is not music of a visual system, it’s more like a fancy lisp front-end.

I would love to see a Fungeoid that can run itself. Heck, Uxntal could be fungeoid, it has a PC and a stack, if I had a visual representation in 2d of a program, written in Uxntal, that could do it. Something like Chipwitz’s Ibol

..

okay.

I have to build this now, thanks gang.

Here’s a little Uxntal program LIT 08 LIT 00 &l INC GTHk ?/lwhere I made the loop by a series of jumps by +/- 1, not really efficient, but easy to parse. The T/F arrows are inverted tho, working on it.

2 Likes

Squeak and its descendants (meaning nearly all Smalltalk systems relevant today) are built on a virtual machine that implements the object lake and a bytecode instruction set that provides fundamental operations such as creating an object, but also a few less fundamental operations for efficiency. The VM is itself written in Smalltalk, with a code generator making C code.

Code running in the Smalltalk image has access to all VM operations, although their direct use is discouraged. Code running in the Smalltalk image can also declare the VM as a dependency and make use of it, for example to generate specialized VMs. There are real-life projects that do this.

But the system is not autonomous. Compiling the VM requires a C compiler and build tools, all of which live outside of the image.

The UI is implemented inside the image, so whatever it takes to make such UIs is available to all Smalltalk code.

However, after reading the other comments in this thread, I wonder whether Smalltalk qualifies as “visual”!

1 Like

I do think there are different ways to be visual. The *Talk family takes the desktop metaphore approach. Tho, you could argue that is simply an IDE with some more immersive UI abilities. Self was a fun one since it was prototype-based. The UI was like a color palette to play with. I could never get started with SmallTalk, but I managed to make a few toy UI tools in Self (which is a far less polished experience. You are very much traveling back in time with Self).

Other tools are more program visualizers where they render code in a graphical form (such as graph-editors or the dreaded UML). Tote lets you define rules to a CA-like system. And, Game-of-Life and CAs restrict you to a fixed rule-set which lets Turing completeness appear as an emergent feature.

I think the fun of CAs and rule-systems is their power to be symbolic and directly manipulated. There is a strong sense of play with them.

Also @neauoire your ability to prototype things so fast is always amazing. I am currently saving that energy for DecAdv.

2 Likes