A Symbol-Oriented GUI

This is a very unformed idea that I’m trying to crystallise into a better shape, or any shape at all.

It’s been haunting me for decades, though. Probably since the era of OS/2 2.0, and my disappointment when I first saw its “Workplace Shell” and found that it wasn’t like a shell at all. That is, you could use graphical objects, even send messages to them… but you couldn’t make your own, at least not interactively.

What I’ve wanted since the 1980s is a GUI where the components are “text-like”, by which I mean:

  • A window, or any other graphical component, is composed entirely from a set of smaller graphical components. Those components can all be readily seen, and they can be also be individually selected, copied and pasted. There is nothing in fact to a window other than all its individual visible components. There are, specifically, no hidden elements: no code, no configuration, no binary files. Everything you see, you get, and only that.

  • Obviously this wouldn’t get us to parity with a full GUI toolkit, because you wouldn’t (at least at first) be able to program your GUI objects. This would just be a view layer. So if there were programmed, reactive components, with configuration or code or hidden state, we’d probably want some kind of “control view” (like right-click, only more powerful) which would let us view this state. And however we viewed it (not necessarily as text - perhaps as some kind of circuit diagram), that view would also be text-like, composed of components each individually selectable.

Perhaps this idea could be called a “symbolic GUI”, because it would be composed of symbols rather than text. But “symbolic” is such an already overloaded term. “Symbol-oriented” maybe?

The important part of this idea is the selectability of all the subcomponents. Probably with some kind of grouping construct (eg a box, as the equivalent of brackets). Components would be not necessarily of a fixed size, but they wouldn’t be arbitrarily sized either. They’d probably expand to fit the size of the components inside them.

To achieve this, we’d need at least three things:

  • A set of symbols, with a basic sort of “visual grammar” on how they can connect (eg: left/right, up/down, inside/outside). The equivalent of Unicode, but for GUI elements.
  • canvases which can display these symbols, and allow selection, grouping, insert, delete, copy and paste, analogous to text areas. Those canvases must be completely unrelated to the software which hosts them. You must be able to copy symbols to and from a canvas, ship them between systems, and have them render recognisably and with the same semantics.
  • probably a strict, one-to-one mapping to and from Unicode, though that wouldn’t be necessary if we had decent interoperability between systems that wasn’t just Unicode, but we don’t.
  • possibly multiple “fonts” which would change how the symbols display, but crucially, NOT their basic identity or behaviour. CSS almost does this - and yet also does too much.

This is the fork in the path of GUIs that I’ve been expecting to happen multiple times, and it didn’t. I first thought Workplace Shell might be it, and then OpenDoc, and then HTML, and then Java Abstract Window Toolkit, and then I thought surely KDE or GNOME would, and then maybe OpenOffice Draw… but that path still hasn’t forked, not in the simple, clean, standard, inter-system, composable way that seemed obvious to me as the next step from the 1980s.

Anyway, that’s the idea. Putting it down as a flag because it feels like something that surely has to happen at some point. Until it does, we won’t have an actual “graphical language” because we won’t have the linguistic equivalents of glyphs from which to build larger components.

HTML gets us halfway there. There is a textlike markup language that defines the things on the screen, but there’s also a lot of invisible stuff, and most annoyingly, you can’t select, copy or paste any HTML element from its visible representation and move it elsewhere. And HTML elements all have to exist inside a “page”, rather than on their own.

Postscript /PDF and TeX and OpenOffice XML also, infuriatingly, almost get there, and yet stop well before the point where they would close the loop.

It might be interesting to think about how many of the HTML elements could be repurposed to be completely free of the context of a “page”, and/or how many Unicode codepoints might be able to be used as graphical drawing elements, if things could be put inside them, or they could be used as links between other elements.

Something you might want to look at is Glamorous Toolkit for Pharo Smalltalk. Its unique feature (as far as I know!) is a way to see the code behind any GUI element. For the most frequently special-purpose element types, inspector views, it’s just Alt-Click. For the GUI of more complex tools, you invoke the “scene driller” via Alt-Shift-Click, and then you can inspect all GUI objects (that’s objects in the Smalltalk sense), including the code behind them, but it takes some experience to navigate the object graph.

You can also change any code and see the impact immediately, which is very useful for GUI work.

2 Likes

Hmm, looking at the Glamorous project page, I agree, I really should take a look. Their home page is actually speaking my language, and that’s very rare!

“a uniform environment made out of visual and interactive operators that can be combined inexpensively in many ways.”

"Glamorous Toolkit is powered by the Lepiter knowledge base engine. Lepiter is made out of snippets grouped into pages and can be used for various use cases. "

“Each snippet defines its own language. Languages can be textual, like GraphQL or JavaScript. They can be visual, like a Wardley Map. Or even widgets.”

“Every page defines a shared scope for variables. Combined with an extensible proxy model, variables can be defined and reused across languages.”

Very interesting indeed. I’ll download it and have a play. Thanks!

Ok so my first impression, just reading through the tutorial, is… someone mashed up Smalltalk, Hypercard and Wordpress Gutenberg and Doug Engelbart’s original NLS.

The way all the text seems to be paragraph-based, gives me that impression.

It’s also a little like Federated Wiki or Tiddlywiki, with its card-like snippets that pop out new ones when you click on a link, but it doesn’t give me motion sickness like those do.

It’s certainly doing a much better job of Being Hypertext ™ than any other hypertext environment I’ve seen since the 1980s. It might even be doing a better job of Being Smalltalk ™.

What is this strange feeling in my chest cavity? Could it be… hope?

Edit: Um. Well. Maybe not quite yet. I was happily clicking and then I seem to have caused an error, just by clicking. “Error: Detected invalid location!” and up comes something that looks like an entire stack trace, and the tutorial slams to a halt because it was live and I broke it.

(Fortunately I only broke that one page, so doing “next page” gets the tutorial continuing again? But I don’t even know what it was that I did that was wrong. Also, I’m not sure if I finished the tutorial or just fell off an unfinished end of it.)

Another observation: (I guess of Lepiter in particular, since that seems to be what all the “books” are written in): There’s a lot of text and after a while it becomes really, really hard to scroll. The Page Up / Page Down keys aren’t implemented. I’m reading on a notebook with a touchpad, no scrollwheel, no touchscreen, so I have to keep pointing to the scroll bar and dragging it and it’s just… not very pleasant to have to do that all the time. Compared to, say, the Web (even this site) where page up/down just work as you’d expect.

Also, I think Smalltalk syntax will never really feel natural to me. I sort of get the colons and the method calls chopped into multiple chunks, but then there’s all these other random punctuation characters: square and curly brackets, vertical bars, carets? I know huge things have been built on Smalltalk, but the core foundation just doesn’t seem as simple as it ought to be. Especially if this is a grammar we want to host arbitrary domain specific languages over.

First of all, this is Smalltalk. Glamorous Toolkit is a new UI layer for Pharo. There is growing support for working with code in other languages (for now: Python, JavaScript, Rust), but the implementation language and best-supported target languages is Pharo Smalltalk.

Next, there are definitely rough edges remaining, even though the authors decided somewhat arbitrarily to put the version number 1.0 on it in August. But it’s very usable in daily practice, once you know your way around a bit. It’s been one of may main platforms for two years, and it’s clearly the one I enjoy most.

1 Like

I have to say, after spending just a few hours with Glamorous and knowing basically no Smalltalk, I am enjoying the experience of trying to bootstrap my way towards Baby’s First Class Definition a lot more than the “classic” Smalltalk IDE experience. While I have some problems with navigating GT’s UI (it especially doesn’t seem to do small laptop screens well, and isn’t keyboard-centric), it is a lot more approachable. Much more like dealing with a modern blog/CMS or a Wiki than with an IDE.

Have you checked out Oberon?
Some versions such as the latest one have a text based UI.