Over on the “Next, OOP, WWW, HTML” thread, Akkartik challenged me with the question:
It’s not clear to me what use cases motivate your search.
I’ve been trying to bite my tongue and I’m failing… I feel some impatience with this thread
I’m less interested in coming up with the perfect tools that make doing any abstract thing a pleasure, and more interested in getting on with the doing of concrete things in rigorous even if unaesthetic ways.
And that brought me up short. I hadn’t thought that the “use case” I have in mind and keep sketching out approaches to was either abstract or frustrating and impatience-causing, but apparently it is!
I mean, that’s not wrong. My search is a source of frustration to me personally. As a use case it must seem “abstract”, since nobody seems to have built something that implements it yet; I keep looking at lots of software and go “no, that’s not it, that’s maybe part of it, but it’s not it. And it is important.”
At least that’s my feeling. It is hard to translate that feeling into “the doing of concrete things”, yes. Or even into a list of desirable qualities. And my own sense of frustration at my own inability to do that translating maybe shows through, where perhaps it shouldn’t.
So yeah. What is the magical it that I’m looking for is the use case?
And why did the Commodore PET 2001 in particular - more than other 1977 8-bit machines - spark partial recognition of that magical it for me?
The short answer, I think is that what I’m looking for is an “idea processor”.
An idea processor might not be what everyone else interested in “malleable software” is looking for, which is probably why there’s confusion over why I want this very specific, perhaps very odd, and probably very hard to obtain, list of wants. Why I think this thing that doesn’t exist is so important.
The name “idea processor” came to me, and has stuck in my head ever since (yes, for 40 years), from a January 1984 Byte Magazine article about Thinktank by Living Videotext. That product was essentially just what we now call an “outliner”, a word processing mode that got quickly subsumed into Microsoft Word and, I suppose, Emacs Org Mode:
To date, most personal computer software has enabled us to do familiar tasks more efficiently. With a word processor, for example, we can prepare documents, alter them, and generate clean copies more easily than is possible using a typewriter, paper, liquid correction fluid, and an eraser. While the Block Move commands in word processors make rearranging text simpler and encourage writers to revise, nothing about word processors makes it easier to create a document when the ideas are still in a formative stage or to analyze a document that is already complete.
Spreadsheets are the great exception. They let us analyze related numerical parameters in a way never before available to individuals. Some may dispute this view, saying that spreadsheets just let us look at more alternatives than we could using paper and calculator. The contribution of the spreadsheet program, in this view, is merely an improvement in efficiency over customary methods.
But spreadsheet software contributes much more than increased efficiency. As you watch the effects of changing one figure ripple through a whole model, you are able to think in larger terms than a calculator and paper allow. Furthermore, the visual rippling itself sometimes is tantamount to a graphical simulation of the problem under study. You gain a better understanding of relationships in the model and a feeling for the probable consequences of different kinds of changes. When a change causes startling results, you realize that either the model doesn’t reflect reality or the reality is different in some important way from what you thought. The spreadsheet software provides a ready means for analyzing the startling result.
The time has come for text-editing software to surpass the power of the paper operations it mimics. To date, word processors have taken conceptually complete documents and printed them out neatly. The most interesting and difficult part of preparing many text documents, however, comes much earlier— when concepts are inchoate and their interrelationships are dimly understood. Text editors for programmers have provided “macros” that permit automatic execution of a series of operations, but the operations chained are usually a series of search-and-replace operations designed to achieve a result that conforms to the rigid syntax of a programming language.
Can software help us grope through the earlier stages of text composition? Can it help us analyze finished documents?
…
Two programs that go beyond word processing are now on the market: Thinktank, from Living Videotext in Palo Alto, California, and Zyindex, from Zylab Corporation in Chicago, Illinois. Both these programs recognize that a collection of words can be much more than a finished document. They give the user new ways of getting hold of information contained in text files.
In the case of Thinktank, available for the Apple IIe for $150 and the IBM PC for $195, you obtain information through a tree structure. A document is a large outline. There are headings and subheadings ad infinitum, but you see only as much of the outline as you wish. The EXPAND and COLLAPSE commands control what you see. Thinktank displays a " + " in front of every heading that has further information under it. You know you have reached the end of a branch when a " - " sign precedes a heading. If you use the EXPAND command on an item that is preceded by a " + " the next level of branches will be displayed. You can expand your way through level after level. When you want to concentrate on the more general levels of the outline, you use the COLLAPSE command to hide the levels that give details. By expanding and collapsing, you can delve into the tree of data or concentrate on only the broad headings. You can also move items about in the structure, promoting them to higher levels or moving them to other parts of the outline. You can insert new items and merge old ones as well.
Thinktank, then, is most useful when a document is in the formative stages. As your ideas change and various aspects of them become more and less important, you can alter the document to reflect the changes. However much rearranging you do, Thinktank leaves you with a neat outline that looks as if you understood the subject when you started out.
In a way, Thinktank is analogous to a spreadsheet. A spreadsheet lets you rearrange numbers and relationships until you find a coherent arrangement that shows how to attain your goals. Thinktank manipulates text in a way that lets you rearrange its internal logical relationships until the structure supports a main idea. A spreadsheet is a number manipulator. Thinktank is a text manipulator.
Living Videotext calls Thinktank an “idea processor” to distinguish it from a word processor. While the term is a bit optimistic, it will have to do until a better one comes along. Thinktank’s specific contribution is to apply tree structures to text. Its more general contribution is to treat a body of text as a database whose parts happen to be text.
…
Zyindex, too, treats text as a database. This program lets you search a body of text for occurrences of any word, or of any two or more words within a specifiable number of words of each other. What makes Zyindex so much more valuable than the Search function of a word processor is the availability of Boolean operators. You can look for any occurrence of either “Johnson” or “Holmes” within 100 words of “embezzlement.” Or any occurrence of both “Johnson” and “Holmes” within 100 words of “embezzlement.”
…
If anything is holding back development of software for text manipulation, it is the failure of those of us who create, edit, and analyze text to specify features that would be useful to us. In the age of the typewriter, there was no point in dreaming of such features. In the age of the personal computer, expressing our dreams may result in products that make such features a reality.
So that was 1984. Some of the “idea processor” (the easy parts - the outliner) moved into Word and other word processors. The free-text search part moved into search engines and, basically, became Google. (Because everyone stopped doing any kind of free-text search on their own systems).
LLMs are now threatening to take the “idea processor” idea and combine it with natural language and bring it fully, lurchingly, to life. I say “threatening” rather than “promising” because the LLM approach deeply disturbs me and is likely to massively change the world for the worse (arguably it already has). LLMs are huge and centralized and are both a hallucination and a privacy disaster. I want something that does something similar, but on a small, local, and predictable level.
This is why my first attempt in this direction is SUSN, which is both a syntax / datamodel, and a tiny Javascript script for Node.js, together which implement an outliner and a tiny search engine. Some of SUSN’s weird features as a syntax (like, why I don’t just use JSON or even Lisp S-expressions - why those aren’t “good enough” for the use case) are specifically to enable freeform text entry, on multiple devices (standard Windows notebook and Android phone) with as little friction as possible.
I’ve used SUSN on a text/database of my own creation, specifically to track semi-structured data which begins as ideas, and I’ve found it very useful. It solves a particular problem that only I have. But it’s not the whole of the solution, because it’s not the whole of the use case. It’s one tiny corner of it.
My idea of an “idea processor” is not the same as Living Videotext’s, which yeah, was just an outliner. I see it as a thing somewhere in the space of Smalltalk and the Dynabook. A little adjacent to the Commodore PET, but not it.
Let me digress first to explain why the PET 2001 in particular sparks a sense of recognition to me. What its use case is and isn’t.
In short, I believe the PET was intended as a Computer Aided Instruction machine for the classroom. There were three contending forces in 1977 for the microcomputer: one, videogames; two, business; three, education. Computers in education, specifically between 1960 and 1984, were seen through the lens of CAI.
The definitive early CAI system was PLATO. Alongside PLATO was PILOT, and then BASIC (which was essentially interactive FORTRAN; the line numbers emulate a deck of punched cards). CAI was based around the idea of educational resources which were programs: which would be both simulations, and tests.
There were dedicated CAI languages, but BASIC was picked because it was also popular in education. Mathematics was in BASIC because it was in FORTRAN. String handling was in BASIC when it was very minimal in FORTRAN because the CAI concept was that the machine would prompt a learner with text and receive text answers.
To standard BASIC, the PET added graphical characters and cursor animations - an otherwise strange and unmotivated thing to add - because, I believe, it was intended to do the sort of thing that Macromedia Authorware and Director would do later: (Adobe Authorware - Wikipedia). Authorware itself was a descendent of the PLATO project.
The PET, then - I think like the Apple, which may have had broader use cases, but which did heavily target education - was intended as a “courseware machine”. Not just “playback” of courseware (simulations and exams), but also its creation. That was another reason for using BASIC - so that teachers could quickly make courseware of their own.
Education wasn’t the only use case. Even by launch time, a name which would have made perfect sense as “Personal Electronic Tutor” became the extremely awkward “Personal Electronic Transactor”, and then Commodore quickly followed the PET with the CBM range, as “Commodore Business Machines”, emphasising a business-friendly typewriter keyboard and printers.
But the education and CAI (and PLATO) DNA remains in PETSCII, those “beautiful” characters. Why are they so beautiful when they don’t have to be? So that it’s quick and simple to create courseware and animations with them.
The early Commodore magazines, like the early Apple magazines, and like Creative Computing and the BASIC scene before them, were full of educational and CAI use-cases. By around 1984, the CAI tide had fully retreated, and gaming and business had taken over. There was still educational software, but it felt “lame” and wasn’t a driving force like it had been in 1977.
But it was that educational vibe of the PET that hit me as a kid, and particularly the sense that a computer could be not just a business or gaming thing, but a true “mental amplifier” or “bicycle for the mind”: not just a distraction or an efficiency machine, but something that could help you explore, develop, amplify and transmit ideas in themselves. Particularly through simulation.
I never had the chance to experience PLATO (only a very few highly privileged college students could, and I was in the wrong country), but I did touch just the edge of that scene through the PET and its associated literature. I could feel the aspirations, if not the concrete achievements, of the CAI drive.
Those aspirations - as well as those of Alan Kay later on - woke something in me which has never slept 40 years later. I’m sorry if that something is “abstract”: it has to be, since it’s not yet concretely realised; or it is, but only in small pieces so far. But it’s not vague either. It has a centre and it has edges. It has things it clearly isn’t, and things that it’s more and less like.
CAI itself is not the “idea processor”. CAI (as is “courseware” today) was structured around the idea of a machine running “modules” that students would interact with. For that use-case, a standalone machine even with a tape drive was fine. PLATO had a network, but the PET couldn’t afford that. It wasn’t so much about document production. It was about students experiencing an exam or simulation - which could be very much interactive, and a little bit tailored to the student (which is why all those BASIC programs asked you your name) but didn’t need to really exist in a networked environment. The output was the learning the student walked away with.
An idea processor would be more than that. Kay’s Dynabook captures more of that dream. What we have in laptops and mobiles today, however, is still not really even the Dynabook. At best, they are substrates on which a Dynabook could one day be built… if Silicon Valley were not currently determined to destroy the ability to build one.
The specific things I want in an idea processor, and the reasons for needing them, are:
- Ideas when they strike us are fleeting and fragile, but nonetheless valuable. We need to be able to capture them quickly and with as little friction as possible. That means we need an instant-on interactive interface. We need something like an “editor” or a “notebook” to be the default mode of interaction. We need to quickly be able to jot down thoughts in their simplest form, unstructured, as they come to us, and expand them later. We need those thoughts to be preserved without loss, without switching or distracting our thoughts.
The idea processor, in other words, needs to capture, enhance, and amplify a person’s “flow state”. That’s why “beauty” (such an abstract principle, yes, but you know it when it’s not there) is required. It’s the elimination of distractions.
That’s why particularly the PET’s “immediate mode” with its “more immediate than immediate mode” ability to just scribble on the screen, resonated. That’s part of that “capture ideas at their source” ability. Smalltalk, I think, should have some of that; I just haven’t been able to make friends with any Smalltalk yet.
Unfortunately, a big requirement of fully doing “no distractions” is managing to somehow eliminate or minimise “modes”. That means creating something that’s like one single app, because every time you switch apps you get that “I just walked through a doorway - what was I doing?” code-switch and memory-blank problem.
And single do-everything apps are hard and probably wrong.
Forth-like OSes, interestingly, don’t give me this problem. Something in the concept of the Forth word resonates with me. Unix scripts also are somewhat similar. Windows Powershell is also a great interface to work for the same reason. Emacs probably has this quality, but again, I have failed to make friends with Emacs; it didn’t like me either. Smalltalk again, probably, but again, not quite actually, unless I can somehow find a way in that isn’t filled with distracting complexity.
The idea processor, then, might need to be something like a “shell” in order to minimise distractions and destructive code-switches. This concept of “beauty” is about integration and simplicity (lack of mental complexity in the interaction model) as much as it’s about “visually appealing design”.
-
Ideas are symbols and are also documents. They can be of any size and complexity. Therefore, we need to be able to start with the smallest possible kind of symbol and cluster, group, and rearrange ideas. There should not be artificial barriers between small ideas and big ideas, other than the inherent complexity of the ideas themselves. That means we shouldn’t have to “code-switch” between multiple languages or multiple applications as we are developing ideas. We should be able to smoothly evolve ideas by adding, combining and linking, until we get a literal network. Because ideas spread through social networks. Ideas which run on machines also implement and augment social networks in an extremely literal fashion.
-
Ideas include text, numbers, graphics, media, equations, simulations. These are still, after decades of compound documents, mostly scattered and siloed. Text editors can format equations, but they’re terrible at evaluating them. Databases can capture rigidly structured and typed row-column data, but break when dealing with unstructured or recursively structured data. Specialised graphical tools are great at photo collections or “fine art” but aren’t so good for quick sketching, especially for people with little artistic development. Programming languages, even ones used for scientific computing like Python, even in “computational notebooks” need elaborate setup and are often incredibly dangerous in a cybersecurity sense. We’ve got some tools like Microsoft OneNote, but again this is a proprietary silo.
The “computational notebook” concept, however, combined with “hypertext” and “zettelcasten”, comes the closest so far to what I mean by an idea processor. Hypercard had a lot of the idea processor DNA. So too did the original WorldWideWeb - the one which allowed writing and hosting as part of the same package. The WikiWikiWeb captured that too.
But the wiki still wasn’t quite It. How many people have personal wikis today? How many personal wikis are pleasant to use and don’t distract from your flow state? How many personal wikis can do computation, enforce data constraints, allow searches and textual/relationship analysis, and create simulations? How many can exchange small chunks of your data safely with other people? If a wiki can’t do these necessary things, how can it be nudged in the direction of doing them?
- Ideas can contain graphics. And yes, we could just embed a raw bitmap or vector canvas, add some line/box/circle primitives and call it a day. It’s a drawing tool; we can still do a lot with Microsoft Paint. But the reason I think graphics needs to be “structured” (probably a very hard thing to do, maybe impossible at this point with so many standards; definitely a hard ask) is because, well ideas are structured. Structure is inherent to the idea of the “idea”. (Mental chunking). If we want to “process” graphics as ideas and not just have them be a raw slab of data, we need to be able to pull a graphic apart, label it, change things, etc. The user needs to be able to do this, and do it quickly, as they would with a pencil and paper.
This part seems like it’s super-simple, but it is hard and finding a way to attack it is also hard. It’s hard because we have so much graphics. The art world is broad and intimidating; it’s also a massive attack surface, both technical and social, because there exist very illegal and destructive images and videos the mere sight of which will put you in prison forever. It also crosses over the the equally intimidating (if not quite so legally hazardous) world of book production. Postscript and TeX have been answers in the past; they’re probably required for access to knowledge, but they’re hard to use as a knowledge capturing tool. Markdown is becoming the knowledge capturing tool of choice, but it’s full of unexpected complexity now. Tools like “Lines”, yes, I can see a flash of something powerful and important there: particularly being able to serialise vector graphics into text and then exchange the text.
Premade shapes like emoticons are popular; they don’t scale up and they don’t cluster very well, but you can see the public conversation shaping toward them. “Image macros”, also. Something like Logo’s turtle graphics, I think, is maybe another useful approach. But exchanging documents containing recursive procedures that draw graphics (even though PDF and TeX do this) requires the next item, even harder:
- Ideas need to contain “code” in the form of equation, simulations and tools, because that’s the special genius of the computer: it’s a machine for examining and testing ideas. Ideas then aren’t just raw text (although they can start out as raw text) or raw graphics, or raw numbers, but relations between these things. These relations can include constraints (type declarations) and algorithms.
We need to be able to transmit ideas, including algorithms, but we need to be able to do so safely. When an idea “executes” it needs to not damage the computer it “executes” on. I put “execute” in quotes because again, this shouldn’t be about just shipping someone a machine and have that machine run. It should be something the user can fully deconstruct and work with. It’s an idea in the form of a machine; it’s not a machine that is a black box and just sits and does work. Machines which are black boxes belong to the “game” and “business” use cases; they aren’t about education, and they specifically aren’t about examinable ideas.
Basic cyber-safety of sharable documents should have been a solved problem decades ago, given that offices rely on them, and for many use-cases (.doc files, PDF files, email) we thought it was. But we’ve still managed to un-solve that solved problem. It remains an issue that’s high up on any todo list: first, don’t get your users rooted.
- Ideas need to be safely transmittable. But they also need to be kept local, secret and uncensored. Ideas can be a threat to the state and to entrenched powers. That means that any “idea processor” which starts out with centralised storage/processing or with submitting anything for evaluation and censorship is dangerous. Information has to be kept at the edges, not routed through a center.
Again, this shouldn’t need to be stated. But sadly email, social media, personal photo libraries, book readers, most Markdown diaries today all sync to cloud. That’s dangerous. Combined with pervasive AI scanning of all documents and authoritarian regimes, and terrible things will result from this. Terrible things probably already have resulted - I mean, that’s the whole AI sales pitch. “Do terribly wrong things at scale without understanding what you’re doing”.
There probably are all-local Markdown diaries, which are at least possible contenders for the core of an idea processor. But I feel like Markdown itself as a syntax is going to become a wall rather than a door sooner than later. It would be nice to have something like Markdown, but which can scale to carry code within it, rather than being the file you write your readme.txt in and then your code is in something else.
- We’re going to need the idea processor, if we need it at all, during a time of rapid social change and upheaval - because that’s what our foreseeable future is going to look like for maybe a century. That means that it needs to be able to cope with chaos in networks, power outages, angry online mobs, hardware shortages, to outline a few very known and immediate problems. We don’t have to panic about this but it means we do need to be practical and pragmatic. It can’t involve custom hardware unless that hardware is somehow magically cheap and pervasive and user-serviceable. It needs to be cross-platform. It needs to be able to run on the machines we have (assuming that we can remove or bypass most of the worst privacy and censorship risks - that in itself may be a big ask).
Worst case, we can use, as we do today, a mismash of text files, scripting languages, emulators, broken social media networks that can’t exchange files, and cobble together personal workflows out of existing pieces. We probably will have to do this anyway whatever happens.
But AI is out there and it’s hungry for data. AI is linking all the data together in ways that our patchwork mishmash can’t, and authoritarian regimes inside and out of governments are sucking on that data and doing terrible things with it.
tldr: I feel like we might need something that isn’t AI but can link data together.
Quite a lot of it resonates with with my personal malleable goals.
It’s probably not any specific language you want, but that essence of a friendly live computing interface.
