Scavenger: Making the browser malleable again

Hi everyone! New here, I hope a project isn’t a bad way to introduce myself.

TL;DR: scav lets you steal and rewire UI from websites and export a portable HTML file.

I ran into malleability as a concept earlier this year, right when I started this side project. It was great to find out there’s a whole field of people already building around something I’d been circling on my own. I still feel like there is a lot of reading I need to do, which I’ll get around to at some point :sweat_smile:

Scavenger is a substrate for portable web tools. The idea underneath it is that the browser already is a malleable substrate, we just stopped treating it like one. The original concepts (Engelbart, Ted Nelson) are still lying there beneath infinite layers of abstraction.

My goal with scav is to remove that abstraction and give you tools to use what’s already there. I also wanted to do it in the simplest way possible, which ended up being four single file, no deps, local first tools. The project is hard to explain in words, so I’d appreciate if you checked out this gif to get a grasp on the concept.

Harvest is the main feature, which lets you lift a live component off any page (not scraped, not screenshotted, the actual computed styles and cascade, rewritten straight from the runtime) as HTML to your clipboard. The tool runs from a bookmarklet command palette that has many other tools, all based on runtime DOM manipulation.

Then you can bring those components into Patchbay (a self contained infinite canvas) and wire them together with nodes, adding logic to turn them into something new. Since all state lives in the DOM as attributes, exporting it is just serializing what’s already there, so the result is a portable quine: the HTML holds both your project and the engine to make it run.

So the loop for using those tools together is:

  1. See UI you like on any page
  2. Open the bookmarklet and run Harvest
  3. Point and click what you want to steal
  4. Paste from clipboard into the Patchbay
  5. Wire it into something new
  6. Export the result as portable HTML

For example, steal Google’s searchbar and wire it to rewrite a Wikipedia card

Those two tools work with static data: Harvesting extracts the component as you see it, keeping the style alive but no connection to anything external. Quests is what brings live data into the flow. It’s a small Node TUI that can scrape either values or full components (by running the harvest on an interval) and serve them locally.

The Patchbay has input and output nodes that bring that data in or send values to the local server. If you bring in components, your wires are kept even on refresh, since updates apply by diff-morphing, which means you can have a media feed that runs logic on “whatever is newest”.

With Compiler you can write your own nodes and components for the Patchbay. For example, you could make a chart component that reads price live from an Amazon product and draws it as a line, updating itself every time a new value comes in from Quests.

An interesting consequence of the architecture is that everything is fractal. You can harvest a Patchbay you made and bring it into a new one, with its node chains alive and working. I’m currently building a library of components for audio chains, with sampling and effects built on the WebAudio API, and portability just comes as a side effect of everything else.

The README sums up everything as quick as possible, with linked in-depth docs too:

Feel free to drop any questions!

4 Likes

Harvest is the main feature, which lets you lift a live component off any page (not scraped, not screenshotted, the actual computed styles and cascade, rewritten straight from the runtime) as HTML to your clipboard.

Interesting! This is something I’ve wanted for a very long time - I’ve been grumbling for years about why on earth we’re reduced to screenshotting images of what is already nested chunks of structured text. Will take a look.

the document isn’t a rendering of the app, the app is a rehydration of the document.

I like this. It feels like the philosophy I’ve been struggling to articulate for a long time. My desire for this increases the older I get, and the number of different systems I have to port my personal data collection between - I just want “my stuff” to be a document, a file or folder, that I can drag/copy to a new device and I’m done. Objecty stuff that’s runtime-dependent enough that it can’t just be copied? That should be the runtime’s business to recreate, not mine.

This thing sure looks… a bit complicated at first glance, though. Hope there aren’t too many moving parts to break.

The clock node is your best friend: it emits a 1 pulse, which is read by other nodes as an act-now instruction

Ah, so there’s an entire dataflow/reactive model behind it? Is this model spelled out somewhere? This seems very good, but, there are subtleties to reactive computation, and every implementation seems to have slightly different semantics.

1 Like

Glad the rehydration thing landed! I’m not a native English speaker so I wasn’t sure if it would. And your phrasing is simpler actually, I might steal it :grin:

I think most of the complexity is actually communicational, and my mistake probably. It’s harder to set up and explain because of the tradeoffs I had to make to keep it zero-deps and local.

For example, choosing a bookmarklet for the kernel instead of an extension makes it a little bit weirder to set up. I believe it does end up less complex, though, since once it’s set up you don’t depend on any service, and it never changes unless you update it manually.

On that, I made a page that simplifies the setup: you just drag the “test the kernel” block onto your bookmarks/favourites bar. Clicking the block also launches it, so you can test how it works right there before grabbing it.

I think it is the simplest and most helpful tool in day-to-day usage, so it’s a good entry point. It has many utilities that I didn’t discuss here.

On the model: yes, it’s spelled out in the patchbay reference. It is deliberately not a reactive model, which honestly does introduce some quirks, but most of them are documented, and they’re what the concepts require.

Thank you for checking it out, and sorry for the delay on the answer!

1 Like