Is anyone aware of workflows and/or support tools for collaborative work on malleable software?
My idea would be for each participant to have their personal repository, with links to other participants’ versions. Tools would support discovering further versions (by following neighbor links), and providing summaries of changes elsewhere that would then be easy to integrate in one’s own version.
Git looks perfectly fine as an infrastructure, but today’s forges are insufficient because they assume asymmetric fork relations between repositories, rather than equal peer relations. I don’t know if federated forges (forgejo but also current GitLab projects) propose a different model.
Also, in general most people change multiple parts of the code, that has side effects. Mutability can be very hard to reason with. It does not allow to separate parts of software, except if you design them in such a way.
From a type theoretic point of view, I like what Idris used to have : Type-directed search, in other words search on the specification of the function. And types keep everyone honest. You can use the function without looking at the implementation.
In other words, it enables to treat functions as black boxes.
Social aspects of the dissemination of software changes should also be considered.
I am also interested to see if there are projects on this.
This is a neat idea. I’m not aware of anything exactly in this direction. A few thoughts in the penumbra of this.
@khinsen is aware of this family tree of my Freewheeling Apps. You can click on any node in this image to go to its repo. Each repo’s Readme has a section called “Mirrors and Forks” which links to its immediate upstream and downstreams (example). However it’s a manual task to keep it updated, and I see now that the example is out of date compared to the image. (And of course the image is out of date since I’ve added repos since May of this year.) Even if it was kept up to date, this isn’t quite what you’re asking for. It’s just a one-line summary for each fork, without a way to drill down into differences.
On the other end of the spectrum, a graphical diff tool can provide an extremely thorough list of all the differences between two forks. Just git remote add multiple forks into the same working directory, then git pull local branches, and something like lazygit will list all the changed lines. But this loses the forest for the trees. The right git log command can help summarize it, but it can still feel noisy, because you see every bugfix for a new feature.
I think this is partly a code representation problem. It’s hard to imagine a way to improve things that doesn’t require a lot of manual work to keep up to date. If you switched your code representation to something like my layers idea, however, a diff viewer would work a lot better. New features would become legible to tools as new files, and it becomes more tractable to drill down into changes to a single common feature/file. Layers probably don’t scale, but I think I get to handwave that away for Malleable software
There is a discovery piece that is independent of summarizing changes once you know about a fork. To answer your question, no I am not aware of any innovation in this area in federated Github clones. My sense is that the market hasn’t yet demonstrated enough demand for it to be worth anyone’s time. Well, at least anyone sane enough to care whether there’s anyone around to see what they’re doing
Your layers remind of what people use rebasing in git for: transforming chronological history into a more comprehensible narrative. Roughly, what you propose is to make the narrative part of the code, rather than keep it as metadata outside of the code. Which I think is a good idea in principle, though I am not convinced that the precise approach you outline would actually work for non-trivial code.
git remote is not good enough in my opinion, because remotes are defined in working copies, not repositories. If I have your repository as a remote in mine, that doesn’t give me access to your remotes for discovering the wider networks. But this is a minor technical issue, some convention for storing remotes in a special branch would be sufficient.
There’s something very interesting here, and it touches on the notion of App Free Computing too. There is the ForgeFed specification that provides guidance how software forges can integrate, and allow working on any repository regardless of its location. First of all I’d like to mention that an implementer is free to implement their own process using this specification. One that is more in line of what’s needed for malleable software collab.
Going from there, more interesting to consider is: What is a software forge?
Well, it is an App, and it provides UI’s and abstractions that expose you to an application domain. This application domain in turn serves to help you be more productive in a particular business domain. This (top-level) business domain is nothing less than Software Development itself.
Now a problem statement:
In code forges typically the application domain has become more dominant than the business domain.
All forge softwares are more or less similar in what they offer. A representation of the file hierarchy of the codebase, a list of issues, a list of patches to review, wiki pages, etc. Now consider this:
Where is the software development process in this software abstraction?
How can I track stakeholder Needs being delivered as concrete features, and keep monitoring them?
In this Developer-oriented tool, how can I involve other stakeholders in a multi-disciplinary fashion?
If we go back to the potential that the Fediverse might unleash, we shouldn’t talk about “forge federation” (application domain), but “federated software development” (business domain). Together with App Free Computing this might unlock a vast ecosystem where many people contribute the building blocks in which malleable software can be delivered.
@jryans passed me this link to Situated Software written by Clay Shirkey. It contrasts, what it calls Web School software (basically the entire Web 2.0 high-scale generic platforms), with Situated Software that is tailored for small groups (“Software for your Mom”). With App Free Computing in mind, situated software would be composed from small building blocks that can be universal and reusable. Component-oriented programming. A choreography of small components. Joyful creation of social experiences.
This fits the family tree and layers idea of @akkartik, as well as the notion of Freewheeling apps. If I look at the layers post, I consider the larger blocks to be similar to the Bounded Contexts of Strategic Design in DDD.
Now, back to forges. It is clear to see that they are typical Web School applications. A lot of flexibility and possibillities to accurately support real-world development processes are lost this way. They provide a straitjacket set of features to conform to. And instead of fully supporting the existing development lifecycle, it must be adjusted to adopt work processes to the capabilities of the software forge.
If we go back to considering the business domain of software development, and how it might be supported in a component-oriented manner, we can ditch the restrictions this app places on our imagination.
Why, for instance, do I need to think about the file hierachy of my codebase? The namespaces (modules, clases, whatever they are called) and logic branches in the code itself are the real structure after all, and they are a graph. There are best-practices for chunking code into files, both in language-specific and project-specific conventions. This could be done by the tool.
And how would my codebase be best represented in terms of UX/UI if it allowed me to at any time drill down from stakeholder Needs into functionality and features delivered?
Update: I must remark here that when it comes to software forges, the big platform providers and innovative startup vendors, are in a shift towards a new paradigm where they offer Web School type one-stop-shop experiences on their platforms that facilitate a broader part of the software development lifecycle and more stakeholder types. This trend is Not Good™ for open ecosystems and software freedoms in general. It cements the already overly dominant position of Big Tech platforms. Github is of course a prime example, and you can check out Github Next for a taste of what’s coming.
I only bumped into this very recently, and do not have any formed opinion on it. Don’t take the provided resources literally, but more as a source of inspiration and brainstorming. It becomes interesting when you consider “Collaborative Moldable Development of Malleable Systems”. I found this article:
And in turn it referenced this toolkit:
Now, I don’t find this directly intuitive, and have my reservations on various aspects of the methododology, but when I see these screenshots I imagine them to be a Federated Wiki (or even a Social knowledge fabric) formed from Living documentation and Living specifications that exist in a large malleable software ecosystem. Where each participant outlines the perspectives and changes that led to creation of their fork, and demonstrates them with dynamic content, runnable code snippets, and entry points to make further extensions and adaptations.
Living documentation aims to remove much the of the burden and chores in keeping up-to-date documentation. Docs and specs are part of the codebase and integrated note-taking, diagramming and other inline editing tools make that work as easy as possible. A nice example is Livebook for Elixir.
I definitely agree with all you say here. The reason why I asked about forge federation is because I see them as a first step into a direction I see as worth pursuing. But I don’t want to stay in the app-centric world either, but rather pursue the largely forgotten direction of computational media.
situated software would be composed from small building blocks that can be universal and reusable.
That’s a sentence I could have used verbatim in a recent article I wrote on scientific computing. That’s a highly specialized corner of software, but it faces the same issues as computing elsewhere: fewer and fewer people write the fewer and fewer software packages that everybody uses, leading to a loss of diversity and a concentration of power.
I have been using Glamorous Toolkit for several years, starting with an early alpha release. It’s one of my main computing platforms today, and I wish I could use it for more. There’s obviously some exaggeration and hype in the communication about it, but it delivers on most of its promises. This video is probably the best high-level overview.
However, the current incarnation of Glamorous Toolkit is made for individuals and small teams, not for larger groups of loosely interacting people. It doesn’t address the question I asked at the start of this thread.
I really like the discussion and the fact that we have very similar points of views. I really enjoy it.
I am in favor of situated software and computational media. ( I believe that everything should be an actor, I believe this is the approach of dynamicland as well)
My personal belief is that the semantic web failed to have truction because it looked at data and computation and disregarded the social processes themselves, the custodians of the data and computation.
So for me the solution is to switch from this point of view and start from the social processes.
If we look at dynamicland, they specifically limited the social interactions to those of a community around a building, interactions with physical presence. This has defined the software they are building. In our case, the repository of software is paper bindings, and software copying and change happens with a printer.
Yes, in Open Science you have quite a challenge on your hands, @khinsen. The way forward as outlined in your paper is a good one, and there are five broad categories where improvements can be introduced. The most crucial part is imho the first paragraph in the conclusion:
My principal goal with this work is to encourage scientists and research software engineers to reflect about their computational practices. Why, and to what degree, do you trust your own computations? How reliable do they have to be to support the conclusions you draw from their results? Why, and to what degree, do you trust the computations in the papers you read and cite? Do you consider their reliability sufficient to support the conclusions made?
And in particular in these words: “encourage scientists and research software engineers”. In order to tackle the challenge, a massive shift in the entire field of Open Science is needed. Mass adoption of good practices and appropriate tools.
The challenge is not dissimilar to where I’m interested wrt grassroots movements and “herding cats” (or rather: not try to herd them). Except, maybe, that in academic circles there’s more likelihood of broad adoption once a set of robust, proven approaches are available, both for the work methods as well as for creating supportive tools.
And here I fully agree with @Apostolis stating that…
As mentioned in Let’s Reimagine Social I borrowed Toshihiko Yamakami’s terminology of Social Experience Design (SX) to indicate an approach to solution design where the people’s needs and their interactions with others are central. Yamakami wrote a number of interesting papers on SX and e.g. contrasts it with UX (a broad field that in practice is more app-centric). Nonetheless SX never took hold as a field in it its own right AFAIK.
(Note: Like with App Free Computing, coined by Chris Gebhardt of infocentral.org, I like these terms for what they convey as a concept, but will not necessarily use them in same context or meaning. Gebhardt for instance used it in combination with a technical direction he is no longer pursuing.)
However, while recognizing that tech can only ever by supportive, on the technological front there are a whole range of innovations and new paradigms that are waiting to be explored and combined in interesting new ways. Malleable systems, small-tech/small web, polyglot development, decentralization, local-first software, component-oriented design, event-driven architectures, actor model, object capabilities. The list goes on…
If we look at Glamorous Toolkit it is clear that the approach is conceptually sound, but too opinionated to find the kind of broad adoption that is needed. It is asking to do things a particular way, in an environment where everyone wants to do things My Way™. It is very tricky to support such heterogenous environment, but broad application of open standards may help.
Aside: Here it is interesting to note Github again and their one-stop-shop vendor lock-in strategy. With Github Blocks they are exploring use of the (industry) standard Block Protocol and that’ll obviate the need of devtool vendors to have their own full-blown SaaS platform. But these standards should be used to create open ecosystems, of course. Btw, I should point to Block Protocol creator hash.ai giving their own twist to Semantic Web for use with live data and interactive simulations.
Stimulating emergence of an ecosystem, where countless people are doing their own thing (wherever their interests lie) and start to explore possibilities, find opportunities and collectively unleash the potential, is itself a major SX undertaking. My pursuit of Prosperity Guilds may prove to be naive in the breadth and scope it entails, yet when it has an elegant simplicity at its heart the formula may be viable (the first practice is to let go of Expectations, which is a good help here).
I doubt that broad adoption is a goal of its authors. feenk is a small consulting company. Glamorous Toolkit is a tool for their own use that they make public, and also the support for an ongoing research project on Moldable Development.
BTW, if all software developers did small-scale research before aiming for broad adoption, the tech world would probably be a better place. Premature growth is a huge issue.
Another thought on forges as apps: a very much underestimated version control system is Fossil. Perhaps its best-known feature is integrating an issue tracker into its Web interface and storing the issues along with the repository. But at that level, it’s still an app. What I find much more interesting is that a repository is just a SQLite database, with a documented schema. That opens the way to a much better integration into various software systems. But as far as I know, nobody has been doing this so far.
Ah, I should have been clearer in my explanation. I was musing on “broad adoption” of a paradigm where people are doing their own research, use their own tech stacks and preferred dev process, create malleable software, operate independently, and yet are still fully able to ‘weave’ into the patchwork of what other people are doing. Given the range of technologies and open standards at our disposal there should be opportunities to provide the “technology substrate” that supports this paradigm, so that a large ecosystem can form organically (no need for scale, decentralized and supporting growth).
And in that picture I see Glamorous Toolkit as a PoC that demonstrates the benefits of Living documentation and Computational media (but on a particular techstack, and not suitable for large ecosystems).
DVCS innovations
I agree. An interesting (but long) read is Problems with Pull Requests and How to Fix Them, which makes an argument that current popular version control systems sort of copied a similar approach and didn’t innovate much. And then continues…
As I’ll argue, the current implementation of pull requests actively discourages the many smaller changes workflow. And since smaller changes result in higher quality and faster reviews, today’s implementations of pull requests are undermining quality and velocity.
Btw, Jujutsu VCS project is exploring a bunch of nice innovations. It was recently discussed on HN here (and also in this older discussion thread).
Yeah I’ve been watching Jujutsu closely. Unfortunately it requires a newer version of Rust, which puts me off. On some level that’s the core problem all my work is trying to combat. Normal people can’t (and shouldn’t be expected to) spend all their time constantly upgrading software.
Rust is a nice illustration of what I see as a fundamental problem in computing: technology that is still under active development is at the same time used as if it were stable infrastructure. In scientific computing, Python and Julia are the two main examples (but Rust is starting to appear as well). When I point out that problem, the usual answer is “oh, just run everything in Docker containers”. But infrastructure is something different than snapshotting: it’s a stable technology layer on which non-stable technology can be built.
That’s exactly my view as well. It’s the ideas and concepts that matter, which will then hopefully be adopted elsewhere, and become part of more mainstream technologies. This is slowly happening, e.g. in the Clojure universe (e.g. https://www.youtube.com/watch?v=3bs3QX92kYA). Yes, I know, only Smalltalkers would call Clojure mainstream
Let us suppose that we have a search engine that searches “similar” software. What we want is a way to detect change with regards to a specific module/function/component. I think there are two different types of changes that need to be detected.
A. Internal change. This is the easiest way to detect. We need a specification and any component that respects the specification is similar to the component we have/search. Thus, we can detect change by filtering to just the components with the same external behavior.
B. External Change. Here we detect the change of the specification or the complete absence of the component from the exterior component/environment.
An example :
A. Upgrading a combustion engine retains the same interface with the rest of the car. Thus we can search for new combustion engines.
B. Building electric vehicles is an exterior change. The way to detect it is to find an upper component that retains the specification of the upper component of our part/component. In this case, a combustion engine is used in a car. Electric vehicles are cars and they do not have a combustion engine.
To put it another way, we need to build a search engine that searches across all levels of abstraction.
This is mostly inspired by my rudimentary knowledge of complex systems and their ability to retain stable structures at different levels of “abstraction”.
In code forges typically the application domain has become more dominant than the business domain.