Let us assume that we have metadata that shows the dependencies of software.
Responding to a change in a specific library, we currently have continuous integration, but it still takes a lot of time, the feedback is not immediate.
I was thinking of the possibility of performing the tests across all the ecosystem that uses this specific software.
Currently, it seems to me that it takes a few months or more for one change to affect the other parts of the ecosystem. What if that feedback was instant?
We do, for useful subsets of all software. What I am most familiar with is the Guix package manager. Its package graph would permit an implementation of what you propose, although “instant” would have to be interpreted in the context of available compute and network resources.
My main worry would be the enormous resource requirements when hubs in the graph are changed.
There’s also Rust’s Crater tool which used for running experiments across many packages in the ecosystem to check whether e.g. some compiler change is safe or if there’s some lurking failure uncovered by some surprising usage in downstream packages.
It does seem quite challenging to produce this feedback “instantly” with today’s infrastructure… I wonder what would need to change… Probably many things!
I am on nixos here… I like the culture of Guix and Ludovic’s interest in scientific reproducibility of software.
( I have to look at its test methods closer, maybe they are already doing this in a way.) I know for sure that there is a very accurate method of determining the dependencies, and stripping unnecessary ones.
Of course, a lot of things need to change. It is an engineering problem. It is not as easy as simply redoing all the tests as soon as a change happens.
One simple solution might be to alter the reoccurrence of a test based based on the degree of separation and and the number of changes, maybe from multiple dependencies. A configurable form of batching.
Tooling for this exists in some large private software ecosystems, such as Google’s primary codebase as powered by Blaze (internal equivalent to the open source Bazel build tool). It achieves reasonable run cost across very large dependency graphs via aggressive, fine-grained compilation caching. As jryans mentions, this is possible due to an emphasis on reproducible builds. So you can query to find all builds (“targets”) that are downstream dependencies (Query guide | Bazel) and then rebuild all of those dependencies, executing their tests, to get ecosystem-wide feedback.
In the original post, the question is about getting faster, ideally instant, feedback on potentially breaking dependency changes. It seems the limit would be the propagation time across partitioned codebases (eg, distributed or different visibility scopes). In an ecosystem where all of the code is, in principle, visible and available at the site of the changing dependency, the same strategy of “find all downstream dependencies and rebuild them” can be applied, although depending on heterogeneity this might be a knotty problem.
