warpforge evolution directions (and recursive expanders for build plans)

This is a working document to snapshot some thoughts about the evolution of Warpforge and its APIs.

Observations (that lead to some desires for changes)

We should seriously consider moving the Catalogs to an even yet plainer format.
- The Catalog format as it stands today is a pain to edit manually, due to its inclusion of hashes. (This has been remarked upon by essentially every user to date.)
- We also derive very little value from the use of hashes in the middle of the Catalog data, because in practice, we find we're continuously embedding the Catalog data in other version control systems, which already contain their own checksums. While there is elegance to having our own hashing scheme provide our own bespoke incrementally verifiable trees, there is little strong demonstration of that providing direct value (and very many demonstrations of it creating problematic friction).
- The Catalog format seems like it could become more generally usable (and thus more appealing for others to interact with) with a small amount of extraction and generalization work.
We should seriously consider further iteration on the build plan / plot format.
- The current system of buildplugs which can generate plans... is workable. But it is still not low friction. And most notably, while it is clear in this framework how to use different buildplug systems in different modules and plans... it is not clear how one could compose distinct buildplug systems into one plan, which seems like a considerable limitation.
- The focus on plans being "fully stamped out" and serialized to disk in that form has severe scalability limitations. It was originally defined in the context of thinking primarily about package management (wherein low granularity is the norm, and the total number of targets/steps/whatyoucallit in a module would be low -- often single digits); in that context, it's viable. In other contexts: if we wished to pursue more granular builds, down to compilation units, or smaller (e.g. individual tests, where languages and tools might support that), then the scalability problem becomes evident: we don't want to (frequently) (re)write a json file of that size.
  - and we should consider how we can move to support those higher levels of granularity. Developer adoption and packagers adoption are two different user stories; it's becoming clear developer adoption is going to remain extremely limited without greater compilation unit granularity. And we do want to increase developer adoption, not just chase the very (very) small population of packagers who are not also developing the things they package.
- We still have no sufficiently clear plan for how to integrate other package management tools (a la the size and scope of cargo, for example) gracefully into the current plots+buildplugs system. The rough hypothesis is that we could build "bifrost" systems that have a buildplug run that foreign packaging system, then generate plot data from it, and so on. As discussed above, this is probably workable, but is very unclear if it will be graceful, and it also seems unlikely to have fully explained how it should work if more than one such system is desired in a single module/plot.

So we have some thinking to do.

(And following is some of it I already have done!)

Proposals

For Catalogs

See the Text Catalog gist.

For Build Plans

I propose three main changes (no, four, no, five!):

First of all, most minorly, but let's get it out of the way: the name "plot" has gotten a lot of flak; let's just drop that now. Let's use "build plans" as the working title for now now (unless another yet better idea comes along).
Second: Let's make a more recursive definition of build plans. (This'll be a big topic.)
- A whole section about this, below.
Third: the "type system" for values passed by build plans should be enhanced and its coersions should be better defined. (There is some existing handling of both WareIDs (which point to full filesystems) and of string literals, but many of the intersections of this are not especially well defined, or if it is, the results are definitely not friendly and intuitive, nor particularly useful.)
- Another whole section about this, below.
Fourth: and rather minorly: the build plan definition should always have a steps map. I'd rather have a default step name ("__main__" or so, perhaps) than have a single-step plan have a different level of indentation than a multi-step plan. It makes a surprisingly large difference in the amount of line noise a user experiences in the Getting Started phases of interaction.

Recursive Defns of Build Plans

The main new idea I'd like to introduce for a next generation of build plan spec is: a type of step called an "expander", which can (recursively!) yield more step definitons, which replace the original step in the build plan when "expanded".

These will introduce considerable complexity, but also considerable power and future features:

Expanders can be used to, for example, invoke other foreign package management tools (e.g. cargo and things of that scale) and use them to produce new, more granular, build plan steps.
Since Expanders fit within a build plan, they are free from a problematic challenge with the buildplug system: it is immediately clear how to use various distinct Expanders within a build plan.
If Expanders are a just another step themselves, and use the usual Warpforge input management systems, then versioning and demanding stability of Expanders becomes much easier than it was for buildplugs (which presumed additional out-of-band management).
Recursiveness of Expanders -- though a source of considerable complexity -- should enable the production of high granularity steps... even in scenarios such as autodetection of them from a filesystem, which is also a notably recursive structure. Done right, this should also lead to granular memoization cachebusting (consider the possibilities of having a recursive expander which also has integration with and understanding of git treehashes!).

The primary new costs of this complexity, aside from the code complexity itself, is:

There must be a "compile"/analyze operation for a build plan (also applicable to each step, recursively) which checks all cross references and syntax, and can emit both validation checks, user-relevant guidance for errors, and (as much as possible) resource requirement estimates (step counts, etc).
- These analyze operations should be able to report various threshholds for "purity" of the plan, based on what type of inputs are used, and in some cases annotations of the step actions.

... and Expander nodes will gravely challenge those high level goals: they're naturally unpredictable in both how many nodes they produce, and offer no good estimates of how much work those nodes might entail to execute.

Still, what this design would enable seems compelling. The challenge is in sufficiently gracefully navigation the communication: a build plan using an expander is not yet fully resolved in the same way a build plan without them is; etc. However, we already have those changes: We already need to validate that ingests and host mounts aren't used, for instance, and that network isn't enabled; and already those are things we are not reasonably able to validate with typesystems and schemas: they're things we handle with logical checks. The more conditional definitions we'll now need for whether or not plans are fully expanded is on a very similar level to these existing challenges: it shouldn't make the system's understandability significantly worse.

We should also still absolutely be mindful of retaining the ability to invoke full expansions without any other executions, and still have a very well-paved path through the Warpforge subcommands which serializes the result.

Types and Coersions for Inputs and Outputs

((The notes following focus mostly on inputs, but the applicability to outputs is pretty obvious. This could use further improvements in the writing in a future draft.))

The Inputs of a step are a map of local names (or paths, for files) to the data source.

(Previously: these have been "/path": "ware:{packType}:{hash} or ""$VAR": "literal:{string}". These will probably remain supported as shorthands, but we'll also be introducing considerably more options for annotating those values, as well as revisiting what the default expectations are for how files may be put in place if there's no specific specification.)

Input Keys

There are roughly three types of input keys:

Paths: they start with a slash, and look like "/path/to/place/content". (Whether this path will become a file or directory depends on the input value assigned to this key.)
Variables: they start with a dollar sign, and look like "$VAR" or "$variable". These generally manifest as shell environment variables (although the step's action type ultimately determines what a variable really means for itself).
References: they start with a caret and describe coordinates in Catalog data, so they look like "^catalog:foo.org/proj:v123:arch-os".

The different types of input keys give some expectation of what type of input value will be associated with them. Input keys that are paths will probably have some sort of filesystem as the input value. Input keys that are variable names are likely to have a string literal as their value. Some coersions are possible (namely, mapping a path key to a string literal will result in the creation of a file with that string as its content); others are an immediate error (assigning a filesystem to a variable).

Input keys that are references are a little special. These often have an empty value assigned to them... because the key is actually both the key and the value. Reference keys are resolved into the value they refer to in the catalog, and resolve into one of the other key types (path, or variable) based on other metadata in the catalog. Reference keys can even unpack into multiple key-value pairs.

(Yes, this means reference keys are quite a bundle of fun! Their expansion may result in map key collisions which aren't immediately visible. However, their expansion process is a Warpforge builtin, and is at least guaranteed to be deterministic given the same catalog to resolve against.)

Input Values

Input values can be of many different types:

String literals
Integer literals
Boolean literals
Filesystem snapshot identifiers
catalog coordinates -- resolves into one of the others!
module-local wiring coordinates -- resolves into one of the others!
ingests -- resolves into one of the others!
mount declarations
directory literals
symlink literals
refinements -- apply to any of the above that result in filesystems
(SPECULATIVE) maps, lists, "any"

Many of these have single-string representations for terseness, which use fixed string prefixes to distinguish their type. Others require a full object of data.

(If comparing this against previous versions of Warpforge APIs: we're proposing quite a few more mechanisms for creating small sections of literal filesystems. The symlink and directories literals in are in this group. The "refinements" proposal is fairly present in older generations of spec, but we may want to change some defaults ("ro" defaults, perhaps? hopefully?) and revisit the exact format.)

(TODO: document a complete table for coersions. This should be pretty obvious, though, in general. Strings can used to become file literals. It usually doesn't go the other way. Etc.) (TODO: we might want to define linters for this from the outside. Sometimes I really do want that string-to-file coersion. Other time I'd rather be warned.)

(SPECULATIVE) Input values can also carry some annotations that are purely informative and for visualization purposes. For example, many catalog entries contain annotations like e.g. "pkgRole: tool", and this is a hint that can be carried through the whole system and used for visualization tools to give that particular data wire a less foreground color. (If we pursue this: It is unclear how we'd make it clear that this does not affect memoization, since everything else near it in the document would. Prior rounds of API work and user testing in Warpforge (and predicessors) indicated that mismashing hash-affecting and non-hash-affecting content together in a single tree created large barriers to comprehensiblity; two sibling trees often fared better, even if that relocated (and added) some other friction.)

Inputs when used by the Effects system

Sometimes it's useful to ask Warpforge to set up a directory bare, on the current host, to match the inputs of a step.

(SPECULATIVE FEATURE, not yet implemented!) This is possible, but remember that a step's inputs can be a mixture of files and variables. If you ask for a directory setup, Warpforge can produce it, but the variable will be ignored. If you want the variables to have any effect, you'll have to ask warpforge to give you a shell, not just the directories.

More about Reference inputs

Reference inputs are a feature that, though complex, is desirable for a couple of reasons:

It's very, very common that the user will be happy to take a package and "install" it at a predictable path chosen by the publisher of that catalog content. (And because we have strong conventions in warpsys for how those paths can be chosen to avoid collisions and synthesize easily, it's also quite possible for the publisher of catalog content to pick reasonable paths in an ahead-of-time way.)
It's also very common for packages to have dependencies, and we aren't particularly enthusiastic to demand that users do resolution processes manually, nor invoke a whole expander or use third-party templating phases in order to handle that, considering how common it is.

While these needs can be satisfied by either expanders or by third-party templating phases, demanding those more complex features for such a common case would produce a lot of user friction. More than we felt was viable.

Of course, we also don't want the complexity of anything we build into the core get too out of hand. We want deterministic results, and we also prize having predictably rapid evaluation. So, there are simple boundaries on what a reference input can do:

They're purely lookup of flat data. Resolving a reference does not afford any opportunity for any kind of code to be evaluated. This satisfies both determinism and concerns of speed.
While reference resolve can produce multiple input key-value pairs, the result if this collides with any other input keys at all (from either the top level map, or other maps produced by other reference input resolution) is an error, unless the values also exactly match.
- This error is reportable during plan analysis -- no step execution is necessary to discover it.
- Despite this harshness, collisions are uncommon in practice -- so long as catalog references you're using have followed the strongly-recommended Warpsys conventions for paths.

Reference inputs are resolved before memoization key hashing is computed. (Memoization keys always need to be extremely literal: they most definitely cannot include references to other data that's not covered by the memoization hash; and catalog roots are clearly not appropriate to fold into memoization hashes, because that would produce absurdly excessive unrelated churn. So, it follows pretty naturally that reference inputs can't be present for memoization key derivation.)

(SPECULATIVE) Reference inputs can resolve to other reference inputs. This can occur recursively. We accept this because it's overall still a rapid lookup process. The recursion depth is capped arbitrarily at 5. We've never seen a reason for it to go above 2 in real practice, and 0 is the norm. (Zero steps of recursion is still enough for to specify runtime library dependencies along with an executable package; one step is sometimes used to offer packages+configuration; two steps is enough to offer package suites with shorthand names.)

(SPECULATIVE) Reference nuzzling is a piece of additional configuration that can be placed in the value of a reference input. It allows remapping paths and variables to have additional prefixes, and thus can be used to fix collisions, or simply to apply your own desires and conventions as you see fit.

Errata about input declaration blocks in recursive plans

Warpforge plot objects previously used a full "inputs" section in every plot and subplot. This offered the ability to alias things in local scopes, but was in some cases also required, for pulling references in from the outer scope.

We'll may want to revisit this part of the spec to make those two tasks clearer.

The usage of local aliasing we observed after letting that syntax loose in the wild for a while was... mixed and minimal. (It's possible this was partially due the interplay with starlark buildplugs, which offered yet another layer of reuse which sublimated it.) I think supporting aliasing may still be useful, but should probably be... called that.

On the other hand, having a subplot section which needs to be supplied with some wiring from the enclosing context... that is a considerably different part of the system design, and much more consequential. We should review this part of the design in context of how we might expect users to try to develop reusable subplots. In that story, subplots should reasonably clearly document and declare what their required parameters are.

Previously, the use of catalog references as a subplot input could be taken to mean that input to the subplot was not a "parameter", whereas "wire" type inputs were parameters. We should see if we can make this more explicit and more clear. Parameterization, in other words, should be a more first class concept, so that it's more possible to make reusable subplots and think and talk clearly about that.

warpfork/warpforge-api-evolution-and-recursive-expanders-in-build-plans.md