workflow 0.0.1-preview.1

Incremental workflow engine: steps, dependencies, fingerprints, caching, and services

Readme Files Dependencies Versions

workflow#

An incremental workflow execution engine for Zena: dependency graphs, content fingerprints, cache keys with early cutoff, fine-grained per-output invalidation, and on-demand service lifecycles.

workflow performs no I/O of its own. Hosts supply leaf fingerprints, a digest function, a step cache, and step callbacks; the engine schedules work in dependency order with bounded parallelism and returns the updated state.

Installation#

zena pkg add zena-lang:workflow

Example#

import { NullCache } from 'zena-lang:workflow/cache';
import { Engine, StepSource, Succeeded } from 'zena-lang:workflow/engine';
import { Fingerprint } from 'zena-lang:workflow/fingerprint';
import { Graph } from 'zena-lang:workflow/graph';
import { OutputEntry, OutputManifest } from 'zena-lang:workflow/manifest';
import { EngineState } from 'zena-lang:workflow/state';
import { LeafInput, StepKey, StepOutputs, TaskStep } from 'zena-lang:workflow/step';

let compile = new TaskStep(
  new StepKey('compile'),
  new Fingerprint('cmd:compile-v1'),
  [new LeafInput('src/main.zena')],
  async (ctx) => {
    return new OutputManifest([
      new OutputEntry('out/main.wasm', new Fingerprint('wasm-hash-1')),
    ]);
  },
);

let bundle = new TaskStep(
  new StepKey('bundle'),
  new Fingerprint('cmd:bundle-v1'),
  [new StepOutputs(new StepKey('compile'))],
  async (ctx) => {
    return new OutputManifest([
      new OutputEntry('dist/app.wasm', new Fingerprint('bundle-hash-1')),
    ]);
  },
);

let graph = new Graph([compile, bundle]);
let leaves = {'src/main.zena' => new Fingerprint('src-hash-1')};
let engine = new Engine((data) => new Fingerprint( sha256(data) ), new NullCache());

// First run executes both steps in topological order.
let first = await engine.run(graph, leaves, EngineState.empty());

// Second run with identical inputs and prior state is a no-op (StepSource.Fresh).
let second = await engine.run(graph, leaves, first.state);

Core Concepts#

  • Steps and inputs (workflow/step):
    • TaskStep: runs to completion and returns an OutputManifest.
    • ServiceStep: starts on demand only when a dependent step actually executes, and stops when its last dependent settles (or stays pinned across watch runs in a ServicePool).
    • LeafInput(name): a named external input resolved from the leaves map passed to Engine.run.
    • LeafSet(name, resolve): a lazy input set resolved after upstream dependencies finish (for example, expanding globs over generated files).
    • StepOutputs(step): a dependency on another step whose output manifest (or service identity) enters this step's cache key. Because dependents key on upstream output fingerprints rather than upstream input fingerprints, an upstream step that reruns and produces identical outputs leaves downstream dependents fresh (early cutoff).
    • StepOrder(step): an ordering-only dependency (like Wireit's "cascade": false) that waits for the upstream step without folding its outputs into the dependent's cache key.
  • Validated graph (workflow/graph): Graph checks that step keys are unique, all referenced steps exist, no step lists duplicate inputs, and the dependency graph is acyclic (reporting the cycle path in WorkflowError), producing a deterministic topological order.
  • Caching and freshness (workflow/engine, workflow/cache, workflow/state):
    • If a step's CacheKey matches the StepRecord in the previous EngineState (and optional RunOptions.verifyOutputs succeeds), the step is StepSource.Fresh without consulting the cache or starting services.
    • Otherwise StepCache.get is checked (StepSource.Cache), falling back to executing the step callback (StepSource.Ran) under the run's parallelism limit.
  • Incremental per-output deltas (workflow/delta, workflow/manifest):
    • Each OutputEntry in an OutputManifest can record dependsOn (the namespaced inputs leaf:..., step:... it was derived from).
    • When a step reruns with the same config fingerprint, RunContext provides delta (changed, added, removed), previous (OutputManifest), and staleOutputs. The engine verifies before caching that any output carried over as non-stale kept its previous fingerprint.

Libraries#

  • workflow/engine — Engine, RunOptions, RunResult, StepOutcome (Succeeded, Failed, Skipped), StepSource, ServicePool, and PinnedService.
  • workflow/step — Step, TaskStep, ServiceStep, StepKey, InputRef (LeafInput, LeafSet, StepOutputs, StepOrder), RunContext, ServiceContext, and ServiceHandle.
  • workflow/graph — Graph validation and deterministic topological sorting.
  • workflow/manifest — OutputManifest and OutputEntry.
  • workflow/state — EngineState and StepRecord for persisting prior-run state across runs.
  • workflow/cache — StepCache interface and NullCache.
  • workflow/fingerprint — Fingerprint, CacheKey, and Digest.
  • workflow/canonical — Canonical length-prefixed encodings (cacheKeyFor, manifestFingerprint, ResolvedInput, compareStrings, sortBy).
  • workflow/delta — InputDelta, computeDelta, and staleOutputNames.
  • workflow/errors — WorkflowError.

License#

MIT

3.4 KiB Raw
/**
 * Canonical encodings for cache keys and manifest fingerprints.
 *
 * Every segment of an encoding is length-prefixed, so distinct inputs
 * never encode alike, and (name, fingerprint) lists are sorted by name,
 * so keys are independent of declaration order. The encoded string is
 * handed to the host's digest function; it is never stored.
 */
import {StringBuilder} from 'zena:core';
import {CacheKey, Digest, Fingerprint} from './fingerprint.zena';
import {OutputManifest} from './manifest.zena';
import {WorkflowError} from './errors.zena';

/** Byte-wise string comparison: negative, zero, or positive. */
export function compareStrings(s1: String, s2: String): i32 {
  let len1 = s1.length;
  let len2 = s2.length;
  let minLen = if (len1 < len2) len1 else len2;

  for (var i = 0; i < minLen; i += 1) {
    let b1 = s1.getByteAt(i);
    let b2 = s2.getByteAt(i);
    if (b1 < b2) {
      return -1;
    }
    if (b1 > b2) {
      return 1;
    }
  }

  if (len1 < len2) {
    return -1;
  }
  if (len1 > len2) {
    return 1;
  }
  return 0;
}

/** In-place insertion sort; the stdlib has no sort. */
export function sortBy<T>(items: GrowableArray<T>, compare: (a: T, b: T) => i32): void {
  let len = items.length;
  for (var i = 1; i < len; i += 1) {
    let item = items[i];
    var j = i - 1;
    while (j >= 0 && compare(items[j], item) > 0) {
      items[j + 1] = items[j];
      j -= 1;
    }
    items[j + 1] = item;
  }
}

/** A step input resolved to its fingerprint for one run. */
export class ResolvedInput(name: String, fingerprint: Fingerprint)

function appendSegment(sb: StringBuilder, s: String) {
  sb.appendI32(s.length);
  sb.append(':');
  sb.append(s);
}

/**
 * Appends sorted (name, fingerprint) pairs, rejecting duplicate names.
 * `what` names the collection in the error message.
 */
function appendPairs(
  sb: StringBuilder,
  pairs: GrowableArray<ResolvedInput>,
  what: String,
) {
  // Annotated params: closure arguments to generic functions don't get
  // contextual types from the inferred type arguments yet.
  sortBy(pairs, (a: ResolvedInput, b: ResolvedInput) => compareStrings(a.name, b.name));
  var prev: String? = null;
  for (let pair in pairs) {
    if (prev != null && compareStrings(prev, pair.name) == 0) {
      throw new WorkflowError(`duplicate ${what} name: ${pair.name}`);
    }
    prev = pair.name;
    appendSegment(sb, pair.name);
    appendSegment(sb, pair.fingerprint.value);
  }
}

/**
 * Derives a step's cache key from its config fingerprint and resolved
 * inputs. Input order does not matter; duplicate input names are an
 * error.
 */
export function cacheKeyFor(
  config: Fingerprint,
  inputs: Array<ResolvedInput>,
  digest: Digest,
): CacheKey {
  let sb = new StringBuilder();
  appendSegment(sb, 'workflow-key-v1');
  appendSegment(sb, config.value);
  appendPairs(sb, GrowableArray.from(inputs), 'input');
  return new CacheKey(digest(sb.toString()).value);
}

/**
 * Fingerprints an output manifest: the digest of its sorted
 * (name, fingerprint) pairs. This is what dependent steps see as the
 * producing step's contribution to their cache keys.
 */
export function manifestFingerprint(
  manifest: OutputManifest,
  digest: Digest,
): Fingerprint {
  let sb = new StringBuilder();
  appendSegment(sb, 'workflow-manifest-v1');
  let pairs = new GrowableArray<ResolvedInput>();
  for (let entry in manifest.outputs) {
    pairs.push(new ResolvedInput(entry.name, entry.fingerprint));
  }
  appendPairs(sb, pairs, 'output');
  return digest(sb.toString());
}
none
VersionPublishedDependenciesApproved by
0.0.1-preview.1 2026-11-19 none