workflow 0.0.1-preview.1

Incremental workflow engine: steps, dependencies, fingerprints, caching, and services

Readme Files Dependencies Versions

workflow#

An incremental workflow execution engine for Zena: dependency graphs, content fingerprints, cache keys with early cutoff, fine-grained per-output invalidation, and on-demand service lifecycles.

workflow performs no I/O of its own. Hosts supply leaf fingerprints, a digest function, a step cache, and step callbacks; the engine schedules work in dependency order with bounded parallelism and returns the updated state.

Installation#

zena pkg add zena-lang:workflow

Example#

import { NullCache } from 'zena-lang:workflow/cache';
import { Engine, StepSource, Succeeded } from 'zena-lang:workflow/engine';
import { Fingerprint } from 'zena-lang:workflow/fingerprint';
import { Graph } from 'zena-lang:workflow/graph';
import { OutputEntry, OutputManifest } from 'zena-lang:workflow/manifest';
import { EngineState } from 'zena-lang:workflow/state';
import { LeafInput, StepKey, StepOutputs, TaskStep } from 'zena-lang:workflow/step';

let compile = new TaskStep(
  new StepKey('compile'),
  new Fingerprint('cmd:compile-v1'),
  [new LeafInput('src/main.zena')],
  async (ctx) => {
    return new OutputManifest([
      new OutputEntry('out/main.wasm', new Fingerprint('wasm-hash-1')),
    ]);
  },
);

let bundle = new TaskStep(
  new StepKey('bundle'),
  new Fingerprint('cmd:bundle-v1'),
  [new StepOutputs(new StepKey('compile'))],
  async (ctx) => {
    return new OutputManifest([
      new OutputEntry('dist/app.wasm', new Fingerprint('bundle-hash-1')),
    ]);
  },
);

let graph = new Graph([compile, bundle]);
let leaves = {'src/main.zena' => new Fingerprint('src-hash-1')};
let engine = new Engine((data) => new Fingerprint( sha256(data) ), new NullCache());

// First run executes both steps in topological order.
let first = await engine.run(graph, leaves, EngineState.empty());

// Second run with identical inputs and prior state is a no-op (StepSource.Fresh).
let second = await engine.run(graph, leaves, first.state);

Core Concepts#

  • Steps and inputs (workflow/step):
    • TaskStep: runs to completion and returns an OutputManifest.
    • ServiceStep: starts on demand only when a dependent step actually executes, and stops when its last dependent settles (or stays pinned across watch runs in a ServicePool).
    • LeafInput(name): a named external input resolved from the leaves map passed to Engine.run.
    • LeafSet(name, resolve): a lazy input set resolved after upstream dependencies finish (for example, expanding globs over generated files).
    • StepOutputs(step): a dependency on another step whose output manifest (or service identity) enters this step's cache key. Because dependents key on upstream output fingerprints rather than upstream input fingerprints, an upstream step that reruns and produces identical outputs leaves downstream dependents fresh (early cutoff).
    • StepOrder(step): an ordering-only dependency (like Wireit's "cascade": false) that waits for the upstream step without folding its outputs into the dependent's cache key.
  • Validated graph (workflow/graph): Graph checks that step keys are unique, all referenced steps exist, no step lists duplicate inputs, and the dependency graph is acyclic (reporting the cycle path in WorkflowError), producing a deterministic topological order.
  • Caching and freshness (workflow/engine, workflow/cache, workflow/state):
    • If a step's CacheKey matches the StepRecord in the previous EngineState (and optional RunOptions.verifyOutputs succeeds), the step is StepSource.Fresh without consulting the cache or starting services.
    • Otherwise StepCache.get is checked (StepSource.Cache), falling back to executing the step callback (StepSource.Ran) under the run's parallelism limit.
  • Incremental per-output deltas (workflow/delta, workflow/manifest):
    • Each OutputEntry in an OutputManifest can record dependsOn (the namespaced inputs leaf:..., step:... it was derived from).
    • When a step reruns with the same config fingerprint, RunContext provides delta (changed, added, removed), previous (OutputManifest), and staleOutputs. The engine verifies before caching that any output carried over as non-stale kept its previous fingerprint.

Libraries#

  • workflow/engine — Engine, RunOptions, RunResult, StepOutcome (Succeeded, Failed, Skipped), StepSource, ServicePool, and PinnedService.
  • workflow/step — Step, TaskStep, ServiceStep, StepKey, InputRef (LeafInput, LeafSet, StepOutputs, StepOrder), RunContext, ServiceContext, and ServiceHandle.
  • workflow/graph — Graph validation and deterministic topological sorting.
  • workflow/manifest — OutputManifest and OutputEntry.
  • workflow/state — EngineState and StepRecord for persisting prior-run state across runs.
  • workflow/cache — StepCache interface and NullCache.
  • workflow/fingerprint — Fingerprint, CacheKey, and Digest.
  • workflow/canonical — Canonical length-prefixed encodings (cacheKeyFor, manifestFingerprint, ResolvedInput, compareStrings, sortBy).
  • workflow/delta — InputDelta, computeDelta, and staleOutputNames.
  • workflow/errors — WorkflowError.

License#

MIT

4.9 KiB Raw
import {ResolvedInput} from './canonical.zena';
import {InputDelta} from './delta.zena';
import {Fingerprint} from './fingerprint.zena';
import {OutputManifest} from './manifest.zena';

/** The stable identity of a step across runs. */
export class StepKey(value: String)

/** A step input: a named leaf value, a lazy leaf set, or a dependency on another step. */
export sealed class InputRef {
  case LeafInput, LeafSet, StepOutputs, StepOrder
}

/**
 * A named leaf input. The host resolves the name to a fingerprint at
 * run time; for a build system a leaf is a file, for other hosts it can
 * be anything fingerprintable.
 */
export final class LeafInput(name: String) extends InputRef

/**
 * A named set of leaf inputs resolved lazily once this step's
 * dependencies have settled. For a build system, this expands and
 * hashes input globs after upstream steps have written generated files
 * into the workspace. Each `(name, fingerprint)` pair enters the cache
 * key in the `leaf:` namespace, just like `LeafInput`.
 */
export final class LeafSet(
  name: String,
  resolve: () => Future<Array<(String, Fingerprint)>>,
) extends InputRef

/**
 * A dependency on another step: an ordering edge in the graph. For a
 * task dependency this contributes the producing step's output
 * manifest to the cache key; for a service dependency it contributes
 * the service's identity fingerprint and requires the service to be
 * ready while the dependent runs.
 */
export final class StepOutputs(step: StepKey) extends InputRef

/**
 * An ordering-only dependency on another step (such as wireit's
 * `"cascade": false`). Like `StepOutputs`, the engine waits for a task
 * dependency to succeed (or skips on failure) and starts and holds a
 * service dependency while this step executes, without adding the
 * upstream step's manifest or identity fingerprint to this step's
 * cache key.
 */
export final class StepOrder(step: StepKey) extends InputRef

/**
 * What a task step's run callback receives. When the last recorded run
 * had the same config fingerprint, the engine supplies the input delta
 * against it; the callback may then regenerate only the stale outputs
 * and carry the rest of the previous manifest over unchanged (the
 * engine verifies the carry-over against the dependency records).
 */
export class RunContext {
  /**
   * Changed/added/removed inputs since the last recorded run, or null
   * when no usable history exists (first run, or the step's config
   * changed) — then the step must run from scratch.
   */
  delta: InputDelta?;

  /** The last recorded run's manifest, when `delta` is non-null. */
  previous: OutputManifest?;

  /**
   * Previous outputs invalidated by the delta, per their dependency
   * records. Null exactly when `delta` is null.
   */
  staleOutputs: Array<String>?;

  /**
   * This run's resolved inputs (`leaf:`, `step:`, `service:`),
   * allowing group or fileset steps to forward input fingerprints.
   */
  inputs: Array<ResolvedInput>;

  new(
    delta: InputDelta? = null,
    previous: OutputManifest? = null,
    staleOutputs: Array<String>? = null,
    inputs: Array<ResolvedInput>? = null,
  ) : delta = delta,
      previous = previous,
      staleOutputs = staleOutputs,
      inputs = inputs ?? new GrowableArray<ResolvedInput>();
}

/** What a service step's start callback receives. */
export class ServiceContext {

}

/**
 * A running service, as seen by the engine. The host's start callback
 * returns one; the engine calls `stop` when the service's last
 * dependent has settled (or when a pinned watch session ends).
 */
export interface ServiceHandle {
  isAlive(): boolean;
  stop(): Future<void>;
}

/**
 * A node in the graph: a task that runs to completion, or a service
 * that stays up while dependents need it.
 */
export sealed class Step {
  case TaskStep, ServiceStep

  key: StepKey;

  /**
   * Covers everything that affects the step's behavior other than its
   * inputs: the command line, rule parameters, rule version.
   */
  config: Fingerprint;

  inputs: Array<InputRef>;

  new(this.key, this.config, this.inputs);
}

/**
 * A unit of work: runs to completion and produces an output manifest.
 * The run callback is supplied by the host and performs all I/O; the
 * library only decides whether and when to invoke it.
 */
export final class TaskStep extends Step {
  run: (ctx: RunContext) => Future<OutputManifest>;

  new(key: StepKey, config: Fingerprint, inputs: Array<InputRef>, this.run)
      : super(key, config, inputs);
}

/**
 * A step that stays running instead of completing. `start` resolves
 * when the service is ready to serve dependents. The engine starts a
 * service only when a dependent actually executes (fresh and
 * cache-restored dependents never demand it) and stops it when its
 * last dependent settles.
 */
export final class ServiceStep extends Step {
  start: (ctx: ServiceContext) => Future<ServiceHandle>;

  new(key: StepKey, config: Fingerprint, inputs: Array<InputRef>, this.start)
      : super(key, config, inputs);
}
none
VersionPublishedDependenciesApproved by
0.0.1-preview.1 2026-11-19 none