zebra 0.0.1-preview.1

Zebra: an incremental build orchestrator for Zena and standalone workspaces

Readme Files Dependencies Versions

zebra#

An incremental script and service runner for Zena (zena-package.json), standalone Zebra (zebra.json), and Wireit (package.json) workspaces, built on zena-lang:workflow.

Zebra runs a dependency graph of scripts and long-running services with declared inputs, outputs, and dependencies, rerunning only what changed. For each script, Zebra content-hashes the declared inputs and asks the workflow engine what to do: nothing (when the previous run recorded the same fingerprints and declared outputs are intact), restore from the local content-addressed cache, or execute the command (sh -c in the package directory with node_modules/.bin on PATH).

Installation#

zebra is built into the zena CLI (zena build <target>). To use it as a library or standalone runner:

zena pkg add zena-lang:zebra

CLI Usage#

zena build [-j <jobs>] [-w|--watch] [-f|--force] [--no-cache] [-v|--verbose] [<target>] [-- <args>...]
zena run [-j <jobs>] [-w|--watch] [-f|--force] [--no-cache] [-v|--verbose] <target> [-- <args>...]
zena test [-j <jobs>] [-w|--watch] [--force] [--no-cache] [-v|--verbose] [-- <args>...]
zena build --migrate [--dry-run] [-f|--force] [<package-dir>...]
  • <target> can be a script name in the current package (zena build test, zena run check), a package directory (zena build ./packages/app, which runs build), or <package-dir>:<script> (zena build ./packages/app:test). With no <target>, zena build runs the current package's build script and zena test runs its test script.
  • -- <args>... appends shell-quoted arguments to the entry target's command only, running that entry step uncached without overwriting its saved freshness record.
  • -j / --jobs <n> bounds parallel command execution (defaulting to ZEBRA_PARALLEL, WIREIT_PARALLEL, or ZENA_AVAILABLE_PARALLELISM).
  • -w / --watch watches input globs, lockfiles, and package manifests for changes, re-running affected steps incrementally while keeping unchanged services alive in a ServicePool across rebuilds.
  • -f / --force bypasses freshness and cached outputs to rerun scripts unconditionally.
  • --no-cache disables restoring outputs from .zebra/blobs/ when a step is not fresh.
  • -v / --verbose prints every step (including fresh and skipped) and any captured stdout/stderr on the rail.
  • --migrate converts Wireit blocks in package.json into zena-package.json (when present) or zebra.json.

Programmatic Usage#

import { buildTarget, reportBuild } from 'zena-lang:zebra/runner';
import { ScriptRef } from 'zena-lang:zebra/wireit-config';

let outcome = await buildTarget('.', new ScriptRef('packages/app', 'build'), {
  parallelism: 4,
});
let exitCode = reportBuild(outcome);

Configuration (zena-package.json and zebra.json)#

Zena packages configure scripts and filesets directly in zena-package.json. Non-Zena projects can use a standalone zebra.json (with an optional "workspace": {"packages": [...]} at the workspace root). Both support // and /* ... */ comments:

{
  "name": "app",
  "dependencies": {
    "lib": {"path": "../lib"},
  },
  "filesets": {
    "assets": ["assets/**"],
  },
  "scripts": {
    "build": {
      "command": "zena build src/main.zena -o out/app.wasm",
      "output": ["out/app.wasm"],
      "dependencies": ["source", "assets"],
    },
    "test": {
      "type": "zena:test",
      "tests": "test/*_test.zena",
    },
    "fmt": "zena fmt --check .",
  },
}
  • Declarative zena:test steps and default test target: Scripts in zena-package.json and zebra.json can set "type": "zena:test" (with optional "tests": true, "tests": "test/*_test.zena", or "tests": ["test/*_test.zena", ...] — defaulting to ["test/**/*_test.zena", "tests/**/*_test.zena"] when omitted — and optional "debug": true for -g) to run Zena tests in-process through TestStepRunner. Test pattern globs merge into files, and zena-package.json "zena:test" steps automatically depend on source. When a zena-package.json omits a test script (and no fallback package.json defines test), Zebra synthesizes a "type": "zena:test" target over test/**/*_test.zena and tests/**/*_test.zena.
  • Automatic source fileset: Every package with zena-package.json exposes an automatic source fileset covering its library .zena files (from "source", defaulting to "src" and excluding *_test.zena, test/**, and tests/**), zena-package.json, any "wit" dependencies, and the source filesets of all "path" dependencies. Scripts opt into it via "dependencies": ["source"] (or "../pkg:source").
  • Named filesets: Entries accept either a glob array shorthand ("wit": ["wit/**/*.wit"]) or {"files": [...], "dependencies": [...]} and compose transitively without spawning commands.
  • Caching and tracking: Task scripts cache automatically whenever files, output, or a fileset dependency is declared, defaulting output to [] when omitted. Setting "cache": false opts a step out of freshness and caching while keeping its files globs active in --watch mode. Omitting files, output, and fileset dependencies on a command step leaves it untracked so it runs every time. String entries ("fmt": "zena fmt --check .") are shorthands for always-run commands.
  • Locks: locks defaults to ["zena-lock.json"] and may be set at the package root or overridden per script ("locks": [] disables lockfile hashing).
  • Services: Scripts with "service": true or "service": {"readyWhen": {"lineMatches": "..."}} start on demand when a dependent executes, wait for readiness when configured, are shared across dependents, and are stopped as a Unix process group (SIGTERM then SIGKILL) when dependents finish. Invoking a service directly as the top-level target runs its dependencies and streams the service in the foreground.
  • Wireit fallback: When a package directory has no zebra.json and no "scripts" block in zena-package.json, Zebra falls back to package.json Wireit scripts while still serving filesets from zena-package.json. Dependencies can cross freely between zena-package.json, zebra.json, and package.json packages.

Cache Layout (.zebra/)#

Each package's <pkg>/.zebra/ directory stores:

  • state.json — freshness records for that package's scripts (the last cache key and output manifest).
  • cache/<key>.json — cache key to output manifest.
  • blobs/ — content-addressed output bytes (sha256).

The workspace root's .zebra/ directory stores:

  • hashes.json — the (mtime, size) to sha256 file hash memo shared across packages (with a 20ms racy-timestamp guard).
  • plans/<hash>.plan — stat-verified cached ScriptConfig closures keyed by target entry.

A dependency contributes the digest of its outputs to dependents' cache keys rather than its input fingerprint, giving early cutoff when a dependency reruns and produces identical outputs.

GitHub Actions Remote Cache#

Setting ZEBRA_CACHE=github layers GitHubActionsCache behind the local .zebra/ DiskCache (TieredCache). On a local cache miss, Zebra queries the GitHub Actions v2 CacheService Twirp JSON API, verifies downloaded blob sha256 digests, populates <pkg>/.zebra/, and materializes outputs into the workspace. On a build step execution, Zebra writes <pkg>/.zebra/ first and uploads the step's entry to GitHub Actions Cache (ZEBRA_CACHE_READ_ONLY=1 disables uploads; ZEBRA_CACHE=none disables both local and remote caching).

In GitHub Actions workflows, run .github/actions/setup-zebra-cache before build steps so ACTIONS_RESULTS_URL and ACTIONS_RUNTIME_TOKEN are served to run: steps over a 127.0.0.1 token custodian (ZEBRA_GITHUB_CUSTODIAN_URL).

Libraries#

  • zebra (zebra/main) — CLI entry point (main).
  • zebra/runner — buildTarget, watchTarget, BuildOptions, WatchOptions, reportBuild, reportMigration, formatBuildReport, and BuildOutcome.
  • zebra/zebra-config — zena-package.json and zebra.json parsing (ZebraWorkspace), synthesized source filesets, and Wireit migration (migrateWorkspaceToZebra, convertPackageJsonToZebraJson, mergeWireitIntoZenaPackageJson).
  • zebra/wireit-config — package.json Wireit parsing (WireitWorkspace) and shared script configuration types (ScriptConfig, ScriptRef, DependencySpec, ServiceConfig, EnvBinding, ConfigError, refKey, resolveDependency).
  • zebra/plan-cache — .zebra/plans/ snapshot serialization and stat-verified warm plan loading (loadOrCollectConfigs, PlanLoadStats, ConfigProbe).
  • zebra/builder — Translates ScriptConfig closures into workflow steps and execution callbacks (planScripts, StepExecution, lockfileCandidates).
  • zebra/disk-cache, zebra/github-cache, and zebra/state-json — .zebra/ blob/manifest storage (DiskCache, TieredCache, stateDirOf), GitHub Actions remote caching (GitHubActionsCache, GitHubCacheConfig, CacheMode, resolveCacheMode, resolveGitHubCacheConfig), and JSON serialization for state, manifests, and hash memos.
  • zebra/fs-walk and zebra/fs-util — Input glob discovery (FileHasher, expandGlobs, hashFile, validateGlobs, splitBase) and workspace path utilities.
  • zebra/shell — POSIX shell argument quoting (shellQuote, appendShellArgs). Zebra's terminal output uses zena-lang:terminal (packages/terminal): colors, the Clack-style rail layout and the spinner.

License#

MIT

6.7 KiB Raw
import {
  exists,
  isFile,
  readFile,
  readFileBytes,
  rename,
  writeFile,
  writeFileBytes,
} from 'zena:fs';
import {HashSet} from 'zena:collections';
import {StepCache} from 'workflow/cache';
import {CacheKey, Fingerprint} from 'workflow/fingerprint';
import {OutputManifest} from 'workflow/manifest';
import {StepKey} from 'workflow/step';
import {ensureDir, ensureParentDir, withRoot} from './fs-util.zena';
import {manifestFromJson, manifestToJson} from './state-json.zena';

/**
 * The package directory of a step key ('pkg/dir:script' — package
 * directories never contain ':', script names may).
 */
export function packageDirOf(stepKey: String): String {
  for (var i = 0; i < stepKey.length; i += 1) {
    if (stepKey.getByteAt(i) == 58) {
      return stepKey.sliceBytes(0, i);
    }
  }
  throw new Error('malformed step key: ' + stepKey);
}

/** A package's build-state directory, '<pkg>/.zebra'. */
export function stateDirOf(stepKey: String): String {
  let pkg = packageDirOf(stepKey);
  return if (pkg == '') '.zebra' else pkg + '/.zebra';
}

/**
 * Entries with content fingerprints are files in the workspace;
 * anything else (e.g. the '$volatile' marker of an untracked script)
 * is virtual and has no bytes to store or restore.
 */
export function isFileEntry(fp: Fingerprint): boolean {
  return fp.value.startsWith('sha256:');
}

export function hasVolatileEntry(manifest: OutputManifest): boolean {
  for (let entry in manifest.outputs) {
    let v = entry.fingerprint.value;
    if (v.startsWith('volatile:') || v.startsWith('ref:volatile:')) {
      return true;
    }
  }
  return false;
}

export function blobName(fp: Fingerprint): String {
  // 'sha256:<hex>' -> 'sha256-<hex>', a safe file name.
  let colonSplit = fp.value.split(':');
  if (colonSplit.length != 2) {
    throw new Error('unexpected fingerprint format: ' + fp.value);
  }
  return `${colonSplit[0]}-${colonSplit[1]}`;
}

/**
 * The local cache, sharded per package like wireit's .wireit/: each
 * package's '<pkg>/.zebra/' holds an action cache mapping cache keys to
 * manifests (one JSON file per key) and a content-addressed blob store
 * for output bytes, so the cache location does not depend on where zebra
 * was invoked. A hit materializes the outputs into the workspace
 * before returning, so dependents can read them; missing blobs make
 * the entry a miss.
 */
export class DiskCache implements StepCache {
  #root: String;
  #onRestore: (() => void)?;
  #onFileRestore: ((path: String, fp: Fingerprint) => Future<void>)?;
  #uncachedSteps: HashSet<String>?;
  var #nextTempId = 0;

  new(
    root: String,
    onRestore: (() => void)? = null,
    onFileRestore: ((path: String, fp: Fingerprint) => Future<void>)? = null,
    uncachedSteps: HashSet<String>? = null,
  ) : #root = root,
      #onRestore = onRestore,
      #onFileRestore = onFileRestore,
      #uncachedSteps = uncachedSteps;

  #tempSuffix(): String {
    let id = this.#nextTempId;
    this.#nextTempId = id + 1;
    return `.tmp-${i32ToString(id)}`;
  }

  #manifestPath(step: StepKey, key: CacheKey): String {
    return withRoot(this.#root, `${stateDirOf(step.value)}/cache/${key.value}.json`);
  }

  #blobPath(step: StepKey, fp: Fingerprint): String {
    return withRoot(this.#root, `${stateDirOf(step.value)}/blobs/${blobName(fp)}`);
  }

  async get(step: StepKey, key: CacheKey): Future<OutputManifest?> {
    let uncached = this.#uncachedSteps;
    if (uncached != null && uncached.has(step.value)) {
      return null;
    }
    let manifestPath = this.#manifestPath(step, key);
    if (!(await isFile(manifestPath))) {
      return null;
    }
    let manifest = manifestFromJson(await readFile(manifestPath));
    for (let entry in manifest.outputs) {
      if (isFileEntry(entry.fingerprint) && !(await isFile(this.#blobPath(step, entry.fingerprint)))) {
        return null;
      }
    }
    let onFileRestore = this.#onFileRestore;
    for (let entry in manifest.outputs) {
      if (!isFileEntry(entry.fingerprint)) {
        continue;
      }
      let target = withRoot(this.#root, entry.name);
      await ensureParentDir(target);
      // Restore via sibling temp file + rename so an existing inode (for
      // example an mmap'd .cwasm or running binary) is never truncated in place.
      let tmp = target + this.#tempSuffix();
      await writeFileBytes(tmp, await readFileBytes(this.#blobPath(step, entry.fingerprint)));
      await rename(tmp, target);
      if (onFileRestore != null) {
        await onFileRestore(entry.name, entry.fingerprint);
      }
    }
    let onRestore = this.#onRestore;
    if (onRestore != null) {
      onRestore();
    }
    return manifest;
  }

  async put(step: StepKey, key: CacheKey, manifest: OutputManifest): Future<void> {
    let uncached = this.#uncachedSteps;
    if ((uncached != null && uncached.has(step.value)) || hasVolatileEntry(manifest)) {
      return;
    }
    let stateDir = withRoot(this.#root, stateDirOf(step.value));
    await ensureDir(stateDir + '/blobs');
    await ensureDir(stateDir + '/cache');
    for (let entry in manifest.outputs) {
      if (!isFileEntry(entry.fingerprint)) {
        continue;
      }
      let blob = this.#blobPath(step, entry.fingerprint);
      if (!(await exists(blob))) {
        let tmp = blob + this.#tempSuffix();
        await writeFileBytes(tmp, await readFileBytes(withRoot(this.#root, entry.name)));
        await rename(tmp, blob);
      }
    }
    let manifestPath = this.#manifestPath(step, key);
    let tmpManifest = manifestPath + this.#tempSuffix();
    await writeFile(tmpManifest, manifestToJson(manifest));
    await rename(tmpManifest, manifestPath);
  }
}

/**
 * Two-level step cache (`local` in front of `remote`).
 *
 * `get` checks `local` first and queries `remote` only when `local` misses.
 * `put` writes `local` first so `.zebra/` holds the manifest and blobs before
 * `remote` uploads them.
 */
export class TieredCache implements StepCache {
  #local: StepCache;
  #remote: StepCache;
  #uncachedSteps: HashSet<String>?;

  new(
    local: StepCache,
    remote: StepCache,
    uncachedSteps: HashSet<String>? = null,
  ) : #local = local,
      #remote = remote,
      #uncachedSteps = uncachedSteps;

  async get(step: StepKey, key: CacheKey): Future<OutputManifest?> {
    let uncached = this.#uncachedSteps;
    if (uncached != null && uncached.has(step.value)) {
      return null;
    }
    let localHit = await this.#local.get(step, key);
    return localHit ?? await this.#remote.get(step, key);
  }

  async put(step: StepKey, key: CacheKey, manifest: OutputManifest): Future<void> {
    let uncached = this.#uncachedSteps;
    if ((uncached != null && uncached.has(step.value)) || hasVolatileEntry(manifest)) {
      return;
    }
    await this.#local.put(step, key, manifest);
    await this.#remote.put(step, key, manifest);
  }
}
  • terminal terminal.zena-lang.pkg.zena-lang.dev ~0.0.1-preview.1
  • workflow workflow.zena-lang.pkg.zena-lang.dev ~0.0.1-preview.1
VersionPublishedDependenciesApproved by
0.0.1-preview.1 2026-11-19
  • terminal terminal.zena-lang.pkg.zena-lang.dev ~0.0.1-preview.1
  • workflow workflow.zena-lang.pkg.zena-lang.dev ~0.0.1-preview.1