zebra 0.0.1-preview.1

Zebra: an incremental build orchestrator for Zena and standalone workspaces

Readme Files Dependencies Versions

zebra#

An incremental script and service runner for Zena (zena-package.json), standalone Zebra (zebra.json), and Wireit (package.json) workspaces, built on zena-lang:workflow.

Zebra runs a dependency graph of scripts and long-running services with declared inputs, outputs, and dependencies, rerunning only what changed. For each script, Zebra content-hashes the declared inputs and asks the workflow engine what to do: nothing (when the previous run recorded the same fingerprints and declared outputs are intact), restore from the local content-addressed cache, or execute the command (sh -c in the package directory with node_modules/.bin on PATH).

Installation#

zebra is built into the zena CLI (zena build <target>). To use it as a library or standalone runner:

zena pkg add zena-lang:zebra

CLI Usage#

zena build [-j <jobs>] [-w|--watch] [-f|--force] [--no-cache] [-v|--verbose] [<target>] [-- <args>...]
zena run [-j <jobs>] [-w|--watch] [-f|--force] [--no-cache] [-v|--verbose] <target> [-- <args>...]
zena test [-j <jobs>] [-w|--watch] [--force] [--no-cache] [-v|--verbose] [-- <args>...]
zena build --migrate [--dry-run] [-f|--force] [<package-dir>...]
  • <target> can be a script name in the current package (zena build test, zena run check), a package directory (zena build ./packages/app, which runs build), or <package-dir>:<script> (zena build ./packages/app:test). With no <target>, zena build runs the current package's build script and zena test runs its test script.
  • -- <args>... appends shell-quoted arguments to the entry target's command only, running that entry step uncached without overwriting its saved freshness record.
  • -j / --jobs <n> bounds parallel command execution (defaulting to ZEBRA_PARALLEL, WIREIT_PARALLEL, or ZENA_AVAILABLE_PARALLELISM).
  • -w / --watch watches input globs, lockfiles, and package manifests for changes, re-running affected steps incrementally while keeping unchanged services alive in a ServicePool across rebuilds.
  • -f / --force bypasses freshness and cached outputs to rerun scripts unconditionally.
  • --no-cache disables restoring outputs from .zebra/blobs/ when a step is not fresh.
  • -v / --verbose prints every step (including fresh and skipped) and any captured stdout/stderr on the rail.
  • --migrate converts Wireit blocks in package.json into zena-package.json (when present) or zebra.json.

Programmatic Usage#

import { buildTarget, reportBuild } from 'zena-lang:zebra/runner';
import { ScriptRef } from 'zena-lang:zebra/wireit-config';

let outcome = await buildTarget('.', new ScriptRef('packages/app', 'build'), {
  parallelism: 4,
});
let exitCode = reportBuild(outcome);

Configuration (zena-package.json and zebra.json)#

Zena packages configure scripts and filesets directly in zena-package.json. Non-Zena projects can use a standalone zebra.json (with an optional "workspace": {"packages": [...]} at the workspace root). Both support // and /* ... */ comments:

{
  "name": "app",
  "dependencies": {
    "lib": {"path": "../lib"},
  },
  "filesets": {
    "assets": ["assets/**"],
  },
  "scripts": {
    "build": {
      "command": "zena build src/main.zena -o out/app.wasm",
      "output": ["out/app.wasm"],
      "dependencies": ["source", "assets"],
    },
    "test": {
      "type": "zena:test",
      "tests": "test/*_test.zena",
    },
    "fmt": "zena fmt --check .",
  },
}
  • Declarative zena:test steps and default test target: Scripts in zena-package.json and zebra.json can set "type": "zena:test" (with optional "tests": true, "tests": "test/*_test.zena", or "tests": ["test/*_test.zena", ...] — defaulting to ["test/**/*_test.zena", "tests/**/*_test.zena"] when omitted — and optional "debug": true for -g) to run Zena tests in-process through TestStepRunner. Test pattern globs merge into files, and zena-package.json "zena:test" steps automatically depend on source. When a zena-package.json omits a test script (and no fallback package.json defines test), Zebra synthesizes a "type": "zena:test" target over test/**/*_test.zena and tests/**/*_test.zena.
  • Automatic source fileset: Every package with zena-package.json exposes an automatic source fileset covering its library .zena files (from "source", defaulting to "src" and excluding *_test.zena, test/**, and tests/**), zena-package.json, any "wit" dependencies, and the source filesets of all "path" dependencies. Scripts opt into it via "dependencies": ["source"] (or "../pkg:source").
  • Named filesets: Entries accept either a glob array shorthand ("wit": ["wit/**/*.wit"]) or {"files": [...], "dependencies": [...]} and compose transitively without spawning commands.
  • Caching and tracking: Task scripts cache automatically whenever files, output, or a fileset dependency is declared, defaulting output to [] when omitted. Setting "cache": false opts a step out of freshness and caching while keeping its files globs active in --watch mode. Omitting files, output, and fileset dependencies on a command step leaves it untracked so it runs every time. String entries ("fmt": "zena fmt --check .") are shorthands for always-run commands.
  • Locks: locks defaults to ["zena-lock.json"] and may be set at the package root or overridden per script ("locks": [] disables lockfile hashing).
  • Services: Scripts with "service": true or "service": {"readyWhen": {"lineMatches": "..."}} start on demand when a dependent executes, wait for readiness when configured, are shared across dependents, and are stopped as a Unix process group (SIGTERM then SIGKILL) when dependents finish. Invoking a service directly as the top-level target runs its dependencies and streams the service in the foreground.
  • Wireit fallback: When a package directory has no zebra.json and no "scripts" block in zena-package.json, Zebra falls back to package.json Wireit scripts while still serving filesets from zena-package.json. Dependencies can cross freely between zena-package.json, zebra.json, and package.json packages.

Cache Layout (.zebra/)#

Each package's <pkg>/.zebra/ directory stores:

  • state.json — freshness records for that package's scripts (the last cache key and output manifest).
  • cache/<key>.json — cache key to output manifest.
  • blobs/ — content-addressed output bytes (sha256).

The workspace root's .zebra/ directory stores:

  • hashes.json — the (mtime, size) to sha256 file hash memo shared across packages (with a 20ms racy-timestamp guard).
  • plans/<hash>.plan — stat-verified cached ScriptConfig closures keyed by target entry.

A dependency contributes the digest of its outputs to dependents' cache keys rather than its input fingerprint, giving early cutoff when a dependency reruns and produces identical outputs.

GitHub Actions Remote Cache#

Setting ZEBRA_CACHE=github layers GitHubActionsCache behind the local .zebra/ DiskCache (TieredCache). On a local cache miss, Zebra queries the GitHub Actions v2 CacheService Twirp JSON API, verifies downloaded blob sha256 digests, populates <pkg>/.zebra/, and materializes outputs into the workspace. On a build step execution, Zebra writes <pkg>/.zebra/ first and uploads the step's entry to GitHub Actions Cache (ZEBRA_CACHE_READ_ONLY=1 disables uploads; ZEBRA_CACHE=none disables both local and remote caching).

In GitHub Actions workflows, run .github/actions/setup-zebra-cache before build steps so ACTIONS_RESULTS_URL and ACTIONS_RUNTIME_TOKEN are served to run: steps over a 127.0.0.1 token custodian (ZEBRA_GITHUB_CUSTODIAN_URL).

Libraries#

  • zebra (zebra/main) — CLI entry point (main).
  • zebra/runner — buildTarget, watchTarget, BuildOptions, WatchOptions, reportBuild, reportMigration, formatBuildReport, and BuildOutcome.
  • zebra/zebra-config — zena-package.json and zebra.json parsing (ZebraWorkspace), synthesized source filesets, and Wireit migration (migrateWorkspaceToZebra, convertPackageJsonToZebraJson, mergeWireitIntoZenaPackageJson).
  • zebra/wireit-config — package.json Wireit parsing (WireitWorkspace) and shared script configuration types (ScriptConfig, ScriptRef, DependencySpec, ServiceConfig, EnvBinding, ConfigError, refKey, resolveDependency).
  • zebra/plan-cache — .zebra/plans/ snapshot serialization and stat-verified warm plan loading (loadOrCollectConfigs, PlanLoadStats, ConfigProbe).
  • zebra/builder — Translates ScriptConfig closures into workflow steps and execution callbacks (planScripts, StepExecution, lockfileCandidates).
  • zebra/disk-cache, zebra/github-cache, and zebra/state-json — .zebra/ blob/manifest storage (DiskCache, TieredCache, stateDirOf), GitHub Actions remote caching (GitHubActionsCache, GitHubCacheConfig, CacheMode, resolveCacheMode, resolveGitHubCacheConfig), and JSON serialization for state, manifests, and hash memos.
  • zebra/fs-walk and zebra/fs-util — Input glob discovery (FileHasher, expandGlobs, hashFile, validateGlobs, splitBase) and workspace path utilities.
  • zebra/shell — POSIX shell argument quoting (shellQuote, appendShellArgs). Zebra's terminal output uses zena-lang:terminal (packages/terminal): colors, the Clack-style rail layout and the spinner.

License#

MIT

13.3 KiB Raw
import {
  FileType,
  FsError,
  FsErrorCode,
  exists,
  isFile,
  listDir,
  readFile,
  readFileBytes,
  rename,
  stat,
  writeFile,
} from 'zena:fs';
import { HashSet } from 'zena:collections';
import {compareStrings, sortBy} from 'workflow/canonical';
import {Fingerprint} from 'workflow/fingerprint';
import {StringBuilder} from 'zena:core';
import {GlobSet, hasMagic} from 'zena:glob';
import {isAbsolute} from 'zena:path';
import {ensureParentDir, joinWorkspace, withRoot} from './fs-util.zena';
import {HashMemoEntry, hashesFromJson, hashesToJson} from './state-json.zena';
import {sha256Hex} from 'zena:crypto';
import {ConfigError} from './wireit-config.zena';

/**
 * Entries modified within 20ms of `.zebra/hashes.json`'s own mtime are
 * treated as racily clean (same principle as Git's index timestamp
 * check, covering coarse 1–4ms kernel filesystem ticks) and re-hashed
 * on the next build before becoming permanent memo hits.
 */
let RACY_WINDOW_NANOS = 20000000 as i64;

/**
 * A pattern split at its first segment with glob syntax in it: the
 * literal directory prefix to walk from, and the pattern for paths
 * relative to it. A fully literal pattern splits into its directory and
 * file name. `''` is the workspace root.
 */
export class GlobBase(base: String, rest: String)

export function splitBase(pattern: String): GlobBase {
  let segments = pattern.split('/');
  var literal = 0;
  while (literal < segments.length && !hasMagic(segments[literal])) {
    literal += 1;
  }
  if (literal == segments.length) {
    // Fully literal: the last segment is the file name.
    literal = segments.length - 1;
  }
  let base = new StringBuilder();
  for (var i = 0; i < literal; i += 1) {
    if (i > 0) {
      base.append('/');
    }
    base.append(segments[i]);
  }
  let rest = new StringBuilder();
  for (var i = literal; i < segments.length; i += 1) {
    if (i > literal) {
      rest.append('/');
    }
    rest.append(segments[i]);
  }
  return new GlobBase(base.toString(), rest.toString());
}

/** Directories never worth walking for build inputs or outputs. */
function skipDir(name: String) {
  return name == 'node_modules' || name == '.git' || name == '.wireit' ||
    name == '.zebra' || name == 'target';
}

function prepareGlobs(
  packageDir: String,
  globs: Array<String>,
): (GlobSet, GrowableArray<String>) {
  let rewritten = new GrowableArray<String>();
  let bases = new GrowableArray<String>();
  let seenBases = new HashSet<String>();
  for (let raw in globs) {
    var pattern = raw;
    var negated = false;
    if (pattern.startsWith('!')) {
      negated = true;
      pattern = pattern.sliceBytes(1, pattern.length);
    }
    let trailingSlash = pattern.endsWith('/');
    let workspaceRelative = joinWorkspace(packageDir, pattern);
    if (workspaceRelative.startsWith('..') || isAbsolute(workspaceRelative)) {
      throw new ConfigError(
        `glob "${raw}" in ${packageDir} escapes the workspace`,
      );
    }
    let prefix = if (negated) '!' else '';
    if (trailingSlash) {
      // A trailing-slash pattern ("test/", "!lib/", "./") names a directory
      // tree only, never a regular file with the same name.
      let treeGlob = if (workspaceRelative == '') '**' else `${workspaceRelative}/**`;
      rewritten.push(prefix + treeGlob);
      if (!negated) {
        let base = if (hasMagic(workspaceRelative))
          splitBase(workspaceRelative).base
        else
          workspaceRelative;
        if (seenBases.add(base)) {
          bases.push(base);
        }
      }
    } else if (!hasMagic(workspaceRelative)) {
      // A bare literal path ("lib", "local.txt", "!src/tmp", ".") may name
      // either a regular file or a directory tree on disk. Since collectFiles
      // only tests non-directory paths against GlobSet, pushing both the exact
      // path and `<path>/**` matches whichever exists without extra stats
      // while preserving negation and re-inclusion order.
      if (workspaceRelative == '') {
        rewritten.push(prefix + '**');
      } else {
        rewritten.push(prefix + workspaceRelative);
        rewritten.push(`${prefix}${workspaceRelative}/**`);
      }
      if (!negated && seenBases.add(workspaceRelative)) {
        bases.push(workspaceRelative);
      }
    } else {
      rewritten.push(prefix + workspaceRelative);
      if (!negated) {
        let base = splitBase(workspaceRelative).base;
        if (seenBases.add(base)) {
          bases.push(base);
        }
      }
    }
  }
  return (new GlobSet(rewritten), bases);
}

async function collectFiles(
  root: String,
  dir: String,
  set: GlobSet,
  out: GrowableArray<String>,
  seen: HashSet<String>,
): Future<void> {
  let full = withRoot(root, dir);
  if (!(await exists(full))) {
    return;
  }
  if (await isFile(full)) {
    if (set.matches(dir) && seen.add(dir)) {
      out.push(dir);
    }
    return;
  }
  for (let entry in await listDir(full)) {
    if (entry.name == '.' || entry.name == '..') {
      continue;
    }
    if (entry.fileType == FileType.Directory) {
      if (!skipDir(entry.name)) {
        await collectFiles(root, joinWorkspace(dir, entry.name), set, out, seen);
      }
    } else {
      let path = joinWorkspace(dir, entry.name);
      if (set.matches(path) && seen.add(path)) {
        out.push(path);
      }
    }
  }
}

/**
 * Expands a package's glob list to sorted workspace-relative file
 * paths. Patterns are interpreted relative to `packageDir` and may
 * reach sibling packages with '../'; escaping the workspace, by climbing
 * out of it or by naming an absolute path, is an error. Negations apply
 * across the whole list in order.
 */
export async function expandGlobs(
  root: String,
  packageDir: String,
  globs: Array<String>,
): Future<Array<String>> {
  let (set, bases) = prepareGlobs(packageDir, globs);
  let out = new GrowableArray<String>();
  let seen = new HashSet<String>();
  for (let base in bases) {
    await collectFiles(root, base, set, out, seen);
  }
  sortBy(out, (a, b) => compareStrings(a, b));
  return out;
}

/** The content fingerprint zebra uses everywhere: 'sha256:<hex>'. */
export async function hashFile(root: String, path: String): Future<Fingerprint> {
  return new Fingerprint('sha256:' + sha256Hex(await readFileBytes(withRoot(root, path))));
}

/** Validates glob patterns up front so escaping globs fail at plan time. */
export function validateGlobs(packageDir: String, globs: Array<String>): void {
  for (let raw in globs) {
    var pattern = raw;
    if (pattern.startsWith('!')) {
      pattern = pattern.sliceBytes(1, pattern.length);
    }
    let workspaceRelative = joinWorkspace(packageDir, pattern);
    if (workspaceRelative.startsWith('..') || isAbsolute(workspaceRelative)) {
      throw new ConfigError(
        `glob "${raw}" in ${packageDir} escapes the workspace`,
      );
    }
  }
}

/**
 * Workspace-level `(modified, size)` file hash memo, persisted across
 * runs under `<root>/.zebra/hashes.json`.
 *
 * Within a run, before any step executes or restores outputs into the
 * workspace, a file verified once needs no second `stat`; across runs
 * (and after workspace mutations begin), `stat` checks `(modified, size)`
 * before re-reading bytes.
 */
export class FileHasher {
  #root: String;
  #memo = new HashMap<String, HashMemoEntry>();
  #missingThisRun = new HashSet<String>();
  var #mutationsPossible = false;
  var #dirty = false;
  /** Number of times file bytes were actually read and hashed in this instance. */
  var hashReads: i32 = 0;
  /** Number of times a `stat` check satisfied `hash` / `hashIfExists` without reading bytes. */
  var statHits: i32 = 0;

  new(root: String) : #root = root;

  #hashesPath(): String {
    return withRoot(this.#root, '.zebra/hashes.json');
  }

  /** Loads `<root>/.zebra/hashes.json` if present, skipping racily-recent entries. */
  async load(): Future<void> {
    let path = this.#hashesPath();
    if (!(await isFile(path))) {
      return;
    }
    try {
      let memoStat = await stat(path);
      let cutoff = memoStat.modified - RACY_WINDOW_NANOS;
      let loaded = hashesFromJson(await readFile(path));
      loaded.forEach((filePath, entry) => {
        if (memoStat.modified > (0 as i64) && entry.modified < cutoff) {
          this.#memo[filePath] = entry;
        }
      });
    } catch (e) {
      // Corrupt or unreadable memo files fall back to hashing cleanly.
    }
  }

  /** Persists updated `(modified, size) -> sha256` entries to `<root>/.zebra/hashes.json`. */
  async save(): Future<void> {
    if (!this.#dirty) {
      return;
    }
    let path = this.#hashesPath();
    let merged = new HashMap<String, HashMemoEntry>();
    if (await isFile(path)) {
      try {
        hashesFromJson(await readFile(path)).forEach((k, v) => {
          merged[k] = v;
        });
      } catch (e) {}
    }
    this.#memo.forEach((k, v) => {
      if (v.modified > (0 as i64)) {
        merged[k] = v;
      }
    });
    await ensureParentDir(path);
    let tmp = path + '.tmp';
    await writeFile(tmp, hashesToJson(merged));
    await rename(tmp, path);
    this.#dirty = false;
  }

  markMutated(): void {
    this.#mutationsPossible = true;
  }

  /** Records a file just restored from the blob cache so the next run hits the memo. */
  async recordRestored(path: String, fingerprint: Fingerprint): Future<void> {
    this.#mutationsPossible = true;
    this.#missingThisRun.delete(path);
    let full = withRoot(this.#root, path);
    try {
      let info = await stat(full);
      if (info.fileType == FileType.RegularFile && info.modified > (0 as i64)) {
        this.#memo[path] = new HashMemoEntry(info.modified, info.size, fingerprint, true);
        this.#dirty = true;
      }
    } catch (e) {}
  }

  async hash(path: String): Future<Fingerprint> {
    this.#missingThisRun.delete(path);
    let full = withRoot(this.#root, path);
    if (let (true, cached) = this.#memo.get(path)) {
      if (cached.verifiedThisRun && !this.#mutationsPossible) {
        return cached.fingerprint;
      }
      let info = await stat(full);
      if (
        info.fileType == FileType.RegularFile &&
        info.modified > (0 as i64) &&
        info.modified == cached.modified &&
        info.size == cached.size
      ) {
        cached.verifiedThisRun = true;
        this.statHits += 1;
        return cached.fingerprint;
      }
      this.hashReads += 1;
      let fp = await hashFile(this.#root, path);
      this.#memo[path] = new HashMemoEntry(info.modified, info.size, fp, true);
      this.#dirty = true;
      return fp;
    }
    let info = await stat(full);
    this.hashReads += 1;
    let fp = await hashFile(this.#root, path);
    this.#memo[path] = new HashMemoEntry(info.modified, info.size, fp, true);
    this.#dirty = true;
    return fp;
  }

  /**
   * Returns `path`'s content fingerprint when it exists on disk as a
   * regular file, or `null` when missing or non-regular.
   */
  async hashIfExists(path: String): Future<Fingerprint?> {
    if (!this.#mutationsPossible) {
      if (let (true, cached) = this.#memo.get(path)) {
        if (cached.verifiedThisRun) {
          return cached.fingerprint;
        }
      }
      if (this.#missingThisRun.has(path)) {
        return null;
      }
    }
    let full = withRoot(this.#root, path);
    var hasStat = false;
    var modified = 0 as i64;
    var size = 0 as i64;
    try {
      let info = await stat(full);
      if (info.fileType == FileType.RegularFile) {
        hasStat = true;
        modified = info.modified;
        size = info.size;
      }
    } catch (e) {
      if (
        !(e is FsError) ||
        (e.code != FsErrorCode.NotFound && e.code != FsErrorCode.NotDirectory)
      ) {
        throw e;
      }
    }
    if (!hasStat) {
      if (!this.#mutationsPossible) {
        this.#missingThisRun.add(path);
      }
      if (this.#memo.delete(path)) {
        this.#dirty = true;
      }
      return null;
    }
    if (let (true, cached) = this.#memo.get(path)) {
      if (
        modified > (0 as i64) &&
        modified == cached.modified &&
        size == cached.size
      ) {
        cached.verifiedThisRun = true;
        this.statHits += 1;
        return cached.fingerprint;
      }
    }
    this.hashReads += 1;
    let fp = await hashFile(this.#root, path);
    this.#memo[path] = new HashMemoEntry(modified, size, fp, true);
    this.#dirty = true;
    return fp;
  }

  async hashGlobs(
    packageDir: String,
    globs: Array<String>,
  ): Future<Array<(String, Fingerprint)>> {
    return await this.hashInputs(packageDir, globs, new GrowableArray<String>());
  }

  /**
   * Expands `globs` relative to `packageDir` and includes any existing
   * workspace-relative `lockCandidates`, deduplicating paths so explicit
   * lockfile globs never collide with `packageLocks`.
   */
  async hashInputs(
    packageDir: String,
    globs: Array<String>,
    lockCandidates: Array<String>,
  ): Future<Array<(String, Fingerprint)>> {
    let pairs = new GrowableArray<(String, Fingerprint)>();
    let seen = new HashSet<String>();
    if (globs.length > 0) {
      let files = await expandGlobs(this.#root, packageDir, globs);
      for (let path in files) {
        if (seen.add(path)) {
          pairs.push((path, await this.hash(path)));
        }
      }
    }
    for (let lockPath in lockCandidates) {
      if (!seen.has(lockPath)) {
        let fp = await this.hashIfExists(lockPath);
        if (fp != null) {
          seen.add(lockPath);
          pairs.push((lockPath, fp));
        }
      }
    }
    return pairs;
  }
}
  • terminal terminal.zena-lang.pkg.zena-lang.dev ~0.0.1-preview.1
  • workflow workflow.zena-lang.pkg.zena-lang.dev ~0.0.1-preview.1
VersionPublishedDependenciesApproved by
0.0.1-preview.1 2026-11-19
  • terminal terminal.zena-lang.pkg.zena-lang.dev ~0.0.1-preview.1
  • workflow workflow.zena-lang.pkg.zena-lang.dev ~0.0.1-preview.1