zebra 0.0.1-preview.1
Zebra: an incremental build orchestrator for Zena and standalone workspaces
zebra#
An incremental script and service runner for Zena (zena-package.json), standalone Zebra (zebra.json), and Wireit (package.json) workspaces, built on zena-lang:workflow.
Zebra runs a dependency graph of scripts and long-running services with declared inputs, outputs, and dependencies, rerunning only what changed. For each script, Zebra content-hashes the declared inputs and asks the workflow engine what to do: nothing (when the previous run recorded the same fingerprints and declared outputs are intact), restore from the local content-addressed cache, or execute the command (sh -c in the package directory with node_modules/.bin on PATH).
Installation#
zebra is built into the zena CLI (zena build <target>). To use it as a library or standalone runner:
zena pkg add zena-lang:zebra
CLI Usage#
zena build [-j <jobs>] [-w|--watch] [-f|--force] [--no-cache] [-v|--verbose] [<target>] [-- <args>...]
zena run [-j <jobs>] [-w|--watch] [-f|--force] [--no-cache] [-v|--verbose] <target> [-- <args>...]
zena test [-j <jobs>] [-w|--watch] [--force] [--no-cache] [-v|--verbose] [-- <args>...]
zena build --migrate [--dry-run] [-f|--force] [<package-dir>...]
<target>can be a script name in the current package (zena build test,zena run check), a package directory (zena build ./packages/app, which runsbuild), or<package-dir>:<script>(zena build ./packages/app:test). With no<target>,zena buildruns the current package'sbuildscript andzena testruns itstestscript.-- <args>...appends shell-quoted arguments to the entry target's command only, running that entry step uncached without overwriting its saved freshness record.-j/--jobs <n>bounds parallel command execution (defaulting toZEBRA_PARALLEL,WIREIT_PARALLEL, orZENA_AVAILABLE_PARALLELISM).-w/--watchwatches input globs, lockfiles, and package manifests for changes, re-running affected steps incrementally while keeping unchanged services alive in aServicePoolacross rebuilds.-f/--forcebypasses freshness and cached outputs to rerun scripts unconditionally.--no-cachedisables restoring outputs from.zebra/blobs/when a step is not fresh.-v/--verboseprints every step (includingfreshandskipped) and any captured stdout/stderr on the rail.--migrateconverts Wireit blocks inpackage.jsonintozena-package.json(when present) orzebra.json.
Programmatic Usage#
import { buildTarget, reportBuild } from 'zena-lang:zebra/runner';
import { ScriptRef } from 'zena-lang:zebra/wireit-config';
let outcome = await buildTarget('.', new ScriptRef('packages/app', 'build'), {
parallelism: 4,
});
let exitCode = reportBuild(outcome);
Configuration (zena-package.json and zebra.json)#
Zena packages configure scripts and filesets directly in zena-package.json. Non-Zena projects can use a standalone zebra.json (with an optional "workspace": {"packages": [...]} at the workspace root). Both support // and /* ... */ comments:
{
"name": "app",
"dependencies": {
"lib": {"path": "../lib"},
},
"filesets": {
"assets": ["assets/**"],
},
"scripts": {
"build": {
"command": "zena build src/main.zena -o out/app.wasm",
"output": ["out/app.wasm"],
"dependencies": ["source", "assets"],
},
"test": {
"type": "zena:test",
"tests": "test/*_test.zena",
},
"fmt": "zena fmt --check .",
},
}
- Declarative
zena:teststeps and defaulttesttarget: Scripts inzena-package.jsonandzebra.jsoncan set"type": "zena:test"(with optional"tests": true,"tests": "test/*_test.zena", or"tests": ["test/*_test.zena", ...]— defaulting to["test/**/*_test.zena", "tests/**/*_test.zena"]when omitted — and optional"debug": truefor-g) to run Zena tests in-process throughTestStepRunner. Test pattern globs merge intofiles, andzena-package.json"zena:test"steps automatically depend onsource. When azena-package.jsonomits atestscript (and no fallbackpackage.jsondefinestest), Zebra synthesizes a"type": "zena:test"target overtest/**/*_test.zenaandtests/**/*_test.zena. - Automatic
sourcefileset: Every package withzena-package.jsonexposes an automaticsourcefileset covering its library.zenafiles (from"source", defaulting to"src"and excluding*_test.zena,test/**, andtests/**),zena-package.json, any"wit"dependencies, and thesourcefilesets of all"path"dependencies. Scripts opt into it via"dependencies": ["source"](or"../pkg:source"). - Named
filesets: Entries accept either a glob array shorthand ("wit": ["wit/**/*.wit"]) or{"files": [...], "dependencies": [...]}and compose transitively without spawning commands. - Caching and tracking: Task scripts cache automatically whenever
files,output, or a fileset dependency is declared, defaultingoutputto[]when omitted. Setting"cache": falseopts a step out of freshness and caching while keeping itsfilesglobs active in--watchmode. Omittingfiles,output, and fileset dependencies on a command step leaves it untracked so it runs every time. String entries ("fmt": "zena fmt --check .") are shorthands for always-run commands. - Locks:
locksdefaults to["zena-lock.json"]and may be set at the package root or overridden per script ("locks": []disables lockfile hashing). - Services: Scripts with
"service": trueor"service": {"readyWhen": {"lineMatches": "..."}}start on demand when a dependent executes, wait for readiness when configured, are shared across dependents, and are stopped as a Unix process group (SIGTERMthenSIGKILL) when dependents finish. Invoking a service directly as the top-level target runs its dependencies and streams the service in the foreground. - Wireit fallback: When a package directory has no
zebra.jsonand no"scripts"block inzena-package.json, Zebra falls back topackage.jsonWireit scripts while still serving filesets fromzena-package.json. Dependencies can cross freely betweenzena-package.json,zebra.json, andpackage.jsonpackages.
Cache Layout (.zebra/)#
Each package's <pkg>/.zebra/ directory stores:
state.json— freshness records for that package's scripts (the last cache key and output manifest).cache/<key>.json— cache key to output manifest.blobs/— content-addressed output bytes (sha256).
The workspace root's .zebra/ directory stores:
hashes.json— the(mtime, size)tosha256file hash memo shared across packages (with a 20ms racy-timestamp guard).plans/<hash>.plan—stat-verified cachedScriptConfigclosures keyed by target entry.
A dependency contributes the digest of its outputs to dependents' cache keys rather than its input fingerprint, giving early cutoff when a dependency reruns and produces identical outputs.
GitHub Actions Remote Cache#
Setting ZEBRA_CACHE=github layers GitHubActionsCache behind the local .zebra/ DiskCache (TieredCache). On a local cache miss, Zebra queries the GitHub Actions v2 CacheService Twirp JSON API, verifies downloaded blob sha256 digests, populates <pkg>/.zebra/, and materializes outputs into the workspace. On a build step execution, Zebra writes <pkg>/.zebra/ first and uploads the step's entry to GitHub Actions Cache (ZEBRA_CACHE_READ_ONLY=1 disables uploads; ZEBRA_CACHE=none disables both local and remote caching).
In GitHub Actions workflows, run .github/actions/setup-zebra-cache before build steps so ACTIONS_RESULTS_URL and ACTIONS_RUNTIME_TOKEN are served to run: steps over a 127.0.0.1 token custodian (ZEBRA_GITHUB_CUSTODIAN_URL).
Libraries#
zebra(zebra/main) — CLI entry point (main).zebra/runner—buildTarget,watchTarget,BuildOptions,WatchOptions,reportBuild,reportMigration,formatBuildReport, andBuildOutcome.zebra/zebra-config—zena-package.jsonandzebra.jsonparsing (ZebraWorkspace), synthesizedsourcefilesets, and Wireit migration (migrateWorkspaceToZebra,convertPackageJsonToZebraJson,mergeWireitIntoZenaPackageJson).zebra/wireit-config—package.jsonWireit parsing (WireitWorkspace) and shared script configuration types (ScriptConfig,ScriptRef,DependencySpec,ServiceConfig,EnvBinding,ConfigError,refKey,resolveDependency).zebra/plan-cache—.zebra/plans/snapshot serialization andstat-verified warm plan loading (loadOrCollectConfigs,PlanLoadStats,ConfigProbe).zebra/builder— TranslatesScriptConfigclosures intoworkflowsteps and execution callbacks (planScripts,StepExecution,lockfileCandidates).zebra/disk-cache,zebra/github-cache, andzebra/state-json—.zebra/blob/manifest storage (DiskCache,TieredCache,stateDirOf), GitHub Actions remote caching (GitHubActionsCache,GitHubCacheConfig,CacheMode,resolveCacheMode,resolveGitHubCacheConfig), and JSON serialization for state, manifests, and hash memos.zebra/fs-walkandzebra/fs-util— Input glob discovery (FileHasher,expandGlobs,hashFile,validateGlobs,splitBase) and workspace path utilities.zebra/shell— POSIX shell argument quoting (shellQuote,appendShellArgs). Zebra's terminal output useszena-lang:terminal(packages/terminal): colors, the Clack-style rail layout and the spinner.
License#
MIT
import {
FileType,
FsError,
FsErrorCode,
exists,
isFile,
listDir,
readFile,
readFileBytes,
rename,
stat,
writeFile,
} from 'zena:fs';
import { HashSet } from 'zena:collections';
import {compareStrings, sortBy} from 'workflow/canonical';
import {Fingerprint} from 'workflow/fingerprint';
import {StringBuilder} from 'zena:core';
import {GlobSet, hasMagic} from 'zena:glob';
import {isAbsolute} from 'zena:path';
import {ensureParentDir, joinWorkspace, withRoot} from './fs-util.zena';
import {HashMemoEntry, hashesFromJson, hashesToJson} from './state-json.zena';
import {sha256Hex} from 'zena:crypto';
import {ConfigError} from './wireit-config.zena';
/**
* Entries modified within 20ms of `.zebra/hashes.json`'s own mtime are
* treated as racily clean (same principle as Git's index timestamp
* check, covering coarse 1–4ms kernel filesystem ticks) and re-hashed
* on the next build before becoming permanent memo hits.
*/
let RACY_WINDOW_NANOS = 20000000 as i64;
/**
* A pattern split at its first segment with glob syntax in it: the
* literal directory prefix to walk from, and the pattern for paths
* relative to it. A fully literal pattern splits into its directory and
* file name. `''` is the workspace root.
*/
export class GlobBase(base: String, rest: String)
export function splitBase(pattern: String): GlobBase {
let segments = pattern.split('/');
var literal = 0;
while (literal < segments.length && !hasMagic(segments[literal])) {
literal += 1;
}
if (literal == segments.length) {
// Fully literal: the last segment is the file name.
literal = segments.length - 1;
}
let base = new StringBuilder();
for (var i = 0; i < literal; i += 1) {
if (i > 0) {
base.append('/');
}
base.append(segments[i]);
}
let rest = new StringBuilder();
for (var i = literal; i < segments.length; i += 1) {
if (i > literal) {
rest.append('/');
}
rest.append(segments[i]);
}
return new GlobBase(base.toString(), rest.toString());
}
/** Directories never worth walking for build inputs or outputs. */
function skipDir(name: String) {
return name == 'node_modules' || name == '.git' || name == '.wireit' ||
name == '.zebra' || name == 'target';
}
function prepareGlobs(
packageDir: String,
globs: Array<String>,
): (GlobSet, GrowableArray<String>) {
let rewritten = new GrowableArray<String>();
let bases = new GrowableArray<String>();
let seenBases = new HashSet<String>();
for (let raw in globs) {
var pattern = raw;
var negated = false;
if (pattern.startsWith('!')) {
negated = true;
pattern = pattern.sliceBytes(1, pattern.length);
}
let trailingSlash = pattern.endsWith('/');
let workspaceRelative = joinWorkspace(packageDir, pattern);
if (workspaceRelative.startsWith('..') || isAbsolute(workspaceRelative)) {
throw new ConfigError(
`glob "${raw}" in ${packageDir} escapes the workspace`,
);
}
let prefix = if (negated) '!' else '';
if (trailingSlash) {
// A trailing-slash pattern ("test/", "!lib/", "./") names a directory
// tree only, never a regular file with the same name.
let treeGlob = if (workspaceRelative == '') '**' else `${workspaceRelative}/**`;
rewritten.push(prefix + treeGlob);
if (!negated) {
let base = if (hasMagic(workspaceRelative))
splitBase(workspaceRelative).base
else
workspaceRelative;
if (seenBases.add(base)) {
bases.push(base);
}
}
} else if (!hasMagic(workspaceRelative)) {
// A bare literal path ("lib", "local.txt", "!src/tmp", ".") may name
// either a regular file or a directory tree on disk. Since collectFiles
// only tests non-directory paths against GlobSet, pushing both the exact
// path and `<path>/**` matches whichever exists without extra stats
// while preserving negation and re-inclusion order.
if (workspaceRelative == '') {
rewritten.push(prefix + '**');
} else {
rewritten.push(prefix + workspaceRelative);
rewritten.push(`${prefix}${workspaceRelative}/**`);
}
if (!negated && seenBases.add(workspaceRelative)) {
bases.push(workspaceRelative);
}
} else {
rewritten.push(prefix + workspaceRelative);
if (!negated) {
let base = splitBase(workspaceRelative).base;
if (seenBases.add(base)) {
bases.push(base);
}
}
}
}
return (new GlobSet(rewritten), bases);
}
async function collectFiles(
root: String,
dir: String,
set: GlobSet,
out: GrowableArray<String>,
seen: HashSet<String>,
): Future<void> {
let full = withRoot(root, dir);
if (!(await exists(full))) {
return;
}
if (await isFile(full)) {
if (set.matches(dir) && seen.add(dir)) {
out.push(dir);
}
return;
}
for (let entry in await listDir(full)) {
if (entry.name == '.' || entry.name == '..') {
continue;
}
if (entry.fileType == FileType.Directory) {
if (!skipDir(entry.name)) {
await collectFiles(root, joinWorkspace(dir, entry.name), set, out, seen);
}
} else {
let path = joinWorkspace(dir, entry.name);
if (set.matches(path) && seen.add(path)) {
out.push(path);
}
}
}
}
/**
* Expands a package's glob list to sorted workspace-relative file
* paths. Patterns are interpreted relative to `packageDir` and may
* reach sibling packages with '../'; escaping the workspace, by climbing
* out of it or by naming an absolute path, is an error. Negations apply
* across the whole list in order.
*/
export async function expandGlobs(
root: String,
packageDir: String,
globs: Array<String>,
): Future<Array<String>> {
let (set, bases) = prepareGlobs(packageDir, globs);
let out = new GrowableArray<String>();
let seen = new HashSet<String>();
for (let base in bases) {
await collectFiles(root, base, set, out, seen);
}
sortBy(out, (a, b) => compareStrings(a, b));
return out;
}
/** The content fingerprint zebra uses everywhere: 'sha256:<hex>'. */
export async function hashFile(root: String, path: String): Future<Fingerprint> {
return new Fingerprint('sha256:' + sha256Hex(await readFileBytes(withRoot(root, path))));
}
/** Validates glob patterns up front so escaping globs fail at plan time. */
export function validateGlobs(packageDir: String, globs: Array<String>): void {
for (let raw in globs) {
var pattern = raw;
if (pattern.startsWith('!')) {
pattern = pattern.sliceBytes(1, pattern.length);
}
let workspaceRelative = joinWorkspace(packageDir, pattern);
if (workspaceRelative.startsWith('..') || isAbsolute(workspaceRelative)) {
throw new ConfigError(
`glob "${raw}" in ${packageDir} escapes the workspace`,
);
}
}
}
/**
* Workspace-level `(modified, size)` file hash memo, persisted across
* runs under `<root>/.zebra/hashes.json`.
*
* Within a run, before any step executes or restores outputs into the
* workspace, a file verified once needs no second `stat`; across runs
* (and after workspace mutations begin), `stat` checks `(modified, size)`
* before re-reading bytes.
*/
export class FileHasher {
#root: String;
#memo = new HashMap<String, HashMemoEntry>();
#missingThisRun = new HashSet<String>();
var #mutationsPossible = false;
var #dirty = false;
/** Number of times file bytes were actually read and hashed in this instance. */
var hashReads: i32 = 0;
/** Number of times a `stat` check satisfied `hash` / `hashIfExists` without reading bytes. */
var statHits: i32 = 0;
new(root: String) : #root = root;
#hashesPath(): String {
return withRoot(this.#root, '.zebra/hashes.json');
}
/** Loads `<root>/.zebra/hashes.json` if present, skipping racily-recent entries. */
async load(): Future<void> {
let path = this.#hashesPath();
if (!(await isFile(path))) {
return;
}
try {
let memoStat = await stat(path);
let cutoff = memoStat.modified - RACY_WINDOW_NANOS;
let loaded = hashesFromJson(await readFile(path));
loaded.forEach((filePath, entry) => {
if (memoStat.modified > (0 as i64) && entry.modified < cutoff) {
this.#memo[filePath] = entry;
}
});
} catch (e) {
// Corrupt or unreadable memo files fall back to hashing cleanly.
}
}
/** Persists updated `(modified, size) -> sha256` entries to `<root>/.zebra/hashes.json`. */
async save(): Future<void> {
if (!this.#dirty) {
return;
}
let path = this.#hashesPath();
let merged = new HashMap<String, HashMemoEntry>();
if (await isFile(path)) {
try {
hashesFromJson(await readFile(path)).forEach((k, v) => {
merged[k] = v;
});
} catch (e) {}
}
this.#memo.forEach((k, v) => {
if (v.modified > (0 as i64)) {
merged[k] = v;
}
});
await ensureParentDir(path);
let tmp = path + '.tmp';
await writeFile(tmp, hashesToJson(merged));
await rename(tmp, path);
this.#dirty = false;
}
markMutated(): void {
this.#mutationsPossible = true;
}
/** Records a file just restored from the blob cache so the next run hits the memo. */
async recordRestored(path: String, fingerprint: Fingerprint): Future<void> {
this.#mutationsPossible = true;
this.#missingThisRun.delete(path);
let full = withRoot(this.#root, path);
try {
let info = await stat(full);
if (info.fileType == FileType.RegularFile && info.modified > (0 as i64)) {
this.#memo[path] = new HashMemoEntry(info.modified, info.size, fingerprint, true);
this.#dirty = true;
}
} catch (e) {}
}
async hash(path: String): Future<Fingerprint> {
this.#missingThisRun.delete(path);
let full = withRoot(this.#root, path);
if (let (true, cached) = this.#memo.get(path)) {
if (cached.verifiedThisRun && !this.#mutationsPossible) {
return cached.fingerprint;
}
let info = await stat(full);
if (
info.fileType == FileType.RegularFile &&
info.modified > (0 as i64) &&
info.modified == cached.modified &&
info.size == cached.size
) {
cached.verifiedThisRun = true;
this.statHits += 1;
return cached.fingerprint;
}
this.hashReads += 1;
let fp = await hashFile(this.#root, path);
this.#memo[path] = new HashMemoEntry(info.modified, info.size, fp, true);
this.#dirty = true;
return fp;
}
let info = await stat(full);
this.hashReads += 1;
let fp = await hashFile(this.#root, path);
this.#memo[path] = new HashMemoEntry(info.modified, info.size, fp, true);
this.#dirty = true;
return fp;
}
/**
* Returns `path`'s content fingerprint when it exists on disk as a
* regular file, or `null` when missing or non-regular.
*/
async hashIfExists(path: String): Future<Fingerprint?> {
if (!this.#mutationsPossible) {
if (let (true, cached) = this.#memo.get(path)) {
if (cached.verifiedThisRun) {
return cached.fingerprint;
}
}
if (this.#missingThisRun.has(path)) {
return null;
}
}
let full = withRoot(this.#root, path);
var hasStat = false;
var modified = 0 as i64;
var size = 0 as i64;
try {
let info = await stat(full);
if (info.fileType == FileType.RegularFile) {
hasStat = true;
modified = info.modified;
size = info.size;
}
} catch (e) {
if (
!(e is FsError) ||
(e.code != FsErrorCode.NotFound && e.code != FsErrorCode.NotDirectory)
) {
throw e;
}
}
if (!hasStat) {
if (!this.#mutationsPossible) {
this.#missingThisRun.add(path);
}
if (this.#memo.delete(path)) {
this.#dirty = true;
}
return null;
}
if (let (true, cached) = this.#memo.get(path)) {
if (
modified > (0 as i64) &&
modified == cached.modified &&
size == cached.size
) {
cached.verifiedThisRun = true;
this.statHits += 1;
return cached.fingerprint;
}
}
this.hashReads += 1;
let fp = await hashFile(this.#root, path);
this.#memo[path] = new HashMemoEntry(modified, size, fp, true);
this.#dirty = true;
return fp;
}
async hashGlobs(
packageDir: String,
globs: Array<String>,
): Future<Array<(String, Fingerprint)>> {
return await this.hashInputs(packageDir, globs, new GrowableArray<String>());
}
/**
* Expands `globs` relative to `packageDir` and includes any existing
* workspace-relative `lockCandidates`, deduplicating paths so explicit
* lockfile globs never collide with `packageLocks`.
*/
async hashInputs(
packageDir: String,
globs: Array<String>,
lockCandidates: Array<String>,
): Future<Array<(String, Fingerprint)>> {
let pairs = new GrowableArray<(String, Fingerprint)>();
let seen = new HashSet<String>();
if (globs.length > 0) {
let files = await expandGlobs(this.#root, packageDir, globs);
for (let path in files) {
if (seen.add(path)) {
pairs.push((path, await this.hash(path)));
}
}
}
for (let lockPath in lockCandidates) {
if (!seen.has(lockPath)) {
let fp = await this.hashIfExists(lockPath);
if (fp != null) {
seen.add(lockPath);
pairs.push((lockPath, fp));
}
}
}
return pairs;
}
}
| Version | Published | Dependencies | Approved by |
|---|---|---|---|
| 0.0.1-preview.1 | 2026-11-19 |
|