# Streaming Push and R2-First Repository Design

## Status

**Streaming is production. Legacy paths removed in closure release.** All repos are streaming. The storage-mode concept, rollback backfill, unpack, hydration, and legacy receive code have been removed. See `MIGRATION-STREAMING-PUSH.md` for upgrade guidance.

This document specifies the streaming push design that replaced the unpack-and-hydrate pipeline.

The new design makes R2 `.pack` and `.idx` objects the only required source of truth for Git object data. The repository Durable Object remains the source of truth for metadata only: refs, HEAD, pack catalog state, and receive/compaction leases.

The design is intentionally conservative about complexity:

- No Git protocol changes.
- No repo metadata stored in R2.
- No exact `pack_objects(pack_key, oid)` correctness dependency.
- No correctness dependency on DO `obj:*` or R2 loose mirrors.
- No new distributed transaction.
- No new tunable env vars beyond a single Queue binding.
- No object-level garbage collection in the current design.

## Why This Exists

The current implementation buffers the entire receive-pack request body in memory, indexes through `isomorphic-git` over an in-memory filesystem, then unpacks to loose objects in the Durable Object and mirrors them to R2. That model does not scale and violates the intended ownership boundary:

- the DO is doing data-plane work instead of metadata-plane work;
- correctness depends on loose object materialization;
- fetch/object reads still depend on loose-object semantics;
- exact pack membership is stored in SQLite even though R2 `.idx` already contains the authoritative mapping;
- background hydration is a second correctness mechanism layered on top of unpacking.

The new model replaces all of that with immutable packs in R2, worker-local range reads, and compact metadata in the DO.

## Goals

1. Streaming-first receive path with bounded memory.
2. R2-first data model: packs in R2 are authoritative for object data.
3. DO metadata-only correctness boundary.
4. No externally visible Git protocol changes.
5. Existing packed repos continue to work.
6. Existing repos can lose all loose objects and still remain correct.
7. Fetch becomes naturally pack-first and streaming.
8. Heavy lifting moves out of the DO and into Workers/Queue consumers.
9. The closure release leaves a single streaming-only end state.

## Non-Goals and Deliberate Simplifications

These are intentional, not omissions:

1. The design does not implement full Git garbage collection of unreachable objects. Compaction preserves the union of source-pack objects. This matches current retention behavior more closely and avoids a repo-wide mark phase.
2. The fetch/read path loads whole `.idx` files into memory when needed. This is acceptable because the representative `.idx` is about 3 MB while the representative `.pack` is about 42 MB. The constraint is “do not buffer the pack”, not “never buffer an idx”.
3. Push concurrency is simplified to one active receive lease per repo. The current one-active plus one-queued unpack model is removed because unpacking is removed. Returning `503 Retry-After: 10` to concurrent pushes is Git-compatible and materially simpler.
4. UI raw/blob reads may buffer an individual packed object when it is delta-resolved and not already present in the optional loose cache. This is acceptable because the streaming-first requirement is about pack ingress/egress paths, not ad hoc UI reads; the UI already enforces size caps.
5. Bloom filters are not introduced initially. Active pack count is bounded by compaction, and the `.idx` fanout table already gives an exact, compact lookup structure.

## Source of Truth

### Data plane

Required for correctness:

- `do/<do-id>/objects/pack/*.pack`
- `do/<do-id>/objects/pack/*.idx`

Optional caches only:

- DO `obj:*`
- R2 loose mirrors under `do/<do-id>/objects/loose/*`

### Metadata plane

Required for correctness:

- DO KV: `refs`, `head`, `refsVersion`, `packsetVersion`, `nextPackSeq`, receive/compaction leases
- DO SQLite: active/superseded pack catalog

Legacy data removed in the closure release (stale keys may remain on disk but are not read):

- `pack_objects`, `hydr_cover`, `hydr_pending` (SQLite tables dropped)
- `lastPackOids`, `lastPackKey`, `packList`, `unpackWork`, `unpackNext` (DO KV keys)
- `obj:*` (DO loose object cache)

## Key Design Decisions

### 1. `.idx` is treated as data-plane state, not repo metadata

Refs, HEAD, pack catalog, and leases remain in the DO. Standard Git `.idx` files live next to `.pack` files in R2 because they are required to address immutable pack contents efficiently. No JSON manifests or repo catalog records are stored in R2.

### 2. Receive writes pack data first, commits metadata last

The Worker only publishes a new pack to the repo catalog after:

- the `.pack` object is fully written to R2;
- the `.idx` is successfully built and written to R2;
- pack integrity and thin-pack base validation pass;
- ref/update validation passes.

This avoids a distributed transaction. R2 objects are immutable blobs; the DO metadata only points to fully written pack pairs.

### 3. Fetch and object reads stop depending on loose objects

All fetch planning, object existence checks, commit/tree/tag/blob reads, raw/blob views, diff, and merge-history traversal must use a shared pack-first object store in Worker code. DO object RPCs become compatibility shims or are removed from callers.

### 4. Hydration is replaced by queue-driven pack compaction

The system no longer unpacks to loose objects or builds “hydration packs”. It keeps a bounded set of immutable active packs and periodically compacts older packs into larger packs in a background Queue consumer.

### 5. Compaction is tiered, not full-repo repack

The initial compaction strategy is LSM-like:

- receive packs enter tier 0;
- when a tier exceeds the fixed fan-in, the oldest packs in that tier are merged into one pack in the next tier;
- active pack count remains bounded;
- the algorithm does not require a full-repo rewrite on each push.

This is simpler than full snapshot compaction and good enough for the target workload.

## Assumptions

1. The representative push remains within the Worker request body limit. The supplied sample is about 42 MB, which is under the documented minimum 100 MB request-body limit on paid Cloudflare plans.
2. Workers and Durable Objects run with a 128 MB memory limit; request/response streaming is available; R2 `put()` accepts a `ReadableStream`; R2 ranged reads are strongly supported in Workers; Queue consumers have 15 minutes of wall time.
3. Active pack count is kept low by compaction, so loading a handful of `.idx` files into memory is safe and simpler than inventing a second metadata structure.
4. Existing repos that already had packs in R2 were the primary cutover migration case. Closure assumes that any loose-only repos already completed the required one-time pack migration before this release.

## Cloudflare Constraints Used By The Design

These are the platform facts the design relies on:

- Workers and Durable Objects have a 128 MB memory limit.
- HTTP request handling has no hard wall-time limit while the client stays connected.
- Queue consumers have a 15 minute wall-time limit.
- Workers Paid supports up to 10,000 subrequests per invocation and up to six simultaneous outgoing connections.
- R2 `put()` is strongly consistent and accepts a `ReadableStream`.
- R2 `get()` supports ranged reads by `offset` and `length`.
- R2 multipart uploads exist, but require 5 MiB parts except the last part.

The receive path prefers a single Worker-to-R2 `put()` when the pack length is known ahead of time. When the incoming Git client uses chunked transfer and the pack length is not knowable up front, the receive path falls back to Worker-to-R2 multipart upload with fixed-size parts.

## New Metadata Model

### New SQLite table: `pack_catalog`

Add a new table via `src/do/repo/db/schema.ts` and DAL helpers in `src/do/repo/db/dal.ts`.

Required columns:

- `packKey TEXT PRIMARY KEY`
- `kind TEXT NOT NULL`
  - `receive`
  - `compact`
  - `legacy`
- `state TEXT NOT NULL`
  - `active`
  - `superseded`
- `tier INTEGER NOT NULL`
- `seqLo INTEGER NOT NULL`
- `seqHi INTEGER NOT NULL`
- `objectCount INTEGER NOT NULL`
- `packBytes INTEGER NOT NULL`
- `idxBytes INTEGER NOT NULL`
- `createdAt INTEGER NOT NULL`
- `supersededBy TEXT NULL`

Indexes:

- active packs ordered by `state`, `seqHi DESC`
- active packs in a tier ordered by `state`, `tier`, `seqLo`

Notes:

- `seqLo` and `seqHi` are catalog ordering markers, not Git semantics.
- Active packs must always represent a disjoint cover of receive-sequence ranges.
- New receive packs start with `seqLo = seqHi = nextPackSeq`.
- A compacted pack gets `seqLo = min(source.seqLo)` and `seqHi = max(source.seqHi)`.

### DO KV keys that remain authoritative

- `refs`
- `head`
- `refsVersion`
- `packsetVersion`
- `nextPackSeq`
- `receiveLease`
- `compactLease`
- `compactionWantedAt`

Lease contents:

- `receiveLease = { token, createdAt, expiresAt }`
- `compactLease = { token, createdAt, expiresAt }`

Fixed code constants, not env vars:

- receive lease TTL: 30 minutes
- compaction lease TTL: 20 minutes

Expired leases are cleared:

- on the next `beginReceive()` / `beginCompaction()` call; and
- by a lightweight DO alarm cleanup path.

### Legacy compatibility mirrors (removed in closure)

`packList`, `lastPackKey`, and `lastPackOids` were maintained as mirrors during rollout for rollback compatibility. They have been removed in the closure release. `pack_catalog` is the sole authority for pack metadata.

## Cross-System Mutation Rules

### Rule 1

Only the DO mutates repo metadata.

### Rule 2

Workers and Queue consumers may write immutable R2 blobs before metadata commit, but those blobs are not visible to the repo until the DO commits the catalog row.

### Rule 3

Deleting unreferenced staged pack blobs and superseded R2 pack pairs is best-effort and retryable. Failure to delete must never affect correctness.

### Rule 4

Compaction never mutates source packs in place. It writes a new pack, then atomically swaps catalog rows in the DO.

### Rule 5

Fetch and UI reads operate on a read-only snapshot of the active catalog. They never mutate repo state.

### Rule 6

Queue delivery is a hint, not the durable record of pending compaction. The durable record is `compactionWantedAt` in DO metadata. Queue delivery failure may delay compaction, but it must not lose the need for compaction.

## Receive Path

### Overview

The Worker handles receive-pack end to end. The DO is used only for:

- acquiring a receive lease;
- reading current refs/HEAD/version metadata;
- committing ref changes and the new pack row;
- clearing the lease.

The DO does not expose any HTTP receive endpoint. All receive coordination uses typed RPCs.

### Client-visible behavior

Unchanged:

- Git Smart HTTP request/response format.
- `report-status`.
- delete-only pushes.
- stale `old-oid` rejection.
- invalid ref rejection.
- thin-pack acceptance when bases exist.

Changed internally:

- concurrent push handling becomes “one active receive lease per repo”; additional pushes get `503 Retry-After: 10`.
- `X-Repo-Changed` and `X-Repo-Empty` are computed in the Worker after DO finalization, not forwarded from a DO HTTP endpoint.

### Step-by-step algorithm

1. Worker calls `stub.beginReceive()` before reading the body.
2. If a valid receive lease already exists, the Worker immediately returns `503 Retry-After: 10`.
3. If the lease is granted, the Worker incrementally parses the pkt-line command section from `request.body` until the flush packet.
4. The remaining bytes are treated as the raw pack stream.
5. The Worker writes the raw pack stream directly to a staged R2 key such as `do/<id>/objects/pack/pack-rx-<leaseToken>.pack`.
6. During the upload stream, the Worker validates only stream-level invariants that are cheap to check in one pass:
   - pack starts with `PACK`
   - version is 2
   - trailer SHA-1 matches the streamed body
   - byte count matches the actual received length
7. After the R2 upload resolves, the Worker runs the new indexer against the staged pack in R2 and writes `pack-rx-<leaseToken>.idx`.
8. The Worker runs thin-pack base validation and ref-target connectivity validation using the new pack-first object store and the active pack catalog snapshot.
9. The Worker calls `stub.finalizeReceive(...)`.
10. The DO atomically:
    - rechecks the lease token;
    - rechecks `old-oid` expectations against current refs;
    - allocates `nextPackSeq`;
    - inserts the new active pack row;
    - updates `refs`, `head`, `refsVersion`, and `packsetVersion`;
    - sets or refreshes `compactionWantedAt` if the catalog now violates the compaction policy;
    - clears the receive lease;
    - returns whether compaction should be queued.
11. The Worker returns the pkt-line `report-status` response.
12. If compaction should run, the Worker enqueues one idempotent Queue message `{ repoId }` in `ctx.waitUntil(...)`.

### Failure handling

If any step before `finalizeReceive()` fails:

- the Worker calls `stub.abortReceive(leaseToken)` best-effort;
- the staged `.pack` and `.idx` are deleted best-effort;
- the response is either `400`, `415`, `500`, or `503` depending on failure class.

If `finalizeReceive()` fails stale-old-oid validation:

- the Worker deletes staged `.pack` and `.idx` best-effort;
- the Worker returns a normal Git `report-status` rejection, not an HTTP 409.

If the client disconnects after `finalizeReceive()` succeeds:

- refs and catalog remain committed;
- a retry from the client is expected to fail stale-old-oid, which is standard Git behavior.

## New Pack Indexer

### Why a new indexer exists

The current `isomorphic-git` path requires the whole pack in memory. The replacement must index from R2 without buffering the entire pack.

### Design

The new indexer lives in `src/git/pack/indexer/` and runs entirely in Worker code.

The indexer is two-stage:

1. `scanPack()`
2. `resolveDeltasAndWriteIdx()`

Implementation note:

- `scanPack()` cannot rely on `DecompressionStream` alone because Git pack entries are concatenated deflate streams and the scanner must know exactly how many compressed bytes each entry consumed;
- the implementation therefore needs a byte-accounting inflate cursor in JS/WASM that exposes end-of-stream position for each packed entry.

### `scanPack()`

Input:

- staged R2 `.pack`
- active pack catalog snapshot for external-base lookup

Behavior:

- reads the pack sequentially from R2 in fixed-size ranges;
- validates header and trailer;
- records, for each object:
  - ordinal
  - offset
  - packed type
  - header length
  - packed span end
  - CRC32 of the raw packed entry
  - base offset for `OFS_DELTA`
  - base oid for `REF_DELTA`
  - result size for delta objects
- computes final oid immediately for non-delta objects by inflating the object payload stream and hashing `"<type> <size>\\0<payload>"`.

Output:

- a compact object-entry table;
- `offset -> index` lookup;
- the list of non-delta resolved objects;
- the list of unresolved delta objects.

Implementation constraint:

- The object-entry table must use typed arrays or chunked binary buffers, not arrays of large JS objects.
- OIDs are stored as raw 20-byte values internally; hex strings are materialized lazily.

### `resolveDeltasAndWriteIdx()`

Behavior:

- resolves unresolved delta objects in dependency order;
- supports:
  - in-pack `OFS_DELTA`
  - in-pack `REF_DELTA`
  - thin-pack external `REF_DELTA` bases
- rejects any delta whose base cannot be resolved from either:
  - an already-scanned in-pack object, or
  - the active pack catalog snapshot.

Memory strategy:

- resolved base payloads are kept in an LRU cache with a hard byte budget;
- evicted base payloads may be recomputed from the pack/object store if needed later;
- correctness does not depend on cache hits.

Delta application:

- parse the delta header varints;
- verify the declared base size;
- materialize the result payload into a pre-sized buffer;
- compute the final oid from `"<baseType> <resultSize>\\0<resultPayload>"`.

After all object oids are resolved:

- sort entries by oid;
- emit a standard Git idx v2 file;
- write it to R2;
- return `objectCount`, `idxBytes`, and the parsed `IdxView`.

### Why this is feasible

For the representative pack:

- pack: about 42 MB
- idx: about 2.7 MB
- objects: about 97k

The design never holds the whole pack in memory. It only holds:

- the compact entry table;
- a bounded payload cache for delta bases;
- the final idx buffer.

That fits comfortably within the 128 MB memory limit for the target workload.

## Connectivity Validation

Validation remains intentionally limited to what the current product already enforces.

### Pack-level validation

Reject the push if any packed object is structurally invalid or any thin-pack external base cannot be resolved from the active catalog snapshot.

### Ref command validation

Keep the existing rules:

- reject `HEAD` updates;
- reject invalid ref names;
- delete requires existing ref and matching `old-oid`;
- create requires zero `old-oid`;
- update requires matching `old-oid`;
- no partial apply: all commands succeed or none apply.

### Target-object connectivity validation

For each non-delete `newOid`:

- resolve tags transitively, with a hard depth limit of 8;
- if the final type is `commit`, require:
  - object exists;
  - root tree exists;
  - each parent exists;
- if the final type is `tree` or `blob`, require it exists.

The design does not add full `fsck` or full tree-walk validation.

## Fetch and Read Path

### New shared object store

Create `src/git/object-store/` and move all correctness-critical object reads to it.

Required APIs:

- `loadActivePackCatalog(repoId)`
- `loadIdxView(packKey)`
- `findObject(oid)`
- `hasObjectsBatch(oids)`
- `readObject(oid)`
- `readObjectRefsBatch(oids)`
- `readBlobStream(oid)`
- `iterPackOids(packKey)`

### Catalog loading

`getPackCandidates()` stops using `pack_objects` and R2 listing as correctness sources.

New behavior:

- ask the DO only for the active pack catalog;
- cache the result in request memo;
- sort active packs by `seqHi DESC`, then by `tier DESC`.

R2 listing remains migration-only fallback, never the primary read path.

### Idx loading

`IdxView` replaces `pack_objects` as the membership source.

Behavior:

- load the entire `.idx` object into memory;
- keep raw fanout, raw name table, and raw offset tables;
- build `oid -> index` lazily or via binary search over the raw name table;
- build `offsetToIndex` and `nextOffset` once per loaded idx.

### Object resolution

`findObject(oid)`:

1. iterate active packs in catalog order;
2. use the loaded `.idx` to check membership;
3. return the first hit as `(packKey, objectIndex, offset, nextOffset)`.

`readObject(oid)`:

1. locate the object by `.idx`;
2. read the object entry header and compressed payload span from the pack via range reads;
3. if base object type, inflate and return payload;
4. if delta, recursively resolve base and apply delta;
5. memoize per request.

### Closure planning

Replace all current dependencies on DO `hasLooseBatch()`, `getPackOids*()`, and `getObjectRefsBatch()`.

New rules:

- `findCommonHaves()` uses `hasObjectsBatch()` over the active pack catalog.
- `buildUnionNeededForKeys()` unions `.idx` membership from the selected packs.
- `computeNeededFast()` uses worker-local `readObjectRefsBatch()` over packed objects.

### Streaming fetch

The existing streaming assembler remains the fetch data path, with two changes:

1. pack discovery comes from the active pack catalog rather than `packList + pack_objects`;
2. all object/membership lookups come from the worker-local object store.

This means:

- single-pack fetch remains streaming;
- multi-pack fetch remains streaming;
- fetch becomes correct even when all loose objects are removed.

### UI and admin read paths

The following routes must stop depending on DO object RPCs for correctness:

- tree
- blob
- raw
- rawpath
- commits
- commit diff
- merge fragment expansion

Required behavior:

- if a loose cache entry exists, it may be used;
- if not, the route must read from packs;
- if a blob is delta-resolved and over the existing UI size cap, preserve the current “too large” behavior.
- `readBlobStream()` must preserve correctness for raw/blob routes, but it is not required to be zero-copy; it may materialize an individual packed object or delta base chain as long as it never requires whole-pack buffering.

## Optional Loose Cache

The DO may continue to cache loose objects, but only as an opportunistic cache.

Rules:

- no fetch or receive correctness may depend on it;
- compaction does not require it;
- deleting all loose data must not break clones, fetches, or UI reads.

Permitted uses:

- hot tree/commit/tag cache
- hot raw/blob cache
- debug-only loose fallbacks

## Background Compaction

### Why compaction exists

Removing unpack/hydration means packs in R2 must themselves remain a manageable serving set. Compaction is the replacement for “loose objects + hydration packs”.

### Queue model

Add one Queue binding, for example `REPO_MAINT_QUEUE`.

Producer:

- the main Worker after successful receive finalization;
- admin compaction trigger;
- migration jobs.

Consumer:

- the same Worker export, with `queue()` implemented in `src/git/compaction/run.ts`.

Recommended Queue config:

- `max_batch_size = 1`
- `max_batch_timeout = 1`
- default retries are acceptable

No new env vars are required.

The DO alarm remains in use, but only for lightweight metadata work:

- expire stale receive and compaction leases;
- retry queue re-arm when `compactionWantedAt` is set and no compaction lease is active;
- never perform pack indexing, unpacking, or compaction itself.

### Compaction policy

Fixed constants in code:

- fan-in: 4
- one compaction lease per repo

Policy:

1. tier 0 contains receive packs;
2. if more than four active packs exist in a tier, compact the oldest four in that tier;
3. output one pack in the next tier;
4. source packs become `superseded` only after DO commit.
5. no compaction lease may be granted while a receive lease is active for the same repo.
6. receive has priority over compaction; if a receive lease becomes active after compaction starts, `commitCompaction()` must fail and the queue worker must retry later.

### Compaction algorithm

1. Queue consumer calls `stub.beginCompaction()`.
2. The DO either returns “no work” or returns:
   - lease token
   - `packsetVersion`
   - selected source packs
   - full active catalog snapshot
   - target tier
3. The worker computes `needed = union(source pack idx membership)`.
4. The worker calls the existing or updated streaming assembler with:
   - `packKeys = full active catalog`
   - `needed = union(source pack membership)`

The full active catalog is passed so that delta bases outside the source set are pulled in automatically. The output pack therefore becomes self-contained enough to replace the source packs.

5. The output stream is written directly to a staged compacted-pack key in R2.
6. The new indexer runs against the staged compacted pack and writes the staged `.idx`.
7. The worker calls `stub.commitCompaction(...)`.
8. The DO atomically:
   - rechecks the lease token;
   - rechecks `packsetVersion`;
   - rechecks that the selected source packs are still active and unchanged;
   - marks the new pack active;
   - marks source packs superseded;
   - updates `packsetVersion`;
   - clears or refreshes `compactionWantedAt` based on the post-commit catalog state;
   - clears the compaction lease.
9. The worker deletes superseded R2 blobs best-effort in `waitUntil(...)`.

### What compaction does not do

It does not:

- rewrite refs;
- delete unreachable objects from within a pack;
- require loose objects;
- require hydration state or `pack_objects`.

## Existing Repositories

### Existing packed repos (pre-closure migration, now completed)

Migration was automatic during the cutover phase:

1. On DO startup or first access, if `pack_catalog` was empty:
   - seeded from the union of `lastPackKey`, `packList`, and the full R2 `.pack` listing under the repo prefix (these keys no longer exist post-closure);
   - ignored `pack_objects` for correctness.
2. For each discovered pack:
   - required `.pack` and `.idx` to exist;
   - parsed `.idx` fanout to get object count;
   - inserted a `legacy` `active` row into `pack_catalog`;
   - assigned synthetic sequential `seqLo = seqHi` values.
3. Legacy `pack-hydr-*` packs were inserted as normal active packs.

### Existing loose-only repos (pre-closure migration, now completed)

These could not switch directly to `repoStorageMode = streaming` during the cutover migration.

Required migration:

1. Traverse reachable objects from current refs using the legacy loose path.
2. Build one pack in R2 with a streaming non-delta pack writer.
3. Index it with the new indexer.
4. Insert it into `pack_catalog` as a `legacy` active pack.
5. Leave loose data untouched.

Only after that migration completed could the repo be promoted during cutover.

### Emergency rollback (removed in closure release)

The emergency backfill tool described here was used during the phase-3 canary rollout and has been removed in the closure release. See `docs/streaming-push-closure-plan.md` for details.

## Compatibility Surface

### Routes and headers

Unchanged public Git routes:

- `GET /:owner/:repo/info/refs`
- `POST /:owner/:repo/git-upload-pack`
- `POST /:owner/:repo/git-receive-pack`

Internal headers:

- `X-Repo-Changed`
- `X-Repo-Empty`

These remain worker-internal and continue to exist if callers still use them.

### RPC compatibility (historical)

The following RPCs were removed across phases 1-5 and the closure release:

- `getObjectStream`, `getObject`, `getObjectSize` — replaced by worker-local pack-first reads
- `hasLooseBatch` — replaced by `hasObjectsBatch` over pack catalog
- `getObjectRefsBatch` — replaced by worker-local packed-object batch reader
- `getPackLatest`, `getPackOids`, `getPackOidsBatch` — replaced by `pack_catalog` as sole authority
- `getUnpackProgress` — replaced by `beginReceive()` lease acquisition
- `getRepoStorageMode`, `setRepoStorageModeGuarded` — storage-mode concept removed

### Progress and activity UI

The current UI renders unpack progress in multiple places. That caller graph must be migrated explicitly.

Replacement contract:

- replace `getUnpackProgress()` in `src/common/progress.ts` with `getRepoActivity()`;
- `getRepoActivity()` returns a neutral idle state plus two optional live states:
  - `receiving`
  - `compacting`
- the existing progress banner remains in place, but its text changes:
  - `receiving`: “Receiving push…”
  - `compacting`: “Compacting packs…”
  - idle: render nothing.

Files that must migrate together:

- `src/common/progress.ts`
- `src/routes/ui/overview.ts`
- `src/routes/ui/tree.ts`
- `src/routes/ui/commits.ts`
- `src/routes/ui/adminPage.ts`
- `src/client/components/ProgressBanner.tsx`
- `src/client/pages/OverviewPage.tsx`
- `src/client/pages/TreePage.tsx`
- `src/client/pages/CommitsPage.tsx`
- `src/client/pages/AdminPage.tsx`

During phase 2, these files may continue to render a banner, but they must stop interpreting `unpackWork`, `queuedCount`, or hydration presence as correctness signals.

### Admin endpoints

- `POST /:owner/:repo/admin/compact` — returns a compaction plan by default and triggers compaction when `dryRun === false`
- `DELETE /:owner/:repo/admin/compact` — clears queued compaction work for the repo

The `/admin/hydrate` aliases and `/admin/storage-mode` endpoints were removed in the closure release.

Current pack-deletion admin behavior must also change:

- `DELETE /:owner/:repo/admin/pack/:packKey`
  - may only delete `superseded` packs;
  - must reject deletion of `active` packs by default;
  - any forced delete of an `active` pack remains an explicit admin-only break-glass path and is outside normal operating assumptions.

### Debug/admin fields

Replace loose/unpack/hydration-centric fields with:

- `receiveLease`
- `compaction`
- `activePacks`
- `supersededPacks`
- `packCatalogVersion`

Legacy debug fields (`unpacking`, `queuedCount`, `lastPackOids`, `repoStorageMode`, `hydration*`) were removed in the closure release.

Current debug endpoints must stay functional, but they must read through the pack-first object store:

- `GET /:owner/:repo/admin/debug-state`
- `GET /:owner/:repo/admin/debug-commit/:commit`
- `GET /:owner/:repo/admin/debug-oid/:oid`

Admin UI migration (completed):

- The admin island uses compaction-derived status exclusively
- `StorageModeCard` and `HydrationCard` components have been removed
- Pack-removal warnings reference compaction supersession, not hydration

## What Was Removed (Closure Release)

The following subsystems were removed in the closure release:

- DO `POST /receive` endpoint
- Background unpacking (alarm-driven loose object extraction)
- Hydration planning and segment building
- `pack_objects`, `hydr_cover`, `hydr_pending` SQLite tables
- `lastPackOids`, `lastPackKey`, `packList` DO KV keys
- `hasLooseBatch()`, `getObjectStream()`, `getObjectSize()`, `getPackLatest()`, `getPackOids()`, `getPackOidsBatch()` RPCs
- `RepoStorageMode` type and all storage-mode switching
- Rollback backfill machinery
- `isomorphic-git` dependency
- Configuration: `REPO_UNPACK_*`, `REPO_KEEP_PACKS`, `REPO_PACKLIST_MAX`, `REPO_DO_MAINT_MINUTES`

## Module Deliverables

The implementation must keep `repoDO.ts` thin by pushing logic into helper modules.

Required new modules:

- `src/do/repo/catalog.ts`
- `src/do/repo/receiveLease.ts`
- `src/do/repo/compactionLease.ts`
- `src/do/repo/legacyCompat.ts`
- `src/git/receive/pktSectionStream.ts`
- `src/git/receive/streamReceivePack.ts`
- `src/git/pack/indexer/scan.ts`
- `src/git/pack/indexer/inflateCursor.ts`
- `src/git/pack/indexer/resolve.ts`
- `src/git/pack/indexer/writeIdx.ts`
- `src/git/object-store/catalog.ts`
- `src/git/object-store/idxView.ts`
- `src/git/object-store/lookup.ts`
- `src/git/object-store/readObject.ts`
- `src/git/object-store/readRefsBatch.ts`
- `src/git/object-store/delta.ts`
- `src/git/compaction/plan.ts`
- `src/git/compaction/run.ts`
- `src/git/compaction/legacyLoosePack.ts`

Primary files to update:

- `src/routes/git.ts`
- `src/routes/admin.ts`
- `src/routes/ui/adminPage.ts`
- `src/routes/ui/overview.ts`
- `src/routes/ui/tree.ts`
- `src/routes/ui/commits.ts`
- `src/routes/ui/raw.ts`
- `src/routes/ui/helpers.ts`
- `src/index.ts`
- `src/common/progress.ts`
- `src/do/repo/repoDO.ts`
- `src/do/repo/debug.ts`
- `src/do/repo/db/schema.ts`
- `src/do/repo/db/dal.ts`
- `src/client/components/ProgressBanner.tsx`
- `src/client/pages/AdminPage.tsx`
- `src/client/pages/CommitsPage.tsx`
- `src/client/pages/OverviewPage.tsx`
- `src/client/pages/TreePage.tsx`
- `src/client/islands/repo-admin/types.ts`
- `src/client/islands/repo-admin/useRepoAdminActions.ts`
- `src/client/islands/repo-admin/RepoOverviewCard.tsx`
- `src/client/islands/repo-admin/HydrationCard.tsx`
- `src/client/islands/repo-admin/PackFilesCard.tsx`
- `src/client/islands/repo-admin/index.tsx`
- `src/git/operations/closure.ts`
- `src/git/operations/fetch/neededFast.ts`
- `src/git/object-store/catalog.ts`
- `src/git/operations/read/diff.ts`
- `src/git/operations/read/objects.ts`
- `src/git/operations/read/tree.ts`
- `src/git/operations/read/commits.ts`
- `src/git/pack/assemblerStream.ts`
- `wrangler.jsonc`

## Phased Implementation Plan (completed)

> **Archival:** All phases are complete and the closure release has removed all legacy code paths. This section is preserved as historical rollout context. See `MIGRATION-STREAMING-PUSH.md` for the current operator guide.

### Phase 1: Pack Catalog and Worker Object Store Shadow Mode (completed)

Added `pack_catalog` schema and DAL helpers, DO helpers for catalog reads/leases/legacy seeding, worker-local `IdxView` and packed-object resolver, read-path shadow validators, and queue binding. Automatic `pack_catalog` backfill for packed repos.

### Phase 2: Read Path Cutover (completed)

Switched fetch planning, object reads, and all UI routes to worker-local pack-first object store. Removed correctness dependence on DO `getObject*`, `hasLooseBatch`, `getPackOids*`, and `getObjectRefsBatch`. Replaced unpack-progress UI with repo-activity UI. Added admin-only storage-mode control for canary validation.

### Phase 3: Streaming Receive Cutover (completed)

Added receive lease RPCs, staged R2 write, pack indexer, thin-pack validation, connectivity checks, and DO finalization. Kept `packList` and `lastPackKey` mirrored for rollback. Added emergency legacy backfill tooling for canary rollback.

### Phase 4: Queue-Driven Compaction (completed)

Implemented `beginCompaction` / `commitCompaction` / `abortCompaction` and queue consumer. Tiered compaction of active catalog. Repurposed `/admin/hydrate` as compaction alias and added `/admin/compact`.

### Phase 5: Legacy Path Retirement (completed)

Removed legacy receive/unpack/hydration methods from active code paths. Moved all repos to streaming. Removed legacy RPCs and tests. The closure release then removed all remaining legacy code, schema, and configuration.

## Automated Testing Plan

Automated tests must not assume `uncommitted-fixture/` is committed.

### New or rewritten worker tests

1. Receive path streams without `request.arrayBuffer()`.
2. Single active receive lease returns `503` to concurrent pushes.
3. Thin-pack base resolution uses existing packed objects, not loose objects.
4. Fetch works after deleting all DO `obj:*`.
5. Fetch works after deleting all R2 loose mirrors.
6. `hasObjectsBatch()` and `readObjectRefsBatch()` are pack-first.
7. Compaction replaces source packs and keeps fetch correct.
8. `DELETE /admin/pack/:packKey` rejects deletion of active packs.
9. Debug endpoints continue to work after all loose data is removed.
10. Test repo seed helpers can build pack-only repos without writing loose objects.
11. Pack-first test seeding replaces `seedMinimalRepo()` loose assumptions in new and migrated tests.

### Legacy tests (removed in closure)

The following test categories were deleted in the closure release:

- hydration clear/delete tests
- unpack progress tests
- one-deep unpack queue tests
- `pack_objects` exact membership tests
- storage-mode transition tests
- rollback compatibility tests

### Performance and memory tests

Add a local-only benchmark script under `scripts/` that:

- uses `uncommitted-fixture/` when present;
- measures peak memory, index time, and receive latency;
- is not required for CI.

CI should instead generate smaller deterministic packs with:

- long OFS-delta chains;
- thin REF_DELTA bases;
- multiple receive packs triggering compaction.

Test seeding must also change:

- stop relying on `seedMinimalRepo()` writing loose objects for packed repos;
- add a packed-only seed helper and use it in all fetch/read tests that are meant to validate the new correctness path.

## Acceptance Criteria

The rollout is successful only if all of the following are true:

1. A repo push with about 8,000 commits, about 70,000 loose objects’ worth of content, a 40 MB pack, and a 3 MB idx completes without buffering the whole pack in memory.
2. After such a push, deleting all DO `obj:*` keys does not break fetch, tree/blob/raw reads, diff, or commit browsing.
3. After such a push, deleting all R2 loose mirrors does not break fetch, tree/blob/raw reads, diff, or commit browsing.
4. Fetch planning no longer depends on `pack_objects`, `lastPackOids`, `unpackWork`, `unpackNext`, or hydration state.
5. Active pack count remains bounded by the compaction policy after repeated pushes.
6. The DO is never required to materialize object data for correctness.
7. The only metadata authority remains the DO.
8. No route order or auth behavior regresses.
9. Existing packed repos migrate without requiring loose objects.
10. Legacy loose-only repos have an explicit migration path before cutover.

## Final Notes For Implementers

1. Do not bloat `src/do/repo/repoDO.ts`. Every new behavior belongs in a helper module and is only wired through thin RPC methods.
2. Do not add raw Drizzle queries outside the DAL.
3. Do not reintroduce exact per-pack SQLite membership as a correctness dependency.
4. Do not make loose objects required again, even as a “temporary shortcut”.
5. Do not store repo metadata in R2.
6. Prefer simple fixed constants over new env vars unless a hard operational need appears during implementation.

---

## Closure Release Implementation Checklist (completed)

All items below were completed in the closure release. See `docs/streaming-push-closure-plan.md` for the full implementation plan.

1. ~~Remove emergency backfill tool~~ -- done
2. ~~Remove `/admin/hydrate` aliases and `startHydration`/`clearHydration` RPCs~~ -- done
3. ~~Remove `mirrorLegacyPackKeys()` and `lastPackKey`/`lastPackOids`/`packList` from `RepoStateSchema`~~ -- done
4. ~~Remove `REPO_UNPACK_*` from `wrangler.jsonc`, `repoConfig.ts`, `vitest.bindings.ts`~~ -- done
5. ~~Delete legacy receive, unpack, hydration code~~ -- done
6. ~~Remove `unpackWork`, `unpackNext`, `hydrationWork`, `hydrationQueue` from `RepoStateSchema`~~ -- done
7. ~~Drop `pack_objects`, `hydr_cover`, `hydr_pending` tables~~ -- done
8. ~~Remove `isomorphic-git` from `package.json`~~ -- done
9. ~~Remove `RepoStorageMode` type and all storage-mode RPCs/admin endpoints~~ -- done
10. ~~Delete remaining legacy tests~~ -- done
11. ~~Final docs cleanup~~ -- done