# One-Shot Plan: Sidecar-Backed Fetch Closure

## Summary

Implement per-pack logical object-reference sidecars and use them for final `git-upload-pack` closure planning. This directly addresses the incident: fetch planning must not read thousands of packed objects from R2 just to discover commit/tree/tag edges.

The pressure test changed the plan in four important ways: sidecar readiness is checked before streaming starts, `.refs` becomes a required artifact for newly active packs, backfill uses the streaming indexer with existing `.idx` help for tricky `REF_DELTA` bases, and tree gitlinks are excluded from closure edges.

## Design Decisions

- Add an R2 sidecar beside every pack: `packKey.replace(/\.pack$/, ".refs")`, exposed through `packRefsKey(packKey)` in `src/keys.ts`.
- Keep sidecars as derived R2 artifacts. Do not add SQLite tables and do not add DO methods unless an existing RPC cannot provide the current active catalog.
- Sidecars are required for final fetches with haves. Initial clone with no haves stays idx-only via `buildInitialCloneNeeded()`.
- Missing/corrupt/stale sidecar on final fetch returns `503 Retry-After: 10` before `packfile\n` is emitted, and queues idempotent backfill.
- The production fetch planner must not fall back to `readObjectRefsBatch()` for active packs. That path is the scalability problem.

## Sidecar Format and Types

- Create a focused module, e.g. `src/git/pack/refIndex.ts`, with named types:
  - `PackRefView`
  - `PackRefBuildResult`
  - `PackRefSnapshotEntry`
  - `PackRefSnapshotLoadResult`
- Binary format v1:
  - magic/version
  - `objectCount`
  - `packBytes`
  - pack checksum from the `.idx` trailer
  - idx checksum from the `.idx` trailer
  - `typeCodes` in idx OID order, using canonical Git object type codes for commit/tree/blob/tag
  - `refStarts` length `objectCount + 1`
  - flat raw 20-byte ref OIDs
- Parser rejects bad magic/version, count mismatch, pack byte mismatch, checksum mismatch, truncation, non-monotonic `refStarts`, and invalid final offset.
- Extend `IdxView` to expose `packChecksum` and `idxChecksum`.
- Extend `RequestMemo` with `packRefViews` and `packRefViewPromises`.
- Add a byte-capped isolate LRU for `PackRefView`, keyed by `packKey + idxChecksum`. Do not keep unbounded global string maps.

## Indexer Changes

- Refactor the pack indexer around a named `PackRefsBuilder`.
- Add final object type storage to `PackEntryTable`; do not overwrite existing pack type fields because rewrite/idx logic still needs pack type codes.
- Feed `PackRefsBuilder` whenever a full logical object payload is available:
  - non-delta objects in `scanPack()`
  - resolved delta objects in `resolveDeltaEntry()`
  - external REF_DELTA fallback objects after delta application
- Add a new tree parser for closure refs that includes mode parsing and excludes gitlinks (`160000`). Keep existing `parseTreeChildOids()` behavior unless all callers are reviewed.
- Extract one shared OID-order helper from `writeIdxV2()` and use it for both `.idx` and `.refs`. Tie-break duplicate OIDs deterministically by pack entry index.
- Replace or wrap `resolveDeltasAndWriteIdx()` with an artifact-writing API that writes `.idx` and `.refs` together and returns `{ objectCount, idxBytes, refIndexBytes, idxView }`.

## Receive and Compaction

- Receive with a non-delete pack:
  - stage `.pack`
  - scan/resolve
  - write `.idx`
  - write `.refs`
  - run connectivity
  - call `finalizeReceive()`
- If receive fails after staging, cleanup deletes `.pack`, `.idx`, and `.refs`.
- Delete-only receive does no sidecar work.
- Compaction writes `.refs` for the compact target before `commitCompaction()`.
- Compaction retry/conflict/error cleanup deletes staged `.pack`, `.idx`, and `.refs`.
- Delayed superseded-pack delete deletes `.refs` with `.pack` and `.idx`.
- Admin `removePack()` deletes `.refs` for superseded packs. Missing `.refs` is a successful no-op.
- Purge remains prefix-based and needs no special-case sidecar logic.

## Fetch Path

- Move final fetch planning before creating the response stream. The stream should only start after snapshot, idx, sidecar, and closure planning are ready.
- Add `loadPackRefSnapshot(env, repoId, snapshot, cacheCtx)`:
  - loads one sidecar per active pack
  - validates against the corresponding `IdxView` and `PackCatalogRow`
  - uses `limiter.run("r2:get-pack-refs", ...)`
  - logs `stream:plan:ref-snapshot`
- Add sidecar-backed closure planner:
  - preserve current have cap behavior unless tests intentionally change it
  - common haves are active-pack membership checks from idx snapshots
  - stop set starts with acknowledged haves and retains the existing first-parent enrichment
  - walk wants through sidecar refs with a cursor queue
  - use request-local maps bounded by active indexed object count
  - compute reachable want closure minus stop-set reachability
- If planner budget is exceeded, return `503 Retry-After: 10` before streaming and log `stream:plan:closure-budget-exceeded`.
- Keep `readObjectRefsBatch()` for non-fetch callers/tests only, or leave it as an explicitly non-production fallback with tests proving fetch does not call it.

## Backfill

- Add queue message `pack-ref-backfill` to `RepoMaintenanceQueueMessage`.
- Fetch queues one message per missing/corrupt/stale active pack using `ctx.waitUntil`; duplicate messages are allowed.
- Queue handler:
  - gets current active catalog through existing DO RPC with limiter/counting
  - acks if target pack is no longer active
  - loads target `.idx`; if valid `.refs` already exists, ack
  - runs the streaming scan/resolve machinery with the target pack excluded from the external-base catalog
  - seeds same-pack `REF_DELTA` dependency edges from the existing `.idx` so backfill does not fall back to recursive packed-object reads for objects already in the target pack
  - tries duplicate external-base candidates in catalog order so a newer unusable duplicate does not hide an older materializable base
  - writes only the `.refs` sidecar; the deterministic `.idx` is already present and must not be rewritten by backfill
  - retries transient R2/DO failures
  - acks deterministic invalid pack/idx failures after logging
- The DO never reads or writes R2 for backfill.

## State Matrix

| State                                       | Behavior                                                    |
| ------------------------------------------- | ----------------------------------------------------------- |
| Initial clone, no haves                     | Existing idx-only path; sidecar not required                |
| Negotiation request, `done=false`           | Existing ACK-only behavior; sidecar not required            |
| Final fetch, all sidecars valid             | Sidecar closure, then stream pack                           |
| Final fetch with duplicate wants            | Queue and send each canonical active object once            |
| Final fetch reaches duplicate OID in packs  | Newest snapshot-order pack is canonical                     |
| Force-push leaves overlapping active packs  | Traverse newest pack first; old overlap does not grow queue |
| Final fetch has no common haves             | Sidecar closure runs with an empty stop set                 |
| Final fetch, sidecar missing/corrupt/stale  | Queue backfill, return `503 Retry-After: 10` before stream  |
| Final fetch exceeds closure timeout         | Return `503 Retry-After: 10` before stream                  |
| Final fetch exceeds missing-ref cap         | Return `503 Retry-After: 10` before stream                  |
| Active pack has idx missing/corrupt         | Existing repository-not-ready behavior                      |
| Newly received pack `.refs` write fails     | Abort receive, cleanup staged artifacts, no catalog commit  |
| Receive connectivity rejects                | Cleanup staged `.pack/.idx/.refs`, no catalog commit        |
| Finalize receive conflict/lease mismatch    | Cleanup staged `.pack/.idx/.refs`                           |
| Compaction target `.refs` write fails       | Abort/retry compaction, cleanup staged artifacts            |
| Superseded pack delete sees missing `.refs` | Log debug/info and continue                                 |
| Superseded pack delete                      | Delete `.pack`, `.idx`, and `.refs` together                |
| Admin remove superseded pack                | Delete `.pack`, `.idx`, `.refs`; report artifact outcomes   |
| Backfill enqueue fails                      | Log warning; still return retry readiness response          |
| Backfill target no longer active            | Ack as stale                                                |
| Backfill sees valid sidecar already present | Ack as duplicate/no-op                                      |
| Backfill transient R2/DO failure            | Retry with delay                                            |
| Backfill deterministic invalid pack/idx     | Log invalid-pack and ack                                    |

## Logging and Limits

- Add structured logs for:
  - `ref-index:write-start`
  - `ref-index:write-complete`
  - `ref-index:write-error`
  - `stream:fetch:ref-index-missing`
  - `stream:plan:ref-snapshot`
  - `ref-index:backfill-start`
  - `ref-index:backfill-complete`
  - `ref-index:backfill-stale-pack`
  - `ref-index:backfill-invalid-pack`
- Every new R2 and DO call goes through the existing limiter/counting pattern:
  - `r2:get-pack-refs`
  - `r2:put-pack-refs`
  - `r2:delete-pack-refs`
  - `do:get-active-pack-catalog`
- Queue sends are logged and scheduled with `waitUntil`; queue worker R2/DO work uses the same limiter rules as compaction.

## Tests

- Unit tests:
  - sidecar encode/decode round trip
  - every validation rejection case
  - tree parser excludes `160000` gitlinks
  - commit/tree/tag/blob edge encoding
  - duplicate OID deterministic ordering
- Worker tests:
  - receive writes `.refs`
  - compaction writes target `.refs`
  - cleanup deletes `.refs` on receive rejection, finalize conflict, lease mismatch, compaction retry, and superseded delete
  - final fetch with valid sidecars does not call per-object `r2:get-range` during closure
  - missing sidecar returns `503` before `packfile\n`
  - backfill creates sidecar and subsequent fetch succeeds
  - corrupt/stale sidecar is rebuilt
  - annotated tag chain closure works
  - submodule gitlink is not required as an object in the superproject
- Large fixture:
  - opt-in test using `uncommitted-fixture/pack-395a180893e59dad8ef9d7fa135ecd8b1b399bb1.pack`
  - assert sidecar generation handles 97,409 objects
  - assert closure planning uses sidecar reads and stays within request subrequest/memory expectations
- Validation commands:
  - `npm run typecheck`
  - targeted fetch/indexer/receive/compaction worker tests
  - opt-in large fixture test when validating scale

## Assumptions

- Temporary `503 Retry-After` is acceptable for incremental fetches while legacy active packs are backfilled.
- No protocol support for shallow/filter fetch is added in this change.
- Sidecars are derived artifacts; correctness still depends on `.pack`, `.idx`, and the DO active pack catalog.