Anvil Spec Resolutions

Date: 2026-03-18

This file records binding resolutions for ambiguities in anvil-spec.md without editing the base spec in place yet. Until the main spec is reconciled, this file is authoritative for the topics below.

1. Authority Boundary For Queueing, D1 Sync, Lock Release, And Queue Advancement

ProjectDO is the sole authoritative owner of project-level concurrency, queue mutation, queue dispatch state, D1 reconciliation state, lock release, and queue advancement.

The Worker and queue consumer may only:

• call ProjectDO RPCs to accept work, claim work, report terminal results, request cancellation, record heartbeats, or kick reconciliation
• act as delivery hints or execution agents around ProjectDO

They must not independently mutate durable queue state or D1 run-summary state as an authoritative source of truth.

2. Public Status Mapping For Accepted Runs

pending and executable are internal ProjectDO queue states only.

Public APIs, RunDO, and D1 run_index use only the canonical public run status enum:

• queued
• starting
• running
• cancel_requested
• canceling
• passed
• failed
• canceled

Both internal pending and internal executable map to public queued. Queue-local ordering is exposed separately through project queue summaries rather than through a separate public run status.

3. Source Revision A Run Executes

Webhook-triggered runs

Webhook-triggered runs execute the exact commit SHA from the verified provider event.

• the canonical accepted snapshot for webhook runs includes commit_sha
• queue delay must not change the revision that executes

Manual runs

Manual runs execute the branch head as resolved when the queue consumer starts execution.

• manual acceptance does not pin a commit SHA up front
• commit_sha may be null at acceptance time
• after checkout resolves HEAD, the queue consumer backfills the actual checked-out commit_sha immediately
• this backfilled SHA must be propagated to ProjectDO, RunDO, and D1 run_index
• this backfilled SHA is metadata describing what ran; it is not an acceptance-time execution pin

4. position Semantics In The Per-Project FIFO Queue

project_runs.position is a queued-only stable ordering token.

• it is assigned when a run is accepted into the per-project queue
• it is monotonically increasing within a project while rows remain queued
• it is not renumbered after cancellation, promotion, or terminalization
• gaps are allowed
• it is NULL once a run is no longer in queued state
• FIFO promotion uses ascending non-null position

This means position is not a mutable "live index" that is continuously compacted.

5. Proof Required Before Promoting The Next Run

ProjectDO may promote the next queued run only after one of these conditions is true:

1. the queue consumer has completed teardown for the active run and reported terminalization back to ProjectDO
2. ProjectDO has declared the runner lost during stale-heartbeat recovery and has taken over terminalization itself

Operationally:

• the normal path is queue-consumer cleanup, sandbox/session teardown, terminalization, then promotion
• the recovery path is watchdog takeover after stale heartbeat, terminalization as runner_lost or the recovered terminal state, then promotion

The system does not require a positive sandbox acknowledgment in the watchdog recovery path before advancing the queue.

6. Idempotent D1 Reconciliation Algorithm

D1 reconciliation is upsert current truth by run_id.

d1_sync_status values in v1

• needs_create
• current
• needs_update
• needs_terminal_update
• done

Allowed d1_sync_status transitions

• needs_create -> current
• needs_create -> needs_update
• needs_create -> needs_terminal_update
• current -> needs_update
• current -> needs_terminal_update
• needs_update -> current
• needs_update -> needs_terminal_update
• needs_terminal_update -> done

done is a terminal D1 sync state. No transition out of done is permitted.

Create sync

When d1_sync_status = needs_create, ProjectDO upserts a queued row into D1 using the best current run truth.

Metadata sync

When d1_sync_status = needs_update, ProjectDO upserts the current non-terminal run truth into D1.

This status exists because a run's metadata may change after initial D1 creation but before terminal completion. The primary case is commit SHA backfill for manual runs (see Resolution 3): the queue consumer resolves the checked-out HEAD and records it back to ProjectDO, which must propagate the updated commit_sha to D1.

Terminal sync

When d1_sync_status = needs_terminal_update, ProjectDO upserts the terminal row into D1 using the latest terminal truth from ProjectDO and RunDO.

Required properties

• retries replay the same upsert behavior and are safe
• needs_create -> needs_terminal_update is valid and means a run became terminal before the initial D1 row was confirmed
• needs_create -> needs_update is valid and means metadata changed before the initial D1 row was confirmed; the metadata sync path uses the same upsert and will create the row
• a terminal run may therefore be reconciled by an initial queued upsert followed by a terminal upsert
• unique-key conflicts on run_id are resolved by the upsert, not by ad hoc insert-then-update branching
• a non-terminal D1 upsert must not regress a row that is already terminal in D1; the upsert must guard status, started_at, finished_at, and exit_code with a conditional that preserves existing terminal values
• commit_sha must use COALESCE semantics: a non-null value already in D1 is never overwritten with null

7. Queue Capacity Limit And Overflow Behavior

The per-project cap remains 20 accepted queued runs.

Manual triggers

• if the project already has 20 accepted queued runs, the private API returns 409 project_queue_full

Verified webhook triggers

• if the project already has 20 accepted queued runs, the public webhook route returns 429
• the verified delivery is recorded with outcome queue_full
• later duplicate deliveries with the same (project_id, provider, delivery_id) within the 72-hour dedupe window do not re-evaluate capacity, do not create a run, and return 2xx
• the stored delivery outcome remains queue_full for audit and UI purposes

Addendum: provider conflict resolution for queue_full webhook deliveries (2026-03-22)

This addendum supersedes the four bullets above and the generic duplicate-replay rule below for the specific case where a verified delivery reaches project queue capacity.

• queue_full is treated as a temporary receiver failure, not a terminal governance result.
• the public webhook route returns 503 Service Unavailable
• the response includes Retry-After: 60
• 429 is reserved for edge or rate-limit style throttling, not per-project queue exhaustion after webhook verification succeeds
• the verified delivery is still recorded durably with stored outcome queue_full
• a later delivery with the same (project_id, provider, delivery_id) may be re-evaluated while the stored outcome is queue_full
• if a later retry is accepted, anvil updates the stored row to accepted and attaches the resulting runId
• all other stored outcomes remain terminal within the 72-hour dedupe window and must replay without reprocessing

Rationale:

• GitLab documents that webhook receivers should use 4xx only for misconfigured webhooks and 5xx for temporary receiver failures; GitLab may permanently disable 4xx receivers but only temporarily back off 5xx failures
• GitHub documents that failed webhook deliveries are not redelivered automatically, but failures can be redelivered manually or through the API; using 503 still expresses temporary capacity exhaustion more accurately than a permanent-looking 4xx
• for v1, anvil prefers one consistent cross-provider contract for temporary queue exhaustion: return 503, keep the delivery id stable, and allow a later resend to settle the durable result

References:

• GitLab webhook receiver requirements and delivery headers: https://docs.gitlab.com/user/project/integrations/webhooks/
• GitHub failed delivery handling and webhook best practices: https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries and https://docs.github.com/webhooks/using-webhooks/best-practices-for-using-webhooks

8. Canonical Webhook Normalization Contract

Backend Phase 5 must use one canonical normalized payload before invoking ProjectDO acceptance logic.

The canonical payload must include at least:

• provider
• deliveryId
• normalized event kind
• raw provider event name
• repositoryUrl
• raw ref
• normalized branch
• commitSha
• beforeSha when the provider supplies it

This payload is the cross-provider contract for GitHub, GitLab, and Gitea.

9. Webhook Dedupe Semantics

Only verified webhook deliveries are written to the dedupe ledger.

All verified deliveries must be recorded in project_webhook_deliveries, not only deliveries that create runs.

The dedupe ledger must support explicit reason-specific stored outcomes, with a small normalized enum such as:

• accepted
• ignored_ping
• ignored_event
• ignored_branch
• queue_full

The implementation may additionally classify an inbound request as a duplicate at handling time, but it must preserve the originally recorded stored outcome for that delivery ID rather than overwriting it with duplicate.

Required behavior:

• dedupe is keyed by (project_id, provider, delivery_id)
• retention remains 72 hours
• ping/test events, filtered non-default-branch pushes, duplicates, and queue-full no-ops are recorded
• if a delivery was already recorded, duplicate handling replays the previously decided durable result rather than reprocessing the event

10. Webhook CRUD And Rotation Contract

The webhook configuration API uses a simple secure upsert model.

PUT /api/private/projects/:projectId/webhooks/:provider

• create the provider row when it does not exist, or update enabled / provider config when it already exists
• on create:
  • request body provides enabled
  • request body may provide a plaintext secret
  • if no secret is provided, Worker generates one and returns it exactly once in the create response
• on update:
  • request body may change enabled
  • request body may change provider config when applicable
  • request body must not include secret
  • inline secret replacement is rejected; secret changes use rotate-secret

Disabled rows

• disabling a webhook does not delete its stored encrypted secret
• disabled rows retain secret material until rotated or deleted

DELETE /api/private/projects/:projectId/webhooks/:provider

• hard-delete the webhook row and encrypted secret material

POST /api/private/projects/:projectId/webhooks/:provider/rotate-secret

• generate and persist a new secret
• return the new plaintext secret exactly once in the response
• there is no overlap window where both old and new secrets verify

11. Webhook Secret Custody And Verification Boundary

ProjectDO owns durable custody of webhook verification material.

This includes:

• encrypted-at-rest storage
• enablement state
• rotation state
• dedupe and post-verification governance decisions

The Worker owns ephemeral cryptographic use of webhook secrets at public ingress.

Required behavior:

• ProjectDO stores webhook secrets only in encrypted form
• Worker loads the minimal encrypted verification material from ProjectDO
• Worker decrypts the secret in memory
• Worker verifies the raw provider request using the provider signature scheme
• Worker does not persist plaintext webhook secret material
• after verification succeeds, Worker calls ProjectDO with only verified normalized payload

Rationale:

• this keeps durable secret custody with the project-local actor
• it keeps raw-request authentication at the Worker edge boundary where headers, body, WAF protection, and provider-specific ingress handling already live
• it avoids serializing decrypt-and-verify work inside ProjectDO for every webhook delivery

Default Implementation Notes

Until the main spec is updated:

• treat this file as authoritative for queue, run, and webhook contract decisions covered above
• prefer the current ProjectDO/RunDO implementation model where it already matches these resolutions