Skip to content
rachel/specter
File

Blob: gateway/endpoints.md

Markdown162 lines

Specter Internal Gateway Endpoints

This document describes the administrative endpoints exposed under /_internal on the apex domain.

These endpoints are:

  • Protected with HTTP Basic Auth (realm: internal).
  • Intended for operators only; do not expose publicly.
  • Available only when INTERNAL_USER and INTERNAL_PASS are configured.

Authentication

  • Configure credentials via environment variables:
    • INTERNAL_USER
    • INTERNAL_PASS
  • Access pattern (example):
    • https://<INTERNAL_USER>:<INTERNAL_PASS>@<apex-domain>/_internal/...
  • Requests without valid credentials receive 401 Unauthorized.
  • If credentials are not configured, the /_internal tree is disabled and requests return 404 Not Found.

Index

GET /_internal/

Local provider, vnode states, neighbors, counters, and observation time. No key scans or peer RPCs; busy membership references show as unavailable.

GET /_internal/overview.json returns the same local observation as JSON. Node identifiers are decimal strings; a null last_stabilized means it has not been observed. predecessor_available and successors_available distinguish an unavailable membership reference from an observed empty one. The observation is sampled across independent fields; it is not an atomic cluster snapshot.

GET /_internal/endpoints returns this documentation as plain text. Any unmatched /_internal/* path also returns this document for compatibility. Installations without an overview handler continue to return the documentation at the index.

Local assets at /_internal/ui/ use the same authentication.

ACME Certificate Management

Mounted under /_internal/acme when ACME is enabled.

  • POST /_internal/acme/clean

    • Triggers background cleanup in the ACME storage:
      • Removes old OCSP staples.
      • Deletes expired certificates (with a grace period).
    • Response: 204 No Content on success.

  • GET /_internal/acme/certs

    • Lists ACME issuers and the count of certificates for each.
    • Response: JSON object (application/json):
      • { "issuers": [{ "issuer": string, "count": number }, ...] }.

  • GET /_internal/acme/certs/{issuer}

    • Lists certificates for a specific issuer.
    • Response: JSON object (application/json):
      • { "issuer": string, "count": number, "certs": [{ "name": string, "domain": string }, ...] }.

  • GET /_internal/acme/certs/{issuer}/{name}

    • Returns detailed information for a single certificate:
      • Subject/issuer, validity period, SANs, key information, and optional metadata.
    • Response: JSON object (application/json).

  • DELETE /_internal/acme/certs/{issuer}/{name}

    • Deletes certificate, private key, and metadata for a specific certificate.
    • Response: 204 No Content whether or not the certificate already exists; 500 Internal Server Error if deletion fails.

Chord Ring Introspection

Mounted under /_internal/chord.

  • GET /_internal/chord/stats

    • Returns statistics about the Chord ring from this node’s perspective.
    • Format handling via chi URLFormat middleware:
      • /_internal/chord/stats or /_internal/chord/stats.txt → plain text (text/plain).
      • /_internal/chord/stats.html → HTML page that wraps the same text output.
    • This deeper diagnostic enumerates and exports local keys to report their ownership and sizes. Use the operator overview for inexpensive local status.
    • Optional query parameter:
      • ?key=<bytes> → returns the raw value for a specific KV key on this node (text/plain), or 404 if the key is not present.

  • GET /_internal/chord/graph

    • Returns a Graphviz/DOT representation of the ring topology:
      • Nodes in the ring.
      • Successor links.
      • Finger table edges.
      • Predecessor link from this node.
    • Response: text/plain DOT graph.

Tunnel Server Introspection

Mounted under /_internal/tun.

These are browser views. JSON data uses /_internal/api/tun/ and /_internal/api/tun/{id}/{address}, with the same authentication.

  • GET /_internal/tun/

    • HTML overview of connected tunnel clients:
      • Node identity/address.
      • Client identifier.
      • Version.
      • Observation timestamp and a link to the client's configuration and registration details.
    • Local connection list; no per-client Chord lookups.
    • Response: text/html page.

  • GET /_internal/tun/{id}/{address}

    • HTML view of tunnels for a specific client node.
    • Fetches running configuration and registered hostnames within one deadline.
    • Response: text/html page describing hostname → target mappings and separate configured/registered states. Names returned by either source are included.
    • Partial results remain visible; failed sources show Unknown.

Configuration Migrator

Mounted under /_internal/migrator when enabled.

  • GET /_internal/migrator/

    • Serves a helper HTML page explaining how to migrate legacy client configuration.

  • POST /_internal/migrator/

    • Accepts a v1 client configuration in YAML.
    • Requires a valid clientId and 44-character token; invalid input returns 400 Bad Request.
    • Returns a v2 configuration with a new client certificate/key and migrated tunnels.
    • Response: application/yaml.

Runtime Debug / Profiling

Mounted under /_internal/debug.

This tree is provided by github.com/go-chi/chi/v5/middleware.Profiler() and includes:

  • GET /_internal/debug/pprof/
  • GET /_internal/debug/pprof/cmdline
  • GET /_internal/debug/pprof/profile
  • GET /_internal/debug/pprof/symbol
  • GET /_internal/debug/pprof/trace
  • GET /_internal/debug/vars

The exact set may vary with the underlying net/http/pprof and expvar handlers, but it generally mirrors the standard Go pprof endpoints.

Internal Proxying Between Nodes

The /_internal tree can be accessed directly on a given node, or proxied via another node in the cluster.

  • When x-internal-proxy-node-address is set on the request headers, the gateway will:

    • Dial the target node over the INTERNAL stream.
    • Forward the HTTP request to that node’s /_internal handlers.
    • If proxying fails (for example, the node is unreachable), respond with 502 Bad Gateway and an error message.

  • The gateway adds x-internal-proxy-forwarded to prevent loops.

This allows you to query a specific node’s internal state (ACME, Chord, tunnels, etc.) through any other node that can reach it.