# Plan v2: public ledger, create-time PIN locks, and expiring pages Target branch: `feat/page-pin-locks`, cut from `master` at `7ad6b01` (PR #15 merged). This plan supersedes v1 of **Plan: Per-page PIN locks**. It incorporates the code review and the subsequent product/security decisions. It is intended to be executable by another agent without needing the design conversation that produced it. --- ## 1. Outcome and product contract Agent-pages becomes a public publish ledger whose page bodies may optionally be protected by a fixed six-digit PIN and whose pages may optionally expire. ### 1.1 Listing and access - The ledger at `/` and `/page/{n}` is public. It lists every active page, including PIN-locked pages. - A locked row exposes the page title, slug, format, update time, expiry (when set), and the fact that it is locked. The PIN protects the body, version history, and raw representation—not the existence or title of the page. - An open page requires no viewer credential. - A locked page uses exactly six ASCII digits (`^[0-9]{6}$`). PIN length is fixed in code and is not configurable. - PINs are set only when a page is created. They cannot be added, removed, or changed later. - Content/title updates preserve the original access state and expiry. ### 1.2 MCP publication contract `publish_page` gains `locked: bool = False`, `pin: str | None = None`, and `expiry` (default `never`). Its behavior is exact: | `locked` | `pin` | Result | |---|---|---| | omitted/false | omitted | Publish an open page | | true | omitted | MCP server generates a cryptographically random six-digit PIN, publishes a locked page, and returns URL **and PIN** | | omitted/false | supplied | Supplying a PIN implies locked; publish using the supplied PIN | | true | supplied | Publish locked using the supplied PIN | The MCP server—not the language model—generates a missing PIN: ```python generated_pin = f"{secrets.randbelow(1_000_000):06d}" ``` This includes leading zeroes. A successful generated-PIN tool result must say, plainly: ```text Published locked page at https://… PIN: 004821 ``` It must also include `generated_pin` in structured tool output. `generated_pin` appears only when MCP generated it; a user-supplied PIN is not echoed redundantly. Later API/MCP reads never return a PIN. Generate once before the HTTP call and do not automatically retry an ambiguous publish transport failure. If the MCP-to-API result is ambiguous, the tool error should say the outcome is unknown and repeat the generated PIN so the operator can recover the newest page through `list_pages`. Idempotent publish remains out of scope. ### 1.3 Direct API publication contract `POST /v1/pages` accepts optional `pin` and `expiry`. The direct API does not generate PINs; callers wanting a locked page must supply one. The MCP server resolves `locked` into a concrete PIN before calling the API. Examples: ```json { "title": "Open", "format": "markdown", "content": "…" } ``` ```json { "title": "Locked", "format": "html", "content": "…", "pin": "004821", "expiry": "1_week" } ``` Accepted expiry values: | Value | Stored deadline | |---|---| | omitted / `never` | `NULL` | | `1_day` | accepted-at time + 24 hours | | `1_week` | accepted-at time + 7 days | | `1_month` | accepted-at time + 30 days | | `1_year` | accepted-at time + 365 days | Store an absolute UTC `expires_at`; do not store the duration label. Updates cannot extend or change it. ### 1.4 Expiration contract As soon as `expires_at <= now`, before physical cleanup runs, a page: 1. Disappears from the public ledger and its counts. 2. Disappears from authenticated API/MCP `get_page` and `list_pages`. 3. Returns the normal styled **404**, not 410, from latest, historical, raw, and unlock URLs. This deliberately does not confirm that the page used to exist. 4. Rejects updates with API 404. 5. Performs no R2 body read on any rejected viewer request. Physical deletion is a separate, retryable CLI operation run by host cron. There is no scheduler or background cleanup loop inside FastAPI. ### 1.5 Editing authority and PIN boundary The PIN is a viewer credential only. It is not an ownership or edit credential. - `PUBLISH_API_TOKEN` remains instance-wide administrative authority. - The MCP client needs `AGENT_PAGES_MCP_AUTH_TOKEN`; the MCP service uses `AGENT_PAGES_API_TOKEN`/`PUBLISH_API_TOKEN` downstream. - Anyone with a usable publishing path can call authenticated `list_pages`, obtain UUIDs, and update any unexpired page without its PIN. - The public ledger exposes slugs, not page UUIDs. - A slug, a PIN, or both never authorize an update. - Updates remain `PUT /v1/pages/{page UUID}` and require the publish bearer. This is accepted for the current single-owner personal instance. Per-user ownership, per-page edit secrets, create-only tokens, and multi-tenant authorization are out of scope and must not be implied by the PIN feature. --- ## 2. Threat model and security choices ### 2.1 Six-digit PINs A uniformly generated PIN has 1,000,000 possibilities (~19.9 bits). User-provided PINs may have lower effective entropy; accept every six-digit value, including `000000`, without a weak-PIN blocklist. Document that generated PINs are preferable for sensitive pages. The online defenses are per-client throttling and bounded cryptographic work. There is deliberately no instance-wide per-page counter: a global page counter would let any anonymous visitor lock everyone out by spending the page budget repeatedly. ### 2.2 Two client-scoped rate limits Use the existing atomic fixed-window SQLite implementation and `publish_rate_limits` table, generalized enough that PIN error messages do not say “publish.” Consume both counters before scrypt verification: 1. **Page + client IP:** 5 attempts per 15 minutes. - Key: `sha256("pin-unlock-page-ip:" + page_id + ":" + client_ip)`. - Limits brute force against one page from one network identity. 2. **Global client IP:** 30 attempts per 15 minutes across all pages. - Key: `sha256("pin-unlock-ip:" + client_ip)`. - Bounds one address attempting many pages and bounds local resource consumption. Both count submitted unlock attempts, including a successful one. Once unlocked, the cookie avoids further submissions. Check the global-IP budget first, then page+IP, so a client sweeping pages cannot create/use a fresh page bucket to bypass the global ceiling. Suggested settings (PIN length itself is not a setting): ```python pin_unlock_rate_limit_max = 5 pin_unlock_rate_limit_global_max = 30 pin_unlock_rate_limit_window_seconds = 900 pin_unlock_hash_concurrency = 4 pin_unlock_hash_wait_seconds = 0.25 pin_unlock_cookie_ttl_seconds = 604800 # 7 days ``` Environment aliases should follow the existing uppercase pattern. Validate every count/window/concurrency value as >= 1 and wait as >= 0. Move `resolve_client_ip()` and trusted-proxy helpers out of `app/viewer/session.py` into a neutral module such as `app/client_ip.py`, because the index session is being retired but PIN limiting still needs the reviewed rightmost-walked XFF logic. - Untrusted peers cannot control XFF. - A trusted proxy with missing/unusable XFF uses its peer address as a shared bucket with the same conservative ceilings. - A missing `request.client` becomes an explicit shared `unknown` bucket; it must not receive a looser allowance. - Preserve/add the production warning when HTTPS is configured while `TRUSTED_PROXY_CIDRS` is empty. Log during application startup/lifespan, not as an import side effect. ### 2.3 Rate-limit row retention Every distinct rotating IP creates counter keys. Expired fixed-window rows do not disappear automatically. Reuse the indexed probabilistic purge and calculate retention using every feature sharing the table: ```python retention_seconds = max( settings.publish_rate_limit_window_seconds, settings.pin_unlock_rate_limit_window_seconds, ) ``` The index sign-in window disappears when index auth is removed. Page+IP keys also rotate, so invoke best-effort cleanup from the PIN attempt path. Cleanup failure must never turn an otherwise valid attempt into 500. ### 2.4 Bounded scrypt work Scrypt is intentionally memory-hard. With the chosen parameters each operation uses roughly 16 MiB. A rate limit controls requests over time but does not prevent many first requests from arriving simultaneously across different IPs/pages. Add one process-wide `threading.BoundedSemaphore(4)` around both hash creation and verification. This deployment is one API process, so four slots bound concurrent expected scrypt memory near 64 MiB. Unlock order: 1. Validate six ASCII digits before any expensive work. 2. Consume global-IP and page+IP counters. 3. Acquire a scrypt slot with a short bounded wait (default 250 ms). 4. If no slot is available, return a retryable 503 with `Retry-After: 1`; the already-consumed attempt remains consumed. 5. Run scrypt and constant-time comparison. 6. Release the semaphore in `finally` on every path. Publishing a locked page also uses the semaphore. An authenticated publish that cannot acquire capacity returns a retryable 503 and must not upload to R2 or create DB metadata. If multi-worker/multi-host deployment is introduced later, the limit is per process and must be divided by worker count or moved to shared infrastructure. ### 2.5 PIN hashing Add `app/pins.py` with a self-describing stored format: ```text scrypt$16384$8$1$$ ``` Use: ```python N = 2**14 R = 8 P = 1 DKLEN = 32 SALT_BYTES = 16 ``` - Use `hashlib.scrypt`, a random per-page salt, strict base64 decoding, and `hmac.compare_digest` on derived bytes. - Reject malformed/unknown stored formats safely rather than raising into a viewer 500. - Bound parsed numeric parameters before invoking scrypt so corrupt/malicious DB values cannot request extreme memory. Accept only the reviewed parameter set in v1. - Never log a supplied PIN, hash, cookie, or generated derived key. - A DB leak still permits offline enumeration of a six-digit PIN; scrypt raises its cost but is not equivalent to a high-entropy secret. ### 2.6 Unlock cookie Use one page-scoped opaque cookie: ```text name: agent_pages_unlock value: {expires_at}.{HMAC(pin_hash, "agent-pages-unlock-v1|" + slug + "|" + expires_at)} path: /p/{slug} flags: HttpOnly; SameSite=Lax; Secure when PUBLIC_BASE_URL is HTTPS TTL: 7 days, no sliding renewal ``` The server enforces the signed absolute expiry, not only `Max-Age`. Changing a PIN is unsupported; physical deletion removes the validating row. Bind the slug into the MAC and compare cookies in constant time. Keep the cookie path without a trailing slash so it reaches `/p/{slug}` as well as nested version/raw routes. Add the RFC 6265 prefix test proving a cookie for `foo-a1b2c3d4` is not sent to `foo-a1b2c3d4-e5f6g7h8`. --- ## 3. Data model ### 3.1 `pages` additions ```python pin_hash: Mapped[str | None] = mapped_column(String(255), nullable=True) expires_at: Mapped[datetime | None] = mapped_column( DateTime(timezone=True), nullable=True, index=True ) purge_started_at: Mapped[datetime | None] = mapped_column( DateTime(timezone=True), nullable=True ) ``` - `pin_hash IS NULL` means open; non-NULL means locked. - PIN is page-level, so every historical version is protected consistently. - `expires_at IS NULL` means never expires. - `purge_started_at` is a cleanup claim/lease and update-race guard. - Do not add `pin_set_at`: PINs are create-only and it has no consumer. The legacy `visibility` column remains physically present for migration compatibility but is deprecated and ignored for listing/access decisions. Add comments/docs making clear that locked pages are still publicly listed. Do not undertake a SQLite table rebuild merely to drop it in this change. ### 3.2 Durable deletion record and attempts Use two tables so the unique slug reservation and append-only attempt history do not conflict. `page_deletion_logs` — one durable record per expired page: ```text id UUID primary key page_id UUID snapshot, no FK (source row is deleted) slug VARCHAR(128) UNIQUE NOT NULL expires_at DATETIME NOT NULL first_attempted_at DATETIME NOT NULL completed_at DATETIME NULL status pending | deleted version_count INTEGER NOT NULL objects_deleted INTEGER NOT NULL default 0 attempt_count INTEGER NOT NULL default 0 last_error_summary TEXT NULL, capped/sanitized ``` `page_deletion_attempts` — append-only details for each CLI attempt: ```text id INTEGER primary key/autoincrement deletion_log_id FK page_deletion_logs.id, RESTRICT attempted_at DATETIME NOT NULL completed_at DATETIME NOT NULL status succeeded | failed objects_attempted INTEGER NOT NULL objects_deleted INTEGER NOT NULL error_summary TEXT NULL, capped/sanitized ``` Do not store page bodies, PINs, PIN hashes, cookies, or full exception traces in either table. Application logs may carry tracebacks; DB error summaries must exclude credentials and be capped. `page_deletion_logs.slug` is the permanent tombstone. Slug allocation checks active `pages.slug` and tombstoned `page_deletion_logs.slug`. An expired URL can therefore never resolve to unrelated future content. ### 3.3 Expiration helper Centralize UTC/SQLite-naive handling and active predicates. A page is active only when: ```python Page.expires_at.is_(None) | (Page.expires_at > now) ``` Use the same definition in viewer lookup, API get/list, index stats/list, and update checks. Cleanup deliberately queries the inverse. Avoid scattering subtly different comparisons across modules. --- ## 4. Migration mechanics `create_all()` does not add columns to existing SQLite tables. Extend `agent-pages init-db` after `Base.metadata.create_all()`: 1. Inspect `PRAGMA table_info(pages)`. 2. Add missing nullable columns individually: ```sql ALTER TABLE pages ADD COLUMN pin_hash VARCHAR(255); ALTER TABLE pages ADD COLUMN expires_at DATETIME; ALTER TABLE pages ADD COLUMN purge_started_at DATETIME; ``` 3. Create the expiry index explicitly for existing volumes: ```sql CREATE INDEX IF NOT EXISTS ix_pages_expires_at ON pages (expires_at); ``` 4. Let `create_all()` create both new deletion tables on fresh and existing databases; explicitly assert required indexes/uniques in tests. 5. Preserve the existing explicit `publish_rate_limits.window_start` index creation. No data backfill is needed: - Existing `pin_hash = NULL` → open. - Existing `expires_at = NULL` → never expires. - Existing `purge_started_at = NULL` → not being purged. - The now-public ledger lists every existing active test page immediately. Run `init-db` twice in the migration test. Both runs must succeed and leave one copy of every column/index/table. This is another hand-written schema evolution. Record in `MEMORY.md` that the next nontrivial schema change should introduce Alembic rather than extending the DDL block again. --- ## 5. API and service changes ### 5.1 Schemas Add: ```python ExpiryOption = Literal["never", "1_day", "1_week", "1_month", "1_year"] ``` `PageCreateRequest` gains: ```python pin: str | None = None expiry: ExpiryOption = "never" ``` Validate PIN with an ASCII-specific full match, not `\d`, which accepts Unicode digits. Continue stripping titles and enforcing UTF-8 body size. `PageSummary` gains: ```python has_pin: bool expires_at: datetime | None ``` `PageResponse` gains the same two fields. No schema contains `pin_hash`, raw PIN, `purge_started_at`, or deletion internals. `PageUpdateRequest` does **not** gain PIN or expiry fields. Unknown fields remain rejected according to the chosen Pydantic model policy; add an explicit test that attempts to mutate either cannot silently succeed. ### 5.2 Create service In `create_page`: 1. Consume the authenticated publish limit. 2. Convert expiry label to one absolute UTC timestamp using a captured `now`. 3. If PIN supplied, acquire the bounded scrypt slot and hash before any R2 upload. Capacity failure is retryable and leaves no object/row. 4. Plan and upload version 1 using the existing unique R2 key and compensation discipline. 5. Generate the slug and check both active page and deletion-log reservations. 6. Insert Page, including `pin_hash` and `expires_at`, plus PageVersion atomically. 7. Retain unique-conflict retries for concurrent slug allocation. Define a specific `SlugAllocationError`. After the existing bounded retry count is exhausted, compensate the uploaded object and return a retryable API response such as 503 with an actionable detail (`Could not allocate a unique page URL; retry the publish`). MCP must preserve that meaning. Do not return a misleading content-validation error. ### 5.3 Update service Updates continue to create immutable content versions. Add lifecycle checks twice: 1. Before the R2 upload, reject missing, expired, or `purge_started_at != NULL` as `PageNotFoundError`. 2. After upload, when reacquiring the page for the version commit, recheck expiry against a fresh `now` and confirm `purge_started_at IS NULL`. If it became expired/claimed during network I/O, roll back and compensate only the newly uploaded unique key. Updates preserve `pin_hash` and `expires_at` exactly. They do not renew the unlock cookie, change access, or extend lifetime. ### 5.4 Authenticated get/list - `GET /v1/pages/{id}` returns 404 for expired/claimed/deleted pages. - `GET /v1/pages` excludes expired/claimed pages and its `total` counts only active pages. - Responses expose `has_pin` and `expires_at`, never secret material. ### 5.5 Authority tests Keep and explicitly test the boundary: - Slug only → cannot update. - Slug + correct viewer PIN → cannot update. - UUID without publish bearer → 401. - UUID with wrong bearer → 401. - UUID with valid publish bearer → update succeeds without viewer PIN while active. - Valid bearer + expired UUID → 404. - Public ledger markup does not contain page UUIDs. --- ## 6. Public ledger migration Remove the opt-in visibility switch and make `/` public in this deployment. ### 6.1 Remove index authentication surface - `/` and valid `/page/{n}` render the ledger without a cookie. - Remove sign-in and sign-out controls/templates from navigation. - Retire `POST /session` and `POST /session/signout`; they should fall through to the styled 404. - Remove `AGENT_PAGES_INDEX_TOKEN`, index cookie signing, and index sign-in rate-limit settings/docs. - Delete index session code after moving reusable client-IP resolution to `app/client_ip.py`. - Preserve path-based pagination, UTC grouping, the `Page.id` secondary sort key, mobile table semantics, and no-JavaScript behavior. ### 6.2 Ledger data and presentation - Index queries and `index_stats` filter active, non-purging pages using the shared predicate. - Locked rows get a mono lock sigil plus real accessible text (`Locked`); do not rely on glyph/color alone. - Show expiry in UTC when present and `No expiry`/omit the marker when absent. Keep the compact mobile layout usable and update explicit grid-row assignments and ARIA roles if a column is added. - The format remains the only format color. A lock/expiry marker should not introduce a competing semantic palette. - Keep `X-Robots-Tag: noindex, nofollow, noarchive`. Publicly reachable does not mean search-engine discovery. - The anonymous page/count secrecy rules are intentionally retired. The physical legacy `visibility` column is not consulted. Update AGENTS/MEMORY language from “unlisted” to “public ledger; optionally body-locked.” --- ## 7. Viewer gating and unlock flow ### 7.1 Gate every body-bearing route All four routes must enforce active lifecycle and PIN access before R2: | Route | Active open | Active locked, no cookie | Active locked, valid cookie | Expired | |---|---|---|---|---| | `/p/{slug}` | 200 | 303 unlock | 200 | styled 404 | | `/p/{slug}/v/{n}` | 200 | 303 unlock | 200 | styled 404 | | `/p/{slug}/raw` | normal raw behavior | raw-styled 403 | 200 for HTML | raw-styled 404 | | `/p/{slug}/v/{n}/raw` | normal raw behavior | raw-styled 403 | 200 for HTML | raw-styled 404 | Check lifecycle before access so an unlock page or cookie cannot reveal an expired page. Check access before `storage.get_object()`. Refactor `load_page_view` into metadata resolution plus explicit body loading, or otherwise provide one auditable boundary; do not fetch metadata twice merely to avoid the refactor. HTML host behavior remains two requests: the unlocked outer shell loads immutable `/v/{n}/raw`; the raw request independently verifies the cookie. ### 7.2 Unlock routes Add `app/viewer/unlock.py`: | Route | Behavior | |---|---| | `GET /p/{slug}/unlock` | Active locked page → form; active open/already unlocked → 303 validated `next` or canonical URL; missing/expired/claimed → styled 404 | | `POST /p/{slug}/unlock` | Correct → cookie + 303 validated same-page target; wrong → 401 form; limited → 429; hash capacity busy → 503; missing/expired → 404 | The form may show title and locked state, since both are already public. It shows no body, versions, PIN hint, hash detail, or expiry-internal state. Use `inputmode="numeric"`, a six-character limit/pattern, and accessible error text. Server validation remains authoritative. Validate `next` more narrowly than the global theme helper: it must be the canonical, historical, raw, or historical raw URL for the same slug. Reject other pages and all schemes, hosts, query strings, fragments, backslashes, and control characters. Default to `/p/{slug}`. Use the existing form-enabled shell CSP after renaming it from home-specific terminology if helpful. Only unlock and any remaining legitimate shell form may use `form-action 'self'`. Raw/artifact CSP remains unchanged: `sandbox`, `script-src 'none'`, `font-src 'none'`, `form-action 'none'`. ### 7.3 Cache safety Every `/p/*` response—open or locked, shell or raw, success/error/redirect/unlock—gets: ```http Cache-Control: private, no-store Vary: Cookie ``` This prevents unlocked content crossing credential states and prevents cached open content surviving `expires_at`. Preserve base security headers and CSP on every response. Ensure global error handlers add the same cache policy for `/p/*` validation, 404, 405, and 500 responses. ### 7.4 Browser verification Before declaring the HTML path complete, verify in real Chrome and Firefox that the page-scoped SameSite=Lax cookie is attached to the same-site request for an iframe with `sandbox=""`. The expected behavior is that sandboxing makes the resulting document origin opaque but does not strip cookies from the same-site network request. If measurement disproves that expectation, stop and redesign; do not silently add unlock credentials to query strings. Any token fallback requires separate review because URLs leak through copying/history/logs. --- ## 8. MCP client and tools ### 8.1 `publish_page` FastMCP signature: ```python async def publish_page( title: str, format: PageFormat, content: str, locked: bool = False, pin: str | None = None, expiry: ExpiryOption = "never", ) -> CallToolResult: ``` Resolution: ```python generated_pin = None effective_pin = pin if locked and effective_pin is None: generated_pin = f"{secrets.randbelow(1_000_000):06d}" effective_pin = generated_pin # effective_pin being supplied implies locked even when locked=False ``` Forward `pin` only when `effective_pin` is non-NULL and forward expiry. API validation remains canonical. Do not implement scrypt or database logic in MCP; it stays a thin HTTP client. Successful structured content includes normal page metadata (`id`, `slug`, `url`, `version`, `has_pin`, `expires_at`) and conditionally `generated_pin`. Ensure the human-readable result prominently returns the generated PIN with the URL. Tool docs warn that supplied/generated PINs appear in the agent transcript and must be shared appropriately. ### 8.2 `update_page` Do not add PIN or expiry parameters. Its docstring explicitly says: - requires UUID and publishing authority; - cannot change PIN/expiry; - fails after expiry; - creates a new immutable version. ### 8.3 `get_page` and `list_pages` Surface `has_pin` and `expires_at`; exclude expired pages according to the API. Keep page UUIDs in authenticated MCP results. Document that publishing credentials are administrative and permit updates without viewer PINs. --- ## 9. Physical cleanup CLI and host cron ### 9.1 Command Add: ```bash agent-pages purge-expired --batch-size 100 --claim-timeout-seconds 3600 ``` Exit behavior: - `0` when the batch completes with no failures, including nothing due. - nonzero when any claimed page fails cleanup, so cron monitoring can alert. - Print concise structured/log-friendly totals: examined, claimed, deleted, failed, objects deleted. - Never print bodies, PINs, hashes, cookies, or R2 credentials. Document a host cron example (frequency may be adjusted operationally): ```cron */15 * * * * cd /path/to/agent-pages && docker compose exec -T app agent-pages purge-expired --batch-size 100 ``` The app container already has the SQLite volume and R2 credentials. No second long-running Compose service and no in-app scheduler are required. ### 9.2 Claim and retry algorithm For each eligible page in deterministic expiry/id order: 1. In a short SQLite transaction, conditionally claim it only when `expires_at <= now` and `purge_started_at` is NULL or older than the claim-timeout cutoff. Set `purge_started_at = now` atomically. A stale lease allows recovery after process death. 2. Create or load the durable `page_deletion_logs` row for page ID/slug. The unique slug begins permanent reservation immediately. Increment/record attempt state without losing prior attempts. 3. Read every committed PageVersion R2 key and expected count after the claim. 4. Delete every key. Treat an already-missing object as success; R2 deletion is idempotent. Continue attempting remaining keys after one failure so the attempt record is complete. 5. Append one `page_deletion_attempts` row with sanitized result counts/error. 6. On any object failure: update the durable log to pending/error, clear `purge_started_at` (or allow lease expiry if the DB write itself fails), commit the attempt, and retain Page/PageVersion metadata for retry. 7. On total R2 success: in one DB transaction mark the durable log deleted/completed, append the successful attempt, and delete Page; versions cascade. The tombstone remains forever. Crash properties: - Crash before claim commit: another run claims normally. - Crash after claim: stale lease reclaims. - Crash after some/all R2 deletes: retry treats missing keys as success. - Crash before Page deletion: metadata retains the authoritative key list. - DB failure after R2 success: retry repeats idempotent deletes, then finalizes. Never delete the Page row before every R2 key is successfully removed; doing so loses the cleanup manifest. ### 9.3 Update/cleanup race The cleanup claim and update’s final lifecycle recheck form the boundary: - Cleanup claims only expired pages. - An update started before expiry may upload outside the DB lock. - At commit, update rechecks current time and `purge_started_at`. If the page expired or was claimed, it aborts and compensates its unique new R2 key. - Cleanup sees only committed PageVersions after its claim. Add a deterministic test that pauses update between upload and commit, claims expiry, resumes update, and proves the new object is compensated and never becomes an orphan/version. --- ## 10. Slug allocation and permanent reservation Current slugs are readable title bases plus eight random lowercase/digit characters (`36^8` possibilities). Preserve this format. Allocation must reject a candidate present in either: ```text pages.slug page_deletion_logs.slug ``` Keep database uniqueness as the final concurrency guard. Because the tables are separate, perform both existence checks during each retry; the deletion-log unique constraint permanently protects tombstones. A creator cannot submit a slug through API/MCP. Tests monkeypatch slug generation to cover: - collision with an active page → retry and succeed; - collision with a pending/deleted audit slug → retry and succeed; - all retries collide → no DB page/version, uploaded object compensated, meaningful retryable API/MCP error; - after physical deletion, requesting the old viewer URL remains 404 while a later same-title page receives a different slug. --- ## 11. Detailed test plan Create focused files rather than one monolith: ### 11.1 `tests/test_pins.py` - Hash round trip for values including `000000` and `999999`. - Different salts produce different stored strings for the same PIN. - Wrong PIN rejects. - Non-six-digit/API Unicode-digit input rejects before hashing. - Malformed, truncated, unknown-algorithm, invalid-base64, and out-of-bounds-parameter stored hashes reject safely. - Comparison uses derived bytes; patch/spy where practical. - Semaphore releases after success and exception. - Fifth concurrent operation cannot enter when four slots are occupied and receives the retryable capacity result. ### 11.2 `tests/test_pin_publish.py` - API no PIN → `has_pin=false`, DB NULL, page open. - API valid PIN → `has_pin=true`, DB stores a hash that contains neither plaintext PIN nor recoverable raw value. - API rejects five/seven digits, whitespace, signs, Unicode numerals, numbers instead of strings. - Every expiry option maps to the exact fixed duration; omitted/never stores NULL. - Response includes `has_pin` and `expires_at`, never PIN/hash. - Hash-capacity failure happens before R2 put and DB insert. - Content/title limits and R2 compensation behavior remain unchanged. ### 11.3 MCP tests (`tests/test_mcp_server.py` and client tests) - false/omitted locked + no PIN forwards no PIN and publishes open. - locked=true + no PIN patches `secrets.randbelow` to 4821, forwards `004821`, and returns URL plus `PIN: 004821` and structured `generated_pin`. - supplied `004821` with locked omitted implies locked, forwards exactly it, and does not add `generated_pin` to output. - locked=true + supplied PIN does not call generator. - expiry is forwarded exactly. - invalid supplied PIN error comes from API without MCP-side storage logic. - generated PIN is generated once and no automatic HTTP retry occurs. - ambiguous transport error identifies unknown outcome and preserves generated PIN in the error guidance. - update tool has no PIN/expiry mutation arguments. - get/list surface `has_pin`/`expires_at` and retain UUIDs only on authenticated tool access. ### 11.4 `tests/test_pin_unlock.py` For each of the four viewer routes: - open active page remains normal; - locked/no cookie produces the specified 303 or raw 403; - locked/forged, expired, truncated, other-page cookie rejects; - locked/valid cookie produces expected content; - page A cookie rejects for page B; - current and every historical version use the same page PIN; - HTML host iframe raw request succeeds with cookie in server tests; - unlock GET open/already unlocked redirects to validated same-page target; - unknown and expired unlock URLs return indistinguishable styled 404s; - invalid next target defaults/rejects safely and cannot redirect to another slug. Instrument `FakeR2Storage` with `get_calls`; `objects` alone does not record reads. Assert `get_calls == []` whenever lifecycle/access gates reject before body fetch. ### 11.5 `tests/test_pin_rate_limit.py` - Page+IP allows exactly five attempts, sixth gets 429 with Retry-After. - Different IP on same page has an independent budget; there is no global per-page lockout. - Same IP on different page has a separate page+IP budget. - Global IP ceiling blocks the 31st attempt across many pages. - Both budgets count a successful submitted attempt. - Rate-limit rejection never calls scrypt. - Hash-capacity rejection returns 503 and releases no extra slot. - Trusted rightmost XFF resolution, spoofed prepended XFF, missing/invalid XFF shared bucket, untrusted peer, IPv4/IPv6, and unknown request client behave as designed. - Rotating-IP rows are purged after the longest active retention window; purge failure is best effort. - Concurrent attempts cannot exceed atomic fixed-window counts. ### 11.6 `tests/test_expiry.py` At `expires_at - epsilon`, `expires_at`, and `expires_at + epsilon`, cover: - public ledger presence/counts; - API get/list/total; - MCP behavior through mocked API; - latest Markdown and HTML host; - historical version; - latest raw and historical raw; - unlock GET/POST; - update acceptance/rejection; - no R2 read on expired viewer paths; - all expired viewer paths are styled 404, never 410 and never reveal expiry. Use injected/captured time helpers rather than fragile wall-clock sleeps. ### 11.7 `tests/test_purge_expired.py` - No-op batch exits success and writes no audit rows. - Active and never-expiring pages are untouched. - Eligible page is claimed, every version object deleted, Page/versions removed, durable log marked deleted, successful attempt appended. - One R2 failure retains metadata/key manifest, records failed attempt/error summary, and retries successfully later. - Already-missing R2 key is treated as success. - Crash/stale `purge_started_at` lease can be reclaimed; fresh lease cannot. - DB finalization failure after R2 success is safely retryable. - Batch size and deterministic order work across multiple runs. - PIN/hash/body/R2 credentials are absent from audit records and captured logs. - Update-versus-cleanup race compensates only the update’s unique key. - CLI exit code nonzero when any page fails and zero otherwise. - Durable log slug is unique; attempt rows are append-only. ### 11.8 `tests/test_slug_reservation.py` - Active collision retry. - Pending/deleted tombstone collision retry. - Retry exhaustion maps to actionable retryable response and R2 compensation. - Deleted slug is never reused. ### 11.9 Public ledger/session tests Rewrite `test_home_index.py`/`test_home_session.py` expectations: - Anonymous `/` renders rows and counts. - Anonymous path pagination works and preserves tie-break ordering. - Existing pages are public immediately after migration. - Locked marker has accessible text; public markup contains slug/title but no UUID, PIN, or hash. - Expired/claimed rows and counts disappear. - `/session` and `/session/signout` are styled 404s. - No index cookie or token environment dependency remains. - Mobile grid/table ARIA structure remains correct. ### 11.10 Security/cache/CSP tests - Every `/p/*` response class includes `Cache-Control: private, no-store` and `Vary: Cookie`: open, locked, unlock, raw, historical, redirects, 401/403/404/405/422/429/500. - Unlock CSP allows `form-action 'self'` but no script. - Raw/artifact CSP remains `sandbox`, `script-src 'none'`, `font-src 'none'`, `form-action 'none'` and never loads shell assets. - Markdown/Mermaid/HTML sandbox boundaries remain unchanged. - Security headers remain on public ledger and cleanup-independent errors. - Correct PIN cannot authorize API update; only valid publish bearer can. ### 11.11 Migration tests Construct a pre-feature SQLite database, insert representative existing Page/PageVersion rows and fake R2 bodies, then run `init_db()`: - New columns/tables/indexes exist. - Existing `pin_hash`, `expires_at`, and `purge_started_at` are NULL. - Existing pages are open, public in ledger, and bodies reachable. - Running `init_db()` twice is idempotent. - New tombstone uniqueness and attempt FK behavior work. ### 11.12 Manual browser acceptance In Chrome and Firefox: - locked Markdown unlock/read; - locked HTML unlock → sandboxed iframe receives cookie and renders body; - fullscreen immutable raw works after unlock; - wrong PIN, rate-limit, expiry, and 404 presentation; - light/dark/system shell themes; - cookie path prefix isolation; - back/refresh does not reveal cached body after expiry (also inspect response headers). --- ## 12. Success criteria The feature is complete only when all are true: 1. Anonymous visitors can browse the active ledger without an index credential; legacy test pages remain open and visible after migration. 2. Open-by-default publication works through direct API and MCP. 3. A supplied six-digit PIN implies a locked page; invalid PIN shapes are rejected. 4. `locked=true` without a PIN causes MCP to generate a zero-padded cryptographically random six-digit PIN and return it prominently with the URL exactly once. 5. No plaintext PIN or PIN hash appears in DB plaintext fields, later responses, application logs, ledger HTML, or deletion audit data. 6. All current/historical/raw body routes gate locked content before R2 and unlock with one page-level cookie. 7. Expiration is enforced logically at the exact deadline across viewer, index, API, MCP, and updates, always presenting missing-page 404 semantics. 8. Every `/p/*` response is `private, no-store` and varies on Cookie; CSP/sandbox protections do not regress. 9. PIN attempts are bounded per page+IP (5/15m) and globally per IP (30/15m), with no globally attacker-consumable page lockout. 10. At most four scrypt operations run concurrently in the single API process and capacity exhaustion is retryable without memory growth. 11. Rate-limit rows from rotating IPs are eventually purged without affecting active counters or request success. 12. `purge-expired` removes all immutable R2 versions before deleting metadata, is crash/retry safe, exits observably, and records durable sanitized audit/attempt rows. 13. Deleted slugs remain reserved forever and can never resolve to different future content. 14. An update/expiry/cleanup race leaves no orphaned R2 object and never resurrects an expired page. 15. Viewer PINs never grant edit authority; update remains protected by instance publish credentials and UUID. 16. MCP remains an HTTP-only thin client with no DB/R2 imports. 17. Full `python -m pytest` passes, new migration tests pass from a pre-feature schema, and Chrome/Firefox iframe-cookie acceptance is recorded. --- ## 13. Recommended implementation sequence Keep commits reviewable and preserve security boundaries at every step: 1. **Browser spike:** verify cookie attachment to opaque sandboxed iframe in Chrome/Firefox; record result. 2. **Models + migration:** Page columns, deletion log/attempt tables, indexes, old-DB idempotence tests. 3. **Lifecycle helpers:** central active/expired predicate/time handling; service query tests. 4. **PIN module:** fixed validation, scrypt serialization/parser bounds, semaphore, unit tests. 5. **Create/API contract:** pin hash + expiry calculation + response metadata + collision/tombstone checks. 6. **MCP contract:** locked/pin decision table, secure generation, one-time result, client forwarding/tests. 7. **Public ledger:** remove index gate/session, move IP resolver, public active filtering, lock/expiry UI. 8. **Unlock route/cookie/CSP:** form, validation, cookies, same-page next policy. 9. **Four-route viewer gate:** lifecycle then PIN then R2; cache policy across all `/p/*` responses. 10. **Rate limiting:** page+IP/global-IP counters, retention purge, trusted proxy warning, semaphore capacity errors. 11. **Update lifecycle:** pre/post upload expiry/purge checks and race compensation. 12. **Cleanup CLI:** claim lease, R2 deletion, durable log + append-only attempts, tombstone reservation, cron docs. 13. **Integration/security tests:** expiry matrix, authority boundary, cache/CSP, cleanup races, manual browsers. 14. **Docs/decisions:** update AGENTS.md, MEMORY.md, README, architecture, `.env.example`, Compose/host cron guidance. 15. **Final verification:** full tests, lint/type checks used by repo, clean secret scan, manual acceptance record. Steps 1–10 are the viewer/security core. Do not expose PIN-locked pages before all four routes, cache headers, and rate/concurrency controls land together. --- ## 14. Documentation and locked-decision updates Update: - **AGENTS.md:** Privacy/Index/Index-auth/limits/retention decisions, routes, expiry CLI, new models, and do-not-regress rules. - **MEMORY.md:** public listing is intentional; PIN is create-only/read-only access control; six fixed digits; MCP generation; per-client limits without global page lockout; four scrypt slots; expiry 404 semantics; permanent slug tombstones; audit structure; instance-wide publish authority remains admin. - **docs/architecture.md:** viewer lifecycle/PIN boundary before R2, public ledger, cleanup state flow, MCP generation boundary, credential diagram. - **README.md / `.env.example`:** rate/concurrency/cookie settings, public index behavior, expiry options, CLI and cron, trusted proxy security, credential authority warning. - **Docker/operations docs:** exact `docker compose exec -T app agent-pages purge-expired` cron command, exit codes, retry behavior, database/R2 backup implications. Add explicit do-not-regress entries: 1. Active/expiry and PIN checks cover all four routes before R2. 2. Every `/p/*` response is private/no-store and varies on Cookie. 3. No instance-wide per-page unlock counter may be introduced. 4. PIN verification remains bounded to four concurrent scrypt operations per process. 5. PIN/expiry remain create-only and page-level. 6. Expired pages are indistinguishable from missing pages externally. 7. Cleanup never deletes DB metadata before all R2 objects and retains retry/audit state. 8. Deleted slugs remain permanently reserved. 9. PINs never authorize writes; publish bearer remains the write boundary. 10. MCP continues to have no storage/model imports and generated PINs are returned with successful URLs. --- ## 15. Explicitly out of scope - Locking an existing open page. - Removing, rotating, recovering, or changing a PIN after publication. - Changing/adding/extending expiry after publication. - Manual page deletion. - Per-version PINs or expiry. - User accounts, SSO, multiple tenants, per-page ownership/edit secrets, create-only tokens. - Making PIN-locked titles/slugs private. - Search-engine indexing or public search beyond the ledger. - CAPTCHA, proof-of-work, distributed rate limiting, multi-process semaphore coordination. - Weak-PIN blocklists. - Idempotent publish/update requests. - Alembic introduction in this feature (but it is the recorded next-schema-change trigger). - In-process/background cleanup scheduler or a second long-running Compose worker. - Reusing expired slugs under any circumstance.