Plan: Page Tags (v1)

Status: Approved for implementation — handed to an implementing agent. Scope owner: operator (design reviewed 2026-08-15). Prerequisite reading: AGENTS.md (locked v1 decisions, do-not-regress list), MEMORY.md (decision register D01–D64), docs/architecture.md.

This document is the complete specification. Where this plan and older docs disagree, this plan wins for tag behavior; for everything else the existing locked decisions win. If you must deviate, record the deviation in MEMORY.md and this file's changelog.

1. What we are building

Every page can carry an ordered array of up to 5 tags. Tags are short, single-token, lowercase labels (e.g. agent-pages, research, kubernetes). The two product uses:

1. Ledger pills — tags render as small pills on each row of the public ledger (/ and /page/{n}) and on /search results. Every tag of a page is shown.
2. Tag search — /search accepts up to 10 tag filters alongside (or instead of) the existing fuzzy title query. Selected tags render as removable pills in the search UI; adding a tag uses a text input backed by a native <datalist> of the existing vocabulary. No JavaScript — all pill interactions are server-rendered links and GET forms.

Agents choose tags at publish time. The MCP exposes the existing vocabulary via a new list_tags tool so agents can prefer existing tags and only mint new ones when nothing fits. The convention (documented in the MCP tool description, not enforceable server-side) is that the first tag is the repo or top-level folder name the artifact was produced from — the MCP server is a thin HTTP client with no filesystem context, so the calling agent supplies this.

2. Locked decisions for this feature

These extend the decision register. When implementation lands, append them to MEMORY.md as D65–D71 (adjust numbers if the register moved) and update AGENTS.md accordingly.

3. Data model

New Alembic revision 0003_page_tags in app/alembic/versions/ (down_revision = 0002_search_corpus_state). Downgrade must cleanly drop both tables (one-revision-back rollback support, do-not-regress #6).

tags
  id          INTEGER PK AUTOINCREMENT
  name        VARCHAR(32) NOT NULL UNIQUE          -- canonical lowercase, indexed via unique
  created_at  DATETIME(timezone) NOT NULL server_default now()

page_tags
  page_id     UUID NOT NULL FK -> pages.id ON DELETE CASCADE, indexed
  tag_id      INTEGER NOT NULL FK -> tags.id, indexed
  position    INTEGER NOT NULL                     -- 0..4, display order
  PRIMARY KEY (page_id, tag_id)
  UNIQUE (page_id, position)

SQLAlchemy models go in app/models.py (Tag, PageTag); add a tags relationship on Page ordered by PageTag.position (association object or secondary + order_by; keep it simple and explicit). SQLite FK pragma is already ON (D18), so the cascade works.

Not a JSON column on pages: both product uses (indexed exact-match filtering; a fetchable vocabulary with counts) want normalized rows.

page_deletion_logs / purge (app/services/purge.py): no changes. page_tags rows die with the page via cascade; tags rows persist harmlessly.

4. API changes (/v1, all publish-bearer authed)

4.1 POST /v1/pages

PageCreateRequest (app/schemas.py) gains:

tags: list[str] = Field(default_factory=list, max_length=5)

Validator: normalize each entry (NFKC → casefold → strip), enforce T1 grammar, reject duplicates post-normalization. Error messages must name the offending tag and the rule ("Tag 'My Tag' is invalid: tags are 1-32 chars of [a-z0-9._-], starting alphanumeric, no spaces").

Inside create_page (app/services/pages.py): after the page row is added and before bump_search_corpus_generation, resolve/insert Tag rows and add PageTag rows in the same transaction. Tag upsert must tolerate a concurrent insert of the same name (catch IntegrityError → re-select; or INSERT OR IGNORE + select). Keep the existing R2-compensation structure untouched.

4.2 PUT /v1/pages/{id}

PageUpdateRequest gains tags: list[str] | None = None (same validator). Semantics: None/omitted → preserve existing tags exactly; present (including []) → full replace. Replacement happens in the same transaction as the conditional UPDATE ... WHERE current_version = expected gate, after the rowcount check succeeds — delete existing PageTag rows, insert the new set. The existing bump_search_corpus_generation call already covers cache invalidation.

4.3 Responses

• PageSummary and PageResponse (app/schemas.py) gain tags: list[str] (canonical names, position order).
• _page_summary in app/services/pages.py populates it from the relationship.
• list_pages must avoid N+1: eager-load (selectinload) or one grouped query for the page IDs on the current page. Keep Page.id as secondary sort key (do-not-regress #9).
• The search summary builder _summary_from_row (app/services/search.py) works from a column Row, not an ORM object — fetch tags for the winning top-N page IDs in one additional query after ranking (and in fetch_active_summaries_by_ids for cache-hit hydration).

4.4 New: GET /v1/tags

New router app/api/v1/tags.py (mounted in app/main.py beside /v1/pages), service logic in a new app/services/tags.py.

GET /v1/tags?limit=100&offset=0      (limit 1..500, default 100)
→ 200 {"items": [{"name": "agent-pages", "page_count": 12}, ...],
       "limit": 100, "offset": 0, "total": 37}

• Auth: require_publish_auth (same as other /v1 routes; wrong token 401, unset 503).
• page_count counts active pages only (page_is_active_clause(now) + purge_started_at IS NULL), joined through page_tags. Tags with zero active pages are omitted.
• Order: page_count desc, name asc.
• Handler is sync def (do-not-regress #22).

5. Search changes

5.1 Query surface

GET /search accepts, in addition to q:

• Repeated tag params: /search?tag=agent-pages&tag=research (optionally with q=...).
• Validation, in this order, all before consuming any rate budget (mirrors the over-64 q rule, do-not-regress #26):
  • more than 10 tag params → 400 "Use at most 10 tag filters."
  • any param failing T1 grammar (after normalization) → 400 naming the bad value (sanitized via sanitize_display_text before reflection).
  • duplicate tags after normalization → deduplicate silently (not an error — trivially reachable via crafted URLs).
• Valid-but-unknown tags are allowed and simply match nothing (they are filters).
• Blank/missing q with valid tags does not redirect home — it runs a tags-only search. Blank q with no tags keeps today's 303 → /.
• q alone: behavior byte-for-byte identical to today.

Add the validation helper in app/search_query.py (e.g. validate_search_tags(raw_values) -> TagsValidation) so viewer code stays thin; reuse the T1 regex from one shared constant (suggest app/services/tags.py exporting TAG_PATTERN, imported by schemas and search_query — mind the import direction: app/ modules may import from app/services/, mcp_server/ may not).

5.2 Execution paths (app/services/search.py, app/search_cache.py)

Three shapes:

1. q only — unchanged code path.
2. tags only — new cheap path: one SQL query joining page_tags/tags with the active/purge predicates, GROUP BY page_id with COUNT(*) AS tag_match_count, ordered (tag_match_count desc, updated_at desc, id desc), limit search_result_limit (50). No RapidFuzz, no scoring semaphore — do not acquire a search slot. Consumes normal search rate budget. Not LRU-cached in v1 (it is one indexed query; caching adds key cardinality for no measured win — note this in code).
3. q + tags — fetch_active_search_candidates gains an optional tag filter: candidates are restricted (SQL join as above) to pages matching ≥1 selected tag, and each candidate carries its tag_match_count. Scoring/admission (D62–D64 passes) is unchanged and runs on the filtered candidate list. Final ranking: (score desc, tag_match_count desc, updated_at desc, id desc) — extend _rank_matches. valid_until derives from the filtered candidate snapshot (same mechanism, do-not-regress #24).

5.3 Cache key

SearchCacheKey (app/search_cache.py) extends to include the sorted tuple of canonical tags (empty tuple when none): (normalized_query, tags_tuple, generation, cutoff, limit). All lookup/store/invalidate helpers and get_or_compute_search take the tag tuple. Generation semantics unchanged: tag edits only happen inside create/update, which already bump — cached tag-filtered results invalidate correctly with zero new machinery (do-not-regress #23). Consider raising search_cache_max_entries default 256 → 512 to absorb the added key cardinality (config change in app/config.py; optional, cheap: entries are small per the 2.9 MiB benchmark note in MEMORY.md).

5.4 Rate limits

Unchanged budgets and atomicity (do-not-regress #26): every accepted /search request — q-only, tags-only, or both, cache hit or miss — consumes burst + sustained in the one existing atomic transaction. Validation failures (over-64 q, >10 tags, bad tag grammar) reject before any budget.

6. Viewer changes

6.1 Ledger pills (/, /page/{n}, /search results)

• app/viewer/rows.py row() adds tags: list[str] from PageSummary.tags.
• app/templates/viewer/_ledger.html: render pills inside the existing Page cell (ledger-cell-page), on a line under the title/slug. Do not add an eighth column — the sub-48rem mobile collapse depends on explicit grid-row per cell and the table is already dense. Keep explicit ARIA roles intact.
• Each pill is a link to /search?tag={name} (URL-encode defensively even though the grammar is URL-safe).
• Styling in app/static/viewer-home.css: muted mono chips (border or faint background, --muted-family ink), wrapping allowed. Format keeps the page's only color (D43) — pills stay ink/muted, no new hues. Verify the 390px mobile collapse still lays out correctly with 5 pills (the known-fragile case from MEMORY.md 2026-07-31).

6.2 Search form and selected-tag pills

• app/templates/viewer/_search_form.html: the form (used on /, /page/{n}, /search) gains a tag text input (name="tag", maxlength="32") with list="tag-options", plus a <datalist id="tag-options"> populated server-side with the active vocabulary (cap at top 100 by page_count; provided via template context from app/services/tags.py). When rendering on /search with active filters, emit <input type="hidden" name="tag" value="..."> per selected tag and a hidden q carrying the current query, so submitting adds the typed tag to the existing state. Ledger pages (/, /page/{n}) render the same form without hidden state.
• app/templates/viewer/search.html: above results, render each selected tag as a pill with a remove link — an <a> to /search re-serialized with that tag omitted (keep q and other tags). Also show a "clear all tags" link when ≥2 tags. Result messaging: extend _search_caption / result_message in app/viewer/search.py to name the filters, e.g. 3 matches for “plan” tagged agent-pages or research.
• app/viewer/search.py: parse+validate tag params (Section 5.1), thread the tag tuple through get_or_compute_search / the tags-only path, build pill/remove-link hrefs server-side (single helper that serializes q + tags to a canonical /search URL; write it once, use it for pills, remove links, and the form).
• app/viewer/index.py: pass the datalist vocabulary into the home/page-n context.

6.3 Theme round-trip

/viewer/theme/{theme} currently re-synthesizes only q for next=/search (set_viewer_theme in app/viewer/routes.py, theme_set_href/parse_theme_search_query in app/viewer/theme.py). Extend both ends so validated tag params round-trip too (revalidate against T1, cap 10, drop invalid silently). validate_next_path itself must not change — it stays path-only and strict (do-not-regress: D40/D60); tags ride as extra query params on the theme URL exactly like q does.

6.4 CSP / headers

No changes. /search, /, /page/{n} already send form-action 'self' (do-not-regress #27); <datalist> needs no script. All new handlers stay sync def (do-not-regress #22).

7. MCP changes (mcp_server/)

Thin HTTP client only — no storage/model imports (do-not-regress #1, #20).

• mcp_server/client.py: publish_page(..., tags=None), update_page(..., tags=None) forward tags when not None; new list_tags(limit=100, offset=0) → GET /v1/tags.
• mcp_server/server.py:
  • publish_page tool gains tags: list[str] | None = None. Validate client-side before dispatch (T1 grammar, ≤5, no dupes) so agents get fast errors. Tool description gains: "Optionally attach up to 5 single-word tags (1-32 chars, lowercase [a-z0-9._-]). Call list_tags first and prefer existing tags; create new ones only when none fit. Convention: the first tag is the repo or top-level folder name the artifact was produced from. Tags are public metadata on the ledger, even for PIN-locked pages."
  • update_page tool gains tags: list[str] | None = None — omitted preserves, present replaces (mirror the API).
  • New read-only tool list_tags(limit: int = 100, offset: int = 0) returning the vocabulary with counts (structured content: {items: [{name, page_count}], total, limit, offset}).
  • Version-skew (T9): after a publish/update that sent tags, if the response payload lacks a matching tags field, append a warning line to the success text (e.g. "Note: the server did not confirm tags; it may predate tag support.") — do not raise.
  • page_ref passes tags through when present so structured results include it.

8. File-by-file change list

Out of scope (do not build): tags-only PATCH endpoint, tag rename/merge/delete admin, fuzzy tag matching, tags in the RapidFuzz corpus, per-version tags, JS-driven pill editing, FTS5 work, /v1/tags public (unauthenticated) exposure.

9. Testing and success criteria

The change is done when all of the following hold. Run pip install -e ".[dev]" then python -m pytest — the full suite must pass, including every existing test (regressions in untouched tests mean the change is wrong, not the test).

New/extended automated tests

• tests/test_tags_validation.py (new): T1 grammar matrix — accepts a, agent-pages, next.js, my_notes, 32-char tag; rejects empty, 33 chars, spaces, leading -/_/., non-ASCII (e.g. café, fullwidth forms after NFKC), control chars; casefold applies (Agent-Pages → agent-pages accepted); >5 tags; duplicates post-normalization (Foo, foo) → 422.
• tests/test_api_tags.py (new): POST with tags persists order and canonical names; PUT omitted preserves; PUT [] clears; PUT replace works; responses (POST/PUT/GET one/GET list) carry tags; GET /v1/tags counts active pages only (publish two pages sharing a tag, expire one via fixture, count drops), ordering, pagination, 401/503 auth behavior; concurrent-ish duplicate tag creation does not 500 (upsert race guard).
• tests/test_search_tags.py (new): tags-only search returns match-count-then-recency ordering; OR semantics (page with either tag matches; match-count ranks 2-tag match above 1-tag); q+tags filters candidates before scoring (a fuzzy-matching title without the tag is excluded); >10 tags → 400 with no rate budget consumed (assert via limiter state, mirroring existing over-64 tests in tests/test_search_rate_limit.py); invalid tag → 400 sanitized reflection; unknown valid tag → zero results, no error; blank q + tags searches (no 303); blank q no tags still 303s.
• tests/test_search_cache.py (extend): cache key separates same q with different tag sets; tag-filtered entries invalidate on create/update generation bump; tags-only path does not acquire a scoring slot (assert semaphore untouched) and is not cached.
• tests/test_home_index.py / tests/test_search_viewer.py (extend): pills render on ledger and search rows with correct hrefs; locked pages show pills; selected-tag pills with correct remove links (removing one keeps q + others); datalist present with vocabulary; hidden inputs carry state so form submits add tags.
• tests/test_viewer_theme.py (extend): theme link on /search?q=x&tag=a&tag=b round-trips both; invalid tags dropped; validate_next_path still rejects query strings (unchanged assertions must stay green).
• tests/test_mcp_server.py (extend): publish_page/update_page forward tags; client-side validation errors; list_tags tool result shape; skew warning appended when response lacks tags (fake client returning legacy payload); structured page_ref includes tags.
• Migration (tests/test_search_migration.py pattern): agent-pages init-db on a fresh DB creates both tables; upgrade on a pre-0003 volume works; alembic downgrade -1 drops them cleanly.
• Purge (tests/test_purge_expired.py extend): purging a tagged page removes its page_tags rows, leaves tags rows, and the vocabulary no longer lists a tag whose only page was purged.

Manual verification (required, with artifacts)

Run the app locally (agent-pages init-db, uvicorn app.main:app --port 8000 with fake/dev R2 or the test harness), publish 3–4 pages with overlapping tags via curl or MCP, then verify in a real browser: pills on /; click a pill → filtered /search; add a second tag via the form (datalist suggestions appear); remove a pill via its link; combine with a fuzzy q; switch theme mid-search and confirm q + tags survive; check the 390px mobile layout with a 5-tag row. Capture screenshots/video per the walkthrough-artifact rules.

Performance guardrail

Run python scripts/benchmark_search.py --pages 1000 before and after; the q-only miss path must not regress materially (>10%). Tag filtering is expected to be neutral-to-faster (smaller candidate sets). The standing FTS5 escalation disposition in MEMORY.md is unchanged by this feature — do not start FTS5 work.

10. Hard constraints (do not regress)

Copy of the rules most likely to be violated by this work — the full list in AGENTS.md still applies:

1. Route handlers stay sync def (AGENTS.md #22).
2. Corpus generation bumps only inside successful create/update transactions; purge never bumps (#23). Tag mutations must not add a second bump path.
3. Search rate budget: burst + sustained in one atomic transaction; validation failures reject before budget; cache hits still count (#26).
4. validate_next_path stays path-only — never teach it query strings (D40/D60).
5. No JavaScript on /, /page/{n}, /search; no CSP loosening anywhere; artifacts//raw untouched.
6. Search scores titles only — tags filter, never enter the fuzzy corpus (#25).
7. MCP stays HTTP-only, no app/services/ or app/models imports (#1).
8. Schema changes via Alembic with working one-revision downgrade (#6).
9. Page.id stays a secondary sort key everywhere pagination exists (#9).
10. PIN/expiry semantics untouched; tags never gate reads or authorize writes.

11. Delivery checklist

1. Implement per Sections 3–8; commit in logical units.
2. Full python -m pytest green; new tests from Section 9 present and green.
3. Manual browser verification done with artifacts.
4. AGENTS.md + MEMORY.md updated (new decisions, layout/API/viewer sections, do-not-regress additions for tags).
5. PR opened with the evidence, marked ready for review when complete (MEMORY.md rule 7 / D35).