# Plan: Page Tags (v1)

**Status:** Approved for implementation — handed to an implementing agent.
**Scope owner:** operator (design reviewed 2026-08-15).
**Prerequisite reading:** `AGENTS.md` (locked v1 decisions, do-not-regress list), `MEMORY.md` (decision register D01–D64), `docs/architecture.md`.

This document is the complete specification. Where this plan and older docs disagree, this plan wins for tag behavior; for everything else the existing locked decisions win. If you must deviate, record the deviation in `MEMORY.md` and this file's changelog.

---

## 1. What we are building

Every page can carry an ordered array of up to **5 tags**. Tags are short, single-token, lowercase labels (e.g. `agent-pages`, `research`, `kubernetes`). The two product uses:

1. **Ledger pills** — tags render as small pills on each row of the public ledger (`/` and `/page/{n}`) and on `/search` results. Every tag of a page is shown.
2. **Tag search** — `/search` accepts up to **10** tag filters alongside (or instead of) the existing fuzzy title query. Selected tags render as removable pills in the search UI; adding a tag uses a text input backed by a native `<datalist>` of the existing vocabulary. **No JavaScript** — all pill interactions are server-rendered links and GET forms.

Agents choose tags at publish time. The MCP exposes the existing vocabulary via a new `list_tags` tool so agents can prefer existing tags and only mint new ones when nothing fits. The convention (documented in the MCP tool description, not enforceable server-side) is that the **first tag is the repo or top-level folder name** the artifact was produced from — the MCP server is a thin HTTP client with no filesystem context, so the *calling agent* supplies this.

## 2. Locked decisions for this feature

These extend the decision register. When implementation lands, append them to `MEMORY.md` as **D65–D71** (adjust numbers if the register moved) and update `AGENTS.md` accordingly.

| # | Decision | Rationale |
|---|----------|-----------|
| T1 | **Tag grammar:** after NFKC → casefold, a tag must match `^[a-z0-9][a-z0-9._-]{0,31}$` (1–32 chars, ASCII only, starts alphanumeric; `-`, `_`, `.` allowed as separators). Anything else → validation error. Never silently strip/split. | Repo/folder names contain `-`, `_`, `.` (`next.js`, `my_notes`). The operator's initial 20-char cap was raised to 32 because real repo names exceed 20 (`agent-artifact-hosting` = 22, `research-agent-artifact-hosting` = 31). ASCII-only keeps pills unambiguous in `?tag=` URLs and avoids Unicode-confusable vocabulary entries. |
| T2 | **Tags are canonical lowercase** — `Agent-Pages` and `agent-pages` are the same tag; stored and displayed lowercase. | Controlled vocabulary; case variants would fragment it. |
| T3 | **Per-page cap 5, order-preserving; duplicates (after normalization) in one request → 422.** | Operator constraint. Agent-chosen order is the display order. |
| T4 | **Tags are page-level and mutable via `PUT /v1/pages/{id}` (full replace), not per-version.** Same shape as `title`. No tags-only PATCH endpoint in v1. | Reusing PUT means every tag mutation rides the transaction that already bumps the search corpus generation atomically (do-not-regress #23). A metadata-only PATCH is a v1.1 follow-up. |
| T5 | **Multi-tag search is OR (match any), max 10 tags per query.** Tags-only results rank by `(tag_match_count desc, updated_at desc, id desc)`. With a fuzzy `q`, rank by `(score desc, tag_match_count desc, updated_at desc, id desc)`. | Forced by arithmetic: pages have ≤5 tags, so AND of 6–10 tags can never match. |
| T6 | **Tag matching is exact (canonical name), never fuzzy, and tag names never enter the RapidFuzz title corpus.** Tags are a SQL prefilter applied before scoring. | Preserves "search titles only" (do-not-regress #25) and avoids re-litigating the D62–D64 admission rules. |
| T7 | **Tag search UI is JavaScript-free**: removable pills are links (same URL minus that tag), adding uses a GET form with hidden inputs carrying current state plus a `<datalist>`-backed text input. | D58 (no-JS search) and the ledger's zero-JS design are locked. Native `<datalist>` gives autocomplete without scripts. |
| T8 | **Tag vocabulary is public.** Pills appear on the public ledger for every active page, including PIN-locked ones (same as titles today; PIN protects the body only). `tags` table rows may orphan after purge; the vocabulary is always computed by joining active pages, so orphans are invisible. Purge does not bump corpus generation (unchanged). | Consistent with D46. Orphan rows are tiny; GC is unnecessary. |
| T9 | **MCP does not fail closed on tag version-skew.** If an older API ignores `tags` (response omits/differs), MCP appends a warning line to the success message instead of raising. | The PIN/expiry fail-closed checks guard security/lifecycle hazards; ignored tags are cosmetic. |
| T10 | **Tag creation is implicit and publish-gated.** Unknown (valid) tags are created inside the create/update transaction. No separate tag CRUD. Vocabulary growth is bounded by the publish bearer + 25/hour publish limit. | No new abuse surface, no new admin surface. |

## 3. Data model

New Alembic revision `0003_page_tags` in `app/alembic/versions/` (down_revision = `0002_search_corpus_state`). Downgrade must cleanly drop both tables (one-revision-back rollback support, do-not-regress #6).

```
tags
  id          INTEGER PK AUTOINCREMENT
  name        VARCHAR(32) NOT NULL UNIQUE          -- canonical lowercase, indexed via unique
  created_at  DATETIME(timezone) NOT NULL server_default now()

page_tags
  page_id     UUID NOT NULL FK -> pages.id ON DELETE CASCADE, indexed
  tag_id      INTEGER NOT NULL FK -> tags.id, indexed
  position    INTEGER NOT NULL                     -- 0..4, display order
  PRIMARY KEY (page_id, tag_id)
  UNIQUE (page_id, position)
```

SQLAlchemy models go in `app/models.py` (`Tag`, `PageTag`); add a `tags` relationship on `Page` ordered by `PageTag.position` (association object or `secondary` + `order_by`; keep it simple and explicit). SQLite FK pragma is already ON (D18), so the cascade works.

**Not** a JSON column on `pages`: both product uses (indexed exact-match filtering; a fetchable vocabulary with counts) want normalized rows.

`page_deletion_logs` / purge (`app/services/purge.py`): **no changes**. `page_tags` rows die with the page via cascade; `tags` rows persist harmlessly.

## 4. API changes (`/v1`, all publish-bearer authed)

### 4.1 `POST /v1/pages`

`PageCreateRequest` (`app/schemas.py`) gains:

```python
tags: list[str] = Field(default_factory=list, max_length=5)
```

Validator: normalize each entry (NFKC → casefold → strip), enforce T1 grammar, reject duplicates post-normalization. Error messages must name the offending tag and the rule (`"Tag 'My Tag' is invalid: tags are 1-32 chars of [a-z0-9._-], starting alphanumeric, no spaces"`).

Inside `create_page` (`app/services/pages.py`): after the page row is added and before `bump_search_corpus_generation`, resolve/insert `Tag` rows and add `PageTag` rows in the same transaction. Tag upsert must tolerate a concurrent insert of the same name (catch `IntegrityError` → re-select; or `INSERT OR IGNORE` + select). Keep the existing R2-compensation structure untouched.

### 4.2 `PUT /v1/pages/{id}`

`PageUpdateRequest` gains `tags: list[str] | None = None` (same validator). Semantics: `None`/omitted → preserve existing tags exactly; present (including `[]`) → full replace. Replacement happens in the same transaction as the conditional `UPDATE ... WHERE current_version = expected` gate, after the rowcount check succeeds — delete existing `PageTag` rows, insert the new set. The existing `bump_search_corpus_generation` call already covers cache invalidation.

### 4.3 Responses

- `PageSummary` and `PageResponse` (`app/schemas.py`) gain `tags: list[str]` (canonical names, position order).
- `_page_summary` in `app/services/pages.py` populates it from the relationship.
- `list_pages` must avoid N+1: eager-load (`selectinload`) or one grouped query for the page IDs on the current page. Keep `Page.id` as secondary sort key (do-not-regress #9).
- The search summary builder `_summary_from_row` (`app/services/search.py`) works from a column `Row`, not an ORM object — fetch tags for the winning top-N page IDs in **one** additional query after ranking (and in `fetch_active_summaries_by_ids` for cache-hit hydration).

### 4.4 New: `GET /v1/tags`

New router `app/api/v1/tags.py` (mounted in `app/main.py` beside `/v1/pages`), service logic in a new `app/services/tags.py`.

```
GET /v1/tags?limit=100&offset=0      (limit 1..500, default 100)
→ 200 {"items": [{"name": "agent-pages", "page_count": 12}, ...],
       "limit": 100, "offset": 0, "total": 37}
```

- Auth: `require_publish_auth` (same as other `/v1` routes; wrong token 401, unset 503).
- `page_count` counts **active** pages only (`page_is_active_clause(now)` + `purge_started_at IS NULL`), joined through `page_tags`. Tags with zero active pages are omitted.
- Order: `page_count desc, name asc`.
- Handler is **sync `def`** (do-not-regress #22).

## 5. Search changes

### 5.1 Query surface

`GET /search` accepts, in addition to `q`:

- Repeated `tag` params: `/search?tag=agent-pages&tag=research` (optionally with `q=...`).
- Validation, in this order, all **before** consuming any rate budget (mirrors the over-64 `q` rule, do-not-regress #26):
  - more than 10 `tag` params → 400 `"Use at most 10 tag filters."`
  - any param failing T1 grammar (after normalization) → 400 naming the bad value (sanitized via `sanitize_display_text` before reflection).
  - duplicate tags after normalization → deduplicate silently (not an error — trivially reachable via crafted URLs).
- Valid-but-unknown tags are allowed and simply match nothing (they are filters).
- Blank/missing `q` **with** valid tags does **not** redirect home — it runs a tags-only search. Blank `q` with no tags keeps today's 303 → `/`.
- `q` alone: behavior byte-for-byte identical to today.

Add the validation helper in `app/search_query.py` (e.g. `validate_search_tags(raw_values) -> TagsValidation`) so viewer code stays thin; reuse the T1 regex from one shared constant (suggest `app/services/tags.py` exporting `TAG_PATTERN`, imported by schemas and search_query — mind the import direction: `app/` modules may import from `app/services/`, `mcp_server/` may not).

### 5.2 Execution paths (`app/services/search.py`, `app/search_cache.py`)

Three shapes:

1. **`q` only** — unchanged code path.
2. **tags only** — new cheap path: one SQL query joining `page_tags`/`tags` with the active/purge predicates, `GROUP BY page_id` with `COUNT(*) AS tag_match_count`, ordered `(tag_match_count desc, updated_at desc, id desc)`, limit `search_result_limit` (50). **No RapidFuzz, no scoring semaphore** — do not acquire a search slot. Consumes normal search rate budget. Not LRU-cached in v1 (it is one indexed query; caching adds key cardinality for no measured win — note this in code).
3. **`q` + tags** — `fetch_active_search_candidates` gains an optional tag filter: candidates are restricted (SQL join as above) to pages matching ≥1 selected tag, and each candidate carries its `tag_match_count`. Scoring/admission (D62–D64 passes) is unchanged and runs on the filtered candidate list. Final ranking: `(score desc, tag_match_count desc, updated_at desc, id desc)` — extend `_rank_matches`. `valid_until` derives from the **filtered** candidate snapshot (same mechanism, do-not-regress #24).

### 5.3 Cache key

`SearchCacheKey` (`app/search_cache.py`) extends to include the **sorted tuple of canonical tags** (empty tuple when none): `(normalized_query, tags_tuple, generation, cutoff, limit)`. All lookup/store/invalidate helpers and `get_or_compute_search` take the tag tuple. Generation semantics unchanged: tag edits only happen inside create/update, which already bump — cached tag-filtered results invalidate correctly with zero new machinery (do-not-regress #23). Consider raising `search_cache_max_entries` default 256 → 512 to absorb the added key cardinality (config change in `app/config.py`; optional, cheap: entries are small per the 2.9 MiB benchmark note in `MEMORY.md`).

### 5.4 Rate limits

Unchanged budgets and atomicity (do-not-regress #26): every accepted `/search` request — q-only, tags-only, or both, cache hit or miss — consumes burst + sustained in the one existing atomic transaction. Validation failures (over-64 `q`, >10 tags, bad tag grammar) reject before any budget.

## 6. Viewer changes

### 6.1 Ledger pills (`/`, `/page/{n}`, `/search` results)

- `app/viewer/rows.py` `row()` adds `tags: list[str]` from `PageSummary.tags`.
- `app/templates/viewer/_ledger.html`: render pills **inside the existing Page cell** (`ledger-cell-page`), on a line under the title/slug. Do **not** add an eighth column — the sub-48rem mobile collapse depends on explicit `grid-row` per cell and the table is already dense. Keep explicit ARIA roles intact.
- Each pill is a link to `/search?tag={name}` (URL-encode defensively even though the grammar is URL-safe).
- Styling in `app/static/viewer-home.css`: muted mono chips (border or faint background, `--muted`-family ink), wrapping allowed. **Format keeps the page's only color** (D43) — pills stay ink/muted, no new hues. Verify the 390px mobile collapse still lays out correctly with 5 pills (the known-fragile case from `MEMORY.md` 2026-07-31).

### 6.2 Search form and selected-tag pills

- `app/templates/viewer/_search_form.html`: the form (used on `/`, `/page/{n}`, `/search`) gains a tag text input (`name="tag"`, `maxlength="32"`) with `list="tag-options"`, plus a `<datalist id="tag-options">` populated server-side with the active vocabulary (cap at top 100 by page_count; provided via template context from `app/services/tags.py`). When rendering on `/search` with active filters, emit `<input type="hidden" name="tag" value="...">` per selected tag and a hidden `q` carrying the current query, so submitting **adds** the typed tag to the existing state. Ledger pages (`/`, `/page/{n}`) render the same form without hidden state.
- `app/templates/viewer/search.html`: above results, render each selected tag as a pill with a **remove link** — an `<a>` to `/search` re-serialized with that tag omitted (keep `q` and other tags). Also show a "clear all tags" link when ≥2 tags. Result messaging: extend `_search_caption` / `result_message` in `app/viewer/search.py` to name the filters, e.g. `3 matches for “plan” tagged agent-pages or research`.
- `app/viewer/search.py`: parse+validate `tag` params (Section 5.1), thread the tag tuple through `get_or_compute_search` / the tags-only path, build pill/remove-link hrefs server-side (single helper that serializes `q` + tags to a canonical `/search` URL; write it once, use it for pills, remove links, and the form).
- `app/viewer/index.py`: pass the datalist vocabulary into the home/page-n context.

### 6.3 Theme round-trip

`/viewer/theme/{theme}` currently re-synthesizes only `q` for `next=/search` (`set_viewer_theme` in `app/viewer/routes.py`, `theme_set_href`/`parse_theme_search_query` in `app/viewer/theme.py`). Extend both ends so validated `tag` params round-trip too (revalidate against T1, cap 10, drop invalid silently). `validate_next_path` itself **must not change** — it stays path-only and strict (do-not-regress: D40/D60); tags ride as extra query params on the theme URL exactly like `q` does.

### 6.4 CSP / headers

No changes. `/search`, `/`, `/page/{n}` already send `form-action 'self'` (do-not-regress #27); `<datalist>` needs no script. All new handlers stay sync `def` (do-not-regress #22).

## 7. MCP changes (`mcp_server/`)

Thin HTTP client only — no storage/model imports (do-not-regress #1, #20).

- `mcp_server/client.py`: `publish_page(..., tags=None)`, `update_page(..., tags=None)` forward `tags` when not None; new `list_tags(limit=100, offset=0)` → `GET /v1/tags`.
- `mcp_server/server.py`:
  - `publish_page` tool gains `tags: list[str] | None = None`. Validate client-side before dispatch (T1 grammar, ≤5, no dupes) so agents get fast errors. Tool description gains: *"Optionally attach up to 5 single-word tags (1-32 chars, lowercase [a-z0-9._-]). Call list_tags first and prefer existing tags; create new ones only when none fit. Convention: the first tag is the repo or top-level folder name the artifact was produced from. Tags are public metadata on the ledger, even for PIN-locked pages."*
  - `update_page` tool gains `tags: list[str] | None = None` — omitted preserves, present replaces (mirror the API).
  - New read-only tool `list_tags(limit: int = 100, offset: int = 0)` returning the vocabulary with counts (structured content: `{items: [{name, page_count}], total, limit, offset}`).
  - Version-skew (T9): after a publish/update that sent tags, if the response payload lacks a matching `tags` field, append a warning line to the success text (e.g. `"Note: the server did not confirm tags; it may predate tag support."`) — do **not** raise.
  - `page_ref` passes `tags` through when present so structured results include it.

## 8. File-by-file change list

| File | Change |
|------|--------|
| `app/models.py` | Add `Tag`, `PageTag`; `Page.tags` relationship ordered by position. |
| `app/alembic/versions/0003_page_tags.py` | New migration (up + down). |
| `app/schemas.py` | Shared tag validator; `tags` on `PageCreateRequest` / `PageUpdateRequest` (optional) / `PageSummary` / `PageResponse`; `TagCount` + `TagListResponse`. |
| `app/services/tags.py` (new) | `TAG_PATTERN`, `normalize_tag`, upsert helper used by pages service, `list_tag_counts` (active-only), datalist vocabulary helper. |
| `app/services/pages.py` | Create/update tag persistence in-transaction; `_page_summary` tags; `list_pages` eager loading. |
| `app/api/v1/tags.py` (new) + `app/main.py` | `GET /v1/tags` router, mounted. |
| `app/api/v1/pages.py` | No structural change (schemas carry the field); verify error mapping for tag validation is 422 via pydantic. |
| `app/services/search.py` | Tag-filtered candidate fetch with `tag_match_count`; tags-only listing path; ranking extension; top-N tag hydration; `fetch_active_summaries_by_ids` tags. |
| `app/search_cache.py` | Cache key gains sorted tag tuple; thread through all helpers. |
| `app/search_query.py` | `validate_search_tags`. |
| `app/search_limits.py` | No change (verify only). |
| `app/viewer/search.py` | Parse tags, three-shape dispatch, pill/remove-URL builder, captions/messages. |
| `app/viewer/index.py` | Datalist vocabulary in context. |
| `app/viewer/rows.py` | `tags` in row dict. |
| `app/viewer/theme.py`, `app/viewer/routes.py` | Theme round-trip carries validated tags for `next=/search`. |
| `app/templates/viewer/_ledger.html` | Pills in Page cell. |
| `app/templates/viewer/_search_form.html` | Tag input + `<datalist>` + hidden state. |
| `app/templates/viewer/search.html` | Selected-tag pills with remove links. |
| `app/static/viewer-home.css` | Pill styles, mobile collapse verification. |
| `app/config.py` | Optional: `search_cache_max_entries` 256 → 512. |
| `mcp_server/client.py`, `mcp_server/server.py` | Tags params, `list_tags` tool, skew warning. |
| `tests/` | See Section 9. |
| `AGENTS.md`, `MEMORY.md` | New decisions T1–T10 as D65+; update locked-decision table, layout, API table, viewer/search sections, do-not-regress additions. |

Out of scope (do **not** build): tags-only PATCH endpoint, tag rename/merge/delete admin, fuzzy tag matching, tags in the RapidFuzz corpus, per-version tags, JS-driven pill editing, FTS5 work, `/v1/tags` public (unauthenticated) exposure.

## 9. Testing and success criteria

The change is **done** when all of the following hold. Run `pip install -e ".[dev]"` then `python -m pytest` — the full suite must pass, including every existing test (regressions in untouched tests mean the change is wrong, not the test).

### New/extended automated tests

- **`tests/test_tags_validation.py`** (new): T1 grammar matrix — accepts `a`, `agent-pages`, `next.js`, `my_notes`, 32-char tag; rejects empty, 33 chars, spaces, leading `-`/`_`/`.`, non-ASCII (e.g. `café`, fullwidth forms after NFKC), control chars; casefold applies (`Agent-Pages` → `agent-pages` accepted); >5 tags; duplicates post-normalization (`Foo`, `foo`) → 422.
- **`tests/test_api_tags.py`** (new): POST with tags persists order and canonical names; PUT omitted preserves; PUT `[]` clears; PUT replace works; responses (`POST`/`PUT`/`GET one`/`GET list`) carry `tags`; `GET /v1/tags` counts active pages only (publish two pages sharing a tag, expire one via fixture, count drops), ordering, pagination, 401/503 auth behavior; concurrent-ish duplicate tag creation does not 500 (upsert race guard).
- **`tests/test_search_tags.py`** (new): tags-only search returns match-count-then-recency ordering; OR semantics (page with either tag matches; match-count ranks 2-tag match above 1-tag); `q`+tags filters candidates before scoring (a fuzzy-matching title *without* the tag is excluded); >10 tags → 400 with no rate budget consumed (assert via limiter state, mirroring existing over-64 tests in `tests/test_search_rate_limit.py`); invalid tag → 400 sanitized reflection; unknown valid tag → zero results, no error; blank `q` + tags searches (no 303); blank `q` no tags still 303s.
- **`tests/test_search_cache.py`** (extend): cache key separates same `q` with different tag sets; tag-filtered entries invalidate on create/update generation bump; tags-only path does not acquire a scoring slot (assert semaphore untouched) and is not cached.
- **`tests/test_home_index.py` / `tests/test_search_viewer.py`** (extend): pills render on ledger and search rows with correct hrefs; locked pages show pills; selected-tag pills with correct remove links (removing one keeps `q` + others); datalist present with vocabulary; hidden inputs carry state so form submits add tags.
- **`tests/test_viewer_theme.py`** (extend): theme link on `/search?q=x&tag=a&tag=b` round-trips both; invalid tags dropped; `validate_next_path` still rejects query strings (unchanged assertions must stay green).
- **`tests/test_mcp_server.py`** (extend): `publish_page`/`update_page` forward tags; client-side validation errors; `list_tags` tool result shape; skew warning appended when response lacks tags (fake client returning legacy payload); structured `page_ref` includes tags.
- **Migration** (`tests/test_search_migration.py` pattern): `agent-pages init-db` on a fresh DB creates both tables; upgrade on a pre-0003 volume works; `alembic downgrade -1` drops them cleanly.
- **Purge** (`tests/test_purge_expired.py` extend): purging a tagged page removes its `page_tags` rows, leaves `tags` rows, and the vocabulary no longer lists a tag whose only page was purged.

### Manual verification (required, with artifacts)

Run the app locally (`agent-pages init-db`, `uvicorn app.main:app --port 8000` with fake/dev R2 or the test harness), publish 3–4 pages with overlapping tags via `curl` or MCP, then verify in a real browser: pills on `/`; click a pill → filtered `/search`; add a second tag via the form (datalist suggestions appear); remove a pill via its link; combine with a fuzzy `q`; switch theme mid-search and confirm `q` + tags survive; check the 390px mobile layout with a 5-tag row. Capture screenshots/video per the walkthrough-artifact rules.

### Performance guardrail

Run `python scripts/benchmark_search.py --pages 1000` before and after; the q-only miss path must not regress materially (>10%). Tag filtering is expected to be neutral-to-faster (smaller candidate sets). The standing FTS5 escalation disposition in `MEMORY.md` is unchanged by this feature — do not start FTS5 work.

## 10. Hard constraints (do not regress)

Copy of the rules most likely to be violated by this work — the full list in `AGENTS.md` still applies:

1. Route handlers stay **sync `def`** (AGENTS.md #22).
2. Corpus generation bumps only inside successful create/update transactions; purge never bumps (#23). Tag mutations must not add a second bump path.
3. Search rate budget: burst + sustained in one atomic transaction; validation failures reject **before** budget; cache hits still count (#26).
4. `validate_next_path` stays path-only — never teach it query strings (D40/D60).
5. No JavaScript on `/`, `/page/{n}`, `/search`; no CSP loosening anywhere; artifacts/`/raw` untouched.
6. Search scores **titles only** — tags filter, never enter the fuzzy corpus (#25).
7. MCP stays HTTP-only, no `app/services/` or `app/models` imports (#1).
8. Schema changes via Alembic with working one-revision downgrade (#6).
9. `Page.id` stays a secondary sort key everywhere pagination exists (#9).
10. PIN/expiry semantics untouched; tags never gate reads or authorize writes.

## 11. Delivery checklist

1. Implement per Sections 3–8; commit in logical units.
2. Full `python -m pytest` green; new tests from Section 9 present and green.
3. Manual browser verification done with artifacts.
4. `AGENTS.md` + `MEMORY.md` updated (new decisions, layout/API/viewer sections, do-not-regress additions for tags).
5. PR opened with the evidence, marked **ready for review** when complete (MEMORY.md rule 7 / D35).