# Fuzzy title search on the ledger — implementation plan v2
**Status:** approved design, not implemented.
**Target:** current `master` after the public-ledger, PIN-lock, and page-expiry work.
**Supersedes:** version 1 of this plan in full. This version incorporates the
codebase review, Docker benchmarks, and the subsequent operator decisions about
normalization, validation, rate limiting, cache invalidation, expiry, CSP,
theme round-trips, and the future FTS5 path.
**Goal:** add a search form to the public ledger that finds pages by current
title, tolerates ordinary typos, preserves substring matching, and remains safe
under unauthenticated load on a small single-process VPS.
Example:
```text
query: kubrenetes
title: Kubernetes migration plan
result: match
```
This document is an implementation contract. An implementing agent should not
need the design conversation that produced it.
---
## 1. Final product contract
### 1.1 Search surface
- Search is available from `/`, `/page/{n}`, and `/search` through the same
ordinary HTML GET form.
- The results URL is `GET /search?q=...` and is linkable, refreshable,
bookmarkable, and compatible with browser history.
- There is no JavaScript, typeahead, debounce, or client-side title corpus.
- Search covers the current `pages.title` value only.
- Search never reads R2 objects and never searches bodies.
- Search never matches slugs, PINs, UUIDs, content hashes, or version bodies.
- PIN-locked pages are included because their current titles and lock state are
already public on the ledger. Clicking a locked result still enters the
existing unlock flow.
- Expired and purge-claimed pages are excluded using the same lifecycle
predicate as the ledger and API.
- Results are relevance-ordered, limited to the best 50, and not paginated.
- The visible result count reports every hit at or above the cutoff before the
top-50 truncation.
### 1.2 Matching contract
- Library: `rapidfuzz>=3.9`.
- Do **not** add NumPy in this phase.
- Scorer: `rapidfuzz.fuzz.WRatio`.
- Processor passed to RapidFuzz: `None`; both query and titles are normalized by
application code before scoring.
- Default cutoff: 70, settings-driven and validated within `0..100`.
- Candidate ordering before scoring: `updated_at DESC, id DESC`.
- Final ordering: score descending, then `updated_at DESC`, then `id DESC`.
- Tie ordering is explicit and must not depend on RapidFuzz iteration order.
- `total_matched` is calculated before slicing to `limit=50`.
- `truncated` is exactly `total_matched > len(hits)`.
Use `process.extract(..., limit=None)` or an equivalently benchmarked
single-query API. Do not use `process.cdist` unless NumPy is explicitly added
and a production-container benchmark demonstrates a material win. The scoping
benchmark in `python:3.12-slim` did not demonstrate one.
### 1.3 Scale and datastore
- Expected corpus: hundreds to roughly 10,000 active pages.
- SQLite remains the source of truth.
- This feature introduces one small durable corpus-generation table.
- Because this is a nontrivial schema change and the repository has recorded
that the next such change must introduce Alembic, this feature also introduces
Alembic. This is an explicit scope decision, not a silent v1.1 expansion.
- No FTS table, external search service, second database, or in-memory title
index is added in Phase 1.
### 1.4 Default protection settings
| Setting | Default |
|---|---:|
| `SEARCH_SCORE_CUTOFF` | `70` |
| `SEARCH_RESULT_LIMIT` | `50` |
| `SEARCH_CACHE_MAX_ENTRIES` | `256` |
| `SEARCH_RATE_LIMIT_BURST_MAX` | `20` |
| `SEARCH_RATE_LIMIT_BURST_WINDOW_SECONDS` | `60` |
| `SEARCH_RATE_LIMIT_SUSTAINED_MAX` | `200` |
| `SEARCH_RATE_LIMIT_SUSTAINED_WINDOW_SECONDS` | `3600` |
| `SEARCH_RATE_LIMIT_SHARED_MULTIPLIER` | `5` |
| `SEARCH_CONCURRENCY` | `2` |
| `SEARCH_BUSY_RETRY_AFTER_SECONDS` | `1` |
`SEARCH_RESULT_LIMIT` is configurable for tests and operations but must remain
bounded to `1..50` in v1. The product contract is never more than 50 visible
results.
---
## 2. Why this design
### 2.1 Why not FTS5 in Phase 1
FTS5's trigram tokenizer is useful for substring recall, not edit-distance
matching. A strict trigram `MATCH` can discard the desired result before a fuzzy
ranker sees it, especially for short strings and transpositions. `spellfix1` is
not present in the standard Python SQLite build. Fuzzy scoring therefore remains
application work in Phase 1.
### 2.2 Why RapidFuzz without NumPy
The production base is `python:3.12-slim`; the project supports Python 3.11+.
RapidFuzz publishes compatible manylinux wheels. A Docker benchmark using the
production base found no useful single-query advantage from NumPy-backed
`process.cdist` over RapidFuzz's extraction APIs. `workers=-1` is also a poor fit:
each request could occupy every core and defeat the meaning of an outer scoring
semaphore.
Use one RapidFuzz dependency, one worker, and the process-wide application
semaphore described below.
### 2.3 Why a database generation instead of hashing 10,000 rows per request
The earlier proposal built a cache fingerprint by fetching every active
`(page_id, current_version)` pair and hashing it on every request. That was
correct but made every cache lookup O(N). It avoided fuzzy scoring but did not
avoid a 10,000-row scan.
The accepted design stores one monotonic database generation:
```text
search_corpus_state
┌────┬────────────┐
│ id │ generation │
├────┼────────────┤
│ 1 │ 419 │
└────┴────────────┘
```
Every create or update that can change search results increments the generation
in the same SQLite transaction as the page mutation. A cache hit therefore
requires one O(1) generation read, not a corpus scan.
### 2.4 Why generation alone is insufficient
Logical expiry changes the active corpus at an exact time without writing to the
database. Cache entries therefore carry `valid_until`, the earliest expiry among
the active candidate corpus used to calculate the entry. A cache entry is usable
only while both are true:
```text
cached_generation == database_generation
now < valid_until # when valid_until is not None
```
At equality the entry is stale because the lifecycle rule is
`expires_at <= now → inactive`.
---
## 3. Exact query validation and normalization
### 3.1 Two query representations
Never conflate display and matching values:
- `display_query`: the validated original spelling and case, with control
characters removed. It is shown in the input, result count, and empty state.
- `match_query`: the normalized value used only for scoring and cache keys.
Example:
```text
raw/display: " Kubrenetes "
match: "kubrenetes"
```
### 3.2 Normalization function
Define one pure function and use it for both incoming queries and every title:
```python
def normalize_match_text(value: str) -> str:
"""NFKC, casefold, replace non-alphanumerics, collapse whitespace."""
```
Required order:
1. Unicode NFKC normalization.
2. Unicode `casefold()`.
3. Replace every maximal run of non-alphanumeric characters with one space.
4. Collapse whitespace runs to one ASCII space.
5. Strip leading and trailing whitespace.
Do not combine this with `rapidfuzz.utils.default_process`; that would create two
normalization contracts. The function must normalize titles and queries
identically, including `Straße`/`strasse` and composed/decomposed characters.
### 3.3 Validation decision table
Use `request.query_params.getlist("q")` so repeated parameters are handled
explicitly.
| Input | Response | DB read | Rate budget | Scoring |
|---|---:|---:|---:|---:|
| No `q` | `303 /` | no | no | no |
| Multiple `q` values | inline `400` | no | no | no |
| Blank after trimming | `303 /` | no | no | no |
| Raw decoded length > 64 | inline `400` | no | no | no |
| Normalized length > 64 | inline `400` | no | no | no |
| Normalized length < 2 | `200` guidance | no | no | no |
| Normalized length 2..64 | continue | yes | yes | maybe |
The normalized-length check is separate because NFKC may expand a character.
The HTML input has `maxlength="64"` as a convenience, but the server is the
authority.
Extract the existing error-display sanitizer into a shared helper with an
explicit maximum, for example:
```python
sanitize_display_text(value: str, *, max_chars: int) -> str
```
The existing error path passes 120; search passes 64. Jinja autoescape remains
mandatory.
---
## 4. Data model and Alembic transition
### 4.1 New model
Add a singleton table:
```python
class SearchCorpusState(Base):
__tablename__ = "search_corpus_state"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
generation: Mapped[int] = mapped_column(Integer, nullable=False)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
)
```
Database invariants:
- Exactly one logical row, `id=1`.
- Initial generation is `1`.
- Generation is positive and monotonic.
- Application code never deletes the singleton row.
- A missing row is an operational/schema error, never silently recreated on a
search request.
Use a SQLite check constraint for `id = 1` and `generation >= 1` if it is cleanly
portable through the chosen Alembic migration.
### 4.2 Atomic generation bump
Provide a service helper that participates in the caller's transaction and does
not commit independently:
```python
def bump_search_corpus_generation(db: Session, *, now: datetime) -> None:
result = db.execute(
update(SearchCorpusState)
.where(SearchCorpusState.id == 1)
.values(
generation=SearchCorpusState.generation + 1,
updated_at=now,
)
)
if result.rowcount != 1:
raise SearchCorpusStateError(...)
```
Call it exactly once in the same transaction when:
- A page is successfully created.
- Any new version is successfully committed, including content-only updates.
Even without a title change, `updated_at`, current version, tie ordering, and
displayed metadata change.
Do not bump it when:
- Validation fails.
- R2 upload fails before the database transaction.
- A create transaction rolls back.
- An update loses the current-version race.
- An update loses the expiry/purge conditional gate.
- A commit fails and is rolled back.
- An already-expired page is claimed or physically deleted by the purge worker;
it was already absent from the logical search corpus.
- A rate-limit row changes.
The bump and page change must commit or roll back together. There must never be a
page mutation visible at generation N when that mutation should have produced
generation N+1.
### 4.3 Reading generation
Use a narrow scalar query:
```python
def get_search_corpus_generation(db: Session) -> int: ...
```
Search cache hits perform this query before looking up the LRU. Do not derive the
generation from `COUNT`, timestamps, `PRAGMA data_version`, or rate-limit table
writes.
### 4.4 Alembic adoption contract
Add:
```text
alembic.ini
alembic/
env.py
script.py.mako
versions/
0001_pre_search_baseline.py
0002_search_corpus_state.py
```
`0001_pre_search_baseline` represents the complete schema immediately before
this search feature. Its upgrade path creates that complete schema for a new
database. `0002_search_corpus_state` creates and seeds the singleton table.
Transition existing installations safely:
1. If `pages` does not exist, run Alembic from an empty database through `head`.
2. If `pages` exists but `alembic_version` does not:
- run the existing compatibility logic needed to reach the known pre-search
schema, excluding the new search table;
- validate required pre-search tables, columns, indexes, and constraints;
- stamp `0001_pre_search_baseline` only after validation succeeds;
- upgrade to `head`.
3. If `alembic_version` exists, run `upgrade head` normally.
Do not stamp an unknown or partially migrated schema. A failed validation must
exit nonzero with an actionable message and leave the database untouched beyond
any transactionally safe compatibility work.
`agent-pages init-db` remains the operator entry point, but becomes the wrapper
for this transition and Alembic upgrade. Update the Docker image to copy
`alembic.ini` and `alembic/`, and add the Alembic dependency.
Tests may continue using `Base.metadata.create_all()` for isolated fresh
databases, but migration tests must exercise the real `init-db`/Alembic path.
---
## 5. Search service and scoring
### 5.1 New service module
Create `app/services/search.py`. Storage/model access remains in `app/services/`;
the MCP server remains HTTP-only.
Suggested immutable types:
```python
@dataclass(frozen=True)
class SearchCandidate:
summary: PageSummary
normalized_title: str
@dataclass(frozen=True)
class SearchHit:
page_id: UUID
score: float
@dataclass(frozen=True)
class SearchResult:
hits: tuple[SearchHit, ...]
total_matched: int
truncated: bool
valid_until: datetime | None
```
The cache value may reuse `SearchResult` if its contents remain immutable.
### 5.2 Candidate query
Fetch only columns needed to score and render a `PageSummary`. Do not load page
bodies, version bodies, or the stored `pin_hash`. Derive `has_pin` in SQL or via a
narrow boolean expression.
Use one captured UTC reference time for the candidate statement:
```text
WHERE (expires_at IS NULL OR expires_at > reference_now)
AND purge_started_at IS NULL
ORDER BY updated_at DESC, id DESC
```
Reuse `page_is_active_clause(reference_now)` and the existing UTC helpers.
`valid_until` is the earliest non-null `expires_at` among the entire active
candidate corpus used for that calculation, not merely the visible top 50. This
is conservative: an irrelevant page expiry may invalidate a result, but an
expiry can never leave a stale result usable.
### 5.3 Scoring pipeline
1. Normalize every candidate title with `normalize_match_text`.
2. Call RapidFuzz with already-normalized strings and `processor=None`.
3. Apply the configured cutoff.
4. Convert scores to ordinary Python floats.
5. Sort explicitly by score, UTC-normalized `updated_at`, and UUID.
6. Record `total_matched`.
7. Keep only the first `limit` hits.
8. Set `truncated`.
The service must not know about HTTP status codes, templates, client IPs, or
rate-limit responses.
---
## 6. Process-local LRU cache
### 6.1 Key and value
Cache key:
```text
(normalized_query, database_generation, cutoff, limit)
```
Example:
```text
("kubrenetes", 419, 70, 50)
```
Cache value:
```python
SearchCacheValue(
hits=(
SearchHit(page_id=page_a, score=90.0),
SearchHit(page_id=page_b, score=81.0),
# top 50 at most
),
total_matched=73,
truncated=True,
valid_until=datetime(...), # or None
)
```
The cache holds at most 256 **search entries**, not 256 page hits. Each entry
contains at most 50 ID/score pairs.
### 6.2 Cache hit
A cache hit requires:
1. The exact key exists.
2. `valid_until is None` or `now < valid_until`.
On a hit:
1. Move the entry to the most-recently-used end.
2. Fetch current summaries for at most the cached 50 page IDs using the active
predicate as a defensive check.
3. Build a dictionary by ID and restore relevance order by walking the cached
hit tuple. SQL `IN (...)` ordering is not relied upon.
4. If an expected ID is missing or inactive, remove this entry and treat the
request as a cache miss. This should be exceptional because generation and
`valid_until` are the primary invariants.
5. Do not acquire a scoring semaphore slot.
6. Do not scan the full corpus or call RapidFuzz.
### 6.3 Cache miss
On a miss:
1. Acquire a scoring slot non-blockingly.
2. Fetch the active corpus.
3. Score, sort, count, and truncate.
4. Re-read the database generation before inserting.
5. If generation changed during the calculation, do not cache the result under
the old generation. The request may render its consistent statement snapshot,
but the next request must use the new generation.
6. If `valid_until` was reached during calculation, do not cache the result;
refetch once with a new reference time before rendering if necessary to avoid
surfacing a page that is already expired at response construction.
7. Store the immutable result only after successful scoring and validation.
### 6.4 Locks and concurrency
Use a small process-local lock around LRU state only:
- lookup;
- stale-entry removal;
- move-to-MRU;
- insertion/replacement;
- capacity eviction;
- generation-change clearing;
- explicit reset.
Never hold the cache lock during SQLite access, title normalization, scoring,
summary lookup, or template rendering.
Duplicate computation after simultaneous misses is acceptable if avoiding it
would require holding the cache lock. The scoring semaphore still bounds the
work. A per-key single-flight mechanism is out of scope unless profiling proves
duplicate misses material.
### 6.5 Eviction and invalidation
The rules are exact:
- **Capacity:** inserting a 257th key evicts the least recently used key.
- **Access:** a valid hit becomes most recently used.
- **Replacement:** recomputing the same key replaces its value and makes it MRU.
- **Generation:** when a request observes a new database generation, clear all
entries belonging to earlier generations under the cache lock. They can never
be valid again.
- **Expiry:** when a requested entry has `now >= valid_until`, remove only that
exact key and treat it as a miss. Other entries are validated lazily on their
own lookups.
- **Restart/deploy:** the process-local cache starts empty.
- **Tests:** expose `reset_search_cache()`.
- **Failure:** an exception or capacity rejection never inserts a partial entry.
Because expiry does not increment generation, a recalculation after expiry may
replace an old entry under the same key with a later `valid_until`.
---
## 7. Exact rate-limit contract
Rate limiting is implemented inside the valid `/search` path. It is not global
middleware and applies to no other route or query parameter. In particular,
theme round-trip `q` parameters consume no search budget.
### 7.1 Client identity
Reuse `resolve_client_ip(request, settings)`:
- Direct/untrusted peer: use the socket peer.
- Trusted proxy with usable XFF: rightmost-walk to the first untrusted hop.
- Trusted proxy without usable XFF: use the proxy address and set `shared=True`.
Search creates two hashed counter keys per resolved identity:
```text
sha256("search-ip-min:" + resolved_ip)
sha256("search-ip-hr:" + resolved_ip)
```
With 100 distinct resolved IPs there can be 200 active search counter rows: one
minute row and one hour row per IP. Rows are dynamic and eventually purged.
For `shared=True`, multiply both maximum request counts by
`SEARCH_RATE_LIMIT_SHARED_MULTIPLIER=5`; windows remain unchanged:
```text
normal: 20/minute, 200/hour
shared: 100/minute, 1000/hour
```
### 7.2 One transaction for both windows
Consume burst and sustained counters in one short independent SQLite session and
one outer transaction:
```text
BEGIN
burst conditional increment
sustained conditional increment
COMMIT
```
If either limit is full, roll back the transaction. Therefore:
- Burst failure never spends sustained budget.
- Sustained failure rolls back the provisional burst increment.
- A rejected request consumes neither counter.
Reuse the existing atomic conditional-update primitive. Do not hold the SQLite
writer lock while fetching candidates or scoring.
If a rejection occurs, inspect both relevant counter rows at the same reference
time and return the longest applicable `Retry-After`. This prevents a client from
waiting for the minute window only to discover that the hour window was also
full.
### 7.3 Processing order and consumption
Exact route order:
1. Validate and normalize `q`; invalid/short inputs stop with no budget.
2. Resolve client identity.
3. Atomically consume burst and sustained budgets.
4. Commit both counters.
5. Probabilistically purge expired rate-limit rows using shared retention.
6. Read database corpus generation.
7. Check the process-local LRU and entry expiry.
8. Cache hit: load at most 50 summaries and render; no scoring slot.
9. Cache miss: acquire scoring semaphore non-blockingly.
10. No slot: inline `503`, `Retry-After: 1`.
11. With a slot: fetch candidates and score inside `try/finally`.
12. Always release the slot.
13. Conditionally cache the immutable result and render.
Consequences that must remain explicit:
- A cache hit still consumes both request counters.
- A `503` caused by scoring capacity still consumes both counters because they
committed earlier. Retrying is itself load and is not refunded.
- A scorer exception still consumes both counters but releases the semaphore.
- Valid `HEAD /search?q=...` follows the same rate/cache/scoring contract as GET;
HEAD cannot become a protection bypass.
### 7.4 Responses
- Burst/sustained rejection: inline search shell, `429`, calculated
`Retry-After`.
- Scoring capacity full: inline search shell, `503`, `Retry-After: 1` by default.
- Both responses keep the search form and a link to the ledger.
- Neither uses the generic dead-end error page.
- All carry the normal search shell security and cache headers.
### 7.5 Shared retention and cleanup
Move rate-limit retention calculation out of the private unlock route and into
shared rate-limit/config code:
```text
retention = max(
publish window,
PIN unlock window,
search burst window,
search sustained window,
)
```
Every consumer must use this one calculation. Retention below any live window
would allow probabilistic cleanup to delete a live counter and grant fresh
budget.
Valid accepted searches call `maybe_purge_expired_rate_limit_rows` after their
counter transaction commits. Cleanup remains best effort and never converts an
otherwise successful request into a 500.
---
## 8. Scoring concurrency
Create `app/search_limits.py` or an equivalently focused module containing the
process-wide `threading.BoundedSemaphore`.
Default capacity: `SEARCH_CONCURRENCY=2`.
Use non-blocking acquisition. Do not queue search scoring work inside the
application:
```python
slot = acquire_search_slot_nonblocking()
try:
result = score(...)
finally:
slot.release()
```
The route remains a sync `def`. FastAPI runs it in the worker threadpool while
RapidFuzz's C++ work can release the GIL. An `async def` route would run
CPU-bearing work on the event-loop thread and is forbidden by the existing
route-handler rule.
The semaphore is per process. This is correct for the locked single-process
deployment. A future multi-worker deployment must multiply total capacity by
worker count or replace this mechanism; that is out of scope.
---
## 9. HTTP route and rendering
### 9.1 Router
Create `app/viewer/search.py` with explicit `GET` and `HEAD` support and include
its router from `app/main.py`.
Keep the handler sync. Dependencies:
- `Request`;
- database session;
- settings.
Missing/blank query redirects use the same private/no-store redirect headers as
the ledger/theme flows.
### 9.2 Headers
Every `/search` response, including 200/400/429/503, carries:
```text
Cache-Control: private, no-store
Vary: Cookie
X-Content-Type-Options: nosniff
Referrer-Policy: no-referrer
X-Robots-Tag: noindex, nofollow, noarchive
Content-Security-Policy: ... form-action 'self' ... script-src 'none' ...
```
Search uses server-side caching only. Browser/intermediary caching remains
disabled.
### 9.3 CSP trust contract
Amend the current enumeration into a capability rule while preserving least
privilege.
`form-action 'self'` is allowed only on shells that actually contain a form:
- `/`;
- `/page/{n}`;
- `/search`;
- `/p/{slug}/unlock`.
These remain `form-action 'none'`:
- Markdown host;
- HTML host;
- Mermaid helper;
- `/raw`;
- published HTML artifacts;
- ordinary error pages without the search form.
Do not flip existing assertions for HTML host or raw responses. Add positive
assertions specifically for ledger/search and preserve negative assertions for
every untrusted surface.
### 9.4 Shared rows and table semantics
Move common row formatting from `app/viewer/index.py` into
`app/viewer/rows.py`:
- `_as_utc`;
- `_expiry_label`;
- `_FORMAT_SIGILS`;
- row construction.
Keep date grouping specific to the index.
Reuse `_ledger.html`, but parameterize:
- accessible caption;
- whether date-band rows render;
- updated timestamp label format.
Ledger behavior remains:
```text
date band: 05 AUG 2026 UTC
Updated: 14:07
caption: Published pages, most recently updated first. Times are UTC.
```
Search behavior becomes:
```text
no date band
Updated: 05 AUG 2026 14:07
caption: Search results for “kubrenetes”, ordered by relevance. Times are UTC.
```
The visible line above results is one of:
```text
12 matches for “kubrenetes”
Showing the top 50 of 73 matches for “migration”
No titles match “zzzzzz”
```
It is ordinary visible content in reading order, not an `aria-live` region.
### 9.5 Templates and CSS
Add:
```text
app/templates/viewer/_search_form.html
app/templates/viewer/search.html
```
Include `_search_form.html` on `/`, `/page/{n}`, and `/search`. The result page
uses `search.html`; do not overload `home.html` with a large set of search-only
branches.
Form contract:
```html
```
Extend `viewer-home.css`, still scoped under the home/search host body class.
Preserve the sub-48rem table grid, explicit rows, and ARIA roles. Test keyboard
focus, long query wrapping, and the full updated timestamp on mobile.
---
## 10. Theme round-trip
`validate_next_path` remains path-only and continues rejecting `?` and `#`.
Add exactly bare `/search` to its allowlist.
Extend theme helpers with a separately validated display query. Build raw URL
separators in Python and let Jinja autoescape:
```python
href = f"/viewer/theme/{theme}?next={quote(next_path, safe='')}"
href += f"&q={quote(display_query, safe='')}"
```
Do **not** insert literal `&` in Python; Jinja would double-escape it.
The theme route may append `?q=` to its redirect only when:
- validated `next` is exactly `/search`;
- `q` passes the same display validation and 64-character bound.
Preserve the original validated spelling and case. Do not replace it with the
normalized match query. A `q` supplied with any other `next` path is dropped.
Theme `q` handling never consumes search rate-limit budget.
---
## 11. Privacy and security boundaries
### 11.1 Permanent search boundary
Search titles only. Bodies are in R2 and may be PIN-protected. Searching bodies
would create a body-existence/PIN-bypass oracle and require object reads. This is
a permanent boundary, not a Phase 1 optimization.
### 11.2 Lifecycle boundary
Candidate and cached-summary reads reuse:
```text
page_is_active_clause(now)
AND purge_started_at IS NULL
```
Locked pages remain included and retain their lock marker. Expired/claimed pages
must never surface from a fresh query or a cache hit.
### 11.3 Reflected input
Display text passes through the shared control-character sanitizer and Jinja
autoescape. Search CSP is `script-src 'none'`. No query is placed into a CSS
class, raw HTML, or script context.
### 11.4 Logging reality
Application/structured logs never explicitly record query text.
Because search uses a GET URL, query text may appear in:
- Uvicorn access logs;
- Traefik access logs if enabled;
- browser history;
- copied/shared links.
Document this honestly. `Referrer-Policy: no-referrer` prevents propagation in
HTTP referrer headers but does not remove the query from first-party access logs
or browser history.
---
## 12. Observability
Use structured key/value logs without query text.
Every `429`:
```text
event=search_rejected
reason=burst|sustained
client_bucket=
current_count=
limit=
window_seconds=
retry_after_seconds=
shared_bucket=true|false
```
Every concurrency `503`:
```text
event=search_rejected
reason=concurrency
capacity=2
retry_after_seconds=1
```
Every cache-miss score:
```text
event=search_scored
duration_ms=
candidate_count=
matched_count=
truncated=true|false
generation=
cached=true|false
```
Cache events needed for operations/tests:
```text
cache_hit
cache_miss
cache_expired
cache_generation_cleared
cache_capacity_eviction
cache_store_skipped_generation_changed
```
Avoid logging page titles as part of search telemetry; they are public but can
still be arbitrary user-controlled text.
The future performance trigger measures cache-miss scoring/candidate duration,
not total HTTP latency. Rate-limit lock contention, template rendering, and
network time must not be misdiagnosed as a need for FTS.
---
## 13. Future FTS5 escalation contract
FTS5 is not part of this implementation. Begin a measured FTS prototype when
either holds:
- active page count exceeds 25,000; or
- cache-miss search computation p95 exceeds 50 ms over at least 1,000 measured
cache misses in the production-shaped environment.
The future implementation must obey:
1. Add it through Alembic.
2. Index the same normalized title representation as Python.
3. Queries shorter than six normalized characters use the full scan.
4. Longer queries use distinct query trigrams joined with `OR`; never use an
all-trigrams strict match as a fuzzy recall filter.
5. Apply active and purge-claim predicates before Python ranking.
6. Do not impose a candidate-count cap before the Python ranker in the first FTS
version.
7. Feed candidates to the unchanged WRatio/cutoff/tie-break implementation.
8. Fall back to a full scan if FTS is unavailable or errors.
9. Validate insertion, deletion, substitution, transposition, substring, word
order, and Unicode cases against full-scan results.
10. Require 100% recall on the mandated typo suite and an explicitly recorded
high-recall target on a larger generated corpus.
11. Shadow-check a sample of production queries against full scan during rollout
without logging query text.
An exact-trigram prefilter cannot mathematically preserve every possible
`WRatio >= 70` result. FTS is therefore a quality-measured escalation, not an
automatic switch at a row-count threshold.
---
## 14. Test plan
### 14.1 Normalization and ranking unit tests
- NFKC-equivalent strings normalize identically.
- `Straße` and `strasse` normalize identically.
- Composed and decomposed accented forms normalize identically.
- Punctuation becomes word boundaries; whitespace collapses.
- Query and title use the same function.
- `kubrenetes` matches `Kubernetes migration plan` above cutoff 70.
- `migra` matches by substring.
- Exact title outranks substring and typo matches.
- Word reordering behaves as expected under WRatio.
- Scores below cutoff are absent.
- `total_matched` is recorded before limit.
- Tie ordering is score, UTC update time, UUID.
- Mixed naive/aware SQLite datetimes do not raise or reorder incorrectly.
- Empty and Unicode-only-normalized values do not crash.
- No NumPy import or dependency is required.
### 14.2 Validation tests
- No `q` → 303 `/`, no DB access, no rate call.
- Blank `q` → 303 `/`, no DB access, no rate call.
- Repeated `q` parameters → inline 400, no budget.
- One-character normalized query → guidance 200, no scan/budget.
- Raw 65-character query → inline 400, no scan/budget.
- Raw ≤64 whose NFKC result exceeds 64 → inline 400, no scan/budget.
- Exactly 64 raw and normalized characters is accepted.
- Control characters are removed from display text.
- `