Fuzzy title search on the ledger — implementation plan v2

Status: approved design, not implemented.

Target: current master after the public-ledger, PIN-lock, and page-expiry work.

Supersedes: version 1 of this plan in full. This version incorporates the codebase review, Docker benchmarks, and the subsequent operator decisions about normalization, validation, rate limiting, cache invalidation, expiry, CSP, theme round-trips, and the future FTS5 path.

Goal: add a search form to the public ledger that finds pages by current title, tolerates ordinary typos, preserves substring matching, and remains safe under unauthenticated load on a small single-process VPS.

Example:

query:  kubrenetes
title:  Kubernetes migration plan
result: match

This document is an implementation contract. An implementing agent should not need the design conversation that produced it.

1. Final product contract

1.1 Search surface

• Search is available from /, /page/{n}, and /search through the same ordinary HTML GET form.
• The results URL is GET /search?q=... and is linkable, refreshable, bookmarkable, and compatible with browser history.
• There is no JavaScript, typeahead, debounce, or client-side title corpus.
• Search covers the current pages.title value only.
• Search never reads R2 objects and never searches bodies.
• Search never matches slugs, PINs, UUIDs, content hashes, or version bodies.
• PIN-locked pages are included because their current titles and lock state are already public on the ledger. Clicking a locked result still enters the existing unlock flow.
• Expired and purge-claimed pages are excluded using the same lifecycle predicate as the ledger and API.
• Results are relevance-ordered, limited to the best 50, and not paginated.
• The visible result count reports every hit at or above the cutoff before the top-50 truncation.

1.2 Matching contract

• Library: rapidfuzz>=3.9.
• Do not add NumPy in this phase.
• Scorer: rapidfuzz.fuzz.WRatio.
• Processor passed to RapidFuzz: None; both query and titles are normalized by application code before scoring.
• Default cutoff: 70, settings-driven and validated within 0..100.
• Candidate ordering before scoring: updated_at DESC, id DESC.
• Final ordering: score descending, then updated_at DESC, then id DESC.
• Tie ordering is explicit and must not depend on RapidFuzz iteration order.
• total_matched is calculated before slicing to limit=50.
• truncated is exactly total_matched > len(hits).

Use process.extract(..., limit=None) or an equivalently benchmarked single-query API. Do not use process.cdist unless NumPy is explicitly added and a production-container benchmark demonstrates a material win. The scoping benchmark in python:3.12-slim did not demonstrate one.

1.3 Scale and datastore

• Expected corpus: hundreds to roughly 10,000 active pages.
• SQLite remains the source of truth.
• This feature introduces one small durable corpus-generation table.
• Because this is a nontrivial schema change and the repository has recorded that the next such change must introduce Alembic, this feature also introduces Alembic. This is an explicit scope decision, not a silent v1.1 expansion.
• No FTS table, external search service, second database, or in-memory title index is added in Phase 1.

1.4 Default protection settings

SEARCH_RESULT_LIMIT is configurable for tests and operations but must remain bounded to 1..50 in v1. The product contract is never more than 50 visible results.

2. Why this design

2.1 Why not FTS5 in Phase 1

FTS5's trigram tokenizer is useful for substring recall, not edit-distance matching. A strict trigram MATCH can discard the desired result before a fuzzy ranker sees it, especially for short strings and transpositions. spellfix1 is not present in the standard Python SQLite build. Fuzzy scoring therefore remains application work in Phase 1.

2.2 Why RapidFuzz without NumPy

The production base is python:3.12-slim; the project supports Python 3.11+. RapidFuzz publishes compatible manylinux wheels. A Docker benchmark using the production base found no useful single-query advantage from NumPy-backed process.cdist over RapidFuzz's extraction APIs. workers=-1 is also a poor fit: each request could occupy every core and defeat the meaning of an outer scoring semaphore.

Use one RapidFuzz dependency, one worker, and the process-wide application semaphore described below.

2.3 Why a database generation instead of hashing 10,000 rows per request

The earlier proposal built a cache fingerprint by fetching every active (page_id, current_version) pair and hashing it on every request. That was correct but made every cache lookup O(N). It avoided fuzzy scoring but did not avoid a 10,000-row scan.

The accepted design stores one monotonic database generation:

search_corpus_state
┌────┬────────────┐
│ id │ generation │
├────┼────────────┤
│ 1  │ 419        │
└────┴────────────┘

Every create or update that can change search results increments the generation in the same SQLite transaction as the page mutation. A cache hit therefore requires one O(1) generation read, not a corpus scan.

2.4 Why generation alone is insufficient

Logical expiry changes the active corpus at an exact time without writing to the database. Cache entries therefore carry valid_until, the earliest expiry among the active candidate corpus used to calculate the entry. A cache entry is usable only while both are true:

cached_generation == database_generation
now < valid_until        # when valid_until is not None

At equality the entry is stale because the lifecycle rule is expires_at <= now → inactive.

3. Exact query validation and normalization

3.1 Two query representations

Never conflate display and matching values:

• display_query: the validated original spelling and case, with control characters removed. It is shown in the input, result count, and empty state.
• match_query: the normalized value used only for scoring and cache keys.

Example:

raw/display:  "  Kubrenetes  "
match:        "kubrenetes"

3.2 Normalization function

Define one pure function and use it for both incoming queries and every title:

def normalize_match_text(value: str) -> str:
    """NFKC, casefold, replace non-alphanumerics, collapse whitespace."""

Required order:

1. Unicode NFKC normalization.
2. Unicode casefold().
3. Replace every maximal run of non-alphanumeric characters with one space.
4. Collapse whitespace runs to one ASCII space.
5. Strip leading and trailing whitespace.

Do not combine this with rapidfuzz.utils.default_process; that would create two normalization contracts. The function must normalize titles and queries identically, including Straße/strasse and composed/decomposed characters.

3.3 Validation decision table

Use request.query_params.getlist("q") so repeated parameters are handled explicitly.

The normalized-length check is separate because NFKC may expand a character. The HTML input has maxlength="64" as a convenience, but the server is the authority.

Extract the existing error-display sanitizer into a shared helper with an explicit maximum, for example:

sanitize_display_text(value: str, *, max_chars: int) -> str

The existing error path passes 120; search passes 64. Jinja autoescape remains mandatory.

4. Data model and Alembic transition

4.1 New model

Add a singleton table:

class SearchCorpusState(Base):
    __tablename__ = "search_corpus_state"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    generation: Mapped[int] = mapped_column(Integer, nullable=False)
    updated_at: Mapped[datetime] = mapped_column(
        DateTime(timezone=True),
        nullable=False,
    )

Database invariants:

• Exactly one logical row, id=1.
• Initial generation is 1.
• Generation is positive and monotonic.
• Application code never deletes the singleton row.
• A missing row is an operational/schema error, never silently recreated on a search request.

Use a SQLite check constraint for id = 1 and generation >= 1 if it is cleanly portable through the chosen Alembic migration.

4.2 Atomic generation bump

Provide a service helper that participates in the caller's transaction and does not commit independently:

def bump_search_corpus_generation(db: Session, *, now: datetime) -> None:
    result = db.execute(
        update(SearchCorpusState)
        .where(SearchCorpusState.id == 1)
        .values(
            generation=SearchCorpusState.generation + 1,
            updated_at=now,
        )
    )
    if result.rowcount != 1:
        raise SearchCorpusStateError(...)

Call it exactly once in the same transaction when:

• A page is successfully created.
• Any new version is successfully committed, including content-only updates. Even without a title change, updated_at, current version, tie ordering, and displayed metadata change.

Do not bump it when:

• Validation fails.
• R2 upload fails before the database transaction.
• A create transaction rolls back.
• An update loses the current-version race.
• An update loses the expiry/purge conditional gate.
• A commit fails and is rolled back.
• An already-expired page is claimed or physically deleted by the purge worker; it was already absent from the logical search corpus.
• A rate-limit row changes.

The bump and page change must commit or roll back together. There must never be a page mutation visible at generation N when that mutation should have produced generation N+1.

4.3 Reading generation

Use a narrow scalar query:

def get_search_corpus_generation(db: Session) -> int: ...

Search cache hits perform this query before looking up the LRU. Do not derive the generation from COUNT, timestamps, PRAGMA data_version, or rate-limit table writes.

4.4 Alembic adoption contract

Add:

alembic.ini
alembic/
  env.py
  script.py.mako
  versions/
    0001_pre_search_baseline.py
    0002_search_corpus_state.py

0001_pre_search_baseline represents the complete schema immediately before this search feature. Its upgrade path creates that complete schema for a new database. 0002_search_corpus_state creates and seeds the singleton table.

Transition existing installations safely:

1. If pages does not exist, run Alembic from an empty database through head.
2. If pages exists but alembic_version does not:
   • run the existing compatibility logic needed to reach the known pre-search schema, excluding the new search table;
   • validate required pre-search tables, columns, indexes, and constraints;
   • stamp 0001_pre_search_baseline only after validation succeeds;
   • upgrade to head.
3. If alembic_version exists, run upgrade head normally.

Do not stamp an unknown or partially migrated schema. A failed validation must exit nonzero with an actionable message and leave the database untouched beyond any transactionally safe compatibility work.

agent-pages init-db remains the operator entry point, but becomes the wrapper for this transition and Alembic upgrade. Update the Docker image to copy alembic.ini and alembic/, and add the Alembic dependency.

Tests may continue using Base.metadata.create_all() for isolated fresh databases, but migration tests must exercise the real init-db/Alembic path.

5. Search service and scoring

5.1 New service module

Create app/services/search.py. Storage/model access remains in app/services/; the MCP server remains HTTP-only.

Suggested immutable types:

@dataclass(frozen=True)
class SearchCandidate:
    summary: PageSummary
    normalized_title: str

@dataclass(frozen=True)
class SearchHit:
    page_id: UUID
    score: float

@dataclass(frozen=True)
class SearchResult:
    hits: tuple[SearchHit, ...]
    total_matched: int
    truncated: bool
    valid_until: datetime | None

The cache value may reuse SearchResult if its contents remain immutable.

5.2 Candidate query

Fetch only columns needed to score and render a PageSummary. Do not load page bodies, version bodies, or the stored pin_hash. Derive has_pin in SQL or via a narrow boolean expression.

Use one captured UTC reference time for the candidate statement:

WHERE (expires_at IS NULL OR expires_at > reference_now)
  AND purge_started_at IS NULL
ORDER BY updated_at DESC, id DESC

Reuse page_is_active_clause(reference_now) and the existing UTC helpers.

valid_until is the earliest non-null expires_at among the entire active candidate corpus used for that calculation, not merely the visible top 50. This is conservative: an irrelevant page expiry may invalidate a result, but an expiry can never leave a stale result usable.

5.3 Scoring pipeline

1. Normalize every candidate title with normalize_match_text.
2. Call RapidFuzz with already-normalized strings and processor=None.
3. Apply the configured cutoff.
4. Convert scores to ordinary Python floats.
5. Sort explicitly by score, UTC-normalized updated_at, and UUID.
6. Record total_matched.
7. Keep only the first limit hits.
8. Set truncated.

The service must not know about HTTP status codes, templates, client IPs, or rate-limit responses.

6. Process-local LRU cache

6.1 Key and value

Cache key:

(normalized_query, database_generation, cutoff, limit)

Example:

("kubrenetes", 419, 70, 50)

Cache value:

SearchCacheValue(
    hits=(
        SearchHit(page_id=page_a, score=90.0),
        SearchHit(page_id=page_b, score=81.0),
        # top 50 at most
    ),
    total_matched=73,
    truncated=True,
    valid_until=datetime(...),  # or None
)

The cache holds at most 256 search entries, not 256 page hits. Each entry contains at most 50 ID/score pairs.

6.2 Cache hit

A cache hit requires:

1. The exact key exists.
2. valid_until is None or now < valid_until.

On a hit:

1. Move the entry to the most-recently-used end.
2. Fetch current summaries for at most the cached 50 page IDs using the active predicate as a defensive check.
3. Build a dictionary by ID and restore relevance order by walking the cached hit tuple. SQL IN (...) ordering is not relied upon.
4. If an expected ID is missing or inactive, remove this entry and treat the request as a cache miss. This should be exceptional because generation and valid_until are the primary invariants.
5. Do not acquire a scoring semaphore slot.
6. Do not scan the full corpus or call RapidFuzz.

6.3 Cache miss

On a miss:

1. Acquire a scoring slot non-blockingly.
2. Fetch the active corpus.
3. Score, sort, count, and truncate.
4. Re-read the database generation before inserting.
5. If generation changed during the calculation, do not cache the result under the old generation. The request may render its consistent statement snapshot, but the next request must use the new generation.
6. If valid_until was reached during calculation, do not cache the result; refetch once with a new reference time before rendering if necessary to avoid surfacing a page that is already expired at response construction.
7. Store the immutable result only after successful scoring and validation.

6.4 Locks and concurrency

Use a small process-local lock around LRU state only:

• lookup;
• stale-entry removal;
• move-to-MRU;
• insertion/replacement;
• capacity eviction;
• generation-change clearing;
• explicit reset.

Never hold the cache lock during SQLite access, title normalization, scoring, summary lookup, or template rendering.

Duplicate computation after simultaneous misses is acceptable if avoiding it would require holding the cache lock. The scoring semaphore still bounds the work. A per-key single-flight mechanism is out of scope unless profiling proves duplicate misses material.

6.5 Eviction and invalidation

The rules are exact:

• Capacity: inserting a 257th key evicts the least recently used key.
• Access: a valid hit becomes most recently used.
• Replacement: recomputing the same key replaces its value and makes it MRU.
• Generation: when a request observes a new database generation, clear all entries belonging to earlier generations under the cache lock. They can never be valid again.
• Expiry: when a requested entry has now >= valid_until, remove only that exact key and treat it as a miss. Other entries are validated lazily on their own lookups.
• Restart/deploy: the process-local cache starts empty.
• Tests: expose reset_search_cache().
• Failure: an exception or capacity rejection never inserts a partial entry.

Because expiry does not increment generation, a recalculation after expiry may replace an old entry under the same key with a later valid_until.

7. Exact rate-limit contract

Rate limiting is implemented inside the valid /search path. It is not global middleware and applies to no other route or query parameter. In particular, theme round-trip q parameters consume no search budget.

7.1 Client identity

Reuse resolve_client_ip(request, settings):

• Direct/untrusted peer: use the socket peer.
• Trusted proxy with usable XFF: rightmost-walk to the first untrusted hop.
• Trusted proxy without usable XFF: use the proxy address and set shared=True.

Search creates two hashed counter keys per resolved identity:

sha256("search-ip-min:" + resolved_ip)
sha256("search-ip-hr:"  + resolved_ip)

With 100 distinct resolved IPs there can be 200 active search counter rows: one minute row and one hour row per IP. Rows are dynamic and eventually purged.

For shared=True, multiply both maximum request counts by SEARCH_RATE_LIMIT_SHARED_MULTIPLIER=5; windows remain unchanged:

normal: 20/minute, 200/hour
shared: 100/minute, 1000/hour

7.2 One transaction for both windows

Consume burst and sustained counters in one short independent SQLite session and one outer transaction:

BEGIN
  burst conditional increment
  sustained conditional increment
COMMIT

If either limit is full, roll back the transaction. Therefore:

• Burst failure never spends sustained budget.
• Sustained failure rolls back the provisional burst increment.
• A rejected request consumes neither counter.

Reuse the existing atomic conditional-update primitive. Do not hold the SQLite writer lock while fetching candidates or scoring.

If a rejection occurs, inspect both relevant counter rows at the same reference time and return the longest applicable Retry-After. This prevents a client from waiting for the minute window only to discover that the hour window was also full.

7.3 Processing order and consumption

Exact route order:

1. Validate and normalize q; invalid/short inputs stop with no budget.
2. Resolve client identity.
3. Atomically consume burst and sustained budgets.
4. Commit both counters.
5. Probabilistically purge expired rate-limit rows using shared retention.
6. Read database corpus generation.
7. Check the process-local LRU and entry expiry.
8. Cache hit: load at most 50 summaries and render; no scoring slot.
9. Cache miss: acquire scoring semaphore non-blockingly.
10. No slot: inline 503, Retry-After: 1.
11. With a slot: fetch candidates and score inside try/finally.
12. Always release the slot.
13. Conditionally cache the immutable result and render.

Consequences that must remain explicit:

• A cache hit still consumes both request counters.
• A 503 caused by scoring capacity still consumes both counters because they committed earlier. Retrying is itself load and is not refunded.
• A scorer exception still consumes both counters but releases the semaphore.
• Valid HEAD /search?q=... follows the same rate/cache/scoring contract as GET; HEAD cannot become a protection bypass.

7.4 Responses

• Burst/sustained rejection: inline search shell, 429, calculated Retry-After.
• Scoring capacity full: inline search shell, 503, Retry-After: 1 by default.
• Both responses keep the search form and a link to the ledger.
• Neither uses the generic dead-end error page.
• All carry the normal search shell security and cache headers.

7.5 Shared retention and cleanup

Move rate-limit retention calculation out of the private unlock route and into shared rate-limit/config code:

retention = max(
  publish window,
  PIN unlock window,
  search burst window,
  search sustained window,
)

Every consumer must use this one calculation. Retention below any live window would allow probabilistic cleanup to delete a live counter and grant fresh budget.

Valid accepted searches call maybe_purge_expired_rate_limit_rows after their counter transaction commits. Cleanup remains best effort and never converts an otherwise successful request into a 500.

8. Scoring concurrency

Create app/search_limits.py or an equivalently focused module containing the process-wide threading.BoundedSemaphore.

Default capacity: SEARCH_CONCURRENCY=2.

Use non-blocking acquisition. Do not queue search scoring work inside the application:

slot = acquire_search_slot_nonblocking()
try:
    result = score(...)
finally:
    slot.release()

The route remains a sync def. FastAPI runs it in the worker threadpool while RapidFuzz's C++ work can release the GIL. An async def route would run CPU-bearing work on the event-loop thread and is forbidden by the existing route-handler rule.

The semaphore is per process. This is correct for the locked single-process deployment. A future multi-worker deployment must multiply total capacity by worker count or replace this mechanism; that is out of scope.

9. HTTP route and rendering

9.1 Router

Create app/viewer/search.py with explicit GET and HEAD support and include its router from app/main.py.

Keep the handler sync. Dependencies:

• Request;
• database session;
• settings.

Missing/blank query redirects use the same private/no-store redirect headers as the ledger/theme flows.

9.2 Headers

Every /search response, including 200/400/429/503, carries:

Cache-Control: private, no-store
Vary: Cookie
X-Content-Type-Options: nosniff
Referrer-Policy: no-referrer
X-Robots-Tag: noindex, nofollow, noarchive
Content-Security-Policy: ... form-action 'self' ... script-src 'none' ...

Search uses server-side caching only. Browser/intermediary caching remains disabled.

9.3 CSP trust contract

Amend the current enumeration into a capability rule while preserving least privilege.

form-action 'self' is allowed only on shells that actually contain a form:

• /;
• /page/{n};
• /search;
• /p/{slug}/unlock.

These remain form-action 'none':

• Markdown host;
• HTML host;
• Mermaid helper;
• /raw;
• published HTML artifacts;
• ordinary error pages without the search form.

Do not flip existing assertions for HTML host or raw responses. Add positive assertions specifically for ledger/search and preserve negative assertions for every untrusted surface.

9.4 Shared rows and table semantics

Move common row formatting from app/viewer/index.py into app/viewer/rows.py:

• _as_utc;
• _expiry_label;
• _FORMAT_SIGILS;
• row construction.

Keep date grouping specific to the index.

Reuse _ledger.html, but parameterize:

• accessible caption;
• whether date-band rows render;
• updated timestamp label format.

Ledger behavior remains:

date band: 05 AUG 2026 UTC
Updated:   14:07
caption:   Published pages, most recently updated first. Times are UTC.

Search behavior becomes:

no date band
Updated:   05 AUG 2026 14:07
caption:   Search results for “kubrenetes”, ordered by relevance. Times are UTC.

The visible line above results is one of:

12 matches for “kubrenetes”
Showing the top 50 of 73 matches for “migration”
No titles match “zzzzzz”

It is ordinary visible content in reading order, not an aria-live region.

9.5 Templates and CSS

Add:

app/templates/viewer/_search_form.html
app/templates/viewer/search.html

Include _search_form.html on /, /page/{n}, and /search. The result page uses search.html; do not overload home.html with a large set of search-only branches.

Form contract:

<form class="ledger-search" role="search" method="get" action="/search">
  <label class="sr-only" for="q">Search page titles</label>
  <input id="q" name="q" type="search" maxlength="64"
         value="{{ display_query }}" placeholder="Search titles"
         autocomplete="off" spellcheck="false">
  <button type="submit">Search</button>
</form>

Extend viewer-home.css, still scoped under the home/search host body class. Preserve the sub-48rem table grid, explicit rows, and ARIA roles. Test keyboard focus, long query wrapping, and the full updated timestamp on mobile.

10. Theme round-trip

validate_next_path remains path-only and continues rejecting ? and #. Add exactly bare /search to its allowlist.

Extend theme helpers with a separately validated display query. Build raw URL separators in Python and let Jinja autoescape:

href = f"/viewer/theme/{theme}?next={quote(next_path, safe='')}"
href += f"&q={quote(display_query, safe='')}"

Do not insert literal & in Python; Jinja would double-escape it.

The theme route may append ?q= to its redirect only when:

• validated next is exactly /search;
• q passes the same display validation and 64-character bound.

Preserve the original validated spelling and case. Do not replace it with the normalized match query. A q supplied with any other next path is dropped.

Theme q handling never consumes search rate-limit budget.

11. Privacy and security boundaries

11.1 Permanent search boundary

Search titles only. Bodies are in R2 and may be PIN-protected. Searching bodies would create a body-existence/PIN-bypass oracle and require object reads. This is a permanent boundary, not a Phase 1 optimization.

11.2 Lifecycle boundary

Candidate and cached-summary reads reuse:

page_is_active_clause(now)
AND purge_started_at IS NULL

Locked pages remain included and retain their lock marker. Expired/claimed pages must never surface from a fresh query or a cache hit.

11.3 Reflected input

Display text passes through the shared control-character sanitizer and Jinja autoescape. Search CSP is script-src 'none'. No query is placed into a CSS class, raw HTML, or script context.

11.4 Logging reality

Application/structured logs never explicitly record query text.

Because search uses a GET URL, query text may appear in:

• Uvicorn access logs;
• Traefik access logs if enabled;
• browser history;
• copied/shared links.

Document this honestly. Referrer-Policy: no-referrer prevents propagation in HTTP referrer headers but does not remove the query from first-party access logs or browser history.

12. Observability

Use structured key/value logs without query text.

Every 429:

event=search_rejected
reason=burst|sustained
client_bucket=<hashed key>
current_count=<n>
limit=<n>
window_seconds=<n>
retry_after_seconds=<n>
shared_bucket=true|false

Every concurrency 503:

event=search_rejected
reason=concurrency
capacity=2
retry_after_seconds=1

Every cache-miss score:

event=search_scored
duration_ms=<n>
candidate_count=<n>
matched_count=<n>
truncated=true|false
generation=<n>
cached=true|false

Cache events needed for operations/tests:

cache_hit
cache_miss
cache_expired
cache_generation_cleared
cache_capacity_eviction
cache_store_skipped_generation_changed

Avoid logging page titles as part of search telemetry; they are public but can still be arbitrary user-controlled text.

The future performance trigger measures cache-miss scoring/candidate duration, not total HTTP latency. Rate-limit lock contention, template rendering, and network time must not be misdiagnosed as a need for FTS.

13. Future FTS5 escalation contract

FTS5 is not part of this implementation. Begin a measured FTS prototype when either holds:

• active page count exceeds 25,000; or
• cache-miss search computation p95 exceeds 50 ms over at least 1,000 measured cache misses in the production-shaped environment.

The future implementation must obey:

1. Add it through Alembic.
2. Index the same normalized title representation as Python.
3. Queries shorter than six normalized characters use the full scan.
4. Longer queries use distinct query trigrams joined with OR; never use an all-trigrams strict match as a fuzzy recall filter.
5. Apply active and purge-claim predicates before Python ranking.
6. Do not impose a candidate-count cap before the Python ranker in the first FTS version.
7. Feed candidates to the unchanged WRatio/cutoff/tie-break implementation.
8. Fall back to a full scan if FTS is unavailable or errors.
9. Validate insertion, deletion, substitution, transposition, substring, word order, and Unicode cases against full-scan results.
10. Require 100% recall on the mandated typo suite and an explicitly recorded high-recall target on a larger generated corpus.
11. Shadow-check a sample of production queries against full scan during rollout without logging query text.

An exact-trigram prefilter cannot mathematically preserve every possible WRatio >= 70 result. FTS is therefore a quality-measured escalation, not an automatic switch at a row-count threshold.

14. Test plan

14.1 Normalization and ranking unit tests

• NFKC-equivalent strings normalize identically.
• Straße and strasse normalize identically.
• Composed and decomposed accented forms normalize identically.
• Punctuation becomes word boundaries; whitespace collapses.
• Query and title use the same function.
• kubrenetes matches Kubernetes migration plan above cutoff 70.
• migra matches by substring.
• Exact title outranks substring and typo matches.
• Word reordering behaves as expected under WRatio.
• Scores below cutoff are absent.
• total_matched is recorded before limit.
• Tie ordering is score, UTC update time, UUID.
• Mixed naive/aware SQLite datetimes do not raise or reorder incorrectly.
• Empty and Unicode-only-normalized values do not crash.
• No NumPy import or dependency is required.

14.2 Validation tests

• No q → 303 /, no DB access, no rate call.
• Blank q → 303 /, no DB access, no rate call.
• Repeated q parameters → inline 400, no budget.
• One-character normalized query → guidance 200, no scan/budget.
• Raw 65-character query → inline 400, no scan/budget.
• Raw ≤64 whose NFKC result exceeds 64 → inline 400, no scan/budget.
• Exactly 64 raw and normalized characters is accepted.
• Control characters are removed from display text.
• <script> query text is escaped in input, count, and empty state.
• Display spelling/case is preserved while matching uses normalized text.

14.3 Lifecycle and corpus tests

• Active open page included.
• Active PIN-locked page included with lock marker.
• Expired page excluded.
• Page expiring exactly at reference time excluded.
• purge_started_at page excluded.
• Search does not call R2.
• Search does not select or expose stored PIN hashes.
• Slug-only match does not produce a result.
• Body-only match does not produce a result.
• Title update changes subsequent matches.

14.4 Generation model/service tests

• Singleton row is present with generation 1 on a fresh DB.
• Generation scalar read returns the row.
• Missing singleton row raises an operational error.
• Successful create increments exactly once.
• Successful title update increments exactly once.
• Successful content-only update increments exactly once.
• R2 upload failure does not increment.
• Slug-allocation/create rollback does not increment.
• Update version conflict does not increment.
• Update rejected by expiry/claim does not increment.
• Database commit failure rolls back page mutation and generation bump together.
• Rate-limit row writes do not increment generation.
• Purging an already-expired page does not increment generation.
• Concurrent successful page writes produce distinct monotonic generations with no lost increment.

14.5 Alembic/migration tests

• Empty database upgraded to head has the complete schema and singleton row.
• Current pre-search database without alembic_version validates, stamps the baseline, upgrades, and preserves all rows.
• Representative older supported database first reaches the pre-search schema, then stamps/upgrades.
• Unknown/partial schema refuses to stamp and exits nonzero.
• Running init-db twice is idempotent.
• Existing page/PIN/expiry/purge/deletion/rate-limit data remains unchanged.
• Existing active pages are searchable after migration.
• Alembic downgrade/upgrade behavior follows the repository's documented support policy.
• Docker image contains Alembic config/revisions and entrypoint initialization reaches head.

14.6 Cache tests

• Key includes normalized query, generation, cutoff, and limit.
• First request misses, scores, and stores immutable tuples.
• Repeated request at same generation and before valid_until hits.
• Hit reads at most cached result IDs and does not fetch the full corpus.
• Hit does not acquire a scoring slot or call RapidFuzz.
• Hit restores cached relevance order regardless of SQL IN return order.
• Missing cached page ID removes the entry and falls back to a miss.
• Generation change clears old-generation entries.
• An in-flight old-generation calculation is not inserted after generation changes.
• valid_until=None remains time-valid.
• now < valid_until hits.
• now == valid_until misses and removes that exact entry.
• now > valid_until misses and removes that exact entry.
• Expiring one entry does not remove an unrelated later-valid entry.
• Recalculation may replace the same key with a later valid_until.
• Access moves a key to MRU.
• Inserting entry 257 evicts the correct LRU key.
• Replacing an existing key does not grow cache length.
• Scoring exception stores nothing.
• Generation mismatch stores nothing.
• Cache reset empties all state.
• Concurrent get/put/reset operations do not corrupt ordering or exceed capacity.

14.7 Rate-limit tests

• Valid search atomically increments burst and sustained counters.
• Burst failure leaves both counters unchanged.
• Sustained failure rolls back the provisional burst increment.
• If both are full, Retry-After reflects the longest applicable wait.
• Twenty valid searches/minute pass; the 21st returns inline 429.
• Hourly limit rejects even when a fresh minute window has room.
• Cache hits still consume both budgets.
• Concurrency 503 still consumes both budgets.
• Missing/blank/short/over-length/repeated-q inputs consume nothing.
• Theme q consumes nothing.
• Shared proxy bucket applies the 5× maxima with unchanged windows.
• Direct/untrusted peers cannot spoof XFF.
• Search uses two keys per resolved client identity.
• HEAD follows the same budget rules as GET.
• Search calls best-effort expired-row cleanup after accepted valid requests.
• Shared retention covers the longest configured publish/PIN/search window.
• Cleanup failure does not change the search response.
• 429 log contains reason/count/limit/bucket but no query.

14.8 Semaphore and failure tests

• Default capacity is two.
• Cache hit bypasses semaphore.
• Cache miss acquires exactly one slot.
• With both slots held, next miss returns inline 503 and Retry-After: 1.
• Non-blocking acquisition does not queue.
• Success releases the slot.
• Scoring exception releases the slot.
• Candidate-fetch exception releases the slot.
• Cache insertion exception cannot leak a slot.
• Route handler is not a coroutine function.
• 503 structured log contains capacity/retry fields and no query.

14.9 Rendering/accessibility tests

• Search form appears on /, /page/{n}, and /search.
• Search form has role=search, a real label, and maxlength=64.
• Search input is prefilled with validated display text.
• Result count is visible before the table.
• Empty state occupies the same outcome slot.
• Truncated state says “Showing the top 50 of N matches”.
• Search table caption says relevance order, not newest-first.
• Search has no date bands.
• Search Updated cells show full UTC date and time.
• Ledger retains date bands and short Updated time.
• Table and mobile-grid ARIA roles remain intact.
• Locked row keeps accessible Locked text.
• Long titles/queries do not overflow at 390 px.
• Keyboard focus is visible on input, button, result links, and theme controls.

14.10 CSP/header/theme tests

• /, /page/{n}, /search, and unlock carry form-action 'self'.
• Markdown host, HTML host, Mermaid helper, raw, and artifacts remain form-action 'none'.
• Search remains script-src 'none'.
• Raw/artifact/Mermaid CSP is byte-for-byte unaffected except intentional shared test refactors.
• Every search response and redirect is private/no-store and varies on Cookie.
• Search carries nosniff, no-referrer, and noindex headers.
• Bare /search passes validate_next_path; /search?q=x still fails it.
• Theme href uses one correctly escaped &q=, never &q=.
• Theme round-trip preserves original validated spelling/case.
• Theme route drops q for every next other than /search.
• Malicious next and q values cannot create an external redirect.

14.11 Observability and privacy tests

• Cache-miss timing event contains no query or title.
• Cache hit/miss/expiry/generation/eviction events are distinguishable.
• 429/503 events contain actionable limit metadata.
• Application error logging uses the route path without appending query text.
• Documentation explicitly notes access-log/browser-history exposure.

14.12 Production-shaped benchmark

Add a repeatable benchmark script or documented command; do not make fragile wall-clock assertions in ordinary unit tests.

Run inside the built Docker image with:

• 10,000 representative short titles;
• typo, substring, exact, common-word, and no-match queries;
• cold cache and warm cache;
• 1 and 2 simultaneous cache misses;
• maximum allowed query length;
• representative 512-character titles.

Record:

• generation-read latency;
• cache-hit summary-fetch latency;
• candidate query/materialization latency;
• normalization latency;
• RapidFuzz latency;
• total search service latency;
• memory used by a full 256-entry LRU.

Confirm the installed RapidFuzz wheel requires no source build and NumPy is not installed transitively.

15. Implementation sequence

Keep commits reviewable and preserve correctness at each stage:

1. Alembic foundation: dependency, config, full pre-search baseline, transition wrapper, Docker copy, migration tests.
2. Corpus state: model, singleton seed, scalar read, atomic bump helper.
3. Mutation integration: create/update bump in the same commit; rollback and concurrency tests.
4. Normalization/scoring service: pure normalization, candidates, WRatio, deterministic sorting, service tests.
5. Cache: generation key, immutable value, valid_until, LRU lock, eviction/invalidation, concurrency tests.
6. Rate settings/shared retention: validated config, shared retention helper, .env.example, README.
7. Atomic two-window limiter: one transaction, longest Retry-After, logs, cleanup integration.
8. Scoring semaphore: non-blocking two-slot limit and reset/test hooks.
9. Shared viewer rows: extract row builder; parameterize caption/date band/ full update label without changing ledger output.
10. Search route/templates: sync handler, validation states, results/degraded states, main router registration.
11. Search form/CSP: form on all ledger/search pages, least-privilege CSP, security tests.
12. Theme query round-trip: bare path allowlist plus separately validated display query.
13. Integration/accessibility tests: full response matrix, lifecycle, cache, mobile semantics, headers.
14. Benchmark: production Docker measurements and cache memory report.
15. Documentation/decisions: AGENTS, MEMORY, architecture, README, .env.example, Alembic operations.
16. Final verification: full test suite, clean worktree review, Docker build, new-volume and upgraded-volume smoke tests.

Do not expose /search until migrations, generation bumping, active filtering, rate limits, semaphore, cache expiry, CSP, and degraded states land together.

16. File map

Expected additions:

alembic.ini
alembic/env.py
alembic/script.py.mako
alembic/versions/0001_pre_search_baseline.py
alembic/versions/0002_search_corpus_state.py
app/services/search.py
app/search_cache.py
app/search_limits.py
app/viewer/search.py
app/viewer/rows.py
app/templates/viewer/search.html
app/templates/viewer/_search_form.html
tests/test_search_service.py
tests/test_search_cache.py
tests/test_search_rate_limit.py
tests/test_search_viewer.py
tests/test_search_migration.py

Expected edits:

pyproject.toml
Dockerfile
app/models.py
app/config.py
app/cli.py
app/main.py
app/rate_limit.py
app/services/pages.py
app/viewer/index.py
app/viewer/errors.py
app/viewer/theme.py
app/viewer/routes.py
app/viewer/shell.py
app/viewer/unlock.py
app/templates/viewer/home.html
app/templates/viewer/_ledger.html
app/static/viewer-home.css
.env.example
README.md
AGENTS.md
MEMORY.md
docs/architecture.md
tests/conftest.py
tests/test_home_index.py
tests/test_home_css.py
tests/test_viewer_security.py
tests/test_viewer_theme.py
tests/test_migration_pin_locks.py

Exact file splits may change if a smaller cohesive boundary is clearer, but the storage rule, MCP boundary, and test responsibilities do not change.

17. Success criteria

The feature is complete only when all are true:

1. A normal HTML search form appears on every ledger page and /search, works with JavaScript disabled, and produces a linkable GET URL.
2. kubrenetes finds Kubernetes migration plan; substring, exact, word-order, and Unicode normalization cases meet the locked scoring contract.
3. Search covers current titles only, never bodies/slugs/R2, and includes locked titles without weakening their body gate.
4. Missing, blank, repeated, short, raw-overlength, and normalized-overlength queries follow the exact status/zero-budget decision table.
5. Active and purge predicates match the ledger/API; expired/claimed pages never appear from a fresh calculation or cache hit.
6. Results are deterministically ordered, count all cutoff hits, show no more than 50, and render a truthful relevance caption plus full UTC update times.
7. Alembic upgrades an empty DB and every supported pre-search DB without data loss; unknown schemas are never blindly stamped.
8. Every successful create/update increments the singleton generation exactly once in the same transaction; every failed/rolled-back mutation leaves it unchanged.
9. A warm cache hit reads one generation row and at most 50 summaries; it never scans the full corpus, calls RapidFuzz, or acquires a scoring slot.
10. Cache entries are immutable, capped at 256, evicted by correct LRU order, cleared on generation change, and removed per-entry at valid_until.
11. Exact-expiry behavior uses now >= valid_until as stale and cannot serve an expired page; valid_until=None remains valid until generation change or capacity/process eviction.
12. In-flight generation changes and scorer failures cannot insert stale or partial cache entries.
13. Burst and sustained counters commit atomically; rejection rolls back both and returns the longest applicable Retry-After.
14. Only valid /search queries consume search budget. Cache hits and capacity 503s do consume it; other routes and theme query parameters do not.
15. Trusted proxy resolution, shared-bucket 5× ceilings, and shared retention are tested and documented.
16. At most two fuzzy calculations run concurrently per process; capacity is non-blocking, returns inline 503, and every success/failure releases its slot.
17. Search/ledger/unlock alone receive form permission as required; Markdown, HTML host, Mermaid, raw, and artifact boundaries remain unchanged.
18. Theme changes preserve the validated display query without weakening the path-only redirect validator or double-escaping the URL.
19. Search responses are private/no-store, vary on Cookie, are non-indexable, send no referrer, and render reflected input only through sanitizer plus autoescape.
20. Structured application logs contain no query text, while documentation accurately warns about GET URLs in access logs, history, and copied links.
21. Production-container benchmarks are recorded for 10,000 pages; RapidFuzz installs from a wheel with no NumPy dependency, and cache memory remains acceptable for the VPS.
22. All existing tests plus the new migration/search/cache/rate/security suites pass; Docker builds; both a fresh volume and upgraded production-shaped volume reach readiness.
23. AGENTS.md, MEMORY.md, README, .env.example, and architecture docs reflect the final contract, including Alembic adoption and the future FTS trigger.

18. Explicitly out of scope

• Body search.
• Slug, UUID, or PIN search.
• Searching historical titles; titles remain current-page metadata until a separate versioned-title decision.
• Search API under /v1 or MCP search tool.
• Typeahead, autocomplete, client-side filtering, or JavaScript enhancement.
• Result pagination or more than 50 visible results.
• Facets for format, lock state, or expiry.
• FTS5 implementation in this phase.
• PostgreSQL, a search daemon, Redis, or a second datastore.
• Distributed rate limiting.
• Cross-process cache or semaphore coordination.
• Multi-worker deployment changes.
• Per-key single-flight suppression.
• Background cache-expiry timers; expiry is validated lazily.
• Hiding public titles of PIN-locked pages.
• Disabling Uvicorn/Traefik access logs as part of this feature.

19. Decision-record updates when implementation lands

Update the proposal entries for fuzzy search in MEMORY.md rather than leaving the v1 scoping text authoritative. Record at minimum:

• one shared normalization function for query/title;
• over-64 queries rejected with no budget;
• RapidFuzz without NumPy;
• database corpus generation plus Alembic;
• per-entry expiry deadline and exact LRU rules;
• atomic two-window rate transaction and shared multiplier;
• scoring concurrency default two;
• CSP form allowlist;
• query-preserving theme round-trip;
• truthful GET/access-log privacy statement;
• measured future FTS recall contract.

Amend the “do not regress” rules to cover generation bump atomicity, cache expiry, search-only title scope, rate transaction atomicity, and the form-action surface allowlist.