applyMMR recomputed every candidate's redundancy against every
already-selected result on each round, re-tokenizing the selected passages
each time. With C candidates and k selections that is O(k^2) calls to
getEnrichedPassage + tokenizeSimple over full chunk bodies, and the
diagnostic average-redundancy loop afterwards re-tokenizes another O(k^2)
times. Since k defaults to len(candidates), this is quadratic in the number
of results that survive dedup/rerank.
On a knowledge base where ~250 candidates reach the MMR stage this burns
tens of seconds of pure CPU inside every knowledge_search call, and no log
line attributes the time to MMR.
Cache each candidate's running max jaccard against the selected set and fold
in only the newly selected result per round, and reuse the already-computed
token sets for the average-redundancy diagnostic. Tokenization now happens
once per candidate. Selection output is unchanged, including tie-breaking;
the added test pins the new implementation against the previous one over a
deterministic corpus, and a benchmark contrasts the two.