# lexsub

An English lexical-substitution tool. **Classic is the default and preferred engine; it never loads or uses a transformer.** Vanilla JavaScript ES modules, no build step, framework, account, API key, or inference server. Text is processed in your browser. Models are downloaded from Hugging Face; transformer code comes from jsDelivr.

## Start

```sh
cd /home/znetsixe/tools/lexsub
./serve.sh
```

Open **http://127.0.0.1:8765** in a recent Chromium browser. Python 3 is the only serving dependency. Do not open `index.html` with `file://`: workers, fetch, and model caching require HTTP. The server binds only to localhost. Stop it with Ctrl+C. Serve the delivered files as-is; no npm installation is needed to use the app.

With `./serve.sh`, Classic works without an Internet connection, including on its first run: all of its data and compromise modules are included. On a public host, first use downloads static files from that host; this is not an offline PWA. No text is sent to an inference service. Transformer modes need Internet on first use. Their files use the browser's Cache API where supported. Browser storage eviction/private mode can require another download. WebGPU is attempted if an adapter is available, with WASM fallback on loading or inference failure. Inference and Classic scoring run in a dedicated module worker. WASM uses one thread, so the plain Python server needs no isolation headers.

## Public sub-path hosting

All local asset URLs are relative (including the worker, modules, CSS, dictionaries and binary tables). The acceptance server exposes the app **only** at `/lexsub/` and rejects requests outside that prefix. Serve it at `https://theoptimal.nl/lexsub/` or another directory with the trailing slash. Use HTTPS publicly for clipboard/WebGPU support. Transformer options retain their external, version-pinned jsDelivr and Hugging Face URLs; Classic makes no such requests.

Prepare a runtime-only directory with:

```sh
python3 tools/package_runtime.py .build/site/lexsub
```

Copy the contents of that directory to the host's `lexsub/` directory. It includes `index.html`, the seven JS/CSS files, `README.md`, `data/`, `vendor/`, and `licences/`. It excludes test evidence, benchmark data, scripts, browser profiles and build dependencies. The packaging script accepts only a new destination within this workspace. Configure HTTP gzip/Brotli on JSON/JS/CSS and binary resources; `.bin` and `.i8` use `application/octet-stream`. Raw files remain the canonical artifacts. Cache content-versioned assets appropriately when deploying updates; keep all data files from the same build together. No upload/deployment was performed.

## Controls

| Control | Behaviour |
|---|---|
| Engine | Explicitly choose Classic, BERT only, T5 only, or Hybrid. The default is Classic. |
| Synonym constraint | Off permits broader vocabulary; Prefer boosts WordNet synonyms; Strict rejects everything outside the original lemma's same-POS synsets, keeping the original if nothing survives. Applies to every engine's word candidates. This is a lexical constraint, not a guarantee of equivalent meaning. |
| Synonym boost | In Prefer, multiply synonym candidate weight by `1 + boost` (default 3). |
| Temperature | 0 chooses the highest-weight candidate; higher values increase variety. Default 0.7. T5 generation has a minimum sampling temperature of 0.1 to retain a sampled candidate set. |
| Selection | Every N eligible words (default 3) or independent probability p (default 0.2). |
| Shift if no candidate | On by default. In Every N mode, try subsequent eligible positions until one succeeds, stopping before the next scheduled position. No backtracking and no relaxed filters; Random mode never shifts. Spans stay inside their scheduled slot. |
| Every N / skip first | After skipping the offset number of eligible words, choose the Nth, 2Nth, etc. N=1 chooses every eligible word. |
| Seed / Re-roll | Seeded mulberry32 controls positions, candidate sampling, T5 generation, and sentence transforms. Re-roll chooses a new seed; press Rewrite to apply it. Results are repeatable for the same settings and runtime, not guaranteed bit-identical across devices/backends. |
| Exclude words | Case-insensitive words separated by spaces, commas, or semicolons. Matches the input word form, not every inflection of its lemma. |
| Fill-mask model | DistilRoBERTa by default; the larger RoBERTa-base is also verified with the actual `fill-mask` pipeline. |
| Span-corruption model | Original T5-small or original T5-base. Neither is FLAN/instruction-tuned. |
| Top-k | BERT's candidate limit; also bounds Classic's ranked pool. T5 generation uses at least top-20 sampling. Default 20. |
| Span length | T5 starts at selected positions and extends across 1–3 eligible adjacent words, never across punctuation, an excluded word, or a sentence boundary. Overlaps are deduplicated. Short sentences reduce the effective length to fit their budget. |
| T5 samples | 2–8 independent samples in one explicit input batch, default 3. Using an input batch avoids the pinned runtime's ineffective `num_return_sequences` setting. |
| Mix α | Hybrid combines separately normalised BERT and T5 weights: `(1−α)B + αT`. 0 is exactly the BERT path; 1 is exactly the T5 path. At either endpoint the unused model is not loaded. |
| Substitute words | Independent Classic toggle. Uses same-POS WordNet lemmas with embeddings, then compromise inflection. |
| Embedding similarity | Weight for candidate/original cosine similarity. |
| Word frequency | Weight for the candidate's English Zipf frequency. |
| Local context fit | Weight for cosine similarity to the mean embedding of up to three words on each side within the sentence. |
| Neighbour similarity threshold | In Classic + Off, also consider the top 30 embedding neighbours above this cosine threshold with the same WordNet POS. Prefer includes WordNet and enabled PPDB candidates; Strict still requires WordNet membership. Enabled sense and fluency guards also apply to embedding neighbours. |
| PPDB paraphrases / Paraphrase weight | Enable the compact PPDB 2.0 S lexical Equivalence list. Merge candidates with WordNet; use PPDB2.0Score as a feature. Weight defaults to 0.6. |
| Contextual word senses / Sense weight | Select the highest-scoring WordNet synset using gloss/example overlap, IDF-weighted GloVe gloss vectors and a sense-rank prior. WordNet synonyms must belong to that synset. Weight 0.5 controls candidate/gloss cosine ranking, not the sense filter. |
| Sense context ± words | Sentence-bounded context window for disambiguation, default 8, range 3–15. |
| N-gram fluency check / Fluency weight | Enable a quantised trigram stupid-backoff scorer (CPU, typed arrays). Compare each replacement with the original in the ±3-word window. Weight defaults to 0.6. |
| Allowed fluency drop | Reject when the mean affected conditional log score falls by more than this value (default 1.5 natural-log units). Lower values keep more original words. This statistical check does not prove grammaticality or equivalent meaning. |
| Quality toggle vs weight | All three toggles start on. Each loads only its own resource on first use. Weight 0 removes ranking influence; **uncheck** to remove candidate generation/filtering. Disabling a loaded layer keeps it in worker memory until Cancel/reload. |
| Restructure sentences | Independent Classic toggle. At most one confident, enabled transform is sampled per sentence after word substitutions. Sentence transforms operate separately from the word-synonym constraint. |
| Restructure probability | Chance to attempt a transform per sentence (default 0.35). |
| Rewrite / Cancel | Rewrite the input using current settings. Cancel terminates the worker and releases models; the previous output remains available. |
| Highlighted word | Tooltip gives original, source B/T/both/C, and normalised sampling weight. Click or Enter/Space cycles through all alternatives, then reverts to the original. |
| Underlined sentence | Tooltip names its transform and original sentence. Click or Enter/Space removes that transform, retaining any word substitutions beneath it. |
| Copy clean text | Copies plain text from the original string plus active edits, without markup. |
| Copy diff | Plain inline diff: `[-original-]{+replacement+}`. Sentence transforms appear as one edit. |
| Stats | Input word/number count, eligible count, active word swaps and percentages of **all input tokens** and **eligible words**, transformed sentences, and rewrite time including any downloads. |

Settings are saved in localStorage with exception handling. Fresh settings use Classic; an existing saved engine choice is preserved. Input and output text are not persisted. Editing settings or input does not alter the previous output until Rewrite. Source labels denote the candidate source; scores are sampling weights, not calibrated confidence.

## How edits work

Word/number tokens retain UTF-16 offsets into the original text. Spaces, tabs, line breaks, and surrounding punctuation are sliced from that text rather than reconstructed. Stopwords, numbers (including letters adjoining digits), words shorter than three characters, user exclusions, and capitalised words away from sentence starts are ineligible. Tokenisation is word-level; transformer subwords never determine selection positions.

BERT replaces a whole selected word with one mask and accepts alphabetic complete-word RoBERTa tokens with a word-boundary marker. T5 inserts `<extra_id_0>`, `<extra_id_1>`, etc., and parses generated sentinels back into spans. A round masks at most floor(15% of its sentence's words), with a one-word minimum. Long sentences are split into bounded context windows; context windows keep the target masks close together. Calls use original surrounding context, so a weak fill cannot cascade into subsequent rounds. Samples that lack an ending sentinel/EOS, contain punctuation/subword artifacts, or change a span's word count are discarded. Surviving per-word T5 weights combine frequency and reciprocal sample rank. BERT supplies softmax probabilities; Hybrid normalises each source before mixing. All modes remove original/empty/punctuation-only/fragment candidates (including isolated letters other than English a/I) and preserve lower/title/upper case. They also reject a candidate whose lemma already occurs anywhere in the original sentence **or** within five input word positions across sentence boundaries. Changed components of multiword T5 fills are checked together; one echo rejects the whole fill, and repeated lemmas within a fill are rejected. Unchanged span components are not proposed edits. Every retained cycling alternative is reserved against later positions, so cycling cannot introduce a duplicate between edits. This conservative policy may discard useful candidates.

Classic uses compromise sentence-context POS tags, WordNet exceptions and lemma rules, cosine scores from quantised GloVe, and frequency scores. Cosines are mapped from [-1,1] to [0,1]; frequency is Zipf/7 clipped to 1; the three weights form a weighted mean. Classic sampling weights are `exp(8 × weighted mean)` before normalisation so weak tail candidates are less likely than under nearly uniform linear weights. The optional sense layer selects a context-fitting synset under the original POS; it is heuristic and can choose incorrectly. Candidates must exist in the embedding vocabulary. Compromise reinflects nouns, past/third-person/gerund verbs, and comparative/superlative adjectives. Every resulting form must be in the trimmed vocabulary/frequency list, and must survive a lemma round trip. Candidate WordNet POS and compromise POS in the replaced context must match the original. Lexical POS evidence (including compromise’s explicit alternative readings) is also required because context tags alone can coerce verbs into nouns. Nouns retain number and reject gerunds; verbs retain past/participle/gerund/base/third-person forms; adjectives retain degree and cannot become numerals or determiners. These same guards apply to BERT/T5 in Prefer and Strict. Classic uses them in every constraint mode. This is conservative heuristic morphology, not a full grammar engine.

### Sentence transforms

| Transform | Default | Confident pattern / limitation |
|---|---|---|
| Clause swap | On | Because/Although/When/If at the front with a comma, or a trailing subordinate clause; both clauses must have recognisable subjects and verbs. Avoids extra internal punctuation. |
| Contractions | On | Explicit unambiguous pairs such as do not/don't and we are/we're. Avoids ambiguous 's and 'd. |
| Adverb movement | On | However/Therefore/Usually with a pronoun subject and a recognised verb. |
| Connective variation | **Off** | but ↔ however between simple clauses; pronoun + also ↔ final as well; in order to → to. Discourse scope may change. |
| Noun pair reorder | **Off** | `the A and B` with two confident plural common nouns. Order can carry emphasis or chronology. |
| Active → passive | **Off** | Only `The singular-noun known-past-verb the singular-noun.` with a small explicit participle dictionary. Voice changes emphasis and broader SVO recognition is unreliable. |
| Number words ↔ digits | **Off** | Zero–twelve directly before a plural noun. Numbers can be names, identifiers, or idioms. |

Sentence transforms intentionally change order and sometimes punctuation. The exact layout guarantee covers word substitutions and unchanged text; it does not prohibit a requested sentence transform from changing its sentence. Browser textareas normalise CRLF/CR input to LF; logical line breaks, blank lines, tabs, and spacing are retained.

## Models, data, provenance, and licences

| Component | Choice and source | Licence |
|---|---|---|
| Transformers.js | [`@huggingface/transformers@3.7.2`](https://www.npmjs.com/package/@huggingface/transformers/v/3.7.2), jsDelivr ES module; published 2025-08-15 | Apache-2.0 |
| Default BERT engine | [`Xenova/distilroberta-base`](https://huggingface.co/Xenova/distilroberta-base), q8 ONNX; compact masked-language model | Apache-2.0 upstream DistilRoBERTa |
| Larger BERT option | [`Xenova/roberta-base`](https://huggingface.co/Xenova/roberta-base), q8 ONNX; full 12-layer encoder | MIT upstream RoBERTa/fairseq |
| T5 options | [`Xenova/t5-small`](https://huggingface.co/Xenova/t5-small) and [`Xenova/t5-base`](https://huggingface.co/Xenova/t5-base), q8 encoder/merged decoder; original sentinel pretraining | Apache-2.0 upstream Google T5 |
| POS/inflection | [`compromise@14.14.4`](https://www.npmjs.com/package/compromise/v/14.14.4), jsDelivr `+esm` snapshot with pinned transitive modules, vendored for first-run offline use; verified in Chromium as native ES modules | MIT, including vendored dependencies |
| Synonyms | [Princeton WordNet 3.0](https://wordnet.princeton.edu/license-and-commercial-use), from NLTK's unchanged archive; 77,197 alphabetic lemmas with POS and separate synonym sets; single-lemma sets retained as POS evidence | WordNet 3.0 licence, copyright Princeton University 2006 |
| Embeddings | [GloVe 6B 50d](https://nlp.stanford.edu/projects/glove/), Jeffrey Pennington, Richard Socher, Christopher D. Manning (2014); first 40,000 lowercase alphabetic words of at least three letters in frequency-ordered vocabulary | Public Domain Dedication and License 1.0; this applies to the pretrained data, separately from GloVe's code licence |
| Frequency | [wordfreq 3.1.1](https://github.com/rspeer/wordfreq), Robyn Speer; English Zipf estimates for the same 40,000 words, rounded to 0.01 | Data derivative **CC BY-SA 4.0**; library Apache-2.0. Attribution and modification notice are embedded in `frequency.json`; retain them and `licences/wordfreq-data-attribution.txt` when redistributing. |

Wordfreq includes freely available SUBTLEX data; credit Marc Brysbaert and Boris New (2009), *Moving beyond Kucera and Francis: A Critical Evaluation of Current Word Frequency Norms and the Introduction of a New and Improved Word Frequency Measure for American English*. It also includes OpenSubtitles, Wikipedia, OPUS, and other sources detailed in the included attribution file. This trimmed frequency database remains under CC BY-SA 4.0. No source text from those corpora is included.

All four model revisions are pinned to repository commit hashes in `model-worker.js`. Model cards identify their upstream bases; the converted repositories do not repeat every upstream licence field. Licence texts are in `licences/`. The app downloads model files only when needed and retains at most one BERT and one T5 model at a time. Choosing a larger model replaces the corresponding loaded model on its next use.

Runtime data totals **48.87 MB raw**; the original resources occupy about 6.1 MB including the **2,000,000-byte** embedding table. New quality resources add 42.82 MB. Vectors are L2-normalised, rounded to int8 at scale 127, stored row-major, and renormalised in the worker. Vocabulary, frequency, and vectors share row order. Source/output hashes and schema notes are in `data/manifest.json`; vendored CDN URLs and original hashes are in `vendor/manifest.json`.

### New runtime resource costs

| File | Raw MB | gzip -9 MB | Loaded when |
|---|---:|---:|---|
| `data/lm-vocab.json` | 0.507 | 0.211 | Fluency enabled |
| `data/ngrams.bin` | 30.100 | 11.333 | Fluency enabled |
| `data/ppdb.json` | 0.145 | 0.044 | PPDB enabled |
| `data/sense-vectors.i8` | 2.313 | 1.784 | Sense enabled |
| `data/senses.json` | 9.751 | 3.120 | Sense enabled |

PPDB 2.0 S: Pavlick, Rastogi, Ganitkevitch, Van Durme, Callison-Burch (2015), [author’s data page](https://cs.brown.edu/people/epavlick/data.html), [paper](https://aclanthology.org/P15-2070/). The 4,775 retained pairs are single-word Equivalence paraphrases with same WordNet POS and vocabulary coverage. **CC BY 3.0** is corroborated by [NLTK distribution metadata](https://github.com/nltk/nltk_data/blob/gh-pages/DATASET-LICENSES.md). NLTK calls its PPDB subset 3.0 Unported; other descriptions specify 3.0 US; both texts are included. The live original licence/download host was unreachable. We used a checksum-pinned [S-all mirror](https://www.kaggle.com/datasets/cudawarrior/ppdb-2-0-s-all); its conflicting Apache metadata is not treated as relicensing. See `licences/classic-layers-attribution.txt` for this verification boundary and changes.

Sense data contains 46,266 WordNet 3.0 synsets, including singleton senses, with glosses/examples. GloVe-weighted means are normalised/quantised to int8. WordNet’s licence and GloVe PDDL apply; both are retained.

Fluency training uses all **86,183,060 regex word tokens** from the two WikiText-103 TRAIN shards, excluding headings. Each order retains the most frequent 1.5 million entries with count ≥3. [Salesforce’s dataset card](https://huggingface.co/datasets/Salesforce/wikitext/raw/main/README.md) identifies Merity, Xiong, Bradbury and Socher (2016), with underlying Wikipedia contributors; metadata lists CC BY-SA 3.0/GFDL while prose says 4.0. The modified n-gram table is distributed **CC BY-SA 4.0**, retaining original attribution and the 3.0 notice. No benchmark text was used for training. Full-corpus vs 20M-token development measurements are in the report. Server gzip/Brotli must be configured; gzip sizes above are measurements, not bundled duplicate files.

### Classic quality scoring

Starting with the original three-feature score `base = 8 × weighted mean`, add:

- PPDB: `3 × paraphraseWeight × clamp((PPDB2.0Score − 2)/3, 0, 1)`; zero for absent pairs.
- Sense: `3 × senseWeight × cosine(candidate, selected gloss vector)`.
- Fluency: `fluencyWeight × clamp(mean log-score delta, −8, 4)`.

Exponentiate, apply the existing Prefer boost, normalise and sample with Temperature. Strict checks WordNet membership regardless of PPDB. An unrepresented sense or a selected synset with no safe candidates does **not** fall back to other WordNet senses; Every-N may shift instead. Novel PPDB/neighbour candidates must fit the selected sense vector better than competing target senses (within 0.04); candidates explicitly belonging only to another WordNet sense are blocked. Glosses, PPDB score and fluency delta appear in changed-word tooltips.

The LM stores exact 48-bit integer trigram keys (float64 arrays), not collision-prone hashes, and int16 quantised natural-log probabilities at scale 1024. Unseen trigrams back off to bigrams then unigrams, with a 0.4 factor each time. Only the replaced word and following two conditional terms can change in a trigram model; these terms use context within ±3 words. Sentence boundaries use start tokens. Single-word inflection/POS/echo guards still run, including for every click-cycle option.

With all three quality toggles off, ranking reproduces the frozen Round-2 Classic implementation. No MiniLM, transformer runtime, neural sentence encoder or LSTM is involved. No LSTM comparison was attempted.

## Rebuild data and verify

These commands keep dependencies/cache under this folder. They do not initialise or modify a Git repository.

```sh
python3 -m pip --cache-dir .build/pip-cache install --target .build/python wordfreq==3.1.1
PYTHONPATH=.build/python python3 tools/build_data.py
python3 tools/vendor_compromise.py
python3 tools/fetch_licences.py
python3 -m pip --cache-dir .build/pip-cache install --target .build/python numpy==2.2.6 pyarrow==19.0.1
python3 tools/build_layers.py all
python3 tools/fetch_layer_licences.py
```

`build_data.py` downloads WordNet and only the beginning of the GloVe ZIP (the 50d member is first), inflates that prefix, selects 40,000 words, and builds the original five runtime resources and their manifest. It verifies WordNet's archive hash and the exact consumed GloVe text prefix. Downloads/build inputs are in `.build/`. The data generator's only semantic Python dependency is pinned wordfreq 3.1.1. The vendoring script fetches pinned jsDelivr ES modules and rewrites imports to local paths.

`build_layers.py` builds PPDB, sense data and the n-gram files and records byte counts/source/output SHA-256 hashes in `data/layers-manifest.json`. It downloads WordNet, a 585 MB compressed PPDB S-all mirror (stream-extracted; never expanded to disk), and WikiText training Parquet shards. Only compact derived files enter `data/`. The PPDB original host was unavailable; the mirror archive is pinned by SHA-256. The initial 20M-token experiment is reproducible with `python3 tools/build_layers.py lm`.

For tests, start `./serve.sh` in one terminal. For sub-path acceptance also run `python3 tools/serve_subpath.py`. Then:

```sh
npm --cache .build/npm-cache install --prefix .build --no-audit --no-fund --ignore-scripts playwright@1.56.1
node tools/test-core.mjs
node tools/test-layers.mjs
node tools/test-lazy.mjs
BASE_URL=http://127.0.0.1:8766/lexsub/ node tools/test-interactions.mjs
node tools/test-browser.mjs
SAMPLE=technical node tools/test-browser.mjs
node tools/test-interactions.mjs
node tools/test-spans.mjs
```

Tests use `/usr/bin/chromium-browser`; adjust that path in the scripts for another installed Chromium. The browser acceptance test genuinely downloads and executes models, exercises all 4 engines × 3 constraints, Classic restructuring on/off, both larger models, clipboard/layout, seeded T5 repeatability, Hybrid endpoints, and phone width. It can take minutes and several hundred MB of downloads. `test-browser.mjs classic` runs only Classic; `SAMPLE=technical` selects the 63-word technical/business paragraph; `bert`, `t5`, or `hybrid` runs that engine's three constraints. Raw results, samples, timings, warnings, and screenshots are in `test-results/`. See `report.md` for the actual run and its limitations.

## Benchmark reproduction

CoInCo (Kremer, Erk, Padó, Thater, EACL 2014) is provided by [IMS Stuttgart](https://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/coinco/) under **CC BY 3.0 US**, explicitly covering the MASC text and substitution annotations. Its source data and derived evaluation instances are downloaded only to `.build/` and removed before delivery; they are never bundled in runtime assets.

```sh
python3 tools/bench/fetch_coinco.py
node tools/bench/run.mjs dev 300
node tools/bench/run.mjs test
python3 tools/bench/assemble.py
```

The benchmark uses all 5,450 official test IDs with gold substitutions. P@1 counts a top-ranked lemma as correct if any annotator supplied it; abstentions/missing alignments count as misses. Reported conditional precision divides hits by attempted substitutions; coverage and weighted gold-vote recall@10 are also reported. This is a target-only candidate-generation evaluation: it bypasses Every-N/eligibility/shift, uses compromise's own POS (not gold POS), and ranks with temperature 0. It uses only the target sentence, not CoInCo's surrounding discourse. Gold multiword alternatives remain in denominators even though the app generates single words. These are clearly defined measures, not claimed official SemEval `best`/`oot` scores. The frozen baseline source is `tools/bench/round2-lexicon.mjs`. Runtime data never uses CoInCo training/test text or labels.

## Limits

This is a lexical exploration tool, not a meaning-preserving paraphraser. WordNet sense selection is approximate and sometimes rejects good options or selects a bad sense; static embeddings include bias and associations; POS tags and sentence boundaries are heuristic. Off can produce unsuitable words, and even Strict can be awkward. T5-small often repeats the original or produces unusable spans; Strict can therefore keep an entire paragraph unchanged. Review the result. Large models consume substantial memory; WebGPU driver compatibility varies. The current acceptance evidence is Chromium/WASM, not a claim that every browser or GPU was tested.

## Cleanup and retained evidence

The final delivery removes `.build/`, browser profiles, downloaded corpora and test/build dependencies after verification. `./serve.sh` needs none of them. `test-results/` retains small measurements, screenshots and earlier Round-2 outputs. `report.md` describes the current run and remaining awkward outputs; `test-results/round2-report.md` preserves the prior report. `tools/write_round3_report.py -o report.md` reproduces the current report from retained evidence.

After cleanup, with `./serve.sh` running: `node --experimental-websocket tools/test-runtime.mjs` uses installed Chromium via CDP without npm packages, asserts `.build/` is absent and tests Classic with external DNS blocked. The smoke test deletes its temporary profile on exit.