Sentinel 2.0: reuse observation embeddings across a sweep, guard result columns, and document the release - #36
Merged
Merged
Conversation
Three related changes for 2.0.0. run_grid_search re-encoded the observation texts on every scoring pass, even though an observation's embedding depends on the encoder and never on the index it is scored against. subsample() shares the parent's sentence model and encoding kwargs, so one set of embeddings is valid for a whole sweep. calculate_rare_class_affinity now accepts sample_embeddings, simulation gains encode_observations(), and run_grid_search hoists the encoding out of its loops. On a 2x2x3 sweep over 320 observations that is 2.99s to 0.52s, a 5.7x saving, with byte-identical rows. Kept behind cache_observation_embeddings so callers short of memory can opt out, and indices that cannot pre-encode fall back rather than failing, which keeps the harness usable with the duck-typed doubles it advertises support for. Result rows are plain dicts so they drop into a DataFrame, but that also means a second write to a key silently destroys the first - which is exactly how an index size once replaced the positive-group count. The grid-search columns are now named constants applied through _add_columns, which raises instead of overwriting. run_grid_search's arguments were ordered by the accident of when each was added, leaving the two index axes separated from the other sweep axes by unrelated plumbing. They are all keyword-only, so regrouping them by role breaks nothing; they now read in the same order as the cost-ordered loop the docstring describes. On SentinelLocalIndex the opposite was true: everything was positional-or-keyword, so a new argument could only be appended. sample_embeddings belongs beside text_samples, so the parameters after it become keyword-only, as do those of from_texts and load. Every call site in the repo already used keywords. Also converts __init__.py from tabs to spaces, the only such file in the repo and the source of all 29 W191 warnings. Co-authored-by: Cursor <cursoragent@cursor.com>
Adds V2_FEATURES.md covering each addition since 1.0 with what it is, why it exists and how to use it, plus migration notes for the two deliberate breaks: keyword-only arguments after the first, and the index_-prefixed grid-search columns. In the README, replaces the "What's New" section with a 2.0 summary table pointing at the relevant sections, adds a section on subsample(), and documents the index sweep axes and the observation embedding cache with its measured saving. Also separates the two things called "metrics" in the tuning section, since the aggregator is swept while the evaluation metrics are all reported. Co-authored-by: Cursor <cursoragent@cursor.com>
Appends a self-contained appendix showing from_texts(), the persisted corpus and seeded loading, subsample(), the grid-search index axes, and the observation embedding cache, with executed outputs. Self-contained deliberately: it builds a small index from a dozen example sentences rather than depending on the data-loading cells above, so it runs in seconds on its own. The shipped example index is used only for the subsample demonstration, where a realistic size is the point - it predates corpus support, so its explanations would show row numbers rather than text. Appended rather than woven in, so the diff is 789 insertions and no deletions and the existing 32 cells are byte-identical. The final aggregator comparison reports the whole table rather than picking a winner: all six separate this example perfectly, which says the example is easy, not that the aggregators are equivalent. Sorting by Cohen's d there produced a number like 8.5e15, because the within-class variance is zero on a fixture this clean. The real comparison is the earlier section, on real data. Co-authored-by: Cursor <cursoragent@cursor.com>
An audit of the public API added since 0e924ae against the appendix found four things demonstrated nowhere: evaluate_groups, DEFAULT_AGGREGATORS, load_corpus, and the sample_embeddings parameter itself, which the appendix only reached indirectly through encode_observations. Adds a section introducing the harness by cost - score once, then evaluate with one aggregator, then all of them, then sweep - which is where evaluate_groups and DEFAULT_AGGREGATORS naturally belong, and which also shows the three metric families on a single row. The group fixture moves there, since that section is now the first to use it. Extends the corpus section with load_corpus, reading the texts back without the embeddings. It returns (None, None) for the shipped example index, which is a concrete demonstration of the pre-2.0 format rather than an assertion about it. Extends the caching section with the underlying parameter, including the refusal when the embeddings do not line up with the text. Co-authored-by: Cursor <cursoragent@cursor.com>
The "Tuning aggregation strategies" section is itself a 2.0 feature - the simulation harness arrived in #31, inside this release - so the appendix opened on a false premise by calling everything above it 1.0. Corrected, and it now points at that section as the worked example on real data. Adds section 7, which sweeps the axes that section could not: three index sizes x three negative ratios x three top_k x two thresholds over the real 30-vs-30 podcast set, 22,885 segments, 324 rows. It runs on the groups built earlier in the notebook rather than the appendix's toy fixture, and is marked as depending on them. That sweep is only practical because of the embedding cache. Encoding those segments takes 118s and a scoring pass afterwards takes 1.4s, so the 27 configurations are 2.6 minutes rather than 54. Three findings, all from the executed output. A larger index is reliably better, with no plateau at the full 1,516 positives. Fewer neighbours beat more, top_k=3 winning on both mean and max. And the aggregator ranking inverts depending on how you read it: top_k_mean has the best average ROC-AUC while skewness has by far the best achievable, 0.996 against 0.890, so skewness is the most configuration-sensitive of the six. Picking by average would have chosen top_k_mean and given up about a tenth of a point. The EditNotebook pass used for the intro correction stripped required fields from 41 stream outputs across 19 of the original cells, which invalidated the notebook. The original 32 cells are restored byte-for-byte from before this release's work, and the remaining outputs repaired, so the file validates again. Co-authored-by: Cursor <cursoragent@cursor.com>
The release range starts at #21, not after it, so three things the notes had omitted or mis-attributed are in scope. 1.0 shipped two summarize metrics, mean_of_positives and skewness. 2.0 has six, adding top_k_mean, percentile_score, softmax_weighted_mean and max_score. An earlier draft of this file claimed the opposite - that all six predated the release - which was wrong, and understated it: the release both widened the choice and supplied the evidence to make it. Also documents the explainability fields on RareClassAffinityResult, and the Dockerfiles, neither of which appeared anywhere in the notes. Replaces the flat list of links with a table of the main changes and a one-line summary of each, so the file opens with what matters rather than an index. Co-authored-by: Cursor <cursoragent@cursor.com>
1.0 declared support for Python 3.9 in both the constraint and the classifiers, but CI has only ever run 3.10 to 3.12, so the claim was never verified. It had also stopped being possible to honour: current torch requires 3.10 or newer, and the constraint here allows anything up to 3.0, so a 3.9 install had to silently resolve to an older torch that nobody tests. An unverified promise that quietly hands users a different dependency set is worse than no promise. The source itself was already 3.9-clean - every file in src/ parses under ast.parse(feature_version=(3, 9)) and there are no 3.10-only runtime APIs - so this is about what the metadata claims rather than about fixing breakage. Raising the minimum is a breaking change, which is why it goes in the major release rather than waiting. Relocking drops three backports whose functionality is in the standard library from 3.10: importlib-metadata, importlib-resources and zipp. Also removes 200-odd python_version < "3.10" markers that can no longer fire. No package versions change, and no packages are added. The lock had to be regenerated because it records the python constraint and a content-hash derived from pyproject.toml; leaving it stale would fail the poetry install that CI runs. Regenerated with Poetry 2.4.1 to preserve lock-version 2.1 - relocking with the older Poetry on this machine silently rewrote it to 2.0. Both versions validate the result. Co-authored-by: Cursor <cursoragent@cursor.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #36 +/- ##
=======================================
Coverage ? 95.20%
=======================================
Files ? 9
Lines ? 667
Branches ? 0
=======================================
Hits ? 635
Misses ? 32
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vcai4071
approved these changes
Aug 6, 2026
vcai4071
left a comment
vcai4071
left a comment
Contributor
There was a problem hiding this comment.
LGTM! I'm wondering if sentinel_against_hate notebook is the right place to show off the new features. Perhaps we can add a examples/feature_tour.ipynb, but up to you.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts the 2.0 release: two code changes worth reviewing carefully, a version bump, and the
documentation and worked example for everything that landed since #21.
The code changes
Observations are encoded once per sweep, not once per pass
run_grid_searchre-ran the sentence model for everytop_kvalue, even though anobservation's embedding depends on the encoder and never on the index it is scored against.
subsample()shares the parent's sentence model and encoding kwargs, so one set ofembeddings is valid for a whole sweep.
calculate_rare_class_affinitynow acceptssample_embeddings,simulationgainsencode_observations(), andrun_grid_searchhoists the encoding out of its loops.Measured on the real evaluation set in the notebook - 60 episodes, 22,885 segments:
Rows are byte-identical either way, which is the guarantee that matters and is asserted by a
test. Behind
cache_observation_embeddings=Trueso callers short of memory can opt out, andan index that cannot pre-encode falls back rather than failing - which keeps the harness
usable with the duck-typed doubles it advertises support for.
Result columns cannot silently overwrite each other
Rows are plain dicts so they drop into a
DataFrame, but that also means a second write to akey destroys the first. That is how an index size once replaced the positive-group count
from
evaluate_groups. The grid-search columns are now named constants applied through_add_columns, which raises instead of overwriting.Non-breaking: rows stay dicts, so
pd.DataFrame(rows)keeps working as documented.Argument order
run_grid_search's arguments were ordered by the accident of when each was added, leavingthe two index axes separated from the other sweep axes by unrelated plumbing. They are all
keyword-only, so regrouping them by role breaks nothing; they now read in the same order as
the cost-ordered loop the docstring describes.
On
SentinelLocalIndexthe opposite was true - everything was positional-or-keyword, so anew argument could only be appended.
sample_embeddingsbelongs besidetext_samples, sothe parameters after it become keyword-only, as do those of
from_textsandload.Every call site in the repository already used keywords, and the docs always taught that
style, so this is a deliberate but narrow break, recorded in the migration notes.
Version
1.0.0->2.0.0, andsentinel.__version__is now exposed (it existed nowhere before).The major bump is what licenses the keyword-only change above.
Also converts
__init__.pyfrom tabs to spaces - the only such file in the repo, and thesource of all 29
W191warnings.Documentation
V2_FEATURES.md(new): every addition since FEAT: Threshold and ratio configuration and testing file for optimal threshold and ratio configuration #21 with what it is, why it exists and howto use it, plus migration notes.
subsample(), the index sweep axes, the embeddingcache, and a clearer split between the two things called "metrics" - the aggregator is
swept, the evaluation metrics are all reported.
outputs. Appended rather than woven in, so the original 32 cells are byte-identical.
What the notebook sweep found
The appendix ends with a real 324-configuration sweep, which is only affordable because of
the caching above. Three findings, all from executed output:
1,516 positives, with no plateau at the top. Useful to know before deciding whether
gathering more examples is worthwhile.
top_k=3wins on both mean and max.top_k_meanhas the bestaverage ROC-AUC (0.809) while
skewnesshas by far the best achievable (0.996 against0.890). Skewness is the most configuration-sensitive of the six.
The library defaults ranked 103rd of 324, at 0.831 against the best configuration's 0.996.
Worth knowing, and the argument for sweeping rather than trusting defaults.
Test plan
pytest tests/- 114 passed, up from 104length mismatch, duplicate column)
flake8- 116 warnings -> 87, with zero new ones introduceddocs/check_docs_sync.pyin syncTwo things left open deliberately
pyproject.tomlstill declares Python 3.9 in both its constraint and classifiers, whileCI tests only 3.10-3.12. The source is 3.9-clean, but the currently resolving
torch(2.9)requires >=3.10, so a 3.9 install must silently backsolve to an untested torch. Narrowing the
supported floor is a product decision rather than a cleanup, and a major release is the right
moment - happy to do it here if reviewers agree.
Known limitations, documented in
V2_FEATURES.md:top_kstill forces a re-score(caching the neighbour search would make it as cheap as the index axes), and
observation_scoresis keyed by observation text, so duplicates within one group collapse.Made with Cursor