FEAT: Threshold and ratio configuration and testing file for optimal threshold and ratio configuration - #21
Merged
vcai4071 merged 21 commits intoJul 16, 2026
Conversation
…SentinelLocalIndex. FEAT: created a testing tool for best threshold and ratio analysis
…e with optional flags. DOCS: Updated relevent documentation with these fixes
Contributor
Author
|
Note: All 20 tests passed with two warnings regarding configuration of the pytests as follows, unsure if this is due to an outdated version of the pytest library, or if these config keys have depreciated. .venv\Lib\site-packages_pytest\config_init_.py:1441 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html" |
…dling and fallback for ratio adjustments
Feat: Adjusts message metrics dynamically
…prevent inclusion
… load time of model exponentially after the first caching. TESTS: Updated embedding tests to include caching and its management functions FEAT: Updated the example script for testing purposes to include caching mechanics
- Apply PEP 8 formatting to Example_Threshold_Script.py - Update embeddings.safetensors - Update sentinel_against_hate.ipynb - Fixed line length violations (max 79 characters) - Corrected indentation and spacing - Enhanced readability while maintaining functionality
…r speech examples
…mpty score arrays, fixes edge case, NaN returns.
…_affinity`, update example file to use path/to/index rather than local path
…omponents in score_formulae and SentinelLocalIndex
…onality, removed redundant exports, added no-cache flag to the testing script
vcai4071
reviewed
Feb 19, 2026
Contributor
Author
|
@vcai4071 all requested changes requested have been made |
vcai4071
approved these changes
Feb 19, 2026
vcai4071
left a comment
vcai4071
left a comment
Contributor
There was a problem hiding this comment.
LGTM, thanks for the great additions!
Contributor
|
@rafainn Can you take a look at the failing test I will merge once the test are passing. |
… have support for PEP 517 builds hence swapped to ^2.0.0 which is compatable - may require further testing however didn't impact functionality of code
Contributor
Author
|
@leoRblx Would you be able to run the tests again, this should fix the build issue it was displaying earlier, however I am unsure if there would be any further conflicts. |
wxiao0421
added a commit
that referenced
this pull request
Aug 4, 2026
The release range starts at #21, not after it, so three things the notes had omitted or mis-attributed are in scope. 1.0 shipped two summarize metrics, mean_of_positives and skewness. 2.0 has six, adding top_k_mean, percentile_score, softmax_weighted_mean and max_score. An earlier draft of this file claimed the opposite - that all six predated the release - which was wrong, and understated it: the release both widened the choice and supplied the evidence to make it. Also documents the explainability fields on RareClassAffinityResult, and the Dockerfiles, neither of which appeared anywhere in the notes. Replaces the flat list of links with a table of the main changes and a one-line summary of each, so the file opens with what matters rather than an index. Co-authored-by: Cursor <cursoragent@cursor.com>
6 tasks
wxiao0421
added a commit
that referenced
this pull request
Aug 6, 2026
…lt columns, and document the release (#36) * Reuse observation embeddings across a sweep, and guard result columns Three related changes for 2.0.0. run_grid_search re-encoded the observation texts on every scoring pass, even though an observation's embedding depends on the encoder and never on the index it is scored against. subsample() shares the parent's sentence model and encoding kwargs, so one set of embeddings is valid for a whole sweep. calculate_rare_class_affinity now accepts sample_embeddings, simulation gains encode_observations(), and run_grid_search hoists the encoding out of its loops. On a 2x2x3 sweep over 320 observations that is 2.99s to 0.52s, a 5.7x saving, with byte-identical rows. Kept behind cache_observation_embeddings so callers short of memory can opt out, and indices that cannot pre-encode fall back rather than failing, which keeps the harness usable with the duck-typed doubles it advertises support for. Result rows are plain dicts so they drop into a DataFrame, but that also means a second write to a key silently destroys the first - which is exactly how an index size once replaced the positive-group count. The grid-search columns are now named constants applied through _add_columns, which raises instead of overwriting. run_grid_search's arguments were ordered by the accident of when each was added, leaving the two index axes separated from the other sweep axes by unrelated plumbing. They are all keyword-only, so regrouping them by role breaks nothing; they now read in the same order as the cost-ordered loop the docstring describes. On SentinelLocalIndex the opposite was true: everything was positional-or-keyword, so a new argument could only be appended. sample_embeddings belongs beside text_samples, so the parameters after it become keyword-only, as do those of from_texts and load. Every call site in the repo already used keywords. Also converts __init__.py from tabs to spaces, the only such file in the repo and the source of all 29 W191 warnings. Co-authored-by: Cursor <cursoragent@cursor.com> * Document the 2.0 features and how to migrate Adds V2_FEATURES.md covering each addition since 1.0 with what it is, why it exists and how to use it, plus migration notes for the two deliberate breaks: keyword-only arguments after the first, and the index_-prefixed grid-search columns. In the README, replaces the "What's New" section with a 2.0 summary table pointing at the relevant sections, adds a section on subsample(), and documents the index sweep axes and the observation embedding cache with its measured saving. Also separates the two things called "metrics" in the tuning section, since the aggregator is swept while the evaluation metrics are all reported. Co-authored-by: Cursor <cursoragent@cursor.com> * Demonstrate the 2.0 features in the hate-speech notebook Appends a self-contained appendix showing from_texts(), the persisted corpus and seeded loading, subsample(), the grid-search index axes, and the observation embedding cache, with executed outputs. Self-contained deliberately: it builds a small index from a dozen example sentences rather than depending on the data-loading cells above, so it runs in seconds on its own. The shipped example index is used only for the subsample demonstration, where a realistic size is the point - it predates corpus support, so its explanations would show row numbers rather than text. Appended rather than woven in, so the diff is 789 insertions and no deletions and the existing 32 cells are byte-identical. The final aggregator comparison reports the whole table rather than picking a winner: all six separate this example perfectly, which says the example is easy, not that the aggregators are equivalent. Sorting by Cohen's d there produced a number like 8.5e15, because the within-class variance is zero on a fixture this clean. The real comparison is the earlier section, on real data. Co-authored-by: Cursor <cursoragent@cursor.com> * Cover the remaining V2 additions in the notebook appendix An audit of the public API added since 0e924ae against the appendix found four things demonstrated nowhere: evaluate_groups, DEFAULT_AGGREGATORS, load_corpus, and the sample_embeddings parameter itself, which the appendix only reached indirectly through encode_observations. Adds a section introducing the harness by cost - score once, then evaluate with one aggregator, then all of them, then sweep - which is where evaluate_groups and DEFAULT_AGGREGATORS naturally belong, and which also shows the three metric families on a single row. The group fixture moves there, since that section is now the first to use it. Extends the corpus section with load_corpus, reading the texts back without the embeddings. It returns (None, None) for the shipped example index, which is a concrete demonstration of the pre-2.0 format rather than an assertion about it. Extends the caching section with the underlying parameter, including the refusal when the embeddings do not line up with the text. Co-authored-by: Cursor <cursoragent@cursor.com> * Add a real-data configuration sweep and its conclusions to the appendix The "Tuning aggregation strategies" section is itself a 2.0 feature - the simulation harness arrived in #31, inside this release - so the appendix opened on a false premise by calling everything above it 1.0. Corrected, and it now points at that section as the worked example on real data. Adds section 7, which sweeps the axes that section could not: three index sizes x three negative ratios x three top_k x two thresholds over the real 30-vs-30 podcast set, 22,885 segments, 324 rows. It runs on the groups built earlier in the notebook rather than the appendix's toy fixture, and is marked as depending on them. That sweep is only practical because of the embedding cache. Encoding those segments takes 118s and a scoring pass afterwards takes 1.4s, so the 27 configurations are 2.6 minutes rather than 54. Three findings, all from the executed output. A larger index is reliably better, with no plateau at the full 1,516 positives. Fewer neighbours beat more, top_k=3 winning on both mean and max. And the aggregator ranking inverts depending on how you read it: top_k_mean has the best average ROC-AUC while skewness has by far the best achievable, 0.996 against 0.890, so skewness is the most configuration-sensitive of the six. Picking by average would have chosen top_k_mean and given up about a tenth of a point. The EditNotebook pass used for the intro correction stripped required fields from 41 stream outputs across 19 of the original cells, which invalidated the notebook. The original 32 cells are restored byte-for-byte from before this release's work, and the remaining outputs repaired, so the file validates again. Co-authored-by: Cursor <cursoragent@cursor.com> * Cover the summarize metrics, explainability and Docker in the 2.0 notes The release range starts at #21, not after it, so three things the notes had omitted or mis-attributed are in scope. 1.0 shipped two summarize metrics, mean_of_positives and skewness. 2.0 has six, adding top_k_mean, percentile_score, softmax_weighted_mean and max_score. An earlier draft of this file claimed the opposite - that all six predated the release - which was wrong, and understated it: the release both widened the choice and supplied the evidence to make it. Also documents the explainability fields on RareClassAffinityResult, and the Dockerfiles, neither of which appeared anywhere in the notes. Replaces the flat list of links with a table of the main changes and a one-line summary of each, so the file opens with what matters rather than an index. Co-authored-by: Cursor <cursoragent@cursor.com> * Require Python 3.10, and resync the lock 1.0 declared support for Python 3.9 in both the constraint and the classifiers, but CI has only ever run 3.10 to 3.12, so the claim was never verified. It had also stopped being possible to honour: current torch requires 3.10 or newer, and the constraint here allows anything up to 3.0, so a 3.9 install had to silently resolve to an older torch that nobody tests. An unverified promise that quietly hands users a different dependency set is worse than no promise. The source itself was already 3.9-clean - every file in src/ parses under ast.parse(feature_version=(3, 9)) and there are no 3.10-only runtime APIs - so this is about what the metadata claims rather than about fixing breakage. Raising the minimum is a breaking change, which is why it goes in the major release rather than waiting. Relocking drops three backports whose functionality is in the standard library from 3.10: importlib-metadata, importlib-resources and zipp. Also removes 200-odd python_version < "3.10" markers that can no longer fire. No package versions change, and no packages are added. The lock had to be regenerated because it records the python constraint and a content-hash derived from pyproject.toml; leaving it stale would fail the poetry install that CI runs. Regenerated with Poetry 2.4.1 to preserve lock-version 2.1 - relocking with the older Poetry on this machine silently rewrote it to 2.0. Both versions validate the result. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Built upon pull request #7
This pull request introduces significant improvements to the Sentinel library, focusing on aggregation flexibility, explainability, and performance optimizations. The README is updated to document new aggregation strategies and explainability features, and the codebase now exposes multiple aggregation functions for scoring, adds per-text explanations, and improves model caching and negative sample ratio handling.
Aggregation and Explainability Enhancements:
skewness,top_k_mean,percentile_score,softmax_weighted_mean,max_score) for combining observation scores, with documentation and usage examples inREADME.md. [1] [2] [3] [4]RareClassAffinityResultdataclass and README usage examples. [1] [2]Performance and Robustness Improvements:
SentenceTransformermodels insrc/sentinel/embeddings/sbert.pyto avoid redundant loading, with cache management utilities. Global caching seems to have reduced load time of ~300 conversations down to 3.5s from the previous 12.3s.Cache_Modelin thecalculate_rare_class_affinitymodel insrc\sentinel\sentinel_local_index.pyto enable and disable caching easily, depending on space constraints and model requirementtest_thresholds_and_ratiosinexamples/Example_Threshold_Script.pyand how different ratios and temperatures affect detection, this shows a high relation with using 0.00 and 0.01 temperature, and ratios of 2-4:1 for optimal accuracy and minimal false positives.API and Documentation Updates:
__init__.pyto expose new aggregation functions in the public API.These changes collectively make Sentinel more configurable, interpretable, and efficient for diverse deployment scenarios.