gh-153804: _remote_debugging: Tachyon Oracle#153806
Conversation
pablogsal
left a comment
pablogsal
left a comment
There was a problem hiding this comment.
I played with this locally and I think this is very useful. The structural part is particularly convincing: with gen_alternating I consistently get 0% impossible stacks in blocking mode and around 38–61% in the live modes across different rates and both fixed and Poisson sampling. I also inspected the raw stacks and these are real impossible combinations, such as bgen being reported with drv_a as its caller and agen being reported with drv_b as its caller.
I am less convinced by the current interpretation of tvd_excess. Independent runs of the exact same blocking mode still produce positive excess, sometimes up to 0.08, so I don't think we can describe this as distance that cannot be explained by noise. Reporting the raw TVD and documenting the floor as an IID heuristic would be more precise.
I left a few comments about this and some cases that can directly affect the measurements. Overall I really like the direction, but I would fix the Poisson scheduling and adjust the TVD presentation before merging. Thanks a lot for working on this!
| ), | ||
| "raw_tvd": raw_tvd, | ||
| "tvd_floor": floor, | ||
| "tvd_excess": ( |
There was a problem hiding this comment.
Subtracting the expected IID floor does not give us the distance that cannot be explained by noise. In an IID simulation the floor was 0.0205, but the 95th percentile was 0.0500. Can we report raw_tvd as well and describe this as a heuristic?
| result["attempts"] += 1 | ||
| if period: | ||
| if args.poisson_sampling: | ||
| time.sleep(random.expovariate(rate_hz)) |
There was a problem hiding this comment.
This can sample well after the deadline. I reproduced a run with duration=0.05 that took 1.87 seconds and still recorded a sample. We also add the unwind time to the exponential delay, so sample starts do not follow the requested Poisson process. Can we schedule absolute arrival times and stop when the next one is beyond the deadline?
| # Remove 1-3 threads | ||
| remove_count = random.randint(1, 5) | ||
| # The threads will terminate naturally since they're daemons | ||
| active_threads = active_threads[remove_count:] |
There was a problem hiding this comment.
This does not remove the threads. mixed_workload() loops forever, so dropping these references only makes thread_count disagree with the real number of threads.
| try: | ||
| proc = subprocess.Popen( | ||
| [sys.executable, tmp_name], | ||
| stdout=subprocess.PIPE, |
There was a problem hiding this comment.
We never drain these pipes while sampling. A snippet that prints enough data will block and then we will profile the blocked pipe write. Can we use DEVNULL or drain them?
|
|
||
| def classify_gen(frames): | ||
| names = {frame.funcname for frame in frames} | ||
| return ("agen" in names and "drv_b" in names) or ( |
There was a problem hiding this comment.
This detects crossed branches, but an orphan or duplicated agen is accepted. I think impossible means “recognized impossible patterns” here, not that every other stack is valid. Can we document that and add a few small classifier tests?
| ) | ||
| parser.add_argument( | ||
| "--runs", | ||
| type=int, |
There was a problem hiding this comment.
Can we require this to be greater than zero? --runs 0 currently crashes with an IndexError in print_results().
|
@maurycy asked me for Windows data: |
3 participants
Please read #153804 for raison d'être.
The ultimate idea is that instead of "I generated a small script with Claude, therefore we're better than we were" we repeatedly and rigorously measure and ensure that our changes to the profiler do not make things worse or, even sometimes, improve the reported data.
The PR introduces a new script suited for Tachyon, named
Tools/inspection/oracle_external_inspection.py, mostly reusing the snippets fromTools/inspection/benchmark_external_inspection.py. For each snippet it reports:[130] 2026-07-16T10:56:25.380112000+0200 maurycy@gimel /Users/maurycy/work/cpython (tachyon-oracle e6a6696?) % sudo ./python.exe Tools/inspection/oracle_external_inspection.py --snippet flat_alternating 3.16.0a0 (heads/tachyon-oracle:e6a6696eec2, Jul 15 2026, 16:39:58) [Clang 21.0.0 (clang-2100.1.1.101)] cases=flat_alternating runs=1 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=False flat_alternating (get_stack_trace) ref_keys=6 mode samples µs errors% impossible impossible% tvd_excess tvd_floor blocking-nocache 291259 8.72 0.00 0 0.00 ref - blocking-cache 299140 8.45 0.00 0 0.00 0.031 0.002 live-cache 251090 6.89 16.30 1857 0.74 0.090 0.002 live-nocache 251006 6.51 16.33 2422 0.96 0.094 0.002I was able to nicely reproduce #151424 with 63.35±2.19 impossible% rate, that matches that finding:
[130] 2026-06-21T21:36:22.319345000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % sudo ./python.exe Tools/inspection/oracle_external_inspection.py --snippet flat_alternating --rate 1 --duration 3 --runs 8 3.16.0a0 (heads/main:6679ac07d88, Jul 16 2026, 11:19:41) [Clang 21.0.0 (clang-2100.1.1.101)] cases=flat_alternating runs=8 duration=3.0 rate_khz=1.0 warmup=0.7 poisson_sampling=False flat_alternating (get_stack_trace) ref_keys=6 mode samples µs errors% impossible impossible% tvd_excess tvd_floor blocking-nocache 24008 19.34±0.30 0.00±0.00 0 0.00±0.00 ref - blocking-cache 24008 18.51±0.33 0.00±0.00 0 0.00±0.00 0.002±0.007 0.017 live-cache 21042 7.70±0.28 12.35±0.61 13333 63.35±2.19 0.319±0.035 0.025 live-nocache 20501 9.31±1.30 14.61±1.14 60 0.29±0.10 0.028±0.004 0.018Please see
Tools/inspection/oracle_external_inspection.py --helpfor all the flags. Not pasting it here for the brevity.Please see
Tools/inspection/oracle_external_inspection.pyfor a run with all the snippets.The first two columns are relatively obvious to read. Important bits:
flat_alternatingwe're usingget_stack_tracebutget_async_stack_traceis also supported for relevant snippets,blocking-nocacheis used as the best available reference. I don't believe it's the ground truth but it's the best one that we have available (we can call it the "oracle").errors%is measured similarly to how the collector does it, that is by catching the exception,impossibleandimpossible%is the number of samples that are simply impossible for a given snippet (eg: leafs simply do not match, they couldn't be called etc.)tvd_flooris how much TVD two samples of the same distribution would show at these sample sizes - like: what is the noise?tvd_excessis the TVD minus the floor, the distance that cannot be explained by the noise. The closer to the zero, the better we're here.In short, the idea behind
tvd_*is to measure the drift (bias? noise?) against the reference.I tried to familiarize myself with the state of the literature what is the best measure of bias, drift etc. in case of statistical samplers. Maybe it's me but I couldn't find any undisputed gold standard here. I'm more than open to any other measure, and it's very likely that we'll find something better.
The PR is big but most of it is copying the snippets to a separate file. The length of the script is on par with the existing benchmarking script.
Closes #153804