feat(autopilot): scaffold signal engine + auto-link backtest metrics (Phase 3)#267
Merged
Merged
Conversation
…(Phase 3) Two new tools close the hypothesis -> backtest -> evidence loop that Phase 1+2 left half-wired: - scaffold_signal_engine: writes a contract-correct code/signal_engine.py stub (no-arg SignalEngine, generate(self, data_map) -> dict of pd.Series) with a runnable flat-signal default and the hypothesis signal_definition embedded as a docstring. Eliminates the runner's most common pre-flight validation failure. Refuses to overwrite an existing engine unless overwrite=true. - link_autopilot_backtest: reads run_card.json metrics from a run dir and links them to the hypothesis via link_backtest in one step, instead of making the agent hand-extract the metrics dict. Both resolve run_dir through safe_run_dir, inheriting the allowed-run-root boundary (fail-closed on path traversal). Adds 12 tests, including a check that the generated stub satisfies the backtest runner contract by importing it and running generate(). Signed-off-by: Robin1987China <fanrenzhige@163.com>
Collaborator
|
Merged — thanks @Robin1987China! 🙏 |
warren618
added a commit
that referenced
this pull request
Jun 23, 2026
…Bridge / Research Delivery) (#297) Mark Research Autopilot (Phase 1-3, #260/#267), the local Data Bridge loader (#252), and the scheduled-research scheduler/executor (#272/#278/#288) as shipped in the Roadmap status column across all 5 READMEs. Feature descriptions left intact so SQL connectors and multi-channel outbound delivery still read as pending. Closes #294.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Research Autopilot Phase 1 (#260) bridged hypothesis → goal and hypothesis → config.json, but left the rest of the loop wired by hand. Two manual steps remained the most error-prone parts of an autopilot run:
code/signal_engine.pyto the runner contract (a no-arg-constructibleSignalEnginewithgenerate(self, data_map)). Getting it wrong is the most common reason a backtest fails its pre-flight check.link_backtestneeds the agent to openrun_card.jsonand hand-extract the metrics dict.This PR adds two deterministic, backend-only tools that close the loop.
Changes
agent/src/tools/autopilot_tool.py:scaffold_signal_engine— writes a contract-correctcode/signal_engine.pystub with a runnable flat (no-position) default and the hypothesissignal_definitionembedded as a docstring for the agent to implement against. Refuses to overwrite an existing engine unlessoverwrite=true.link_autopilot_backtest— readsrun_card.jsonfrom a run dir, extracts the scalarmetrics, and links them to the hypothesis via the existinglink_backtest, returning the metrics for thesis evaluation. Degrades to empty metrics with a warning if the key is absent.run_dirthroughsafe_run_dir, inheriting the allowed-run-root boundary (fail-closed on path traversal, consistent with the backtest tool).agent/tests/test_autopilot_phase3.py: 12 tests.Out of scope (deliberate)
signal_definitioninto working signal logic (LLM territory; the stub is a starting point only).backtesttool — execution stays an explicit agent step so run cost and data-source side effects remain visible.goal_evidencerows — the existingadd_goal_evidencetool handles that and needs the per-turnexpected_goal_id.Test Plan
ruff check agent/src/tools/autopilot_tool.py agent/tests/test_autopilot_phase3.py— cleanpytest agent/tests/test_autopilot_phase3.py agent/tests/test_autopilot_tool.py -q— 19 passedsignal_engine.pyand runsgenerate(), asserting no-arg construction + dict-of-Series outputRisk / rollback
Additive only; both tools auto-register via
BaseTool.__subclasses__(). No schema, config, or behavior changes to existing tools. Revertable with a singlegit revert.