Skip to content

feat(autopilot): scaffold signal engine + auto-link backtest metrics (Phase 3)#267

Merged
warren618 merged 1 commit into
HKUDS:mainfrom
Robin1987China:research-autopilot-phase3
Jun 20, 2026
Merged

feat(autopilot): scaffold signal engine + auto-link backtest metrics (Phase 3)#267
warren618 merged 1 commit into
HKUDS:mainfrom
Robin1987China:research-autopilot-phase3

Conversation

@Robin1987China

Copy link
Copy Markdown
Contributor

Summary

Research Autopilot Phase 1 (#260) bridged hypothesis → goal and hypothesis → config.json, but left the rest of the loop wired by hand. Two manual steps remained the most error-prone parts of an autopilot run:

  1. Writing code/signal_engine.py to the runner contract (a no-arg-constructible SignalEngine with generate(self, data_map)). Getting it wrong is the most common reason a backtest fails its pre-flight check.
  2. Recording results — link_backtest needs the agent to open run_card.json and hand-extract the metrics dict.

This PR adds two deterministic, backend-only tools that close the loop.

Changes

  • agent/src/tools/autopilot_tool.py:
    • scaffold_signal_engine — writes a contract-correct code/signal_engine.py stub with a runnable flat (no-position) default and the hypothesis signal_definition embedded as a docstring for the agent to implement against. Refuses to overwrite an existing engine unless overwrite=true.
    • link_autopilot_backtest — reads run_card.json from a run dir, extracts the scalar metrics, and links them to the hypothesis via the existing link_backtest, returning the metrics for thesis evaluation. Degrades to empty metrics with a warning if the key is absent.
    • Both resolve run_dir through safe_run_dir, inheriting the allowed-run-root boundary (fail-closed on path traversal, consistent with the backtest tool).
  • agent/tests/test_autopilot_phase3.py: 12 tests.

Out of scope (deliberate)

  • Compiling the free-text signal_definition into working signal logic (LLM territory; the stub is a starting point only).
  • Auto-invoking the backtest tool — execution stays an explicit agent step so run cost and data-source side effects remain visible.
  • Writing goal_evidence rows — the existing add_goal_evidence tool handles that and needs the per-turn expected_goal_id.

Test Plan

  • ruff check agent/src/tools/autopilot_tool.py agent/tests/test_autopilot_phase3.py — clean
  • pytest agent/tests/test_autopilot_phase3.py agent/tests/test_autopilot_tool.py -q — 19 passed
  • Full suite excl. e2e — 3268 passed, 1 skipped, no regressions
  • Stub-contract test imports the generated signal_engine.py and runs generate(), asserting no-arg construction + dict-of-Series output

Risk / rollback

Additive only; both tools auto-register via BaseTool.__subclasses__(). No schema, config, or behavior changes to existing tools. Revertable with a single git revert.

…(Phase 3)

Two new tools close the hypothesis -> backtest -> evidence loop that
Phase 1+2 left half-wired:

- scaffold_signal_engine: writes a contract-correct code/signal_engine.py
  stub (no-arg SignalEngine, generate(self, data_map) -> dict of pd.Series)
  with a runnable flat-signal default and the hypothesis signal_definition
  embedded as a docstring. Eliminates the runner's most common pre-flight
  validation failure. Refuses to overwrite an existing engine unless
  overwrite=true.
- link_autopilot_backtest: reads run_card.json metrics from a run dir and
  links them to the hypothesis via link_backtest in one step, instead of
  making the agent hand-extract the metrics dict.

Both resolve run_dir through safe_run_dir, inheriting the allowed-run-root
boundary (fail-closed on path traversal). Adds 12 tests, including a check
that the generated stub satisfies the backtest runner contract by importing
it and running generate().

Signed-off-by: Robin1987China <fanrenzhige@163.com>
@warren618 warren618 merged commit 1b71b13 into HKUDS:main Jun 20, 2026
1 check passed
@warren618

Copy link
Copy Markdown
Collaborator

Merged — thanks @Robin1987China! 🙏 scaffold_signal_engine + link_autopilot_backtest close the two most error-prone manual steps of an autopilot run, and the generated stub matches the runner's SignalEngine contract exactly.

warren618 added a commit that referenced this pull request Jun 23, 2026
…Bridge / Research Delivery) (#297)

Mark Research Autopilot (Phase 1-3, #260/#267), the local Data Bridge loader (#252), and the scheduled-research scheduler/executor (#272/#278/#288) as shipped in the Roadmap status column across all 5 READMEs. Feature descriptions left intact so SQL connectors and multi-channel outbound delivery still read as pending. Closes #294.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants