Feedback from local testing + thoughts on scanner reliability and possible autonomous trading roadmap

[zafahl]

First off, I really appreciate you putting this project out there. I have been working on my own trading-agent project and came across this repo while looking at different AI stock-analysis workflows. I wanted to test this one locally as more of a research/scanner tool, not as something I would let trade live.

For safety, I set it up locally, only added API keys, did not connect it to a broker, and treated the outputs as research only. I also started small with just a few tickers and one analyst before trying a larger watchlist with more analysts selected.

A few things I noticed from testing:

1. The multi-agent flow itself is really cool

   • Being able to select Technical, Fundamentals, Growth, News Sentiment, Sentiment, and Valuation analysts is useful.
   • The CLI flow was pretty easy to get running once the API keys were set up.
   • The output tables are helpful for quickly scanning a bunch of tickers.

2. The news sentiment behavior was confusing

   • In my runs, the News Sentiment agent returned total_articles: 0, bullish_articles: 0, and bearish_articles: 0 for a lot of very common tickers.
   • The result still came through as NEUTRAL, but from a user perspective that feels different from true neutral sentiment.
   • It might be clearer if the system labeled this as something like NO_NEWS_DATA, INSUFFICIENT_NEWS_DATA, or DATA_UNAVAILABLE instead of neutral.

3. The HOLD 100% confidence output can be misleading

   • A lot of tickers came back as HOLD with 100.0% confidence and reasoning like No valid trade available.
   • I understand what the system is probably doing there, but as a non-expert user, that can look like the model is 100% confident the best trading decision is to hold.
   • It may be better to separate “confidence in a trade signal” from “confidence that no valid signal was found.”

4. Some trade decisions seemed stronger than the evidence behind them

   • In one run, the system suggested shorting certain tickers even though the technical analyst was neutral and the news agent had found zero articles.
   • That made me hesitant to treat the output as anything more than a watchlist flag.
   • It would help if the final decision showed whether the evidence was strong, weak, missing, or conflicting.

What I think would make this better:

• Add data-quality flags per agent.

  • Example: data_status: complete, partial, missing, stale, or insufficient.

• Separate true neutral from missing data.

  • Neutral should mean the data was actually checked and did not lean bullish or bearish.
  • Missing news/articles should probably be a separate state.

• Separate confidence types.

  • Signal confidence: how strong the buy/sell/short signal is.
  • Data confidence: how complete/reliable the input data is.
  • Decision confidence: how confident the portfolio manager is after combining signals.

• Add a “why this could be wrong” field.

  • This would be especially helpful for newer users who might otherwise overtrust the final action.

• Add evidence requirements before showing a trade action.

  • For example, before outputting BUY or SHORT, maybe require at least 2 or 3 categories to support the decision:
    • technicals
    • fundamentals/growth/valuation
    • sentiment/news
    • insider activity
    • risk management

• Consider using labels like:

  • TRADE_CANDIDATE
  • WATCHLIST_ONLY
  • NO_VALID_SETUP
  • DATA_INSUFFICIENT

  instead of always forcing the output into BUY, SELL, SHORT, or HOLD.

• Add duplicate ticker cleanup.

  • I accidentally included duplicate tickers in one run, and it would be nice if the CLI automatically deduped them before running.

• Add a cheaper/default model option if possible.

  • The model selector defaulted me into GPT-5.5, which works, but it makes larger watchlist scans more expensive.
  • It would be helpful to have a cheaper OpenAI model option exposed directly in the menu for casual scanner runs.

One other thought: if the long-term goal of this project is to eventually become more of an autonomous trading agent, I think it would need another layer beyond the analyst agents.

Right now, from my testing, it feels more like a really interesting research/scanner tool. That is useful, but I would be nervous about treating the final output as an actual trade command without more safety and execution infrastructure around it.

If autonomous trading is part of the future direction, I think it would need things like:

1. A staged operating mode system

   • Research only
   • Paper trading only
   • Paper trading with simulated approval
   • Live trading with human approval
   • Small autonomous live trades with strict limits
   • Larger autonomy only after enough validation

2. A stronger risk engine

   • Max dollars per trade
   • Max daily loss
   • Max weekly loss
   • Max position size
   • Max trades per day
   • Max exposure by sector/theme
   • Hard stop if data is missing, stale, or conflicting
   • Hard stop if the LLM response is malformed or uncertain

3. Better trade-state management

   • Entry reason
   • Invalidation level
   • Stop loss
   • Profit target or trailing exit
   • Position monitoring after entry
   • Exit logic
   • Re-entry rules
   • Cooldown periods after losses

4. Clear distinction between signals and executable trades

   • A ticker could be a WATCHLIST_CANDIDATE without being a trade.
   • A ticker could have a bullish thesis but still fail the risk engine.
   • A ticker could have missing news data and therefore be blocked from autonomous execution.

5. Data-quality gates

   • Before the system outputs BUY, SHORT, or any live-trade action, it should probably prove that the required data was present.
   • For example, if news articles are zero, that should not be treated the same as true neutral news sentiment.
   • If technicals are neutral but sentiment is bearish, the system should maybe flag that as conflicting rather than escalating straight to a short idea.

6. Backtesting and forward testing

   • Backtests with fees/slippage
   • Walk-forward testing
   • Paper-trading logs
   • Comparison against simple baselines like buy-and-hold or SPY/QQQ
   • Performance by agent, ticker, sector, and market regime

7. Broker/execution safety

   • No live broker connection by default
   • Read-only broker mode first
   • Order preview mode
   • Strictly limited live mode
   • Kill switch
   • Emergency flatten/close-all option
   • Full audit log of every decision, prompt, response, data source, and order

8. Better final decision output

   • Instead of only BUY, SELL, SHORT, or HOLD, maybe the final output could include:
     • NO_VALID_SETUP
     • WATCHLIST_ONLY
     • PAPER_TRADE_CANDIDATE
     • LIVE_TRADE_BLOCKED
     • LIVE_TRADE_ALLOWED
   • That would make it much clearer whether the system is just analyzing or whether the setup actually passes execution requirements.

To me, the analyst-agent layer is the cool part of the project, but autonomous trading would require a separate control/risk/execution layer. The agents can generate opinions, but something else needs to decide whether those opinions are safe, complete, and actionable enough to trade.

Overall, I think the project is already really interesting as an AI stock research sandbox. The biggest improvement, in my opinion, would be making the output more explicit about when the system has a real signal versus when it simply does not have enough usable data.

Thanks again for publishing this. I hope this feedback is useful.