Alpha Trace

Retail and institutional "AI predicts the market" products backtest well and then fail live, and nobody ever discloses when the underlying signal stops working — so risk and compliance teams can't adopt them. Separately, there is no accessible tool that connects a specific news event or public figure's statement to a measurable, evidenced market move; that connection lives in analysts' heads as folklore, redone by hand in spreadsheets every time it's needed.

Free-tier-only data: GDELT and Yahoo Finance rate-limit aggressively (GDELT returns HTTP 429 within minutes of sustained use), so every scraper had to be resume-safe and tolerate multi-day partial completion. The project ran as a solo-architected build against a hard hackathon/demo deadline, on a single Windows development machine, with no dedicated infra budget — which ruled out a database server and forced a file-based (CSV/parquet) architecture that still had to serve a live dashboard without restarts.

The system is split into three layers that don't share process boundaries on purpose: a pipeline layer that writes plain files (CSV/parquet/JSON), a GraphQL layer that reads those files fresh on every request, and a Next.js terminal that queries GraphQL. The reason for the file-based store instead of a database is explicit in the code: at this scale (~1.65M rows) a database server is operational overhead with no query the files can't already answer in milliseconds, and file mtimes give "the API always serves what's on disk" for free — no cache-invalidation logic, no restart-on-deploy step. The tradeoff is real: there's no concurrent-write safety, which is why the live ledger (live_signals.parquet) is strictly append-only and every write path (predict_live.py, refresh_market.py) treats existing rows as immutable once an outcome resolves.

The second architectural decision that shapes everything downstream is the market registry (markets.json): every pipeline script, the API, and every dashboard dropdown reads which markets exist from that one file instead of hardcoding a list. That decision was made after discovering — repeatedly, across several commits — that a hardcoded market list in one file and not another caused dashboards to silently disagree about how many markets existed. onboard_market.py turns adding a market into one CLI command that runs the whole chain (scrape → features → train → validate → verdict) against that registry.

This started as a five-market academic sentiment-mining exercise and turned into something closer to a real trading terminal partway through, once it became clear that "does news sentiment correlate with returns" wasn't an interesting enough claim on its own — the interesting question was whether the model could be trusted to know when it didn't know. That reframing is what pulled in the validation protocol, the confidence intervals, and eventually the honest abstention framework that ended up being the actual product.

The UK market cost the most time and taught the most. Every single lever I could find — new features, recalibrated gates, sector-level sub-models, a longer prediction horizon, mid-cap tickers instead of large-cap, hourly bars instead of daily — looked like a fix on the training folds and then fell apart on the validation folds, one after another, across eight separate attempts. It would have been easy to ship the sixth or seventh attempt's dev-fold number and call it done; the fact that none of them held up on data the model hadn't seen is the reason I trust the markets that did pass.

The moment it clicked was building the weekly regression guard and watching it actually fire on its first real run — a retrain came back objectively worse on three markets, and the system caught it and rolled itself back before anything bad reached the dashboard, with no one watching. That's when the project stopped feeling like a model and started feeling like a system with a safety net.

If I did it again, I'd write the canonical-stats module and the shared market registry on day one instead of after discovering the disagreement bugs they were built to fix — both were retrofits onto code that had already grown five or six independent copies of logic that should have been one function from the start. I'd also add automated tests around the statistics functions and the ledger's append-only guarantee; right now that correctness is enforced by convention and by me checking the numbers by hand, which doesn't scale and didn't catch the shell-escaping bug until it was visible in the UI.

A working system covering 5 markets (NYSE, TSX, TSE, ASX, KOSPI) after two (UK, Hong Kong) were deliberately removed following documented, repeated validation failures. The production model store holds roughly 1.65M walk-forward predictions across 468 companies; the market-level engine's frozen out-of-sample validation reaches 79.7% accuracy on its strongest market (n=59) and a pooled 62.3% across all five markets (n=738) against a 50% baseline. The experiment log (Community/Output/Model/experiments.md) records 23 dated attempts, the majority of them rejections. The live forward-signal ledger and the weekly guarded retrain were both running unattended on a schedule as of this writing, with the regression guard having already triggered one real rollback.

AlphaTrace investigates whether community-level signals — GDELT news sentiment, World Bank macroeconomic indicators, OECD data, Reddit discourse — causally influence stock returns and volatility across NYSE, LSE, TSE, TSX, and HKEX over 5 years (2020–2025).

The pipeline: async multi-source scrapers with rate-limiting and checkpoint-resume → 7-module Neo4j ETL (2,500+ nodes: Company, Country, Event, CommunityIndicator) → 80+ engineered features per country → 8 data mining algorithms → FastAPI + Strawberry GraphQL API (20+ query types) → Next.js 14 dashboard (9 pages, Nivo / ECharts / Recharts / Tableau).

Algorithms: Granger Causality, VAR Impulse Response, Event Study (CAR), K-Means (stocks + market regimes), XGBoost (binary next-day direction, TimeSeriesSplit validation), SHAP explainability, PCA (52 macro indicators → 5 components).