Research

Decisions under uncertainty.

Five research programs, one question: when a system reports a probability, can you trust it? We build the decision systems, the data pipelines under them, and the evaluations that try to break them. Then we publish the results, including where they break.

Multi-agent decision systems

Agents that have to earn the right to reproduce.

An evolving population of about 500 autonomous agents, each defined by a genome of strategy parameters. About every two minutes, every agent is re-scored with a Bayesian-shrunk fitness, so a short lucky streak isn't mistaken for skill. The weakest are removed. Replacements are bred by crossover (about 70%) or by mutating a top parent (about 30%).

Selection is statistically gated. An agent becomes a parent only if it passes a Benjamini–Hochberg false-discovery test, so luck alone can't carry a lineage. Agents' probability estimates are pooled as a trimmed mean in logit space, and down-weighted when they disagree.

~500agents live at a time
3,157distinct agents evaluated
1,090evolution cycles in 34 days
38named strategy families

Evaluation methodology

Try to break it before you believe it.

A result counts only if it survives an attempt to explain it away. Our evaluation method is built around that:

  • Permutation placebos. Every effect is measured against a distribution of shuffled runs, which shows what chance alone would produce.
  • Within-agent identification. Fixed effects for each agent and each day, so a result can't come from which agents happened to trade on which days.
  • Time-split holdouts. Models are tested only on data from after their training window.
  • Persistence checks. Skill has to show up again in the second half of the sample, not just the first.
  • Published limits. We report the controls under which an effect weakens, and trace the failure to its cause.

The thesis applies this to about 113,000 resolved paper trades. The main effect cleared its placebo test within agents. The paper also reports where it weakens and why.

0σ Placebo runs Observed
Isometric wireframe of a sandbox where many agents exchange messages while a red probe agent tests them.
A probe agent tests the rest of the population from the inside.

Publication

Agha, S. A. “Algorithmic Trading in Idiosyncratic-Payoff Markets: A Multi-Agent System for On-Chain Prediction Contracts.” CMC Senior Thesis, Claremont McKenna College, 2026.

Calibrated forecasting

Seventy percent should mean seventy percent.

Raw model probabilities are rarely honest. We fit three calibration families, Platt, beta and temperature scaling, and deploy whichever has the lowest expected calibration error on a held-out window split by time, not at random, so the test never sees the future.

On the held-out window, calibration cut expected calibration error from 0.252 to 0.145 and the Brier score from 0.259 to 0.211. A separate 41-feature gradient-boosted classifier gates entries, and another gates exits.

The model's training cutoff is about 21 months before the evaluation window, so it could not have memorized the outcomes.

Predicted probability Observed frequency 01 Perfectcalibration Underconfident Overconfident
0.252→0.145expected calibration error
0.259→0.211Brier score
41features, entry classifier
~21 motraining cutoff before test
Ridgeline of forecast distributions: each ridge is the range of possible outcomes at a later horizon, growing wider and flatter as uncertainty increases.
A forecast is a range of outcomes, and the range widens with time.

Real-time market data engineering

One clock. Every book.

We record order books in real time across five venue feeds, covering prediction markets and spot and perpetual crypto exchanges, and track more than 15,000 prediction-market instruments. Timestamps are aligned across sources and microstructure is precomputed, so research starts from features, not raw feeds.

Alongside the books: an implied-volatility regime feed, perpetual funding, open-interest and basis series, and national-team football Elo that stores pre-match ratings, so training data never leaks the result.

FeedFeedFeed CaptureTicks + books Timestampalignment Micro-structure Researchstore

Self-hosted models & agent reliability

Own the model. Assume it fails.

The research system ran a 72-billion-parameter open-weights model, 5-bit quantized and served with vLLM on a single GPU server with 128 GB of unified memory.

We assume inference will fail. A health check runs a real end-to-end completion, which catches a model that hangs while its liveness probe stays green. If no model is available, the system drops to a mode that rejects every entry. About 30 long-lived processes run under supervision.

These are the same controls we bring to production AI: a safe default when the model is gone, and no silent failure.

72Bopen-weights, self-hosted
5-bitquantized, vLLM
128 GBunified memory, one server
~30supervised processes

Research → Practice

Each method has a job in production.

In researchIn the systems we deliver
Permutation placebos and time-split holdoutsEvaluation gates before anything reaches production
Calibration checked against real outcomesSafe escalation when a system can't answer with confidence
False-discovery gating on who gets to reproduceNo lucky pass: a system reaches production only after it clears every gate
Robustness checks that show where a result weakensDecision support that shows what would change a recommendation
A no-model mode that rejects every entrySafe defaults: read-only connectors, human approval before writes
End-to-end health checks, not liveness probesOpenTelemetry traces for audit and drift detection

Applied

The same rigor, in production.

Our AI systems and our public sector practice are built on these methods.