ml4t-diagnostic

Statistical validation and diagnostics for quantitative trading strategies: signal analysis, backtest evaluation, and overfitting detection.

Documentation: https://ml4trading.io/docs/diagnostic/

Part of the ML4T Library Ecosystem

This library is one of six interconnected libraries supporting the machine learning for trading workflow described in Machine Learning for Trading:

Together they cover data infrastructure, feature engineering, modeling, signal evaluation, strategy backtesting, and live deployment.

What This Library Does

Evaluating whether a signal or strategy has genuine predictive power requires statistical rigor. ml4t-diagnostic provides:

• Information coefficient (IC) analysis with HAC-adjusted standard errors
• Deflated Sharpe Ratio (DSR) with correlation-adjusted K_eff plus other multiple-testing corrections (RAS, PBO, FDR)
• Combinatorial purged cross-validation (CPCV) with calendar-aware splitting
• Feature importance analysis (MDI, PFI, MDA, SHAP) with consensus ranking
• Trade-level diagnostics with SHAP-based error pattern discovery
• Backtest reporting: BacktestProfile, report metadata, and template-based HTML tearsheets
• Portfolio analysis: 16 performance metrics (Sharpe, Sortino, Calmar, VaR, CVaR, ...)
• Systematic feature selection with IC, importance, correlation, and drift filtering
• 65+ Plotly visualizations with 4 themes (default, dark, print, presentation)

The library implements methods from the academic finance literature, particularly those addressing backtest overfitting and false discovery in strategy research.

Installation

pip install ml4t-diagnostic

Optional dependencies:

pip install ml4t-diagnostic[ml]   # SHAP, importance analysis
pip install ml4t-diagnostic[viz]  # Plotly visualizations
pip install ml4t-diagnostic[backtest]  # ml4t-backtest bridge
pip install ml4t-diagnostic[dashboard]  # Streamlit dashboard
pip install ml4t-diagnostic[all]  # Everything

Quick Start

Signal Analysis

import numpy as np
import polars as pl

from ml4t.diagnostic import analyze_signal

rng = np.random.default_rng(42)
dates = pl.date_range(pl.date(2025, 1, 1), pl.date(2025, 2, 28), eager=True)[:40]
assets = [f"asset_{index:02d}" for index in range(20)]

factor_rows = []
price_rows = []
prices = np.full(len(assets), 100.0)
for date in dates:
    scores = rng.normal(size=len(assets))
    factor_rows.extend(
        {"date": date, "asset": asset, "factor": score}
        for asset, score in zip(assets, scores, strict=True)
    )
    price_rows.extend(
        {"date": date, "asset": asset, "price": price}
        for asset, price in zip(assets, prices, strict=True)
    )
    prices *= 1 + 0.002 * scores + rng.normal(scale=0.005, size=len(assets))

result = analyze_signal(
    factor=pl.DataFrame(factor_rows),
    prices=pl.DataFrame(price_rows),
    periods=(1, 5),
)

assert result.ic["1D"] > 0.1

print(f"IC (1D): {result.ic['1D']:.4f}")
print(f"IC t-stat (1D): {result.ic_t_stat['1D']:.2f}")
print(f"Q5-Q1 spread (1D): {result.spread['1D']:.2%}")

Deflated Sharpe Ratio

import numpy as np

from ml4t.diagnostic.evaluation.stats import deflated_sharpe_ratio

rng = np.random.default_rng(42)
strategy_returns = rng.normal(
    loc=[0.0003, 0.0005, 0.0002],
    scale=0.01,
    size=(252, 3),
)

dsr_result = deflated_sharpe_ratio(
    returns=strategy_returns,
    benchmark_sharpe=0.0,
    correlation_method="effective_rank",
    min_k_eff=2.0,
    periods_per_year=252,
)

print(f"Sharpe: {dsr_result.sharpe_ratio:.2f}")
print(f"Deflated Sharpe: {dsr_result.deflated_sharpe:.2f}")
print(f"Raw trials: {dsr_result.n_trials_raw}")
print(f"Effective trials: {dsr_result.n_trials_effective:.2f}")
print(f"Significant: {dsr_result.is_significant}")

Diagnostic Framework

Tier 1: Feature Analysis (Pre-Modeling)
├── Time series diagnostics (stationarity, ACF, volatility)
├── Distribution analysis (moments, normality, tails)
├── Feature importance (MDI, PFI, MDA, SHAP)
└── Feature interactions (conditional IC, H-stat)

Tier 2: Signal Analysis (Model Outputs)
├── IC analysis (time series, histogram, decay)
├── Quantile returns (spreads, monotonicity)
├── Turnover analysis
└── Multi-signal comparison

Tier 3: Backtest Analysis (Post-Modeling)
├── Trade analysis (win/loss, holding periods)
├── Statistical validity (DSR, RAS, PBO)
├── Trade-SHAP diagnostics
└── Excursion analysis (TP/SL optimization)

Tier 4: Portfolio Analysis (Production)
├── Performance metrics (Sharpe, Sortino, Calmar)
├── Drawdown analysis
├── Rolling metrics
└── Risk metrics (VaR, CVaR)

Statistical Methods

Method	Purpose
DSR (Deflated Sharpe)	Corrects for multiple testing bias
CPCV (Combinatorial Purged CV)	Leak-free time series validation
RAS (Rademacher Anti-Serum)	Backtest overfitting detection
PBO	Probability of backtest overfitting
HAC-adjusted IC	Autocorrelation-robust information coefficient
FDR Control	Multiple comparisons (Benjamini-Hochberg)

Cross-Validation

See the executable cross-validation guide for walk-forward and combinatorial purged cross-validation examples.

Backtest Tear Sheets

The tearsheet pipeline supports direct rendering from normalized surfaces, BacktestResult, or saved run artifacts.

Four presets covering different analysis needs:

Template	Focus	Sections
quant_trader	Trade-level analysis	overview, trading, performance, validation, ML, factors
hedge_fund	Performance and costs	overview, performance, trading, validation, factors, ML
risk_manager	Statistical credibility	overview, validation, performance, trading, factors, ML
full	Comprehensive presentation	overview, performance, trading, validation, factors, ML

The backtest tearsheet guide contains a complete example with synthetic trades and returns.

Portfolio Analysis

import numpy as np

from ml4t.diagnostic.evaluation import PortfolioAnalysis

rng = np.random.default_rng(42)
daily_returns = rng.normal(loc=0.0004, scale=0.01, size=252)
pa = PortfolioAnalysis(daily_returns)
metrics = pa.compute_summary_stats()

print(f"Sharpe: {metrics.sharpe_ratio:.2f}")
print(f"Sortino: {metrics.sortino_ratio:.2f}")
print(f"Max Drawdown: {metrics.max_drawdown:.2%}")
print(f"VaR (95%): {metrics.var_95:.2%}")

PortfolioMetrics exposes total_return, annual_return, annual_volatility, sharpe_ratio, sortino_ratio, calmar_ratio, omega_ratio, tail_ratio, max_drawdown, skewness, kurtosis, var_95, cvar_95, stability, win_rate, profit_factor, avg_win, and avg_loss. When a benchmark is provided, it also exposes alpha, beta, information_ratio, up_capture, and down_capture.

Feature and Trade Diagnostics

The user guides contain executable workflows for feature selection, feature diagnostics, and trade analysis.

Documentation

• Docs Site - deployed documentation
• Backtest Tearsheets - BacktestResult, artifact, and profile-driven reporting
• Book Guide - chapter and case-study map
• Workflows - end-to-end analysis patterns
• Validation Tiers - four-tier diagnostic framework
• Cross-Validation - CPCV and walk-forward splitting
• CV Configuration - JSON/YAML config and fold persistence
• Feature Diagnostics - importance and interaction analysis
• Feature Selection - systematic multi-criteria selection
• Statistical Tests - DSR, RAS, PBO, HAC
• Trade Analysis - trade-level diagnostics and SHAP

Technical Characteristics

• Polars-based: Native Polars DataFrames throughout
• HAC standard errors: Newey-West adjustment for autocorrelated data
• Time-aware validation: Purged and embargoed cross-validation splits
• Calendar-aware: NYSE, CME, crypto calendars for trading-day gaps
• 65+ visualizations: Plotly-based with 4 themes (default, dark, print, presentation)
• PDF/HTML export: Institutional-grade tear sheets
• Type-safe: 0 type diagnostics (ty/Astral), full type annotations
• Release-blocking examples: public scripts and documentation execute in CI

Related Libraries

• ml4t-data: Market data acquisition and storage
• ml4t-engineer: Feature engineering and technical indicators
• ml4t-backtest: Event-driven backtesting
• ml4t-live: Live trading with broker integration

Development

git clone https://github.com/ml4t/diagnostic.git
cd ml4t-diagnostic
uv sync
uv run pytest tests/ -q -n auto
uv run ty check

References

• Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley.
• Bailey, D., & Lopez de Prado, M. (2012). "The Sharpe Ratio Efficient Frontier."
• Bailey, D., et al. (2014). "The Deflated Sharpe Ratio."
• Bailey, D., et al. (2016). "The Probability of Backtest Overfitting."
• Lopez de Prado, M. (2020). "Combinatorial Purged Cross-Validation."

License

MIT License - see LICENSE for details.