Data Snooping: Robustness Testing with SPA, Reality Check, StepM, and Model Confidence Set
Info
The Data Snooping: Robustness Testing with SPA, Reality Check, StepM, and Model Confidence Set is a Expert Advisor for MetaTrader 5 that datasnooping. mqh addresses a critical question after strategy optimization: if the best strategy or parameter set outperforms the benchmark, is it genuinely superior, or merely the luckiest.
Usage
This tool is typically used for automated trading on major forex pairs and gold.
Platform
This Expert Advisor works exclusively on MetaTrader 5 (both build 600+ and newer versions).
Setup
Place the downloaded file in MQL5/Experts folder via File ? Open Data Folder in MetaTrader 5.
How to Install and Use Data Snooping: Robustness Testing with SPA, Reality Check, StepM, and Model Confidence Set
1. Installation: Open the "File" menu, select "Open Data Folder," navigate to MQL/Experts, paste your file, and restart the terminal.
2. Activation: Drag the EA from the Navigator onto a chart, ensure "Allow live trading" is checked in the Common tab, and verify the AutoTrading button is green.
3. Optimization: Right-click your chart, choose "Expert List," click "Properties" to adjust inputs, and save your preferred setup as a set file for future use.
4. Maintenance: Regularly check the "Experts" tab in the terminal window to monitor trade logs and potential execution errors.
Frequently Asked Questions
Q: Why is my EA not opening trades? A: Check the "AutoTrading" button, ensure "Allow live trading" is enabled, and verify your broker allows automated trading on your account type.
Q: Can I run multiple EAs on one chart? A: No, each chart can only host one active EA; however, you can open multiple charts for different currency pairs to run several EAs.
Q: What does the "smiley face" icon mean? A: A smiley face in the top-right corner of the chart indicates the EA is successfully running; a frowny face means it is disabled.
What this tool does
DataSnooping.
Typical Use Case
This Expert Advisor excels in automated trading and technical analysis on MetaTrader 5.
Compatible Platform & Setup
This Expert Advisor works on MetaTrader 5. Place the file in the MQL5/Experts folder and restart the terminal.
Description & Settings
Related: Session Reality Auditor: Verifying Broker Session Data Against Historical Price Activity - another powerful expert for MetaTrader 5 traders.
DataSnooping.mqh addresses a critical question after strategy optimization: if the best strategy or parameter set outperforms the benchmark, is it genuinely superior, or merely the luckiest?Also recommended: tick data recorder - similar expert with strong performance on MetaTrader 5.
This library includes Hansen's test for superior predictive ability (SPA), White's Reality Check, the Romano-Wolf StepM procedure to identify all strategies outperforming the benchmark, and the model confidence set developed by Hansen, Lunde, and Nason. It employs a stationary bootstrap to preserve the dependence of daily results and allows for a user-defined seed. No DLLs are required.
Usage
#include
Library Contents
CSuperiorAbility
: Provides lower, consistent, and upper p-values for the SPA test, the Reality Check (upper p-value without studentization), critical values, strategies exceeding these values, and StepM results. It supports both studentized (Hansen 2005, considering long-run variances from stationary bootstrap) and non-studentized statistics. A higher mean with high volatility is penalized in the studentized version.
CModelConfidenceSet
: Calculates the p-value for each strategy and identifies the best set at any confidence level using either the 'R' statistic (comparing pairs of strategies) or the 'max' statistic. The elimination process efficiently manages memory, retaining the largest pair from each draw, making it performant even with hundreds of strategies.
CStationaryBootstrap
: Implements the stationary bootstrap by Politis and Romano, including indices for resampling and means of resamples derived from running sums of blocks. It supports a user-defined seed.
Note on ARCH Compatibility
The tests in this library align with the Python package arch, producing identical p-values when using the same bootstrap draws. While arch's SPA class has a studentize option, version 8.0's implementation did not apply standard deviations, effectively disabling studentization. This library's Options(..., studentize = false, ...) replicates arch's non-studentized output, while studentize = true provides Hansen's statistic, which accounts for result volatility.
Example
The included script SnoopingExample.mq5 demonstrates the library's functionality. It can generate 100 moving average strategies (long positions when the fast MA is above the slow MA, flat otherwise, with 10 variations for each MA length) or import strategy returns from a CSV file. It then compares the p-value of the best rule tested in isolation against the results from SPA, Reality Check, StepM, and the model confidence set (see Picture 1). For US30 on a Daily timeframe, the best rule shows a t-statistic of 1.72. Tested alone, its p-value is 0.043; however, as the best among 100 tested, its SPA p-value rises to 0.19, indicating no statistically significant outperformance over a null strategy (Picture 2).
Picture 3 illustrates the t-statistic for each rule, categorized by fast and slow average MA settings. Neighboring rules often yield similar results due to shared trading days. The SPA and Reality Check bootstraps account for this dependence, offering a more accurate assessment than a strict Bonferroni correction, which would ignore it.
Picture 4 highlights the same best rule (SMA 5/160). Individually, its p-value is 0.043 (below 5%). When considered as the best of 100 tested rules, its Reality Check p-value becomes 0.244 and its SPA p-value is 0.187, suggesting no evidence of outperformance against staying out of the market.
Validation
The library's results have been validated against arch 8.0's bootstrap indices (stationary bootstrap, 1000 draws) across three datasets: 50 strategies (3 with a genuine edge), 200 correlated parameter sets (no edge), and daily results from 9 sample Expert Advisors.
- SPA without studentization: Lower, consistent, and upper p-values match arch's exactly. Critical values are within 2e-15, long-run variances are identical, and the same superior strategies and StepM set are identified.
- SPA studentized: P-values are identical to an independent NumPy implementation of Hansen's test.
- Model confidence set ('R' and 'max' statistics): P-values for all strategies match arch's.
Files Included
- DataSnooping.mqh: Place in MQL5\Include\DataSnooping.
- SnoopingExample.mq5: Place in MQL5\Scripts\DataSnooping.
You may also like: Writing Array Data to a File - excellent alternative for expert users on MetaTrader 5.
Source Code
#property copyright "RobotFX"
#property version "1.00"
#property script_show_inputs
#include <DataSnooping\DataSnooping.mqh>
input string InpSymbol = "";
input ENUM_TIMEFRAMES InpPeriod = PERIOD_D1;
input int InpBars = 2000;
input bool InpBuyHold = false;
input int InpReps = 2000;
input uint InpSeed = 1;.......
⚠ Limitations & Risk Warning
- This tool is provided for educational and testing purposes only.
- Past performance does not guarantee future results.
- Trading involves substantial risk of loss. Use on a demo account first.
- Results may vary depending on market conditions, broker, and settings.
- We recommend thorough backtesting and forward testing before using with real funds.