# Futures data manifest (1-minute files)

Recorded 2026-10-08, before any strategy was run on the NQ and GC files. Anything that changes a file below (re-export, repair of a hole) must create a new dated manifest entry and a new hash; the old entry stays.

## How the files were made

- Source: NinjaTrader 8.1 historical data (NinjaTrader Historical Data Server), exported by `FuturesDataExporter.cs` (ES: the earlier `ESDataExporter.cs`, same columns) through Strategy Analyzer, 1-minute bars, price based on Last, instrument's default trading hours (CME US Index Futures ETH for ES and NQ, Nymex Metals - Energy ETH for GC).
- Continuous series: **Merge back adjusted** (additive, in index points or dollars per ounce, at each roll). Verified in the NQ file: the first bar moved from 8,776.75 (unadjusted export) to 12,373.00 with the same volume (offset +3,596.25), and the +1% jump at each quarterly roll in 2022-2025 disappeared. Checked bar by bar against a raw export of the single contract NQ 03-20 (NinjaTrader Historical Data export, UTC time stamps, 74,148 rows): from 2020-01-01 18:01 to 2020-03-11 17:00 New York, 67,537 bars match, every open, high, low and close is shifted by exactly +3,596.25 and the volumes are identical (0 mismatches); at 2020-03-11 18:01 the file continues on the next contract (the roll). The 11 raw bars before the roll that the merged file lacks are stray prints on weekends (for example Saturday 2020-01-04 11:54 New York) outside the trading-hours template. Consequence: past prices are not the prices that traded; results in points, R and dollars are right, percentage changes in early years are not (the NQ series starts 41% above its traded level; ES and GC are shifted the same way, by amounts not measured here). Signals that depend on the sign of a crossing are almost unaffected; any rule with a percentage threshold needs care.
- Time stamp: `ny_time` is the **bar close time** converted from the chart time zone (GTB Standard Time) to New York time. Sessions open 18:00 New York the evening before.
- Window used by every study: 2020-01-01 to **2026-09-25 17:00 New York**. The GC file runs to 2026-10-08 and the NQ file to 2026-10-06 (the exporter end date); the days after 2026-09-25 are not used by any test and stay as fresh data for confirming survivors.

## Files

| File | Rows | First bar (NY) | Last bar (NY) | Tick / point value | SHA-256 |
|---|---|---|---|---|---|
| `ES_1min.csv` (exported 2026-09-27) | 2,378,027 | 2020-01-01 18:01 | 2026-09-25 16:59 | 0.25 / $50 | `ce7d4408d45595b2c431fb21833b6728e24ce153edcbc34b3fcde1956cb9c13e` |
| `NQ_1min.csv` (re-exported 2026-10-08 17:12) | 2,383,438 | 2020-01-01 18:01 | 2026-10-06 16:59 | 0.25 / $20 | `7980d8351b388fb1daed4905dc445a1e7a6764f4072013e36a1e21605f5f15af` |
| `GC_1min.csv` (exported 2026-10-08 15:39) | 2,385,099 | 2020-01-01 18:01 | 2026-10-08 08:38 | 0.1 / $100 | `b0eccf7ad06eacab6f48471278ad27611fff7df130967051b1030bbd7bf62fd6` |
| `YM_1min.csv` (exported 2026-10-22 21:25, YM DEC26, merge back adjusted) | 2,372,800 | 2020-01-01 18:01 | 2026-10-22 14:24 | 1 / $5 | `e434dac407685f409016014e137022bf21748624b2c1d765bea7d1860ad1a2e4` |

Checks passed on NQ and GC: no malformed rows, no duplicate or backward time stamps, no OHLC inconsistencies, no zero volume, no non-positive prices; after back-adjustment the quarterly roll weeks show ordinary weekend gaps of both signs.

## YM (added 2026-10-22)

Exported by the owner with `FuturesDataExporter.cs` (Strategy Analyzer, YM DEC26, 1 minute, Last, instrument trading hours CME US Index Futures ETH, global merge policy "Merge back adjusted" checked in the owner's screenshot first). Checks (scratchpad ym_check.js): no duplicate or backward time stamps, no OHLC inconsistencies, no off-tick prices, no zero volume, no non-positive prices. Roll check against the back-adjusted ES file: on 1,734 evenings the session-open gaps of YM and ES differ by a standard deviation of 0.073%; on the 216 roll-week evenings the largest difference is 0.761% (Sunday 15 March 2020, the emergency Fed cut, not a roll jump), so the rolls are adjusted. Holes by `find_holes_ym.js` (the rules of `find_holes.js`, unchanged, comparing YM with ES, NQ and GC), written to `data_holes_ym.json` so that `data_holes.json` stays as the published tests used it: 27 holes in 20 windows to 2026-09-25 17:00, the largest **2023-04-05 14:03 to 2023-04-16 18:01, the same window as the GC hole below** (NinjaTrader's server has no files for those days), then 2022-11-06 19:00 to 11-07 05:34, 2025-11-27 21:45 to 11-28 08:31, 26 to 27 April 2020 evening, the March 2020 overnight gaps, and five 05:49-06:49 gaps on 19 to 23 October 2020.

## Known holes

The holes are listed per market in `data_holes.json`, found by `find_holes.js` with one fixed rule: a gap of more than 30 minutes between two bars of the same session; a session that opens more than 30 minutes after 18:00; a session that ends before 16:30 at a time that fewer than 5 sessions of the file share; a session with fewer than 690 bars; or a weekday session that has no bars in this file but has bars in another of the three (Good Fridays excepted). Early closes that follow the exchange calendar (the Good Friday jobs-report days, 9 January 2025, the day after Thanksgiving in GC) are listed as class "schedule" and are not holes. Smaller gaps of 10 to 30 minutes exist in thin minutes and are not treated as holes. Result: ES 11 holes (merged into 10 windows), NQ 20 (19 windows), GC 17 (11 windows). The causes were not established (the exchange, the data feed or the download); nothing was filled.

Present in all three files (so also in the published ES studies):

- 2020-10-19 to 2020-10-22: 05:49 to 06:49 New York missing every morning (60 minutes a day); 2020-10-23: the session ends at 05:49 (about 11 hours missing).
- 2021-10-03: the session opens at 19:51 instead of 18:00.
- 2022-11-06 19:00 to 2022-11-07 05:34 (634 minutes).
- 2025-11-27 21:45 to 2025-11-28 about 08:00 to 08:46 (10 to 11 hours).

NQ only (ES has bars on the same minutes):

- 2023-04-05 14:03 to 20:01 (297 minutes of ES bars). NinjaTrader files `NQ 06-23` 20230405 and 20230406 are shorter than normal.
- 2025-08-28 20:01 to 2025-08-29 17:00 (1,257 minutes). `NQ 09-25` 20250829 holds 812 bytes against about 9,800 normal.
- 2026-01-22 19:01 to 2026-01-25 18:01: the whole 2026-01-23 session (1,319 minutes). `NQ 03-26` 20260123 holds 437 bytes.
- 2026-01-18 evening open at 19:03; 2026-06-11 10:49 to 11:23; several gaps of 33 to 106 minutes on 2020-03-15/16, the first days of the March 2020 limit moves (ES has one gap of 37 minutes then).

GC only:

- 2023-04-05 14:03 to 2023-04-16 18:01 (11 days; the 06-23 contract has no files for 2023-04-06 to 2023-04-14).
- 2026-02-25 13:01 to 14:46 (105 minutes).

ES and GC but not NQ: 2026-07-24 10:48 to 11:45 (57 minutes).

Not holes: the 15 weekday exchange holidays with no bars are the same in ES and NQ; GC also has no bars on Good Friday 2021-04-02 and 2026-04-03 (metals closed, equity index futures traded a partial day). ES has a daily gap at 16:15-16:30 New York (equity index trading halt) that GC and NQ do not need to match.

## Attempted repair (2026-10-08)

The four damaged NQ day files were removed from the NinjaTrader database (backed up) and the exporter re-run; NinjaTrader did not re-download them (it only fills the ends of a stored range, and it reads a bar cache in `db\cache` that kept the truncated days). A manual download in Historical Data was then tried by the owner and reported as not possible. The four files were restored from the backup, so the NinjaTrader database is back to its original state. The holes above stay in the data.

## Gold and EUR/USD (MetaTrader 4 history of an IC Markets demo account)

The gold and EUR/USD studies read the 1-minute history files of the MetaTrader 4 terminal (`XAUUSD1.hst`, `EURUSD1.hst`, server time = New York + 7 hours, bar time converted to close time). The XAUUSD file keeps growing because the terminal appends live bars, so the published gold studies are reproduced on the part of the file up to their last bar, **2026-10-07 16:57 New York** (7,067,434 one-minute bars from 2004; the minutes 16:58 and 16:59 were appended to the file after those runs; corrected 2026-10-08); the EUR/USD file ends at 2026-09-28 15:46 (close time of its last bar; corrected 2026-10-09, earlier written as 15:47). A check against the 08:30 New York US data releases (`fb_newsclock.js`, 2026-10-09) found the EUR/USD time stamps about one hour early in most weeks of 2005-2014 and in many US/European clock-change weeks after that; the studies whose rules depend on the time of day were audited on 2026-10-09 by re-running them with corrected time stamps (`clock_audit_map.js`, `clock_audit_fix.js`, `clock_audit_compare.txt`): no verdict changed; the gold opening-range breakout and the FX study posts show the comparison. SHA-256 of that part of each file is in `mt4_prefix_hashes.json` (recorded 2026-10-08; the hashes fix the data from this date, the earlier runs were made on files whose earlier part is assumed not to have changed). Holes in these two series were audited at the level of whole days only (days with fewer than 30 one-minute bars, listed in `ma_cross_spec.md`: 34 weekdays for gold); gaps inside a day were **not** audited, and the hole sensitivity test of rule 3 has not been run on them.

## Rule for the strategy studies (to be copied into the study template before the first run)

1. The files above are used as they are, cut at 2026-09-25 17:00 New York. Nothing is interpolated or filled.
2. A hole behaves like an extended weekend: no bars, so a stop or target that would have been touched inside it can only fill on the first bar after it (at the open if the level was gapped through).
3. Every study reports how many trades were open across a hole, and every verdict is recomputed without those trades. If the verdict changes, it is reported as Inconclusive and the reason is stated.
4. The list of holes (`data_holes.json`), the hashes above and the result of rule 3 are published with each post that uses the data.