When Financial Transformers Look Predictive
Separating temporal forecasting skill from static cross-sectional structure
The model looked predictive when all assets were pooled together. A simple asset-identity baseline did even better, and performance fell to roughly random when the model had to predict changes within the same asset.
Central findingStrong pooled discrimination did not establish temporal forecasting skill.
Evidence
The result in context
- Transformer pooled ROC-AUC
- 0.790
- Discrimination across all asset windows combined.
- Training-only static asset prior
- 0.824
- A baseline using persistent asset-level information.
- Pair-weighted within-asset ROC-AUC
- 0.492
- Roughly random when comparisons stay within the same asset.
- Learned models passing the temporal-skill gate
- 0 / 5
On this page
Question
Was the model learning how markets change through time, or mostly recognising persistent differences between assets?
A controlled investigation into whether a strong pooled Transformer result represented genuine temporal forecasting skill.
Initial evidence
A result that looked convincing
Across a pooled test set, the Transformer reached a ROC-AUC of 0.790. ROC-AUC measures how well a model separates two outcomes across possible decision thresholds; 0.5 is random and 1.0 is perfect ranking.
The dataset covered 79 assets, 60-session input windows and a 10-session prediction horizon. The apparent signal was large enough to deserve a more difficult question: what exactly was the model using?
Challenge to the result
A static baseline did better
A training-only prior based on persistent asset identity reached 0.824, higher than the neural model. That did not prove the Transformer used asset identity, but it showed that strong pooled ranking could arise without forecasting changes through time.
When evaluation was restricted to comparisons within the same asset, pair-weighted ROC-AUC fell to 0.492, approximately random.
Temporal evidence explorer
What survives the next control?
Move from the attractive pooled score to the test that the research question actually requires.
Across every asset window combined, the model appeared to separate the two outcomes well.
Promising initial evidence, but not yet evidence of temporal skill.
- Pooled0.790
- Static prior0.824
- Within asset0.492
- Asset identityDetected
- ChronologyHeld out
- Simulation1,040
- Conclusion0 / 5
Interactive evidence
Pooled discrimination did not survive within-asset evaluation
ROC-AUC by evaluation frame. Higher is better; 0.50 is chance-level discrimination.
Read: The static asset prior led on pooled ROC-AUC, while the conditioned Transformer fell below chance within assets.
Data table · 3 verified rows
| Model | Pooled Roc Auc | Within Asset Roc Auc |
|---|---|---|
| Transformer | 0.789814 | 0.491638 |
| Transformer, No Asset ID | 0.715477 | 0.47257 |
| Static Asset Prior | 0.823906 | 0.5 |
Diagnostics
The evidence was tested from several directions
Chronology checks, identity controls and 1,040 controlled simulation runs separated temporal signal from static cross-sectional structure. None of five learned models passed the pre-specified temporal-skill gate.
The conclusion is deliberately narrow: this experiment did not establish temporal forecasting skill. It is not a claim that Transformers never work in finance.
Interactive evidence
Static heterogeneity lifted pooled scores without temporal skill
Controlled simulation at zero dynamic signal and persistence 0.7.
Read: As cross-asset prior heterogeneity rose, the static-prior pooled score climbed to 0.899 while within-asset classification stayed near 0.50.
Data table · 4 verified rows
| Prior Heterogeneity | Asset Prior Pooled Roc Auc | Classifier Within Roc Auc |
|---|---|---|
| 0 | 0.499118 | 0.496634 |
| 0.75 | 0.691917 | 0.496796 |
| 1.5 | 0.817073 | 0.504159 |
| 2.5 | 0.899138 | 0.503338 |
Interactive evidence
The same diagnostic responded when temporal signal was present
Controlled simulation at zero prior heterogeneity and persistence 0.7.
Read: Within-asset ROC-AUC rose monotonically with dynamic signal, validating the diagnostic rather than the original headline model.
Data table · 4 verified rows
| Dynamic Signal | Classifier Within Roc Auc | Reversal Auc Drop |
|---|---|---|
| 0 | 0.496634 | -0.002615 |
| 0.5 | 0.611119 | 0.213772 |
| 1 | 0.715264 | 0.425535 |
| 1.5 | 0.787103 | 0.571246 |
Interpretation
Model evaluation begins after the headline metric
A model can rank pooled observations well while failing at the prediction task a practitioner actually cares about. Baseline design and evaluation structure were therefore more informative than another round of architecture tuning.
The broader lesson is to ask what information a metric permits the model to exploit before treating that metric as evidence of a useful forecasting mechanism.
Reproducibility
Frozen evidence and explicit provenance
The final dissertation repository is frozen. Its committed reports and figures form the preserved research record.
Limitations
What this evidence does not establish
- Repeated inspection of the historical test period made later analysis adaptive rather than a pristine untouched confirmation.
- The result is specific to the assets, horizon, labels and experimental design studied; it does not imply universal Transformer failure or universal asset-identity leakage.
Source and reproducibility
Trace the evidence
Source code, evaluation outputs and supporting material are available in the repository.
View repository- Frozen dissertation repositoryassets/readme/graphical_abstract.svgCommit / evidence ID: 81d129902521303c264b6c05f7a7a344c5f39d54
- Headline resultsreports/tables/ifddrp_identity_dynamic_information_decomposition.csvCommit / evidence ID: 81d129902521303c264b6c05f7a7a344c5f39d54
- Controlled simulationreports/tables/prp1_study_a_independent_simulation_results.csvCommit / evidence ID: 81d129902521303c264b6c05f7a7a344c5f39d54