Gabriel Preda

Large Language Models (LLMs) are trained on heterogeneous corpora that include not only factual information but also narratives shaped by nationalism, ideology, and popular culture. Historical domains are particularly affected, as 19th-century nation-building and later political regimes curated the past into simplified, often heroic narratives. These include proto-national reinterpretations of medieval figures, inflated military victories, and moralized accounts of complex geopolitical events.

This talk introduces a benchmark designed to evaluate LLM robustness to historiographical bias, using Romanian history as a case study. The dataset targets topics where popular narratives diverge from mainstream scholarship, covering periods from medieval to modern history. Rather than focusing solely on factual recall, the benchmark evaluates multiple dimensions: event outcome accuracy (tactical vs. strategic), chronological and contextual reasoning, detection of anachronisms, and stability under different prompt framings (neutral vs. nationalistic). It also assesses the model’s ability to acknowledge uncertainty, reflect historiographical debates, and correct common misconceptions.

Evaluation is based on accuracy against curated reference answers, complemented by qualitative analysis of reasoning patterns and failure modes. Results across multiple LLMs highlight systematic tendencies to reproduce dominant but biased narratives, particularly under leading prompts.

The benchmark is implemented and publicly available on Kaggle (https://www.kaggle.com/benchmarks/gpreda/romanian-history
), supporting reproducible evaluation and community contributions. This work contributes to LLM evaluation by emphasizing epistemic robustness in politically and historically sensitive contexts.