Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models

Official implementation of Trainable Attention-Based Dependency (TAD) - a supervised method for uncertainty quantification (UQ) of large language models (LLMs), introduced in the paper:
Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models (EMNLP 2025)

Overview

TAD learns conditional dependencies between generation steps directly from LLM attention maps, token probabilities, and recurrently computed uncertainty scores from previously generated tokens. It trains a lightweight regression model that recurrently adjusts uncertainty scores using attention-based and recurrent features, enabling unconditional uncertainty estimation for each generated token or sequence.

TAD achieves state-of-the-art results in hallucination detection across 11 datasets and 3 LLMs (LLaMA-3.1, Gemma-2, Qwen-2.5), with 100x less computational overhead than sampling-based UQ methods.

Installation

git clone https://github.com/mbzuai-nlp/llm-tad-uncertainty.git
cd llm-tad-uncertainty
pip install -r requirements.txt

Method

Implementation of the TAD model can be found in tad.py. It supports feature generation from raw attention maps and token probabilities, a two-stage recurrent training procedure as described in the paper, and an efficient inference pipeline.

Benchmark

To run one experiment with LM-polygraph benchmark:

CUDA_VISIBLE_DEVICES=0 HYDRA_CONFIG=./configs/polygraph_eval_truthfullqa.yaml python run_polygraph.py \
    ignore_exceptions=False use_density_based_ue=False batch_size=1 \ 
    subsample_train_dataset=2000 subsample_background_train_dataset=500 subsample_eval_dataset=2000 \
    model.path=meta-llama/Llama-3.1-8B +model.attn_implementation=eager cache_path=./workdir/output \
    +generation_params.samples_n=5 use_seq_ue=True +run_baselines=True +run_supervised_baselines=True \
    +target_train_metric=AlignScore +topns='[1,2,5,10]' +n_steps='[2]'

Reproducing Experiments

Code for reproducing the main results from the EMNLP 2025 paper can be run as follows:

bash scripts/run_tad_exps_norec_aug25_p1.sh & \
bash scripts/run_tad_exps_norec_aug25_p2.sh & \
bash scripts/run_tad_exps_norec_aug25_p3.sh & \
bash scripts/run_tad_exps_norec_aug25_p4.sh

To reproduce the generalization experiments, run:

bash scripts/generalization_sep25/run_tad_exps_final_p1.sh & \
bash scripts/generalization_sep25/run_tad_exps_final_p2.sh

Use generate_series_of_exps.ipynb to regenerate the scripts for run several experiments on different number of GPUs with various datasets and models.

Use results.ipynb to generate tables in pandas and latex with the results.

Citation

@inproceedings{vazhentsev-etal-2025-unconditional,
    title = "Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models",
    author = "Vazhentsev, Artem  and
      Fadeeva, Ekaterina  and
      Xing, Rui  and
      Kuzmin, Gleb  and
      Lazichny, Ivan  and
      Panchenko, Alexander  and
      Nakov, Preslav  and
      Baldwin, Timothy  and
      Panov, Maxim  and
      Shelmanov, Artem",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.1807/",
    pages = "35661--35682",
    ISBN = "979-8-89176-332-6",
    abstract = "Uncertainty quantification (UQ) has emerged as a promising approach for detecting hallucinations and low-quality output of Large Language Models (LLMs). However, obtaining proper uncertainty scores is complicated by the conditional dependency between the generation steps of an autoregressive LLM, because it is hard to model it explicitly. Here, we propose to learn this dependency from attention-based features. In particular, we train a regression model that leverages LLM attention maps, probabilities on the current generation step, and recurrently computed uncertainty scores from previously generated tokens. To incorporate the recurrent features, we also suggest a two-staged training procedure. Our experimental evaluation on ten datasets and three LLMs shows that the proposed method is highly effective for selective generation, achieving substantial improvements over rivaling unsupervised and supervised approaches."
}