Authors:
Yuankun Xie, Xiaoxuan Guo, Jiayi Zhou, Tao Wang, Jian Liu, Ruibo Fu, Xiaopeng Wang, Haonan Cheng, and Long Ye
Where published:
arXiv preprint, 2026.
Abstract:
The paper explores using Audio Large Language Models (ALLMs) for all-type audio deepfake detection (speech, music, environmental sounds, etc.) while also improving interpretability. The authors first evaluate standard training approaches—supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT)—and show their limitations: SFT leads to black-box predictions, while RFT can produce unreliable or hallucinated explanations. To address this, they build a large annotated dataset (~340K samples) with frequency-time structured chain-of-thought (CoT) explanations, generated through an automatic annotation pipeline. They then propose a new training method called FT-GRPO (Frequency Time–Group Relative Policy Optimization), which combines SFT with reinforcement learning guided by frequency-time constraints to ensure meaningful and grounded reasoning. Their approach achieves strong performance across different types of audio deepfakes while also producing interpretable, physically grounded explanations for its decisions.
Dataset names (used for):
- Speech-19LA for speech deepfake detection;
- Sound-ESDD for environmental sound deepfake detection;
- Singing Voice-CtrSVDD for singing voice deepfake detection;
- Music-FakeMusicCaps for music deepfake detection.
Some description of the approach:
The paper proposes FT-GRPO, an interpretable Audio LLM-based framework for all-type audio deepfake detection. It first builds frequency–time structured chain-of-thought rationales through an automatic captioning and polishing pipeline, then trains an ALLM with a two-stage process: SFT cold start followed by GRPO with rewards for accuracy, output format, and frequency–time reasoning completeness.
Some description of the data:
The data cover four audio types: speech, environmental sound, singing voice, and music.
Keywords:
audio deepfake detection; audio large language models; all-type ADD; frequency-time reasoning; reinforcement fine-tuning.
Instance Represent:
Each instance is an audio clip paired with a binary label, real or fake. For the interpretability pipeline, each audio instance is further represented with a structured output containing <think> rationales in Frequency Domain and Time Domain, followed by an <answer> label.
Dataset Characteristics:
The all-type dataset combines four sources and provides training, development, and evaluation splits. The combined sources contain 199,023 training, 84,438 development, and 189,951 evaluation samples. The datasets are selected to be relatively clean and to exclude partially spoofed cases.
Subject Area:
Audio deepfake detection, audio large language models, interpretable AI, reinforcement fine-tuning, speech/sound/singing/music forensics, and multimedia security.
Associated Tools:
Qwen3-Omni-Captioner, Qwen3-235B, Qwen2-Audio-Chat-7B, Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, LoRA, ms-swift, GRPO, and 8 NVIDIA A100 GPUs. Baselines include W2V2-AASIST, WPT-W2V2-AASIST, and ALLM4ADD.
Feature Type:
The key feature representation is frequency–time structured rationales generated by ALLMs. The model is encouraged to provide evidence from both the frequency domain and time domain.
Number of Instances:
For the original combined all-type dataset: 473,412 total samples = 199,023 train + 84,438 development + 189,951 evaluation.
The paper also reports approximately 340K cold-start demonstrations after rationale construction.
Number of Features:
The paper doesn’t explicitly enumerate a fixed “number of features” in the traditional sense. Instead, it uses two feature dimensions for its reasoning framework: Frequency Domain — analyzing spectral artifacts (e.g., unnatural patterns in upper mid and high frequencies); Time Domain — analyzing temporal irregularities (e.g., abrupt pauses, unnatural cadence).