Authors:
Xiang Li, Pin-Yu Chen, and Wenqi Wei
Where published:
ACM Transactions on Internet Technology, Vol. 25, August 2025.
Abstract:
This paper introduces SONAR, a comprehensive framework and benchmark for detecting AI-synthesized audio, and focuses on evaluating how well current detection methods generalize across diverse and realistic scenarios. The authors build a new dataset collected from 9 different audio synthesis platforms (including modern TTS systems) and use it to uniformly evaluate both traditional deep learning models and newer foundation models. Through extensive experiments, they show that existing detection methods suffer from poor generalization, while foundation models perform better, especially due to large-scale pretraining. They also demonstrate that these models have strong cross-lingual capabilities, maintaining good performance even when trained only on English data, suggesting that detection challenges are more related to audio realism than language differences. Additionally, the paper explores few-shot fine-tuning as an efficient way to adapt models for specific use cases. Overall, the study provides a standardized benchmark, reveals key limitations in current systems, and highlights promising directions for improving generalizable audio deepfake detection.
Dataset names (used for):
- SONAR: newly collected evaluation dataset.
- Wavefake: training/fine-tuning and cross-dataset testing.
- LibriSeVoc and In-the-Wild: cross-dataset generalization testing.
- MLAAD: cross-lingual generalization.
- ASVspoof2019 VC subset: voice-conversion generalization.
- LibriTTS clean-test: real speech samples for balanced SONAR evaluation.
Some description of the approach:
The paper proposes SONAR, a benchmark framework for evaluating audio deepfake detectors against modern AI-generated audio. It builds a new evaluation dataset from nine synthesis sources and compares traditional detectors with foundation-model-based detectors across dataset, language, TTS, environmental sound, and voice-conversion generalization settings.
Some description of the data:
It includes both generated audio from APIs/models and collected samples from demo pages or provided test sets; an equal number of real speech samples from LibriTTS clean-test are also provided for evaluation.
Keywords:
Audio; deepfake detection; benchmark framework.
Instance Represent:
Audio is resampled to 16 kHz and converted to raw waveforms of 64,000 samples, about 4 seconds; longer clips are trimmed and shorter clips are repeated/padded.
Dataset Characteristics:
SONAR is an evaluation dataset focused on modern TTS/audio generation systems. It includes 2,274 fake audio samples from nine sources, with varied average durations, pitches, languages, and model years. It is balanced for evaluation by adding 2,274 real LibriTTS clean-test samples.
Subject Area:
Audio deepfake detection, AI-synthesized speech detection, benchmark evaluation, TTS/VC detection, model generalization, cross-lingual detection, and security/privacy.
Associated Tools:
SONAR GitHub repository, TTS/audio sources including OpenAI, xTTS, AudioGen, Seed-TTS, VALL-E, PromptTTS2, NaturalSpeech3, VoiceBox, FlashSpeech; detection models including AASIST, RawGAT-ST, RawNet2, Spec.+ResNet, LFCC-LCNN, Wav2Vec2, Wav2Vec2BERT, HuBERT, CLAP, Whisper-small, and Whisper-large.
Feature Type:
The benchmark covers multiple feature/input types: raw waveform, linear spectrogram, LFCC, and learned foundation-model embeddings. Traditional models include raw-waveform systems, spectrogram-based ResNet, and LFCC-LCNN; foundation models use learned representations from Wav2Vec2, Wav2Vec2BERT, HuBERT, CLAP, and Whisper.
Number of Instances:
4,548 total SONAR evaluation samples: 2,274 AI-synthesized fake samples + 2,274 real speech samples from LibriTTS clean-test.
Number of Features:
The paper doesn’t explicitly state a specific number of features used. It evaluates 11 detection models across various datasets