Authors:
Yuankun Xie, Ruibo Fu, Xiaopeng Wang, Zhiyong Wang, Songjun Cao, Long Ma, Haonan Cheng, and Long Ye.
Where published:
The Fortieth AAAI Conference on Artificial Intelligence (AAAI-26)
Abstract:
The paper tackles the challenge of detecting all types of audio deepfakes (speech, sound, singing, and music) instead of focusing on just one category. The authors first build a comprehensive benchmark dataset that enables evaluation across these different audio types, addressing the lack of unified testing in prior work. They then propose a new training approach called prompt tuning self-supervised learning (PT-SSL), which adapts pre-trained models efficiently using far fewer parameters than traditional fine-tuning. To further improve cross-type detection, they introduce wavelet prompt tuning (WPT-SSL), which captures shared frequency-domain patterns across different audio types without adding extra training cost. Finally, they train a universal detector using all types of deepfake audio together and show that their method (WPT-XLSR-AASIST) significantly improves performance, achieving low error rates and better generalization across diverse audio deepfake scenarios.
Dataset names (used for):
- Speech-19LA for speech deepfake detection;
- Codecfake-A3 for sound deepfake detection;
- CtrSVDD for singing voice deepfake detection;
- FakeMusicCaps for music deepfake detection.
- The combined version is used for all-type co-training.
Some description of the approach:
The paper proposes an all-type audio deepfake detection benchmark covering speech, sound, singing voice, and music. It introduces PT-SSL-AASIST and WPT-SSL-AASIST, where prompt tokens are learned for SSL front-ends while most SSL parameters remain frozen.
Some description of the data:
The benchmark contains four audio types: speech, sound, singing voice, and music.
Keywords:
all-type audio deepfake detection; wavelet prompt tuning; prompt tuning; self-supervised learning; AASIST.
Instance Represent:
Each instance is an audio sample. For model input, audio is downsampled to 16 kHz and trimmed or padded to 64,600 samples, corresponding to about 4 seconds. SSL features for 4-second audio have shape (201, 1024).
Dataset Characteristics:
The all-type benchmark has four categories and three splits. The combined benchmark contains 199,023 training, 84,438 development, and 189,951 evaluation samples. Individual sources are Speech-19LA, Codecfake-A3, CtrSVDD, and FakeMusicCaps.
Subject Area:
Audio deepfake detection, multimedia security, self-supervised learning, prompt tuning, cross-type detection, speech/sound/singing/music forensics.
Associated Tools:
Five baseline models: Spec-Resnet, AASIST, and three SSL-enhanced variants: MERT-AASIST, WavLM-AASIST, and XLSR-AASIST—defined by their front-end and AASIST back-end combinations. Four SSL-AASIST training paradigms: FR (frozen), FT (fine-tuned), PT (prompt tuning), and WPT (wavelet prompt tuning).
Feature Type:
The paper studies spectrogram-based handcrafted features, raw waveform inputs, and SSL-based features. Spec-ResNet uses spectrograms, AASIST operates on raw waveforms, and SSL-AASIST uses MERT, WavLM, or XLSR features. WPT-SSL further applies DWT to prompt tokens to add frequency-band wavelet prompts for type-invariant deepfake detection.
Number of Instances:
473,412 total samples across the combined all-type benchmark: 199,023 train + 84,438 development + 189,951 evaluation.
Number of Features:
For SSL-based models, the paper reports feature shape (201, 1024) for 4-second audio. For Spec-ResNet, spectrograms are computed with 512 FFT points, 160 hop length, and 512 window length. For prompt tuning, the best PT setting uses 10 prompt tokens, and the best WPT setting uses 4 wavelet prompt tokens + 6 standard prompt tokens.