Authors:
Yuankun Xie, Yi Lu, Ruibo Fu, Zhengqi Wen, Zhiyong Wang, Jianhua Tao, Xin Qi, Xiaopeng Wang, Yukun Liu, Haonan Cheng, Long Ye, Yi Sun
Yuankun Xie, Yi Lu, Ruibo Fu, Zhengqi Wen, Zhiyong Wang, Jianhua Tao, Xin Qi, Xiaopeng Wang, Yukun Liu, Haonan Cheng, Long Ye, Yi Sun
Abstract:
The paper focuses on detecting audio deepfakes generated by Audio Language Models (ALMs), which produce highly realistic and diverse audio. The authors first create a large-scale dataset called Codecfake, containing over 1 million audio samples (real and fake in English and Chinese), specifically targeting ALM-based audio generated through neural codec-to-waveform processes. They then propose a new training strategy called CSAM (a modified Sharpness-Aware Minimization) to improve model generalization and avoid domain bias. Using this dataset and training approach, they show that detectors can better identify ALM-generated deepfakes and achieve very low error rates, significantly outperforming existing methods.
The paper focuses on detecting audio deepfakes generated by Audio Language Models (ALMs), which produce highly realistic and diverse audio. The authors first create a large-scale dataset called Codecfake, containing over 1 million audio samples (real and fake in English and Chinese), specifically targeting ALM-based audio generated through neural codec-to-waveform processes. They then propose a new training strategy called CSAM (a modified Sharpness-Aware Minimization) to improve model generalization and avoid domain bias. Using this dataset and training approach, they show that detectors can better identify ALM-generated deepfakes and achieve very low error rates, significantly outperforming existing methods.
Data Creation Method:
Seven neural codec models were trained on LibriTTS, then used to re-encode and decode real audio from VCTK and AISHELL3 to create codec-based fake audio. The neural codec models were trained on LibriTTS, a 585-hour, 24 kHz multi-speaker English corpus, using default or ALM-adopted parameter settings until convergence.
Seven neural codec models were trained on LibriTTS, then used to re-encode and decode real audio from VCTK and AISHELL3 to create codec-based fake audio. The neural codec models were trained on LibriTTS, a 585-hour, 24 kHz multi-speaker English corpus, using default or ALM-adopted parameter settings until convergence.
Number of Speakers:
- 328 speakers inferred from source datasets
Total Size:
- 1,058,216 audio samples
Number of Real Samples:
- 132,277 real samples total: 44,242 from VCTK and 88,035 from AISHELL3
Number of Fake Samples:
- 925,939 fake samples generated by seven codec methods
Description of the Dataset:
- Codecfake is an open-source, large-scale collection comprising over 1 million audio samples in both English and Chinese, focusing on ALM-based audio detection.
Extra Details:
F07 is held out as an unseen fake method for generalization testing. F07 is DAC and is excluded from training/development to test unseen-method generalization. The dataset also includes varied test conditions for codec methods, parameter changes, and ALM-based audio.
F07 is held out as an unseen fake method for generalization testing. F07 is DAC and is excluded from training/development to test unseen-method generalization. The dataset also includes varied test conditions for codec methods, parameter changes, and ALM-based audio.
Data Type:
- Audio samples
- Speech and sound
- Codec-based reconstructed waveform audio
Average Length:
- Not specified
Keywords:
- Audio deepfake dataset, audio deepfake detection, audio language model, neural codec
When Published:
- 2025
Annotation Process:
No manual annotation process is described. The real/fake labels come from dataset construction: original source audio is real, and codec-generated audio is fake.
No manual annotation process is described. The real/fake labels come from dataset construction: original source audio is real, and codec-generated audio is fake.
Usage Scenarios:
Training and evaluating audio deepfake detection models, especially for ALM-based and codec-based deepfake audio; testing generalization to unseen codec methods, varied codec parameters, and in-the-wild ALM-based audio.
Training and evaluating audio deepfake detection models, especially for ALM-based and codec-based deepfake audio; testing generalization to unseen codec methods, varied codec parameters, and in-the-wild ALM-based audio.
Data Accessibility:
The dataset and code are available online through the Zenodo and GitHub links printed in the paper.
The dataset and code are available online through the Zenodo and GitHub links printed in the paper.
Miscellaneous Information:
Limitations and future work include adding more diverse audio types, more acoustic conditions, stronger generalized ADD methods, and source-tracing methods.
Limitations and future work include adding more diverse audio types, more acoustic conditions, stronger generalized ADD methods, and source-tracing methods.
Credits:
Datasets Used:
Datasets Used:
- For Codecfake creation: LibriTTS, VCTK, and AISHELL3
- For ALM A3 testing: AudioCaps
- For experimental comparison/co-training: ASVspoof2019LA / 19LA
Speech Synthesis Models Referenced:
- Codec models used for Codecfake: SoundStream, SpeechTokenizer, FunCodec, EnCodec, AudioDec, AcademicCodec, DAC
- ALM-based generation models referenced/tested include VALL-E, VALL-E X, AudioGen