Authors:
Yassine El Kheir, Younes Samih, Suraj Maharjan, Tim Polzehl, and Sebastian Möller.
Where published:
Findings of the Association for Computational Linguistics: NAACL 2025
Abstract:
This paper performs a detailed layer-wise analysis of self-supervised learning (SSL) models for audio deepfake detection across multiple contexts, including different languages (English, Chinese, Spanish) and scenarios such as partial speech, singing, and real-world scenes. Instead of proposing a new model, the authors investigate which transformer layers contribute most to detection performance. They find that lower layers capture the most useful and discriminative features, while higher layers are less relevant for this task. Importantly, they show that using only a subset of lower layers still achieves competitive performance, which means models can be made more efficient with reduced computation and faster inference. Overall, the study provides insights into how SSL models work internally and offers practical guidance for building more efficient and effective audio deepfake detection systems.
Dataset names (used for):
- ASVspoof 2019 LA / LA19: English full-fake train/eval.
- ASVspoof 2021 LA / LA21 and ASVspoof 2021 DF / DF21: English evaluation.
- ADD23 Track 1.2: Chinese full-fake train/eval.
- HABLA: Spanish full-fake train/eval.
- PartialSpoof: English partial-fake train/eval.
- Half-Truth / HAD: Chinese partial-fake training.
- CtrSVDD: multilingual song deepfake train/eval.
- SceneFake: English scene-fake train/eval.
Some description of the approach:
The paper performs a layer-wise analysis of frozen SSL models for audio deepfake detection. SSL models are used as front-end feature extractors, their transformer-layer outputs are weighted and aggregated, and a back-end classifier predicts real vs. fake.
Some description of the data:
The datasets cover multilingual and contextual audio deepfake settings: full utterance, partial utterance, song-based, and scene-based deepfakes across English, Chinese, Spanish, and multilingual datasets.
Keywords:
self-supervised learning; SSL models; audio deepfake detection; layer-wise analysis; multilingual deepfake detection; partial fake; song deepfake; scene fake.
Instance Represent:
Each instance is an input audio signal. The SSL model extracts hidden states from transformer layers, and the weighted layer representation is used for classification. Audio is cropped or concatenated into about 4-second segments / 64,600 samples, except partial-fake data where full audio is used.
Dataset Characteristics:
The datasets cover Full Fake, Partial Fake, Song, and Scene categories. Full Fake means the whole utterance is generated by TTS/VC; Partial Fake replaces only specific segments; Song includes utterances with background music; Scene keeps speech real but changes the acoustic environment.
Subject Area:
Audio deepfake detection, self-supervised speech models, speaker verification security, multilingual detection, and detection of full, partial, song, and scene deepfakes.
Associated Tools:
- SSL front-ends: Wav2Vec2, HuBERT, and WavLM, each in small/base and large versions.
- Back-end classifiers: FFN and AASIST.
- Training uses cross-entropy loss, Adam optimizer, and one NVIDIA H100 GPU.
Feature Type:
Learned SSL hidden-state representations from different transformer layers. The paper compares layer-wise features from Wav2Vec2, HuBERT, and WavLM, rather than handcrafted features like MFCC/LFCC.
Number of Features:
Based on the paper, the feature dimensionality depends on the model size: Base/Small models: hidden state dimension of 768; Large models: hidden state dimension of 1024. These are the dimensions of each layer’s output vectors (denoted as h_t in the paper), which get weighted and aggregated before being passed to the backend classifier.