Multilingual Spoken Language Identification (SLID) using Acoustic Features & Machine Learning

An end-to-end Machine Learning and Audio Signal Processing pipeline for Spoken Language Identification (SLID) across four distinct human languages: German, Italian, Korean, and Spanish.

The system implements automated audio cleaning, silence removal, acoustic feature extraction (MFCCs, spectral moments, temporal zero-crossing rates), dimensionality reduction (PCA & LDA), unsupervised clustering analysis, and high-accuracy supervised classification reaching $99.48%$ test accuracy.

---

1. System Pipeline Architecture

graph TD
    A[Raw Speech Audio .mp3 / .wav] --> B[Audio Preprocessing & Resampling 22.05 kHz]
    B --> C[Silence Removal & Amplitude Normalization]
    C --> D[Acoustic Feature Extraction 36 Dimensions]
    
    subgraph Feature Space
        D --> D1[13 MFCCs Mean & Std = 26 features]
        D --> D2[Spectral Centroid, Bandwidth, Rolloff = 6 features]
        D --> D3[Zero-Crossing Rate & RMS Energy = 4 features]
    end

    D1 & D2 & D3 --> E[StandardScaler Normalization]
    E --> F[Dimensionality Reduction PCA / LDA]
    
    subgraph Modeling & Evaluation
        F --> G1[Unsupervised Clustering: K-Means & Agglomerative]
        F --> G2[Supervised Classifiers: KNN, SVM-RBF, Random Forest]
    end
    
    G2 --> H[Final Prediction: German / Italian / Korean / Spanish]

Loading

---

2. Audio Processing & Feature Engineering

2.1 Acoustic Preprocessing Pipeline

1. Monophonic Resampling: All recordings are converted to single-channel mono and resampled to $f_s = 22,050\text{ Hz}$.
2. Silence Removal: Trimming low-energy silence segments using an adaptive threshold set $20\text{ dB}$ below peak frame amplitude (librosa.effects.trim).
3. Amplitude Normalization: Peak normalization scaling amplitude vectors to $[-1.0, +1.0]$ to eliminate volume bias across microphone hardware.
4. Length Validation: Discarding truncated audio segments below $1.0\text{ second}$ after trimming.

2.2 Feature Representation (36 Extracted Dimensions)

• Mel-Frequency Cepstral Coefficients (MFCCs): 13 cepstral coefficients capturing vocal tract resonance shapes; both mean ($\mu$) and standard deviation ($\sigma$) computed across temporal frames ($26\text{ features}$).
• Spectral Centroid ($\mu, \sigma$): Frequency center of mass, reflecting perceived spectral sharpness/brightness ($2\text{ features}$).
• Spectral Bandwidth ($\mu, \sigma$): Spectral spread around the centroid ($2\text{ features}$).
• Spectral Rolloff ($\mu, \sigma$): Frequency below which $85%$ of total spectral energy is contained ($2\text{ features}$).
• Zero-Crossing Rate (ZCR) ($\mu, \sigma$): Rate of sign alternation per frame, separating voiced vowels from unvoiced fricatives ($2\text{ features}$).
• Root-Mean-Square (RMS) Energy ($\mu, \sigma$): Signal power envelope over time ($2\text{ features}$).

---

3. Dimensionality Reduction & Clustering Analysis

3.1 Principal Component Analysis (PCA)

Standardized feature vectors ($36\text{ dims}$) were projected via PCA:

• The first 9 principal components capture $> 85%$ of cumulative variance.
• The first 16 principal components capture $95.32%$ of cumulative variance.
• Projecting into reduced orthogonal subspaces effectively eliminates multi-collinearity and suppresses noise.

3.2 Unsupervised Clustering Evaluation

Clustering algorithms were evaluated without class labels to examine natural phonetic grouping:

• K-Means Clustering ($k = 4$):
  • Silhouette Score: $0.2733$
  • Overall Cluster Purity: $61.38%$
  • Spanish exhibited an isolated, pure cluster ($100%$ purity, $90/90$ samples).
• Agglomerative Hierarchical Clustering ($k = 4$):
  • Silhouette Score: $0.2547$
  • Overall Cluster Purity: $58.99%$

---

4. Supervised Classification Benchmarks

Models were trained on 520 balanced samples and evaluated on a holdout test set of 192 samples:

Model Architecture	Hyperparameters	Test Accuracy	Macro Precision	Macro Recall	Macro F1-Score
K-Nearest Neighbors (KNN)	$k=5$, Euclidean distance	$99.48%$	$1.00$	$0.99$	$0.9946$
Support Vector Machine (SVM)	RBF Kernel, $C=1.0, \gamma=\text{scale}$	$98.96%$	$0.99$	$0.99$	$0.9898$
Random Forest	$n_{trees}=100$, Gini criterion	$95.31%$	$0.96$	$0.95$	$0.9530$

Per-Language Classification Report (Best Model: KNN)

Language	Precision	Recall	F1-Score	Support
German	$1.00$	$0.98$	$0.99$	42
Italian	$0.98$	$1.00$	$0.99$	51
Korean	$1.00$	$1.00$	$1.00$	53
Spanish	$1.00$	$1.00$	$1.00$	46
Overall / Macro Avg	$1.00$	$0.99$	$0.99$	192

---

5. Repository Structure

Spoken-Language-Identification-Audio-ML/
├── README.md
├── .gitignore
├── data/
│   ├── dataset_train.csv
│   ├── dataset_test.csv
│   └── features.csv
├── models/
│   ├── knn_model.pkl
│   ├── svm_model.pkl
│   ├── rf_model.pkl
│   ├── classes.npy
│   ├── X_test_processed.npy
│   └── y_test.npy
├── notebooks/
│   ├── 1_Data_Cleaning_and_Feature_Extraction.ipynb
│   ├── 2_Clustering_and_Unsupervised_Analysis.ipynb
│   ├── 3_Supervised_Classification.ipynb
│   └── 4_Model_Evaluation_and_Comparison.ipynb
└── src/
    └── clip.py

---

6. Getting Started & Reproduction

1. Installation

Clone the repository and install required dependencies:

pip install numpy pandas scipy scikit-learn librosa soundfile matplotlib seaborn tqdm

2. Running Notebooks

Navigate to the notebooks/ directory and execute sequentially:

1. 1_Data_Cleaning_and_Feature_Extraction.ipynb: Cleans audio files and extracts 36 acoustic features.
2. 2_Clustering_and_Unsupervised_Analysis.ipynb: Runs PCA variance sweeps, K-Means, and Agglomerative clustering.
3. 3_Supervised_Classification.ipynb: Trains KNN, SVM, and Random Forest models and serializes checkpoints.
4. 4_Model_Evaluation_and_Comparison.ipynb: Computes confusion matrices, precision/recall curves, and cluster purity.

---

7. Author & Academic Context

• Author: Danial Ghorbani
• LinkedIn: Danial Qorbani
• Degree: B.Sc. in Electrical Engineering (Electronics), Minor in Computer Engineering
• Course: Machine Learning
• Institution: School of Electrical and Computer Engineering, University of Tehran
• Contact: daniel.ghorbani.work@gmail.com