PyTorch Seq2Seq English-to-Hindi Translator

A modular, from-scratch Sequence-to-Sequence (Seq2Seq) English-to-Hindi neural machine translation system built using PyTorch.

---

1. Project Overview

This project is built to understand and implement the classical Seq2Seq Encoder-Decoder architecture before moving to Attention mechanisms and Transformers.

Core Design Philosophy & Constraints

• Jointly Learned Embeddings: We explicitly do not use separately pretrained word embeddings (like Word2Vec or GloVe). The English and Hindi embedding matrices are part of the Seq2Seq model itself and are learned jointly with the encoder and decoder through backpropagation.
• VANILLA Seq2Seq Baseline: The current stage is deliberately focused on the vanilla Seq2Seq architecture (without Attention) to observe and document its limitations first-hand.

---

2. Current Architecture

English Sentence
       │
       ▼
Tokenizer / Vocabulary
       │
       ▼
English Token IDs
       │
       ▼
English Embedding (256-dim)
       │
       ▼
LSTM Encoder (512-dim, packed) ──(final hidden/cell state)──► LSTM Decoder (512-dim)
                                                                    ▲
                                                                    │
                                                             Hindi Embedding (256-dim)
                                                                    │
                                                                    ▼
                                                            Linear Output Layer
                                                                    │
                                                                    ▼
                                                          Hindi Vocab Predictions

• No Attention Mechanism: The encoder summarizes the entire source sentence into a single, fixed-size context vector (its final hidden and cell states) which is passed directly to initialize the decoder.
• Packed Sequences: Source sequences are padded for batching but packed before passing to the LSTM Encoder, ensuring the final encoder state represents the last real token rather than padding tokens.

---

3. Project Structure

translator/
├── pyproject.toml
├── uv.lock
├── README.md
├── main.py
├── data/
│   ├── raw/                 # Original parallel corpus files (en_30k.txt, hi_30k.txt)
│   ├── processed/           # Split and cleaned text files
│   │   ├── train/           # 24,000 sentence pairs
│   │   ├── validation/      # 3,000 sentence pairs
│   │   └── test/            # 3,001 sentence pairs
│   └── vocab/               # Serialized vocabularies (english_vocab.txt, hindi_vocab.txt)
├── checkpoints/
│   ├── best_model.pt        # Model checkpoint with lowest validation loss
│   └── final_model.pt       # Model checkpoint after the final epoch (Epoch 25)
├── scripts/
│   ├── 01_prepare_data.py   # Cleans and splits raw text
│   ├── 02_build_vocab.py    # Builds language-specific vocabularies
│   ├── 03_train.py          # Trains the network and checkpoints progress
│   ├── 04_translate.py      # CLI for interactive free-running translation
│   ├── 05_diagnose.py       # Detailed teacher-forced diagnostic CLI
│   └── inspect_lengths.py   # Inspects sequence length distribution statistics
└── src/
    ├── data/                # Data loading, cleaning, filtering, tokenizing, splitting
    ├── dataset/             # Dataset classes & collation helpers (handling sequence lengths)
    ├── encoder/             # LSTM Encoder (with packed sequences)
    ├── decoder/             # LSTM Decoder (autoregressive step-by-step decoding)
    ├── seq2seq/             # Seq2Seq container orchestrating encoder and decoder
    ├── training/            # Custom training loop & cross-entropy loss
    ├── inference/           # Translation helper and diagnostics
    └── utils/               # Checkpoint helpers

---

4. Pipeline & Setup

Setup with uv

# Create and activate environment
uv venv
source .venv/bin/activate

# Install dependencies
pip install torch

Running the Pipeline

Run each script sequentially from the repository root:

1. Prepare Data:

   python -m scripts.01_prepare_data

2. Build Vocabularies:

   python -m scripts.02_build_vocab

3. Train the Model:

   python -m scripts.03_train

4. Interactive Free-running Translate:

   python -m scripts.04_translate

5. Run Teacher-Forced Diagnostics:

   python -m scripts.05_diagnose

---

5. Implementation Details

File Responsibilities & Preprocessing

• src/data/cleaner.py: Normalizes text rather than modifying it aggressively.
• src/data/filter.py: We purposefully do not filter out sentence pairs based on length difference since English and Hindi naturally have different token lengths.
• src/data/tokenizer.py: Tokenizes text at the word-level (e.g. "I am happy." -> ["I", "am", "happy"]).
• src/data/vocabulary.py: Builds separate vocabularies with reserved tokens: <PAD>=0, <UNK>=1, <SOS>=2, <EOS>=3.

Dataset & Tokenization Lengths

The dataset (derived from a 30k subset of the IIT Bombay corpus) has a dictionary/definition style rather than conversational text:

Source: Something that is about to happen. Target: कुछ है जो होने वाला है।

The token length distribution is as follows:

Language	Min	Max	Mean	Median	P95	P99
English	1	83	12.20	11.00	27	39
Hindi	1	59	13.49	12.00	30	45

Major Fixes Applied

1. Source Length Tracking & Packing (src/dataset/collate.py, src/encoder/encoder.py): Originally, sentences were padded, but the Encoder processed the <PAD> tokens. This meant the encoder's final hidden/cell states represented the state after processing padding rather than the final real token (e.g. after <EOS>).
   • Fix: The collator now tracks and returns source_lengths. The encoder uses pack_padded_sequence() so the LSTM stops processing at the actual sentence length. This improved best validation loss dramatically from 5.9321 to 5.5282.
2. Decoder Target Alignment (src/training/trainer.py, src/seq2seq/model.py): Outputs are generated for the entire sequence, but there is no prediction matching <SOS>. The trainer excludes the first output timestep, computing cross-entropy loss against targets[:, 1:] and predictions[:, 1:, :].
3. Correct Checkpointing (scripts/03_train.py): Corrected the checkpoint saving logic. The model now exports two distinct files:
   • checkpoints/best_model.pt: Saved when validation loss reaches a new low.
   • checkpoints/final_model.pt: Saved exactly after the final epoch (e.g., Epoch 25).

---

6. Experimental Trajectory & Diagnostics

The following controlled experiments were conducted to evaluate different architecture/training configurations:

Experiment	Embedding / Hidden Dim	Teacher Forcing Ratio	Key Change / Feature	Best Val Loss (Epoch)	Final Epoch Train/Val Loss	Inference Quality / Observations
1. Original	128 / 256	1.0	Baseline (Uncorrected Hindi Vocab)	-	-	Extremely poor. Output dominated by repeating Hindi characters (e.g., क ा ् ा ् ...) due to tiny vocab (1,258 tokens).
2. Corrected Vocab	128 / 256	1.0	Vocab rebuilt (Eng: 5653, Hi: 5485)	5.9321 (23)	4.8788 / 5.9567	Poor. Simple sentences mapped to random output words (e.g., "I am happy." $\rightarrow$ "जो के लिए ।").
3. Packed Encoder	128 / 256	1.0	Encoder uses packed sequences	5.5282 (16)	2.5260 / 5.7247	Substantial improvement. Grammatical structure emerged, but translations were still semantically incorrect.
4. Mixed Teacher Forcing	128 / 256	0.5	Lowered teacher forcing to counter exposure bias	5.7892 (17)	4.0599 / 5.8712	Degraded validation loss compared to Exp 3. Repetitions remained prevalent.
5. Capacity Boost	256 / 512	0.5	Doubled embedding & hidden sizes	5.7341 (11)	1.9402 / 6.2553	Severe overfitting after Epoch 11. Did not solve the semantic translation issue.

Diagnostic Investigations (Using Epoch-25 Model)

We analyzed the epoch-25 checkpoint (final_model.pt) using both free-running and teacher-forced diagnostics:

1. Free-Running Diagnostic (Training Set)

Testing the final model on its own training sentences yielded mixed results:

• Short sentences were generated exactly or nearly exactly:
  • "To take somebody into legal custody." $\rightarrow$ "किसी को कानूनी हिरासत में लेना ।"
• Longer, information-dense definitions degraded into generic high-frequency Hindi tokens/repetitions.

2. Teacher-Forced Diagnostic (scripts/05_diagnose.py)

To isolate whether decoding errors were purely due to exposure bias, we fed correct target tokens step-by-step to check output token probabilities:

• Short Sentence ("The effect of soil on organism."): All 7/7 target tokens predicted correctly. However, the probability for the correct word "प्रभाव" was only 0.0934, highlighting that top-1 correctness can hide high uncertainty.
• Long Sentence ("An amount of assistance granted which indirectly..."): Predicted only 13/18 tokens correctly. Several target positions had extremely weak probabilities (e.g., "भार" at 0.0066 and "अप्रत्यक्ष" at 0.0539), even when given the correct preceding target tokens.

Conclusion: The model's failure in free-running autoregressive decoding is not purely due to exposure bias; it has weak target-token probabilities at several key positions. Autoregressive decoding then amplifies these local errors when a wrong prediction is fed forward.

---

7. Key Rules for Future Iterations

To prevent blind hyperparameter tuning, future steps must adhere to this cycle:

Identify Problem ──► Formulate Hypothesis ──► Make ONE Controlled Change ──► Train ──► Evaluate Loss/Translations ──► Interpret

What NOT to do next:

• Do not blindly increase hidden layers or embedding dimensions again.
• Do not increase training epochs indefinitely.
• Do not randomly filter the dataset.
• Do not jump directly to Transformers or Attention without understanding the exact failure points of this Seq2Seq baseline.