Feat: Replace TF-IDF with NLP Semantic Search Embeddings (#1181)

[desireddymohithreddy0925]

Summary

This PR resolves #1181 by completely replacing the legacy custom TF-IDF keyword-matching algorithm with a state-of-the-art NLP Semantic Search. By leveraging sentence-transformers and the lightweight all-MiniLM-L6-v2 model, the recommendation engine can now correctly pair synonymous and contextually related inputs (e.g., matching "Frontend" with "UI") that previously would have scored 0.

Related Issue

Closes #1181

Type of Change

• Bug fix — resolves a broken behaviour
• Feature — adds new functionality
• Data — adds new projects to data/projects.json
• Documentation — updates docs, README, or code comments only
• Style — CSS or visual changes only, no logic change
• Refactor — restructures code without changing behaviour
• Test — adds or updates tests

What Was Changed

File	Change made
requirements.txt	Added sentence-transformers==3.0.1.
src/utils/recommender.py	Removed _tf, _idf, _tfidf_vector, _cosine_similarity, and _tokenize. Integrated get_nlp_model() to lazily load the all-MiniLM-L6-v2 model. Rewrote ml_similarity_score() to compute continuous vector cosine similarity. Introduced LRU caching for user query encoding and dictionary caching for project encodings to prevent redundant processing.
tests/test_basic.py	Updated test_ml_similarity_score_returns_float to match the new ml_similarity_score signature since it no longer requires the full all_projects payload for IDF computation.

How to Test This PR

1. Checkout the branch: git checkout feat/issue-1181-nlp-embeddings
2. Run pip install -r requirements.txt. (Note: This will install PyTorch and download the model weights on the first execution).
3. Boot the server python src/app.py.
4. Submit a profile on the homepage using synonymous terminology (e.g., Interest: "UI", Skills: "React, Node").
5. Verify that projects using related terms (e.g., "Frontend") are correctly recommended with a high relevancy score due to semantic similarity.

Test Results

============================= test session starts ==============================
platform darwin -- Python 3.9.6, pytest-8.2.2, pluggy-1.6.0
rootdir: /Users/desireddymohithreddy/GSSOC2026DevPath/DevPathMRD
collected 80 items

tests/test_basic.py .................................................... [ 65%]
............................                                             [100%]

======================== 80 passed, 1 warning in 6.40s =========================

Self-Review Checklist

• I have read CONTRIBUTING.md and followed all guidelines
• My branch name follows the convention: feat/, fix/, docs/, data/, style/, test/
• I have run pytest tests/test_basic.py and all tests pass
• I have run flake8 . locally and there are no errors
• I have not introduced any print() or console.log() debug statements
• Every new function I wrote has a docstring
• I have not modified files outside the scope of the linked issue
• If I changed the UI, I tested it at 375px (mobile) and 1280px (desktop)
• If I added a project to the dataset, it has all required JSON fields

[vercel]

Someone is attempting to deploy a commit to the komalsony234-1530's projects Team on Vercel.

A member of the Team first needs to authorize it.

[komalharshita]

Thanks for the contribution! This is an interesting enhancement, but I think it needs a few changes before it can be merged.

• Please resolve the current merge conflicts with the latest main branch.
• Since this replaces the recommendation engine with sentence embeddings, please document the deployment impact (model download, startup time, memory usage, and any additional runtime requirements).
• Please include benchmark results or comparison examples showing how the new semantic search performs relative to the previous TF-IDF implementation.
• Clarify the fallback behavior if the NLP model fails to load. It should be clear whether recommendations continue to work or gracefully degrade.
• Consider adding tests for semantic matching and model initialization/failure scenarios to ensure the new behavior remains reliable.

Once these points are addressed, I'd be happy to review the PR again.

[desireddymohithreddy0925]

@komalharshita done mam

[komalharshita]

Thank you for your contribution to DevPath!

Great work, and I'm happy to approve this PR. Keep contributing! 🚀