Sentiment analysis on 1.6 million tweets using NLP, Apache Spark, and machine learning.
This project performs sentiment analysis on the Sentiment140 dataset containing 1.6 million tweets. The objective is to classify tweets as positive or negative using natural language processing (NLP) techniques and machine learning models.
- Text preprocessing
- Tokenization and text cleaning
- TF-IDF feature extraction
- Machine learning classification
- Large-scale data processing with Apache Spark
- Model evaluation
- Python
- Apache Spark
- Scikit-learn
- Pandas
- NumPy
- TF-IDF
- NLP
The project uses the Sentiment140 dataset, containing approximately 1.6 million labeled tweets for binary sentiment classification.
The dataset was preprocessed using NLP techniques before converting text into numerical features using TF-IDF. Machine learning models were trained and evaluated to classify tweet sentiment efficiently on a large-scale dataset.
- Natural Language Processing (NLP)
- Text preprocessing techniques
- TF-IDF vectorization
- Machine learning for text classification
- Large-scale data processing with Apache Spark
- Model evaluation and comparison