Instagram Video Downloader Lambda

A serverless AWS Lambda function that downloads Instagram videos and stores them in S3. The project uses Supabase for metadata management and AWS SQS for message queuing.

Features

• Downloads Instagram videos at specified resolutions
• Stores videos in AWS S3
• Uses Supabase for metadata management
• Implements AWS SQS for message queuing
• Includes CloudWatch metrics for monitoring
• Handles retries and error cases
• Supports batch processing of videos

Prerequisites

• AWS Account with appropriate permissions
• Supabase account and project
• Python 3.10 or higher
• AWS CLI configured with appropriate credentials

Environment Variables

Create a .env file with the following variables:

SUPABASE_URL=your_supabase_url
SUPABASE_KEY=your_supabase_key
S3_BUCKET=your_s3_bucket_name
MAX_RETRIES=3
DOWNLOAD_TIMEOUT=300
CHUNK_SIZE=1048576

Installation

1. Clone the repository:

git clone <repository-url>
cd InstagramVideoDownloader-lambda

2. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

Project Structure

• lambda_function.py: Main Lambda function code
• trigger_sqs_batches.py: Script to trigger SQS message batches
• scripts/pack_lambda.sh: Script to package Lambda function
• requirements.txt: Python dependencies
• .env: Environment variables (not committed to git)

Usage

Deploying the Lambda Function

1. Package the Lambda function:

./scripts/pack_lambda.sh

2. Upload the lambda_function.zip to AWS Lambda

Triggering Video Downloads

Run the batch processor to send videos to the queue:

python trigger_sqs_batches.py

CloudWatch Metrics

The following metrics are available in CloudWatch under the namespace 'InstagramCollector':

• ProcessingStarted: Number of videos processing started
• ProcessingSuccess: Number of successfully processed videos
• ProcessingFailed: Number of failed video processing attempts
• DownloadTime: Time taken to download videos
• UploadTime: Time taken to upload videos to S3
• TotalProcessingTime: Total processing time per video
• SupabaseVideoFound: Number of videos found in Supabase
• SupabaseVideoNotFound: Number of videos not found in Supabase
• SupabaseError: Number of Supabase errors
• DownloadTimeout: Number of download timeouts
• DownloadError: Number of download errors
• S3UploadError: Number of S3 upload errors
• InvalidInput: Number of invalid input messages

Error Handling

The system implements comprehensive error handling:

• Retries for failed downloads
• Error logging to CloudWatch
• Metrics tracking for various failure scenarios
• Graceful cleanup of temporary files

Contributing

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

Support

For issues and feature requests, please create an issue in the repository.

---

Maintained by Oriane XYZ