🌐 DaNetworkCode Streamer Scraper

A high-performance, multi-platform livestream monitoring service.

Actively tracks live status, viewer counts, metadata, and VODs across Kick, Twitch, YouTube, TikTok, and Pump.fun.

---

✨ Key Features

• Multi-Platform Integration: Connects to Kick, Twitch, YouTube, TikTok, and Pump.fun via API and web scraping to build a unified streamer database.
• Real-Time Going-Live Notifications: An intelligent frontend alert system that tracks status transitions between polling cycles and handles immediate popups when an offline streamer goes live.
• Pump.fun Crypto Streaming: Tracks streams tied to Solana mint addresses, complete with intelligent WebSocket/React rendering bypass using Puppeteer to scrape live viewer counts.
• Advanced YouTube Quota Management: Features a built-in admin dashboard (/youtube-dashboard) to track daily/monthly usage, archive history, and prevent API exhaustion. Operates on a strict budget target (default 85%).
• Kick OAuth2 & PKCE: Fully automated Kick token generation, client credentials usage, and refreshing for reliable data extraction.
• Intelligent Puppeteer Fallback: Uses headless browser scraping for platforms that require JS rendering, specifically tuned for low-core VPS environments (auto-restarts to prevent memory leaks).
• Crash-Proof Data: Utilizes better-sqlite3 in WAL mode for persistent, lightning-fast reads and writes.
• Security Hardened: Routes are protected using helmet and express-rate-limit.
• Interactive Frontend: Serves a fully responsive UI displaying real-time live status updates, avatars, premiere badges, and viewer counts.

🔔 Live Notification Architecture

The client-side interface includes a built-in notification stack optimized to alert users without causing announcement fatigue:

• State Persistence: Streamer statuses are cached in localStorage (ip2_streamer_statuses) across page refreshes to eliminate false-alarm notifications when re-opening the application.
• Anti-Spam Cooldowns: Employs an explicit per-streamer cooldown tracker (ip2_notif_cooldown) to ensure a streamer transitioning back and forth rapidly won't flood the UI with alert cards.
• Sequential Animation Queue: Multi-streamer online events are caught in a sliding array stack, allowing notice elements to display and fade smoothly over a configured 7-second lifecycle.

🚀 Installation & Setup

Ensure you have Node.js version 20 or higher installed on your system, or use Docker for a containerized deployment.

Option 1: Docker Compose (Recommended)

# 1. Clone and navigate to the directory
git clone https://github.com/Riotcoke123/danetworkcode.git
cd danetworkcode
2. Configure environment variables (see below)

cp .env.example .env
3. Build and run the container

docker-compose up -d --build

Option 2: Standard Node.js / PM2

# 1. Clone the repository
git clone https://github.com/Riotcoke123/danetworkcode.git
cd danetworkcode
2. Install dependencies

npm install
3. Configure environment variables

cp .env.example .env
4. Start the server

npm start
OR start with PM2 for production deployments

npm run pm2-start

⚙️ Environment Variables (.env)

You will need to create a .env file in the root directory. Below are the required configurations:

# Server Settings
PORT=3000
CHECK_INTERVAL_SECONDS=75
CONCURRENT_LIMIT=4
SCRAPE_CONCURRENCY=6
ADMIN_TOKEN=your_secure_random_hex_string_here

# Database & Session
DB_USERNAME=admin
DB_PASSWORD=your_secure_db_password
SESSION_SECRET=your_long_session_secret_here

# Concurrency Limits (Tune based on your VPS cores/RAM)
PUPPETEER_CONCURRENT_LIMIT=2

# API Base URLs
KICK_AUTH_BASE=https://id.kick.com
KICK_API_BASE=https://api.kick.com
KICK_VOD_BASE=https://vod.kick.com
KICK_WEB_BASE=https://kick.com
TWITCH_AUTH_BASE=https://id.twitch.tv
TWITCH_API_BASE=https://api.twitch.tv/helix
TWITCH_WEB_BASE=https://twitch.tv
YOUTUBE_API_BASE=https://www.googleapis.com

# Streamer Lists (Comma separated)
KICK_USERNAMES=streamer1,streamer2
TWITCH_USERNAMES=streamer1,streamer2
YOUTUBE_USERNAMES=streamer1,streamer2
TIKTOK_USERNAMES=streamer1,streamer2
PUMPFUN_MINTS=21rKrtBzibPAZHAHQRzGiGDSh7XimCKB2a8VgsjZpump,AnotherMint

# API Keys & Auth Setup
YOUTUBE_API_KEY=your_google_api_key
YOUTUBE_QUOTA_DAILY_LIMIT=10000
YOUTUBE_MONTHLY_BUDGET_PERCENT=0.85

TWITCH_CLIENT_ID=your_twitch_client_id
TWITCH_CLIENT_SECRET=your_twitch_client_secret

KICK_CLIENT_ID=your_kick_client_id
KICK_CLIENT_SECRET=your_kick_client_secret
KICK_REDIRECT_URI=http://localhost:3000/auth/kick/callback

🛡️ Admin Dashboard & Endpoints

This project exposes several endpoints. Admin routes require your ADMIN_TOKEN passed as a query parameter (?token=YOUR_TOKEN).

Endpoint	Description	Access
/api/streamers	Returns JSON list of all tracked streamers & their current live status.	Public
/api/stats	Returns aggregated viewer counts and total active streams.	Public
/login/kick	OAuth entry point to authorize the Kick API integration.	Public (Rate Limited)
/youtube-dashboard	Visual HTML dashboard showing YouTube API quota consumption.	Admin Only
/api/youtube/quota	Returns current YouTube quota usage and budget status data.	Admin Only
/api/youtube/audit	Generates JSON audit history for API compliance checks.	Admin Only
/healthz	System health, active scrapers, quota snapshots, and error logs.	Admin Only

---

Developed & Maintained by Riotcoke123. Released under the GNU General Public License v3.0 (GPL-3.0).