diff --git a/analytics/README.md b/analytics/README.md new file mode 100644 index 0000000..6a9ffd4 --- /dev/null +++ b/analytics/README.md @@ -0,0 +1,146 @@ +# Zero-Cost Serverless Analytics for Adventures in Deep Space + +This directory contains a privacy-focused and cookie-less analytics tracking system. It runs entirely on **AWS Lambda (via Function URLs)** and **Amazon DynamoDB**, ensuring that your operating costs are **permanently $0.00** for low-to-moderate traffic. + +--- + +## Architecture Overview + +``` +[ Visitor Browser ] + │ + ├── (POST /event) ──► [ AWS Lambda Function URL ] + │ │ + │ (Writes to) + │ ▼ + │ [ Amazon DynamoDB ] + │ + └── (GET /) ────────► [ Beautiful HTML Dashboard ] +``` + +1. **Tracker (`docs/assets/analytics.js`):** Injected automatically on every page of your site via `common.js`. Tracks pageviews, referrers, constellation anchor clicks, and CSV exports. It does not set cookies or use localStorage. +2. **Lambda Ingestion (`analytics/lambda_function.py`):** Runs on AWS Lambda. It extracts geographic country codes from CloudFront network headers for free, classifies traffic into human vs. bot categories using the User-Agent, hashes IP addresses using a daily-rotated cryptographic salt for privacy-preserving anonymity, and stores data. +3. **Database (Amazon DynamoDB):** Stores pageviews and event records. It has no open ports, is secured via IAM, and fits entirely in the perpetual AWS Free Tier. +4. **Dashboard:** Served directly from your Lambda URL when accessed in a browser. It presents human-only metrics, geographical breakdowns, top pages, referrer sources, constellation click counts, and bot traffic breakdowns in a sleek glassmorphic UI. + +--- + +## Deployment Steps + +Follow these steps to deploy your serverless analytics backend in 10 minutes: + +### Step 1: Create the DynamoDB Table +1. Open the [Amazon DynamoDB Console](https://console.aws.amazon.com/dynamodb/). +2. Click **Create table**. +3. Configure the table settings: + * **Table name:** `ads_analytics` + * **Partition key:** `PK` (Type: `String`) + * **Sort key:** `SK` (Type: `String`) +4. Under **Table settings**, select **Customize settings**: + * **Read/write capacity settings:** Choose **Provisioned**. + * Set **Read capacity units (RCU)** to `5`. + * Set **Write capacity units (WCU)** to `5`. + * Turn **Auto-scaling** **OFF** (this keeps you inside the perpetual Free Tier and caps any cost at exactly $0, since DynamoDB will throttle excess requests if spammed). +5. Click **Create table**. + +--- + +### Step 2: Create the AWS Lambda Function +1. Open the [AWS Lambda Console](https://console.aws.amazon.com/lambda/). +2. Click **Create function**. +3. Select **Author from scratch**: + * **Function name:** `ads-analytics-tracker` + * **Runtime:** `Python 3.12` (or current Python 3.x version) + * **Architecture:** `x86_64` +4. Click **Create function**. +5. Once created, in the **Code** tab, replace the boilerplate code in `lambda_function.py` with the complete contents of the [lambda_function.py](file:///home/akarsh/devel/adventures.github.io/analytics/lambda_function.py) file in this directory. +6. Click **Deploy** at the top of the code editor. + +--- + +### Step 3: Enable the Function URL (Perpetually Free Endpoint) +1. In the Lambda function page, go to the **Configuration** tab. +2. Select **Function URL** in the left sidebar and click **Create Function URL**. +3. Configure the Function URL: + * **Auth type:** `NONE` (so public web browsers can send events). + * **CORS:** Leave *disabled* or set to defaults (since our Python code handles CORS headers dynamically inside the Lambda handler to restrict access to `adventuresindeepspace.com` and `localhost`). +4. Click **Save**. +5. Copy the newly generated **Function URL** from the top right of the page (it will look like `https://xxxxxxxxx.lambda-url.us-east-1.on.aws/`). + +--- + +### Step 4: Configure IAM Database Permissions +By default, the Lambda function does not have permission to write to your DynamoDB database. Let's authorize it: +1. In the Lambda function page, go to the **Configuration** tab. +2. Select **Permissions** in the left sidebar. +3. Under **Execution role**, click on the blue link representing your Role name (e.g., `ads-analytics-tracker-role-xxxx`). This opens the IAM Console. +4. In the IAM Role page, click **Add permissions** -> **Create inline policy**. +5. Go to the **JSON** tab and paste the following policy: + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "DynamoDBReadWriteAccess", + "Effect": "Allow", + "Action": [ + "dynamodb:PutItem", + "dynamodb:Query" + ], + "Resource": "arn:aws:dynamodb:*:*:table/ads_analytics" + } + ] + } + ``` +6. Click **Review policy**. +7. Name the policy `DynamoDBAnalyticsAccess` and click **Create policy**. +8. Close the IAM console tab. + +--- + +### Step 5: Configure Environment Variables +1. Back in the Lambda function page, go to the **Configuration** tab. +2. Select **Environment variables** in the left sidebar and click **Edit**. +3. Add the following variables: + * **Key:** `DYNAMODB_TABLE` | **Value:** `ads_analytics` + * **Key:** `ANALYTICS_SECRET` | **Value:** (Type a random 32-character string. This key is used as a cryptographic salt for hashing IP addresses. *Keep this private!*) +4. Click **Save**. +5. Increase the function timeout slightly by going to **General configuration** -> **Edit** -> Change **Timeout** to `5 seconds` -> Click **Save**. + +--- + +### Step 6: Update the Client Tracker Endpoint +1. Open the file [analytics.js](file:///home/akarsh/devel/adventures.github.io/docs/assets/analytics.js) in your text editor. +2. Replace the placeholder endpoint at line 8: + ```javascript + const DEFAULT_ENDPOINT = 'https://analytics.your-personal-server.com/api/event'; + ``` + with your actual AWS Lambda Function URL: + ```javascript + const DEFAULT_ENDPOINT = 'https://xxxxxxxxx.lambda-url.us-east-1.on.aws/'; + ``` + +--- + +### Step 7: Activate the Tracker on Your Site +1. Open the file [common.js](file:///home/akarsh/devel/adventures.github.io/docs/assets/common.js) in your editor. +2. Find the commented-out dynamic import at the end of the script inclusions block: + ```javascript + // To enable analytics tracking once your AWS backend is deployed, uncomment the line below: + // import('./analytics.js').catch(err => console.warn('Analytics failed to load:', err)); + ``` +3. Uncomment the import line: + ```javascript + // To enable analytics tracking once your AWS backend is deployed, uncomment the line below: + import('./analytics.js').catch(err => console.warn('Analytics failed to load:', err)); + ``` +4. Commit and push the changes to GitHub. Your website will now dynamically load the tracker and start recording analytics! + +--- + +## Accessing Your Dashboard + +Your HTML analytics dashboard is served directly from the Lambda function URL: +1. Open your web browser. +2. Navigate to your Lambda Function URL: `https://xxxxxxxxx.lambda-url.us-east-1.on.aws/` +3. You will immediately see your live traffic, top pages, referrers, country origins, constellation click-throughs, and AI crawler statistics! diff --git a/analytics/lambda_function.py b/analytics/lambda_function.py new file mode 100644 index 0000000..0d3ee4c --- /dev/null +++ b/analytics/lambda_function.py @@ -0,0 +1,561 @@ +import json +import os +import hashlib +import hmac +import datetime +import uuid +import boto3 +from boto3.dynamodb.conditions import Key + +# Configuration +ALLOWED_ORIGIN = "https://adventuresindeepspace.com" +DEVELOPMENT_ORIGIN = "http://localhost:4000" # for local testing if needed +TABLE_NAME = os.environ.get("DYNAMODB_TABLE", "ads_analytics") +SECRET_KEY = os.environ.get("ANALYTICS_SECRET", "default_secret_please_change") + +dynamodb = boto3.resource("dynamodb") +table = dynamodb.Table(TABLE_NAME) + +def get_header(headers, name): + # Case-insensitive helper to retrieve headers + name_lower = name.lower() + for k, v in headers.items(): + if k.lower() == name_lower: + return v + return "" + +def get_cors_headers(origin): + allowed = [ALLOWED_ORIGIN, DEVELOPMENT_ORIGIN] + origin_to_return = ALLOWED_ORIGIN + + if origin in allowed: + origin_to_return = origin + + return { + "Access-Control-Allow-Origin": origin_to_return, + "Access-Control-Allow-Methods": "POST, GET, OPTIONS", + "Access-Control-Allow-Headers": "Content-Type, X-Requested-With", + "Access-Control-Max-Age": "86400", + "Vary": "Origin" + } + +def get_visitor_hash(ip, user_agent, secret, date_str): + daily_salt = hmac.new(secret.encode(), date_str.encode(), hashlib.sha256).hexdigest() + # Visitor Hash = SHA256(IP + UserAgent + Daily Salt) + raw_str = f"{ip}|{user_agent}|{daily_salt}" + return hashlib.sha256(raw_str.encode()).hexdigest() + +def classify_referrer(referrer): + if not referrer: + return "Direct" + ref_lower = referrer.lower() + + # Search engines + if any(x in ref_lower for x in ["google.com", "bing.com", "yahoo.com", "duckduckgo.com", "baidu.com", "yandex"]): + return "Search Engine" + # AI Tools + if any(x in ref_lower for x in ["chatgpt.com", "openai.com", "claude.ai", "anthropic.com", "gemini.google.com", "perplexity.ai"]): + return "AI Tool" + # Social Media + if any(x in ref_lower for x in ["facebook.com", "t.co", "twitter.com", "x.com", "reddit.com", "instagram.com", "linkedin.com"]): + return "Social Media" + + try: + domain = referrer.split("//")[1].split("/")[0] + return domain + except Exception: + return "Other" + +def classify_visitor_type(user_agent): + ua_lower = user_agent.lower() + + # 1. AI Agents + ai_agents = [ + "gptbot", "chatgpt-user", "claudebot", "claude-web", "anthropic", + "google-extended", "perplexitybot", "imagesiftbot", "cohere-ai", + "omgilibot", "facebookexternalhit", "bytespider", "diffbot" + ] + if any(bot in ua_lower for bot in ai_agents): + return "AI Bot" + + # 2. Search Engine Crawlers + search_crawlers = [ + "googlebot", "bingbot", "yandexbot", "baiduspider", "duckduckbot", + "slurp", "sogou", "ia_archiver" + ] + if any(crawler in ua_lower for crawler in search_crawlers): + return "Search Bot" + + # 3. Generic Crawlers / Scrapers / Headless Browsers + generic_bots = [ + "headless", "selenium", "playwright", "puppeteer", "phantomjs", + "scrapy", "curl", "wget", "python-requests", "http-client", + "bot", "spider", "crawler", "scrape" + ] + if any(bot in ua_lower for bot in generic_bots): + return "Generic Bot" + + return "Human" + +def handle_options(event): + origin = get_header(event.get("headers", {}), "origin") + return { + "statusCode": 204, + "headers": get_cors_headers(origin), + "body": "" + } + +def handle_post(event): + headers = event.get("headers", {}) + origin = get_header(headers, "origin") + + # Strict Origin / Referer check + allowed_origins = [ALLOWED_ORIGIN, DEVELOPMENT_ORIGIN] + if origin not in allowed_origins: + referer = get_header(headers, "referer") + if not any(x in referer for x in allowed_origins): + return { + "statusCode": 403, + "headers": {"Content-Type": "application/json"}, + "body": json.dumps({"error": "Forbidden Origin"}) + } + + try: + body = json.loads(event.get("body", "{}")) + except Exception: + return { + "statusCode": 400, + "body": json.dumps({"error": "Invalid JSON"}) + } + + event_type = body.get("event", "pageview") + path = body.get("path", "/") + raw_referrer = body.get("referrer", "") + details = body.get("details", "") + + # Get request IP + ip = event.get("requestContext", {}).get("http", {}).get("sourceIp", "0.0.0.0") + user_agent = get_header(headers, "user-agent") or "unknown" + + # Country detection from CloudFront headers passed to Function URL + country_code = get_header(headers, "cloudfront-viewer-country") or "Unknown" + + # Classify Visitor Type (Human, AI Bot, Search Bot, Generic Bot) + visitor_type = classify_visitor_type(user_agent) + + # Unique ID and date + now = datetime.datetime.utcnow() + date_str = now.strftime("%Y-%m-%d") + timestamp_str = now.isoformat() + "Z" + unique_id = str(uuid.uuid4()) + + # hash calculation + visitor_hash = get_visitor_hash(ip, user_agent, SECRET_KEY, date_str) + + # Save to DynamoDB + if event_type == "pageview": + referrer_type = classify_referrer(raw_referrer) + table.put_item( + Item={ + "PK": f"PAGEVIEW#{date_str}", + "SK": f"TIME#{timestamp_str}#{unique_id}", + "path": path, + "referrer": raw_referrer or "Direct", + "referrer_type": referrer_type, + "country": country_code, + "visitor_hash": visitor_hash, + "visitor_type": visitor_type, + "timestamp": timestamp_str + } + ) + else: + table.put_item( + Item={ + "PK": f"EVENT#{date_str}#{event_type}", + "SK": f"TIME#{timestamp_str}#{unique_id}", + "path": path, + "details": details, + "visitor_hash": visitor_hash, + "visitor_type": visitor_type, + "timestamp": timestamp_str + } + ) + + return { + "statusCode": 200, + "headers": get_cors_headers(origin), + "body": json.dumps({"status": "success"}) + } + +def handle_get_dashboard(event): + # Query last 7 days of data for the dashboard + today = datetime.date.today() + dates = [(today - datetime.timedelta(days=i)).strftime("%Y-%m-%d") for i in range(7)] + + pageviews = [] + events_data = [] + + for d in dates: + try: + # Query Pageviews + resp = table.query( + KeyConditionExpression=Key("PK").eq(f"PAGEVIEW#{d}") + ) + pageviews.extend(resp.get("Items", [])) + + # Query CSV Export Events + resp_csv = table.query( + KeyConditionExpression=Key("PK").eq(f"EVENT#{d}#csv_export") + ) + events_data.extend(resp_csv.get("Items", [])) + + # Query Anchor Clicks + resp_anchor = table.query( + KeyConditionExpression=Key("PK").eq(f"EVENT#{d}#anchor_click") + ) + events_data.extend(resp_anchor.get("Items", [])) + except Exception as e: + print(f"Error querying date {d}: {e}") + + # Process aggregates (Human Only vs All Traffic) + human_views = [pv for pv in pageviews if pv.get("visitor_type", "Human") == "Human"] + total_human_views = len(human_views) + unique_human_visitors = len(set(x["visitor_hash"] for x in human_views)) + + total_raw_views = len(pageviews) + unique_raw_visitors = len(set(x["visitor_hash"] for x in pageviews)) + + # Visitor type breakdown counts + visitor_types = {"Human": 0, "AI Bot": 0, "Search Bot": 0, "Generic Bot": 0} + for pv in pageviews: + vt = pv.get("visitor_type", "Human") + visitor_types[vt] = visitor_types.get(vt, 0) + 1 + + # Referrer breakdown (Human Only) + referrers = {} + for pv in human_views: + ref_type = pv.get("referrer_type", "Direct") + referrers[ref_type] = referrers.get(ref_type, 0) + 1 + + # Country breakdown (Human Only) + countries = {} + for pv in human_views: + c = pv.get("country", "Unknown") + countries[c] = countries.get(c, 0) + 1 + + # Top Pages breakdown (Human Only) + pages = {} + for pv in human_views: + p = pv.get("path", "/") + pages[p] = pages.get(p, 0) + 1 + + # CSV Exports (Human Only) + csv_count = sum(1 for e in events_data if "csv_export" in e.get("PK", "") and e.get("visitor_type", "Human") == "Human") + + # Top clicked anchors (Human Only) + anchors = {} + for e in events_data: + if "anchor_click" in e.get("PK", "") and e.get("visitor_type", "Human") == "Human": + anc = e.get("details", "unknown") + anchors[anc] = anchors.get(anc, 0) + 1 + + summary = { + "total_human_views": total_human_views, + "unique_human_visitors": unique_human_visitors, + "total_raw_views": total_raw_views, + "unique_raw_visitors": unique_raw_visitors, + "visitor_types": visitor_types, + "referrers": referrers, + "countries": countries, + "pages": pages, + "csv_exports": csv_count, + "anchors": anchors + } + + # Return HTML Dashboard + html_content = get_dashboard_html(summary) + return { + "statusCode": 200, + "headers": { + "Content-Type": "text/html", + "Cache-Control": "no-cache, no-store, must-revalidate" + }, + "body": html_content + } + +def get_dashboard_html(data): + # Generate bot list details for table + bot_rows = "".join(f"{bot_type}{count}" + for bot_type, count in data["visitor_types"].items() if bot_type != "Human") + + # Render a premium, beautiful dashboard with glassmorphism and modern colors + return f""" + + + Adventures in Deep Space - Analytics + + + + + +
+
+
+

Analytics Dashboard

+

Adventures in Deep Space • Serverless (Last 7 Days)

+
+
+ + Active Tracking +
+
+ +
+
+
Unique Human Visitors
+
{data["unique_human_visitors"]}
+
+
+
Human Pageviews
+
{data["total_human_views"]}
+
+
+
CSV Exports (Humans)
+
{data["csv_exports"]}
+
+
+
Total Bot Traffic
+
{data["total_raw_views"] - data["total_human_views"]}
+
+
+ +
+
+
Top Visited Pages (Human Only)
+ + + + + + {"".join(f"" for path, count in sorted(data["pages"].items(), key=lambda x: x[1], reverse=True)[:10]) or ""} + +
PathViews
{path}{count}
No human pageviews yet
+
+ +
+
Referrer Channels (Human Only)
+ + + + + + {"".join(f"" for ref, count in sorted(data["referrers"].items(), key=lambda x: x[1], reverse=True)) or ""} + +
SourceViews
{ref}{count}
No referrers recorded
+
+
+ +
+
+
Geography (Countries - Human Only)
+ + + + + + {"".join(f"" for country, count in sorted(data["countries"].items(), key=lambda x: x[1], reverse=True)) or ""} + +
CountryVisits
🌍 {country}{count}
No geographics recorded
+
+ +
+
Top Constellations / Anchors Clicked
+ + + + + + {"".join(f"" for anchor, count in sorted(data["anchors"].items(), key=lambda x: x[1], reverse=True)[:10]) or ""} + +
Anchor NameClicks
#{anchor}{count}
No anchors clicked yet
+
+
+ +
+
+
Crawler & Scraping Bot Breakdown
+ + + + + + {bot_rows} + + + + + +
Bot TypeRequests Intercepted
Total Bot Requests{data["total_raw_views"] - data["total_human_views"]}
+
+
+
+ + +""" + +def lambda_handler(event, context): + method = event.get("requestContext", {}).get("http", {}).get("method", "GET") + + if method == "OPTIONS": + return handle_options(event) + elif method == "POST": + return handle_post(event) + elif method == "GET": + return handle_get_dashboard(event) + + return { + "statusCode": 405, + "body": json.dumps({"error": "Method Not Allowed"}) + } diff --git a/docs/assets/analytics.js b/docs/assets/analytics.js new file mode 100644 index 0000000..665bf89 --- /dev/null +++ b/docs/assets/analytics.js @@ -0,0 +1,102 @@ +/** + * Adventures in Deep Space - Privacy-Focused Analytics Tracker + * Cookie-less and Privacy-Respecting. + */ +(function () { + // Configurable endpoint. The user should replace this with their own API/webhook URL. + // Can also be overridden by setting window.ADS_ANALYTICS_ENDPOINT before common.js loads. + const DEFAULT_ENDPOINT = 'https://analytics.your-personal-server.com/api/event'; + const endpoint = window.ADS_ANALYTICS_ENDPOINT || DEFAULT_ENDPOINT; + + // Helper to send data to the backend + function sendPayload(payload) { + // Skip tracking if running locally or tracking is disabled + if (window.location.hostname === 'localhost' || window.location.hostname === '127.0.0.1') { + console.log('[Analytics-Dev]', payload); + return; + } + + try { + const data = JSON.stringify(payload); + if (navigator.sendBeacon) { + navigator.sendBeacon(endpoint, data); + } else { + fetch(endpoint, { + method: 'POST', + body: data, + headers: { 'Content-Type': 'application/json' }, + keepalive: true + }).catch(err => console.warn('Analytics send failed:', err)); + } + } catch (e) { + console.warn('Analytics error:', e); + } + } + + // Helper to track pageview + function trackPageView() { + sendPayload({ + event: 'pageview', + path: window.location.pathname, + referrer: document.referrer || '', + title: document.title + }); + } + + // Helper to track anchor/constellation clicks + function trackAnchorClick(hash) { + if (!hash) return; + sendPayload({ + event: 'anchor_click', + path: window.location.pathname, + referrer: '', // Referrer is empty for internal anchor clicks + details: hash + }); + } + + // --- 1. Track Page View on Load --- + if (document.readyState === 'complete') { + trackPageView(); + } else { + window.addEventListener('load', trackPageView); + } + + // --- 2. Track Initial Hash (if user navigated directly to an anchor) --- + window.addEventListener('load', () => { + if (window.location.hash) { + // Wait slightly to ensure page is loaded + setTimeout(() => { + trackAnchorClick(window.location.hash.substring(1)); + }, 500); + } + }); + + // --- 3. Track Hash Changes --- + window.addEventListener('hashchange', () => { + if (window.location.hash) { + trackAnchorClick(window.location.hash.substring(1)); + } + }); + + // --- 4. Intercept Clicks on Anchor Links --- + document.addEventListener('click', (e) => { + const link = e.target.closest('a'); + if (link) { + const href = link.getAttribute('href'); + if (href && href.startsWith('#')) { + const hash = href.substring(1); + trackAnchorClick(hash); + } + } + }); + + // --- 5. Intercept CSV Export Custom Event --- + document.addEventListener('ads_csv_export', (e) => { + sendPayload({ + event: 'csv_export', + path: e.detail.path || window.location.pathname, + referrer: '', + details: 'CSV Download' + }); + }); +})(); diff --git a/docs/assets/common.js b/docs/assets/common.js index 1871301..f792f62 100644 --- a/docs/assets/common.js +++ b/docs/assets/common.js @@ -76,6 +76,8 @@ customElements.define("x-dso-link", XDsoLinkElement); script.src = 'assets/csv_maker.js'; document.head.appendChild(script); } +// To enable analytics tracking once your AWS backend is deployed, uncomment the line below: +// import('./analytics.js').catch(err => console.warn('Analytics failed to load:', err)); // Font Awesome v4.7.0 { let link = document.createElement('link'); diff --git a/docs/assets/csv_maker.js b/docs/assets/csv_maker.js index 451c4ca..40af228 100644 --- a/docs/assets/csv_maker.js +++ b/docs/assets/csv_maker.js @@ -108,9 +108,10 @@ window.onload = function() { button.className = 'floating'; button.title = `Download table containing SIMBAD data on the ${N} underlined objects on this page`; - // Style the button // Add an event listener to the button (optional) button.addEventListener('click', async () => { + // Dispatch custom event for tracking + document.dispatchEvent(new CustomEvent('ads_csv_export', { detail: { path: window.location.pathname } })); const prev_src = icon.src; icon.src = 'assets/loading.gif'; await generateCSV();