Skip to content

StickKB/Instagram-Reel-Heatmap

BranchesTags

Repository files navigation

Reel Heatmap — Instagram Reels Consumption Tracker

A local-only, personal-use tool that counts how many Instagram Reels you watched on any given day by reading Instagram's built-in Watch History screen via Android UI automation. It stores daily totals in a local SQLite database and visualises them as an interactive, GitHub-style heatmap dashboard served on your machine.

⚠️ Disclaimer

This is an unofficial, personal-use tool and is not affiliated with, endorsed by, or connected to Instagram or Meta in any way. Automating the Instagram app may violate Instagram's Terms of Use. Use at your own risk — the authors take no responsibility for any action taken against your account. The tool is strictly read-only (it only reads Watch History; it never likes, follows, posts, comments, or sends DMs), which minimises risk, but risk is not zero.

Status

Phase What it covers Status
Phase A Storage layer (SQLite + JSONL), dashboard (serve), CLI (run, backfill, report, export, doctor), test suite Working — no emulator required for the dashboard; use seed-demo or --mock-count for dev/demo.
Phase B Live Android emulator scraper (drives Instagram via uiautomator2) Working — implemented and verified end-to-end. run/backfill start the emulator automatically, navigate to Watch History, apply a single-day date filter, and count reels via perceptual-hash de-duplication.

How It Works — Architecture

Android emulator (AVD)
  └─ uiautomator2 → instagram_nav.py → watch_history.py
                                              │
                                              ▼
                                    ScrapeResult (models.py)
                                              │
                              ┌───────────────┴────────────────┐
                              ▼                                ▼
                    SQLite (canonical)              JSONL audit trail
                    data/reel_tracker.sqlite        data/*.jsonl
                              │
                    ┌─────────┴──────────┐
                    ▼                    ▼
              Dashboard             CSV / JSON
              (serve)               (exports/)

The scraper drives a pre-configured Android emulator, opens the Instagram Watch History screen, applies a single-day date filter, and scrolls through the grid counting reels via perceptual-hash de-duplication. Counts come from the in-app Watch History — the authoritative record with ~30-day retention. Instagram's "Download Your Information" export is intentionally not used because it proved incomplete and under-reports consumption.

SQLite is the canonical source of truth. JSONL files are append-only event logs (audit trail; never modified). CSV and JSON exports are derived from SQLite and regenerated after every successful run.

For domain terminology see CONTEXT.md. For design decisions see docs/adr/.

Limitations

  • Today can't be counted. Instagram's date filter is end-exclusive and its picker won't select tomorrow, so a day can only be counted once it's complete. Run for yesterday (the default) — that's the earliest countable day.
  • 30-day retention window. Watch History only retains ~30 days of data. Run daily (or use backfill over a range) to capture history before it ages out. Once a count is in SQLite, it's kept forever.
  • Heavy usage means slower runs. The scraper sees ~9 reels per scroll, so hundreds of reels in a day means a multi-minute scrape. If runs consistently come back at "medium" confidence (cap hit), raise MAX_SCROLLS in .env.
  • UI-automation fragility. Selectors target Instagram's current Android UI and may need updating when the app changes. Pinning the Instagram APK version can help.

Privacy

By default this project stores no personally identifying information about the content you watched:

  • No author handles, Reel captions, URLs, or screenshots are ever written to disk.
  • Each Watch Entry is reduced to a non-reversible, salted fingerprint (SHA-256 over handle + timestamp + caption snippet, or a perceptual image hash). The fingerprint detects if the same card appears twice while scrolling but cannot be reversed into the original content.
  • The STORE_CARD_METADATA flag defaults to false. Changing it to true is opt-in and requires a deliberate edit to .env.

What stays on your machine and never gets committed:

File / dir Role
data/reel_tracker.sqlite Canonical source of truth — daily counts + observation fingerprints.
data/run_log.jsonl Append-only audit trail of every run (start, success/failure, metadata).
data/daily_counts_history.jsonl Append-only event log for daily counts.
data/reel_observations.jsonl Append-only event log for individual fingerprints.
exports/*.csv, exports/*.json Convenience exports derived from SQLite; regenerated each run.
debug_runs/ Screenshots and hierarchy dumps captured on scraper failure.
.env Your local config (AVD name, timezone, paths). Never committed.

All of the above are gitignored. If you use Instagram's "Download Your Information" feature for comparison purposes, keep those files outside the repo directory.

Setup

Prerequisites: Python 3.11+, git.

Primary / tested platform: macOS.

# 1. Clone and enter the project
git clone <repo-url> Reel-Heatmap
cd Reel-Heatmap

# 2. Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# 3. Install dependencies (includes this package in editable mode)
pip install -r requirements.txt

# 4. Configure environment
cp .env.example .env
# Edit .env to set your LOCAL_TIMEZONE and, for Phase B, ANDROID_HOME and AVD_NAME.

Quick Start — Dashboard (no emulator required)

# Confirm the environment looks good
python -m ig_reel_tracker doctor

# Populate 60 days of sample data so the dashboard has something to show
# (DEV ONLY — uses random fake counts, not real data)
python -m ig_reel_tracker seed-demo --days 60

# Open the heatmap dashboard in your browser (http://127.0.0.1:8787)
python -m ig_reel_tracker serve

# Print the last 30 days of counts in the terminal
python -m ig_reel_tracker report --last 30

# Write convenience CSV/JSON exports to ./exports/
python -m ig_reel_tracker export

Phase B Setup — Android Emulator (real data)

The run command drives a real Android emulator to read your Watch History. Set this up once:

  1. Install Android Studio (includes the Android SDK): https://developer.android.com/studio

  2. Add tools to your PATH. Set ANDROID_HOME (the SDK root) and add both platform-tools and emulator directories to your shell's PATH, then reload your shell:

    # Example for zsh — add to ~/.zshrc
export ANDROID_HOME="$HOME/Library/Android/sdk"
export PATH="$ANDROID_HOME/platform-tools:$ANDROID_HOME/emulator:$PATH"

  3. Create a Google Play-enabled AVD. Open Android Studio → Device Manager → Create Virtual Device. Choose a Pixel model (e.g. Pixel 8) with a Google Play system image (API 34 recommended — labelled "Google Play", not "Google APIs"). Name it Pixel_8_API_34 (or update AVD_NAME in .env).

  4. Start the emulator once and log in to Instagram manually:

    emulator -avd Pixel_8_API_34

    Inside the running emulator: open the Play Store, install Instagram, launch it, and sign in with your account. Navigate to Profile → Menu (≡) → Your activity → Watch history to confirm it exists.

  5. Close the emulator. The login session is now saved inside the AVD's virtual disk.

Important — never wipe the AVD. The login session persists in the AVD's app data across automated runs. Wiping the AVD (e.g. emulator -avd ... -wipe-data) destroys the session and requires a manual re-login. The scraper never passes -wipe-data. See docs/adr/0001-emulator-over-real-device.md.

After setup, python -m ig_reel_tracker doctor should show the AVD as available. run (without --mock-count) will start the emulator automatically, scrape Watch History, and shut it down when done (controlled by SHUTDOWN_EMULATOR_AFTER_RUN).

CLI Reference

doctor

python -m ig_reel_tracker doctor

Checks the local environment: Python version, database file, data directories, adb/emulator availability (if ANDROID_HOME is set), and the configured AVD.

run

python -m ig_reel_tracker run [--date DATE] [--visible | --headless]
                               [--open-dashboard] [--mock-count N]
Flag Description
--date DATE today, yesterday, or YYYY-MM-DD (default: yesterday)
--visible Force the emulator to run with a window
--headless Force the emulator to run headless
--open-dashboard Start the dashboard after the run completes
--mock-count N DEV ONLY: skip the emulator and store N fake reels — exercises the storage pipeline without requiring an AVD

Note: --date today is rejected — see Limitations. Real data requires the emulator; omit --mock-count for a live scrape.

backfill

python -m ig_reel_tracker backfill --start YYYY-MM-DD --end YYYY-MM-DD [--mock-count N]

Counts every date in the range in order. Dates older than the Watch History retention window (~30 days) are skipped with a warning. Today is also skipped (for the same reason as run). --mock-count is DEV ONLY.

report

python -m ig_reel_tracker report [--last N]

Prints a table of the most recent N daily counts (default: 30) from the local database.

export

python -m ig_reel_tracker export

Writes exports/daily_counts.csv and exports/daily_counts.json derived from the SQLite database.

serve

python -m ig_reel_tracker serve [--host HOST] [--port PORT] [--no-open]

Starts the local dashboard at http://127.0.0.1:8787 by default. The --no-open flag suppresses automatic browser launch.

Security note: the dashboard binds to 127.0.0.1 only (localhost). If you override WEB_HOST in .env to a non-loopback address, anyone on your network can reach it — there is no authentication.

seed-demo (DEV ONLY)

python -m ig_reel_tracker seed-demo [--days N]

Populates N days of plausible random daily counts (default: 60). Useful for trying the dashboard before any real data exists. This writes fake counts — not real Instagram data.

Dashboard

serve starts a Flask server that listens only on 127.0.0.1:8787 (never on a public interface). Open http://127.0.0.1:8787 in your browser.

The dashboard shows:

  • Heatmap — a calendar grid coloured by daily Reel count (darker = more watched).
  • Summary cards — total, average, peak, and most recent counts.
  • Runs table — history of scraper runs with status, confidence, and source.
  • Trend chart — rolling view of daily counts over time.
  • Export links — download the CSV/JSON exports directly from the browser.

If the default port is occupied: python -m ig_reel_tracker serve --port 8788.

Project Structure

Reel-Heatmap/
├── .env.example                    # Copy to .env and fill in
├── CONTEXT.md                      # Domain glossary (Watch Entry, Fingerprint, etc.)
├── LICENSE
├── pyproject.toml
├── requirements.txt
│
├── src/ig_reel_tracker/
│   ├── __init__.py
│   ├── __main__.py                 # `python -m ig_reel_tracker` entry point
│   ├── append_log.py               # JSONL append helpers
│   ├── cli.py                      # CLI parser and run orchestration
│   ├── config.py                   # Config dataclass loaded from .env
│   ├── date_utils.py               # Target-date resolution, timezone helpers
│   ├── diagnostics.py              # `doctor` command checks
│   ├── emulator.py                 # AVD lifecycle manager
│   ├── errors.py                   # Project-specific exception types
│   ├── exports.py                  # CSV/JSON export writer
│   ├── fingerprint.py              # Salted SHA-256 + perceptual dHash
│   ├── instagram_nav.py            # AndroidDevice + navigation + date filter
│   ├── models.py                   # Observation, ScrapeResult dataclasses
│   ├── storage.py                  # SQLite schema + query helpers
│   ├── watch_history.py            # scrape_watch_history + counter
│   ├── web.py                      # Flask dashboard server
│   ├── web_static/
│   │   ├── app.js
│   │   └── styles.css
│   └── web_templates/
│       └── index.html
│
├── tests/
│   ├── test_append_log.py
│   ├── test_date_utils.py
│   ├── test_fingerprint.py
│   ├── test_storage.py
│   └── test_web_api.py
│
├── data/                           # Created on first run (gitignored)
│   ├── reel_tracker.sqlite
│   ├── run_log.jsonl
│   ├── daily_counts_history.jsonl
│   └── reel_observations.jsonl
│
├── exports/                        # CSV/JSON exports (gitignored)
├── debug_runs/                     # Screenshots/hierarchy dumps on failure (gitignored)
└── docs/
    └── adr/
        ├── 0001-emulator-over-real-device.md
        └── 0002-total-watch-events-over-distinct-reels.md

Tests

pytest

The test suite covers storage, fingerprinting, date utilities, the JSONL append log, and the web API. All tests run without an emulator.

Configuration Reference

Key variables in .env (see .env.example for the full list):

Variable Default Description
AVD_NAME Pixel_8_API_34 Name of the Android Virtual Device to use.
ANDROID_HOME (none) Path to the Android SDK root directory. Required for Phase B.
LOCAL_TIMEZONE Australia/Sydney IANA timezone for resolving today/yesterday.
DB_PATH ./data/reel_tracker.sqlite SQLite database path.
HEADLESS false Run emulator headless by default.
MAX_SCROLLS 150 Safety cap on scrolls per run (~9 reels/scroll). Raise for heavy usage.
SCROLL_PAUSE_SECONDS 0.7 Pause between scrolls; reduce only if the device is consistently fast.
SHUTDOWN_EMULATOR_AFTER_RUN true Shut down the emulator after a run (only if this run started it).
FINGERPRINT_SALT (blank) Salt for one-way fingerprints. Keep stable; changing it breaks comparisons.
WEB_HOST 127.0.0.1 Dashboard bind host. Do not change unless you understand the security implications.
WEB_PORT 8787 Dashboard port.
STORE_CARD_METADATA false Opt-in to storing handles/captions. Keep false for privacy.

Troubleshooting

Problem Fix
Address already in use when running serve Use a different port: serve --port 8788
Instagram shows a login screen during a run Open the emulator manually (emulator -avd <name>), log in, then rerun.
Watch History not found in the app Ensure Instagram is updated to the latest version from the Play Store; confirm the menu path exists: Profile → ≡ → Your activity → Watch history.
Run returns "medium" confidence The scroll cap was hit before reaching end-of-list. Raise MAX_SCROLLS in .env.
adb: command not found Add $ANDROID_HOME/platform-tools to your PATH and reload your shell.
emulator: command not found Add $ANDROID_HOME/emulator to your PATH and reload your shell.
ANDROID_HOME is not set Add export ANDROID_HOME="$HOME/Library/Android/sdk" (or your SDK root) to your shell profile.

Contributing

Issues and PRs are welcome. This is a personal project maintained on a best-effort basis, so response times may vary. When filing a bug, include the output of python -m ig_reel_tracker doctor and the relevant lines from data/run_log.jsonl (redact anything personal).

License

MIT — see LICENSE.

About

Local python script + Android simulator to scrape watch history and show number of reels consumed in a heatmap format

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages