Skip to content

Commit 848c384

Browse files
devilscrapesdevilscrapes
committed
GitHub user scraper for JavaScript and Node.js
0 parents  commit 848c384

11 files changed

Lines changed: 1401 additions & 0 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Your Apify API token: https://console.apify.com/settings/integrations
2+
APIFY_TOKEN=

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
test:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
node-version: [20, 24]
16+
steps:
17+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
18+
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
19+
with:
20+
node-version: ${{ matrix.node-version }}
21+
cache: npm
22+
- run: npm ci
23+
- run: node --check index.js
24+
- run: npm test

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
.env
2+
results.json
3+
results.csv
4+
node_modules/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Devil Scrapes
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
<p align="center"><a href="https://apify.com/DevilScrapes/github-user-scraper"><img src="assets/banner.svg" alt="GitHub User Scraper — Profiles, Repos &amp; Orgs — JavaScript example" width="100%"></a></p>
2+
3+
# GitHub User Scraper for JavaScript and Node.js: account and org lookups for your backend
4+
5+
<p align="center">
6+
<a href="https://apify.com/DevilScrapes/github-user-scraper"><img alt="Run on Apify" src="https://img.shields.io/badge/Run%20on-Apify-DC2626?logo=apify&logoColor=white"></a>
7+
<img alt="JavaScript" src="https://img.shields.io/badge/Node.js-20%2B-339933?logo=nodedotjs&logoColor=white">
8+
<a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/badge/license-MIT-0A0606"></a>
9+
<a href="https://github.com/devilscrapes/github-user-scraper-javascript/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/devilscrapes/github-user-scraper-javascript/actions/workflows/ci.yml/badge.svg"></a>
10+
</p>
11+
12+
Use this repository to call the **[GitHub User Scraper](https://apify.com/DevilScrapes/github-user-scraper)** from Node.js, using a single ES module and the `apify-client` package that Apify maintains. Send a list of logins or profile links and await an array of objects, one per account, carrying follower and repository totals, profile links, timestamps and, optionally, a nested array of recently pushed public repositories. Apify runs the lookups and deals with retries, so your Express handler, worker or scheduled function never has to manage GitHub's request budget itself. Rows end up in `results.json` and `results.csv`, or stay in memory when you import the call into your own code.
13+
14+
> **Actor:** [GitHub User Scraper — Profiles, Repos & Orgs](https://apify.com/DevilScrapes/github-user-scraper) · **Publisher:** [Devil Scrapes](https://apify.com/DevilScrapes) · **Also in Python:** [github-user-scraper-python](https://github.com/devilscrapes/github-user-scraper-python)
15+
16+
## 🔥 What you get
17+
18+
- **People and organizations in one schema**: both kinds of account come back with identical keys, and `type` says `User` or `Organization` so you can split them later.
19+
- **Paste logins or links**: entries in `usernames` may be bare slugs or full profile URLs; the host part is removed before lookup.
20+
- **Counts as integers**: `followers`, `following`, `public_repos` and `public_gists` are numbers, ready to sort, sum or chart.
21+
- **Recent repositories on request**: set `includeRepos` and each row gains a `repos` list of the account's most recently updated public repositories, each with name, description, language, stars, forks, fork and archived flags and last push time. `maxReposPerUser` caps it at up to 100.
22+
- **Bring a token if you need volume**: without one, GitHub allows 60 requests an hour; an optional personal access token lifts that to 5,000.
23+
- **Back-off built in**: on throttling or server errors the Actor waits and retries, and it reports a clear status message instead of finishing with a silently empty dataset.
24+
- **Only what is public**: `email` is filled only when the account owner chose to publish one, and private repositories or memberships are never touched.
25+
26+
## 🎯 Use cases
27+
28+
### Show live GitHub stats in a product page
29+
30+
Refresh the numbers for your own organization on a timer, cache the row and render follower and repository totals in a footer or an about page. The page stays current without the browser calling GitHub.
31+
32+
### Enrich sign-ups that include a GitHub handle
33+
34+
When a user connects or types a GitHub handle, queue a run for it and store `public_repos`, `followers` and `created_at` next to the account. A support or sales tool can then tell a brand-new account from a long-standing maintainer.
35+
36+
### Post a weekly webhook digest
37+
38+
Run the Actor on an Apify schedule, diff the rows against last week's copy and send the deltas to Slack, Discord or any webhook endpoint. DevRel teams get follower and repository changes for themselves and peers without opening a browser.
39+
40+
### Feed an internal developer directory
41+
42+
For an employee directory or an open-source program office, keep each team account's `blog`, `location` and recent `repos` in your database and serve them through your own API, keyed by `login`.
43+
44+
## ⚡ Quick start
45+
46+
You need an Apify account and an API token from [Console → Settings → Integrations](https://console.apify.com/settings/integrations). Apify's free plan includes monthly usage credit; this Actor's event prices are listed under Pricing below.
47+
48+
```bash
49+
git clone https://github.com/devilscrapes/github-user-scraper-javascript.git
50+
cd github-user-scraper-javascript
51+
npm install
52+
export APIFY_TOKEN=your_token_here
53+
node index.js # uses input.json
54+
node index.js my-input.json --out data --max-charge 2
55+
```
56+
57+
The script starts the [GitHub User Scraper — Profiles, Repos & Orgs](https://apify.com/DevilScrapes/github-user-scraper), waits for it to finish, prints a link to the run in Apify Console and saves every result to `results.json` and `results.csv`. It exits with code 1 and the run's status message if the run fails.
58+
59+
## 🧩 The core call
60+
61+
Here is the contract in four statements. `call()` resolves once the run has stopped; `listItems()` then returns the dataset rows as plain JavaScript objects.
62+
63+
```javascript
64+
import { ApifyClient } from 'apify-client';
65+
66+
const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
67+
const run = await client.actor('DevilScrapes/github-user-scraper').call({"usernames": ["apify", "microsoft"], "includeRepos": true, "maxReposPerUser": 5, "proxyConfiguration": {"useApifyProxy": false}});
68+
const { items } = await client.dataset(run.defaultDatasetId).listItems();
69+
console.log(items);
70+
```
71+
72+
## 📥 Input
73+
74+
`index.js` loads `input.json` and passes it along unchanged. The sample checks two organizations and keeps five recent repositories each, which comes back as two objects in a short run. Only `usernames` is mandatory. In a service, build the input per request and inject `githubToken` from an environment secret instead of writing it into a file.
75+
76+
```json
77+
{
78+
"usernames": [
79+
"apify",
80+
"microsoft"
81+
],
82+
"includeRepos": true,
83+
"maxReposPerUser": 5,
84+
"proxyConfiguration": {
85+
"useApifyProxy": false
86+
}
87+
}
88+
```
89+
90+
| Field | Type | Required | Default | Allowed values |
91+
|---|---|---|---|---|
92+
| `usernames` | array | yes | `["torvalds", "apify"]` | — |
93+
| `githubToken` | string | no | — | — |
94+
| `includeRepos` | boolean | no | `false` | — |
95+
| `maxReposPerUser` | integer | no | `30` | — |
96+
| `concurrency` | integer | no | `6` | — |
97+
| `proxyConfiguration` | object | no | `{"useApifyProxy": false}` | — |
98+
99+
## 📤 Output
100+
101+
Expect one object per account. `login`, `type` and `html_url` are always set; other profile properties may be null. Counts are numbers, timestamps are ISO strings, and `repos` is either null or an array of objects with fields such as `full_name`, `language`, `stargazers_count` and `pushed_at`.
102+
103+
| Field | Type | Always present |
104+
|---|---|---|
105+
| `login` | string | yes |
106+
| `type` | string | yes |
107+
| `name` | string or null | no |
108+
| `company` | string or null | no |
109+
| `blog` | string or null | no |
110+
| `location` | string or null | no |
111+
| `email` | string or null | no |
112+
| `bio` | string or null | no |
113+
| `twitter_username` | string or null | no |
114+
| `public_repos` | integer | no |
115+
| `public_gists` | integer | no |
116+
| `followers` | integer | no |
117+
| `following` | integer | no |
118+
| `html_url` | string | yes |
119+
| `avatar_url` | string | no |
120+
| `hireable` | boolean or null | no |
121+
| `created_at` | string | no |
122+
| `updated_at` | string | no |
123+
| `repos` | array or null | no |
124+
| `scraped_at` | string | no |
125+
126+
## 💰 Pricing
127+
128+
| Event | Price (USD) |
129+
|---|---|
130+
| Actor start | $0.2 |
131+
| Result emitted | $0.003 |
132+
133+
Prices as configured on 2026-08-20. The [Actor page](https://apify.com/DevilScrapes/github-user-scraper) always shows the current price. Runs are billed to the Apify account that owns the token. Use `--max-charge` to cap what a single run can cost.
134+
135+
## 🛠️ Working with the data in JavaScript
136+
137+
This script compares today's `results.json` with the copy saved last time, reports follower and repository changes per account plus the latest repository push, and then stores the current file for the next comparison. It uses only Node.js built-ins.
138+
139+
```javascript
140+
import { readFile, writeFile } from 'node:fs/promises';
141+
142+
const current = JSON.parse(await readFile('results.json', 'utf8'));
143+
let previous = [];
144+
try {
145+
previous = JSON.parse(await readFile('previous.json', 'utf8'));
146+
} catch {
147+
// First run: nothing to compare against yet.
148+
}
149+
150+
const before = new Map(previous.map((acc) => [acc.login, acc]));
151+
const report = current.map((acc) => {
152+
const old = before.get(acc.login);
153+
const pushes = (acc.repos ?? []).map((repo) => repo.pushed_at).filter(Boolean).sort();
154+
return {
155+
login: acc.login,
156+
type: acc.type,
157+
followers: acc.followers,
158+
followerChange: old ? acc.followers - old.followers : null,
159+
repoChange: old ? acc.public_repos - old.public_repos : null,
160+
lastPush: pushes.at(-1) ?? null,
161+
};
162+
});
163+
164+
console.table(report);
165+
await writeFile('previous.json', JSON.stringify(current, null, 2));
166+
```
167+
168+
## ❓ FAQ
169+
170+
### How many accounts can one request handle?
171+
172+
That depends on GitHub's allowance rather than the Actor. Anonymous access gets 60 requests an hour per address; a token in `githubToken` raises it to 5,000. Each account is one request, and `includeRepos` adds a second.
173+
174+
### Can my API look up a handle on demand?
175+
176+
Yes. Start a run with a one-item `usernames` array from your route handler and await it, or, if your platform has short timeouts, use `start()` and pick up the dataset from an Apify webhook when the run succeeds.
177+
178+
### What does `concurrency` control?
179+
180+
How many lookups the Actor keeps in flight at once, from 1 to 32 with 6 as the default. Lower it if you share a token with other jobs and want to leave headroom in its hourly budget.
181+
182+
### Are private repositories or org members included?
183+
184+
No. Rows contain public profile data only, and the `repos` array lists public repositories. Private membership and private code need scopes this read-only lookup does not use.
185+
186+
## 🧯 Troubleshooting
187+
188+
- **`Set APIFY_TOKEN first`**: export the token in the same shell, or put it in a local `.env` file your shell loads. Never commit it.
189+
- **Run finished with `FAILED` or `ABORTED`**: open the printed Console link; the run log says why. Input validation errors name the field.
190+
- **Fewer rows than expected**: a `--max-charge` cap also limits how many results can be charged. Raise the cap or lower the limits in your input.
191+
- **Something looks wrong in the data**: open an issue on the [Actor page](https://apify.com/DevilScrapes/github-user-scraper) so the maintainers see it.
192+
193+
## 🔗 More from Devil Scrapes
194+
195+
- [GitHub User Scraper — Profiles, Repos & Orgs on Apify Store](https://apify.com/DevilScrapes/github-user-scraper)
196+
- [Python version of this example](https://github.com/devilscrapes/github-user-scraper-python)
197+
- [All Devil Scrapes Actors](https://apify.com/DevilScrapes)
198+
- [Apify web scraping cookbook](https://github.com/devilscrapes/apify-web-scraping-cookbook)
199+
200+
Related Actors:
201+
202+
- [GitHub Organization Scraper](https://apify.com/DevilScrapes/github-org-scraper) — organization metadata plus its public repository list, including member counts.
203+
- [GitHub Repo Scraper](https://apify.com/DevilScrapes/github-repo-scraper) — deep metadata for individual repositories, such as topics, license and latest release.
204+
- [GitHub Trending Scraper](https://apify.com/DevilScrapes/github-trending-scraper) — what is climbing GitHub's trending page today.
205+
206+
---
207+
208+
MIT licensed example code. The Actor itself runs on [Apify](https://apify.com/DevilScrapes/github-user-scraper) and is maintained by [Devil Scrapes](https://apify.com/DevilScrapes). Not affiliated with GitHub or Microsoft.

‎assets/banner.svg‎

Lines changed: 41 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)