Skip to content

Rank languages by source-role files rather than raw extension counts #200

Description

@andrew

sortLanguages in detect/detect.go orders report.Languages by summing fileExts for each language's detect.files extension patterns. The only thing keeping noise out of that count is defaultSkipDirs, a fixed list of 19 directory names. It has testdata, vendor and third_party, but not fixtures, spec, examples, samples or features. Being keyed on directory names, it also cannot describe individual files such as *_test.go, *.pb.go or zz_generated_*.

The result is that fixture and helper code counts as evidence of what a project is written in. scrutineer is a worked example. Every one of its 19 Python files sits under evals/fixtures/ or skills/*/scripts/, and brief . on a clean checkout reports Python as the repository's second language behind Go. Its vendored JavaScript is excluded only because that directory happens to be called vendor.

Consumers that read the ranking as dominance inherit this. scrutineer picks a per-ecosystem runner image from languages[0], so a repository carrying a large fixture corpus in another language can be routed to a container without its own toolchain.

git-pkgs/roles classifies paths into additive roles from the path alone, with no file reads and no checkout, and is deterministic for a given corpus revision. Running its CLI over the paths above:

path roles
internal/worker/profile.go source
evals/fixtures/sqli-basic/app.py fixture
skills/semgrep/scripts/scan.py tooling
internal/web/static/vendor/htmx-2.0.6.min.js source, vendor, generated, minified
internal/worker/profile_test.go source, test

Scoring only paths whose roles include source while excluding vendor, generated, fixture, example and test would drop all of scrutineer's Python from the ranking. defaultSkipDirs then becomes largely redundant, and the file-level cases it cannot express are covered by the same rule.

Issue #43 describes the same root cause from the sinks side, where secondary languages add noise because every file counts toward a language being present. Weighting by role addresses both.

The library README lists brief/distill + roles as an intended composition, and roles is a Go module with no dependency on brief.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions