Conversation
The scrapers reading the MDN content repository need the data packages MDN generates its compatibility tables out of alongside the documents.
Redcarpet flattens the lists MDN writes its definition lists as, and reads two consecutive GitHub alerts as a single blockquote. kramdown's GFM parser nests both the way CommonMark does.
MdnGit reads the markdown of https://github.com/mdn/content rather than the pages developer.mozilla.org renders out of it, which are the same documents with their sidebars, compatibility tables and specifications filled in. It downloads the content repository and the packages MDN builds those sections out of — browser-compat-data, web-features, web-specs and mdn-data — and expands the KumaScript macro calls its authors leave in the markdown itself. Every page is known upfront, so nothing is crawled and no request is sent while a documentation is built. Mdn, which crawls, stays behind for the documentations that haven't moved over.
The 1300 pages of the reference are read off disk rather than crawled, and their compatibility tables built from a local copy of browser-compat-data rather than from one request per page to bcd.developer.mozilla.org. The entries come out unchanged. The clean_html filter goes: the markup it was undoing is MDN's, and the documents no longer carry any of it.
All 8108 pages, none of them requested over HTTP. The entries filter used to name the part of the web platform a page documents after the specification table MDN rendered into it. That table has been gone for a while, which left 2501 entries under Miscellaneous; the specifications are now looked up in browser-compat-data and web-specs, which puts 852 of them back where they belong.
The formal syntax of a property and the table of its characteristics are generated out of mdn-data, the way MDN generates them. As with the web APIs, the entries filter was reading the part of CSS a page belongs to off a table that no longer exists, leaving 912 of 1107 entries under Miscellaneous; there are 3 left. Note that MDN has since moved its reference under Web/CSS/Reference, so the pages move with it.
Four things kept a call from being expanded or a link from resolving:
- A call inside an inline code span was skipped along with the ones in
the code blocks, which are the only ones meant to read literally.
- The tables generated out of mdn-data are written with the same
cross-references as the prose, so an expansion can hold a call itself.
- An argument is written the way it reads, quoted with backticks as
often as not, escaped as <color> rather than <color>, and left
out altogether in {{rfc("7002",,"3.2")}}.
- Neither the fragments of browser-compat-data nor the ones MDN's
authors write are escaped, and URL.parse rejects both.
The JavaScript and CSS documentations come out with no call left
unexpanded and no dead link; the two remaining in the web APIs are a
code fence upstream forgot to close.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These three documentations are scraped off developer.mozilla.org one page at a
time, plus one request per page to
bcd.developer.mozilla.orgfor the browsercompatibility table. MDN publishes the same documents as markdown in
mdn/content, so this reads them from there
instead: the scrapers download the repository once and build from disk, sending
no request while a documentation is generated.
-j 8)Web APIs used to be 8108 page requests and 7839 compatibility requests.
How it works
Docs::MdnGitis aFileScraperthat downloads the content repository alongwith the packages MDN generates the rest of a page out of —
@mdn/browser-compat-data,web-features,web-specsand, for CSS,mdn-data— and prunes them to what the documentation queries (25MB of JSONdown to 2MB for JavaScript).
lib/docs/mdn_content/holds the page index, themarkdown renderer and an implementation of the 50 KumaScript macros these
documentations use, including the ones MDN renders server-side:
{{Compat}},{{Specifications}},{{CSSSyntax}}and{{CSSInfo}}.Every page is known upfront, so nothing is crawled and
FixInternalUrlsBehavioris no longer needed.
Output
MiscellaneousNo entry is renamed. The CSS and Web APIs entries filters used to name the part
of the platform a page belongs to after a table of specifications that MDN
rendered into every page; that table has been gone for a while, which is why
912 of 1107 CSS entries sat under
Miscellaneous. The specifications are nowlooked up in
browser-compat-dataandweb-specs, which puts 909 CSS entriesand 852 Web APIs entries back where they belong.
The pages also come out smaller — 13.5MB → 11.7MB for JavaScript — now that
MDN's own wrappers aren't part of them, and a few long-standing scraping
artifacts are gone (the stray
>in every specification table, dead links fromunescaped fragments).
Breaking change
MDN has moved its CSS reference under
Web/CSS/Reference, so the CSS pages movewith it:
css/animationbecomescss/properties/animation. This would happenon any re-scrape from today's MDN, but it lands when this is published.
Unchanged
Docs::Mdnand its filters are untouched; HTML, SVG, Web Extensions, XSLT &XPath and HTTP still crawl developer.mozilla.org. Those can move over the same
way later, at which point
Docs::Mdnandmdn/compat_tablescan go.Known limitations
alignment or a link on every multiplier.
{{SubpagesWithSummaries}}renders the index without the summaries (2 pages)."Learn more" / "See full compatibility" links.
web/api/webtransport_api/index.mdforgets to close; MDN renders that pagethe same way.
New dependency
kramdownandkramdown-parser-gfm, in thedocsgroup. Redcarpet flattensthe nested lists MDN writes its definition lists as, and reads two consecutive
GitHub alerts as a single blockquote.