Manifest files for unixfs loaded via HTTP Gateway

[lidel]

This is a placeholder issue for creating a spec for "content root's manifest files" which would be part of the data published on IPFS (eg. /ipfs/CID/_something or /ipfs/CID/.well-known/ipfs/foo) that provides hints to HTTP Gateway around things like redirects, content types and custom HTTP headers.

Related reading:

• Proposal: manifest file for the IPFS gateway
• Proposal: native redirects config
• Content Type set by HTTP Gateway

Possible reuse of:

• _redirects (prior art in netlify docs
  • DONE
    • IPIP: https://specs.ipfs.tech/ipips/ipip-0002/
    • Specification: https://specs.ipfs.tech/http-gateways/web-redirects-file/
    • Docs: https://docs.ipfs.tech/how-to/websites-on-ipfs/redirects-and-custom-404s/
• _headers file (prior art in netlify docs and cloudflare docs)
  • TBD

Feel free to discuss/post ideas below, or open PRs with IPIP drafts that could be reviewed independently.

[BigLep]

Related are notes from the 2021-11-16 IPFS operators meeting: https://hackmd.io/MiYYUCR-RWqhAKfJ_B15wA

[justindotpub]

Hi @lidel. I started working at Fission this year and have recently been tasked with implementing redirects support in go-ipfs. @cbrake started working on this in this PR 🙏 and I plan on continuing the work in this PR.

If you don't mind, I'd like to make sure I understand the process here as far as specs and issue management go so I don't cause any annoying friction.

From the issues I've read through it appears that it was decided to split out redirects support from manifests support when Cloudflare announced Cloudflare Pages with _redirects support, similar to Netlify's _redirects support. This issue I'm commenting on appears to be specific to spec'ing out manifest support. Should redirects support get its own spec and therefore spec issue or were you thinking it would be included in the same spec?

Regardless of whether or not manifests and redirects are separate specs, my assumption is that the redirects portion of the spec will both inform and be informed by the go-ipfs redirects support tentative implementation. Does that sound right? Is there any preferred order to tackling this (spec vs implementation first)? I'm currently assuming I should start with essentially implementing what Cloudflare supports and see what feedback others provide, adjust for feedback, and then eventually work backward from there into text for the spec.

Assuming I'm on the right track, I was planning on submitting 1) an issue in go-ipfs to track this implementation work vs just my draft PR, and 2) possibly an issue in this specs repo to track the redirects portion of the spec, unless it should all be included in this spec issue.

Thank you in advance for any feedback and guidance. If you have any other tips on the code base or any plans you had for how the code should be reorged (I see you've been doing some refactoring recently) I'd love to hear them.

Thank you! 🙏

[BigLep]

Hi @justincjohnson. Thanks for reaching out here. @lidel is great at seeing GitHub notifications and processing them. I don't know how many others are though. If you don't get a response in the next day or so, feel free to start a thread in #ipfs-operators in FIL Slack or #ipfs-dev in IPFS Discord.

[lidel]

Hi @justincjohnson, really appreciate reaching out!

I believe tldr from the prior discussions is that we translated requirements into two sibling features that will improve the devexp around unixfs website hosting using IPFS gateway:

• _redirects (prior art in netlify docs, also discussed in Proposal: native redirects config kubo#7392 (comment))
  • does not seem to be controversial, ready to be worked on
  • mvp should support:
    • relative and absolute redirects
    • "catch-all" redirect for PWAs
• _headers file (prior art in netlify docs)
  • bit more tricky, as we need to be very careful which headers are allowed where
  • comes with some open questions around which headers could be specified without X-foo prefix.

Feel free to work on both, but focusing on _redirects first is a good idea. I am happy to guide / help with reviews. 👍

the redirects portion of the spec will both inform and be informed by the go-ipfs redirects support tentative implementation. Does that sound right? Is there any preferred order to tackling this (spec vs implementation first)? I'm currently assuming I should start with essentially implementing what Cloudflare supports and see what feedback others provide, adjust for feedback, and then eventually work backward from there into text for the spec. [..]
I was planning on submitting 1) an issue in go-ipfs to track this implementation work vs just my draft PR, and 2) possibly an issue in this specs repo to track the redirects portion of the spec, unless it should all be included in this spec issue.

Yes, this plan sounds good.

There are caveats around both headers and redirects that we will fully understand while writing sharness tests for them, so specs may change multiple times before we finish the mvp implementation. It is fine to keep discussion around go-ipfs PR and wait with specs until the dust settles.

Should redirects support get its own spec and therefore spec issue or were you thinking it would be included in the same spec?

We have various gateway improvements in flight, so it is better to have separate specs for now.
When you feel like writing the spec, just open a PR against ipfs/specs that adds it as a standalone file under gateways/REDIRECTS_FILE.md – this way we can get it done without being blocked on anything.

ps. I left some quick comments in ipfs/kubo#8816