BigSemantics does not detect URL redirects

[teschty]

Currently we assume that one URL will always lead to one page, but this isn't always the case, seen with such URLs as https://en.wikipedia.org/wiki/Special:Random, which will almost never return the same page.

[quyin]

Potential solutions:

1. A special mmd attribute marking a location that can redirect to different places;
2. A collection of special URL or URL patterns that can redirect to different places.

I think we need to find more use cases to inform which solution is better.

[quyin]

I'm losing track on the context of this issue. Is this about the XHR downloading process? If so, I believe the current XHRDownloader is able to detect redirections through HTTP 3xx responses. Or is this about the cache?