How to build/run the project? under directory ulrvalidator (project root) please run command in terminal
sbt run
open browser at http://localhost:9000/
Main modules are:
- reactApp.jsx - does UI stuff
- HomeController - checks that url is valid and reachable plus calls HtmlDocumentAnalyzer
- HtmlDocumentAnalyzer - calculates number of headings, number of links, etc. No tests included.
Design decisions: Implementation is stateless plus immutable objects only Requests usually take more than 0.5sec so UI must display some Loading... message Urls are validated in parallel with scala Futures ReactJS is chosen as it has well known advantages over AngularJS In case if something error happens during analysis of single result e.g. html version, it should not bother other parameters calculation, unfortunately this might not be true for parallel links validation in some cases. In order to check that url is accessible, the solution loads whole page into memory.
Limitations: By default on my machine it starts with Xms=1g of memory. This leads to memory crash during http://deutsche-post.de processing due to full pages downloading Overall response (processing) time is limited to 60 seconds timeout either on gui and on server There is 5 seconds timeout to download url Solution validates only links that it recognizes as "external". Solution considers http://sportal.spiegel.de/ as an external link for spiegel.de host (as well as https://services.github.com/ for http://github.com host) but it counts http(s)://spiegel.de/smth or http(s)://www.spiegel.de/smth as internal link Also when it checks page http://google.com, it will count google.ie as an external if redirected. External links count may differ with number of links shown due to possible links duplication on page. 307 Temporary Redirects are not fully checked (according to jsoup documentation) "External" links must begin with http://, https://, or // like //www.google.com Links like whatsapp://send?text=smth having protocol not http(s) are considered as internal and not validated
Implementation comments (comments are fails):
Spray library could be used to download only httpStatuses, not the whole pages.
https://stackoverflow.com/questions/20674448/is-there-an-easy-way-in-scala-to-check-whether-resource-by-some-url-respond-with
No possiblility to show results as soon as they are available. Probably, HTML version, title, headings and links count
should appear first with some 'Loading...' message under it.
Once actual long running urls validation is done it should be displayed.
WebJars were the simplest solution to include the JS libraries.
At last moment I preferred not to use react-bootstrap
because it requires a lot of babel/npm setup and it does not work well with sbt-reactjs sbt plugin
sbt-babel plugin by zenato can transpile bootstrap well but it's build and run were unstable
(it required some npm magic which I haven't learned yet)
I used tables instead of divs because it was easier and I had no luck with react bootstrap