Publish PDF version of documentation alongside html

[btashton]

Right now we only export the html versions of the docs. There have been asks to also publish the PDF which should be low effort for us to do.

[cederom]

+1 :-) im here to help :-)

[btashton]

I took a stab at this today and have something I think is workable. Let me know if this pdf attached seems workable as a first pass.

nuttx.pdf

@cederom @patacongo @hartmannathan

For my own reference

• Update build packages for tex support
• Update latex image to deal with unicode issues SPHINXOPTS='-D latex_engine=lualatex' pipenv run make latexpdf
• covert webp and gif images to png (there are only two) since they cannot be rendered to tex.

[cederom]

Looks awesome first step! Congratulation @btashton :-)

We could create a tar.gz packages for each release along with full pdf documentation. Then save it in various places and we are ready for zombie apocalypse! :-)

Some tweaking that comes to my mind at first look:

• headers, introduction, etc.
• re-formatting long tables that does not fit into a single page can be converted to a long list, etc
• moving long parts (i.e. Apache License) to Appendix section.
• Table of Contents tree embedded into PDF.
• clickable table of contents.
• images/charts reference index.

We need to shrink the full document size from 24MB to as small as possible. Would that be possible to see how small PDF can be with no images at all? I was using LaTeX for some time for documentation / presentation / automation so I have some experience we can use. From what I remember the best format for graphics was vector (where possible) as it results best quality and smallest size (LaTeX converted those to eps/ps and then attached to ps version of the document that was then coverted to pdf).. so we may try using SVG for the diagrams? As for PNG I also had to convert JPG to PNG.. but there may be some smart way around that.. or simply reworking final PDF with the trick below.

Quick trick I usually do to make PDF smaller without noticeable graphics quality loss is:

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dBATCH  -dQUIET -dDownsampleColorImages=true -dColorImageResolution=200 -dColorImageDownsampleType=/Bicubic -sOutputFile=nuttx-small.pdf nuttx.pdf

Tuning the image resolution parameter I got (quality assessed on terminal screenshots modern 4K monitor):

• 200dpi: 24MB -> 7.6MB (best quality: nuttx-small-200dpi.pdf).
• 150dpi: 24MB -> 6.9MB (optimal quality: nuttx-small-150dpi.pdf).
• 100dpi: 24MB -> 6MB (poor quality: nuttx-small-100dpi.pdf).
• 75dpi: 24MB -> 4.7MB (unacceptable quality: nuttx-small-75dpi.pdf).

Please let me know how can I help? :-)

[cederom]

• Okay so I was able to build HTML documentation on FreeBSD. Images on the website take around 22.8MB.

• I was able to quick-and-dirty try out the PDF build based on example sphinx conf [1] with some small adjutments.

  • sphinx-build -b latex . _build_latex -j8; cd _build_latex; gmake all.
  • there are issues with some UTF characters that cannot be converted to LaTeX.
  • all images needs to be converted first to PNG (this can be done with convert utility from ImageMagic).
  • due to LaTeX build errors I had to add -f switch to pdflatex in _build_latex/Makefile. Also batch mode helps in build no questions asked (to be confirmed by Enter anyways). (PDFLATEX = latexmk -pdf -dvi- -ps- -f --interaction=batchmode).

• Resulting PDF is here: main.pdf :-)

  • It is 25MB in size, so probably text takes around 3MB.
  • The template used is kinda old.
  • There is a list of figures and tables (settings for LaTeX document).
  • There is a ToC navigation in PDF (settings for LaTeX document).

[1] https://sphinxguide.readthedocs.io/en/latest/sphinx_basics/settings.html

[btashton]

Tbh I'm probably not going to make too many tuning changes and what we can do are somewhat limited as this is generated via sphinx and some formatting is going to be opinionated from that.

What I will do is wire this up to be linked in the website for download.

As for the size. In this day and age is it really worth making it more optimized? 25MB is tiny compared to even the repo and the PDF is something like 300+ pages with images.

I personally expect minimal usage of this.

[btashton]

• Okay so I was able to build HTML documentation on FreeBSD. Images on the website take around 22.8MB.
• I was able to quick-and-dirty try out the PDF build based on example sphinx conf [1] with some small adjutments.
  • sphinx-build -b latex . _build_latex -j8; cd _build_latex; gmake all.
  • there are issues with some UTF characters that cannot be converted to LaTeX.
  • all images needs to be converted first to PNG (this can be done with convert utility from ImageMagic).
  • due to LaTeX build errors I had to add -f switch to pdflatex in _build_latex/Makefile. Also batch mode helps in build no questions asked (to be confirmed by Enter anyways). (PDFLATEX = latexmk -pdf -dvi- -ps- -f --interaction=batchmode).

You are reinventing what I have already done. Already resolved the Unicode issue and also all the images outside of two of them. There is no need to change the Unicode characters or bulk change images to png.

[cederom]

• It was just a quick test of sphinx + html + latex on FreeBSD on my side :-)
• Regarding LaTeX optimizations / tuning I can handle that no worries I use LaTeX myself for years :-)
• Regarding PDF size optimization I have updated the gs conversion results above for a comparison. This trick comes handy quite often in various areas working with PDF. No commercial closed source converters involved :-)
• We may worry about PDF size optimizations when we have PDF working, cool :-)
• Regarding Terminal Graphics / Screenshots, these need good read, we can use Asciinema [1] that even allows animated terminal in the online documentation!
• Asciinema frame can be converted to SVG for instance with asciicast2vector [2]. That will result in a relatively small and always perfect quality terminal screenshot :-)

[1] https://asciinema.org/
[2] https://github.com/misc0110/asciicast2vector

[cederom]

@btashton: Tbh I'm probably not going to make too many tuning changes and what we can do are somewhat limited as this is generated via sphinx and some formatting is going to be opinionated from that.

It can be easily tuned in conf.py and the LaTeX section from what I can see :-)

What I will do is wire this up to be linked in the website for download.

Awesome! Thank you! :-)

As for the size. In this day and age is it really worth making it more optimized? 25MB is tiny compared to even the repo and the PDF is something like 300+ pages with images.

Well, old habits, my butt hurts when the overall size can be easily decreased 3x+ and the text to gfx ratio is 1:9. I will try to help with gfx optimization + documentation edit / update no worries :-)

I personally expect minimal usage of this.

Even more kudos for doing that @btashton thank you ! :-)

[cederom]

Here goes the ePub test build: NuttX.epub.zip. I had to zip it for GH to accept the file, please unzip after download. You can use Calibre reader to open it [1] :-)

[1] https://github.com/kovidgoyal/calibre