Add developer and user guides for JIT#1876
Add developer and user guides for JIT#1876rapids-bot[bot] merged 19 commits intorapidsai:release/26.04from
Conversation
divyegala
added
doc
Co-authored-by: Kyle Edwards <kyedwards@nvidia.com>
| 1. In-memory cache is valid for the lifetime of the process. | ||
| 2. On-disk cache is valid until a CUDA driver upgrade is performed. This is stored in the user's home directory under the path ``~/.nv/ComputeCache/``, and can be portably shared between machines in network or cloud storage. | ||
|
|
||
| Thus, the JIT compilation is a one-time cost and you can expect no loss in real performance after the first compilation. We recommend that you run a "warmup" to trigger the JIT compilation before the actual usage. |
There was a problem hiding this comment.
Do you want to make it super mega obvious to people who deploy services based on cuvs in containers that "We really really strongly recommend you make sure the cache is stored in a persistent location so that containers don't have to warm up the cache after each restart"
Is it possible to include something that warms up the cache in my Dockerfile? So that the cache is built into the image?
I am not sure if I'd make the connection from reading the current docs, hence wondering if a really explicit "hit people over the head with it" call out would be useful.
There was a problem hiding this comment.
I think that's a great idea. Let me add some phrasing to convey that very clearly.
Is it possible to include something that warms up the cache in my Dockerfile? So that the cache is built into the image?
You mean automatically?
There was a problem hiding this comment.
Sounds good now. Let's see if people get it, if not can always tune this later.
Wasn't thinking of something automatic, more a command I can include in my Dockerfile as a RUN command
There was a problem hiding this comment.
If you see the link that I added now, you can control where the cache is written with an environment variable. I'm hoping docker savvy users can now figure out the volume mount and environment variable connection.
cjnolet
left a comment
cjnolet
left a comment
There was a problem hiding this comment.
Approving so I don't block this, but I think you should address the comments about the links. Those can help users help themselves by reading more in the official documentation, so that they can have a base level knowledge to follow your new uides.
|
|
||
| ## Using Just-in-Time Link-Time Optimization | ||
|
|
||
| cuVS is moving to using link-time optimization for new kernels, and this requires some changes to the way kernels are written. Instead of compiling all kernel variants at build time (which leads to binary size explosion), JIT LTO compiles kernel fragments separately and links them together at runtime based on the specific configuration needed. |
There was a problem hiding this comment.
Can we link somewhere in the cuda docs in this paragraph? Maybe for "link time optimation"?
There was a problem hiding this comment.
Also, can you provide an ever so brief summary of the perf implications? Maybe link to the cuda docs where appropriate for expectations?
There was a problem hiding this comment.
First run perf implications are very kernel and hardware dependent.. the CUDA docs make no guarantees about that.
Co-authored-by: Corey J. Nolet <cjnolet@gmail.com>
|
/merge |
github-project-automation
bot
moved this to Done
in Vector Search, ML, & Data Mining Release Board
4 participants
closes #1522 closes #1523