CUDA vs cuDNN vs tiny-cuda-nn

 These terms usually confuse the beginners. Let's explain them in a simple and clear way.

CUDA vs cuDNN:

NVIDIA CUDA Deep Neural Network (cuDNN) is a GPU-accelerated library of primitives for deep neural networks.

The cuDNN library is a library optimized for CUDA containing GPU implementations. Think of cuDNN as a library for Deep Learning using CUDA and CUDA as a way to talk to the GPU.

In summary, CUDA provides the programming interface and runtime environment for general-purpose computing on a GPU, while cuDNN is a library that specifically optimizes deep learning operations on a GPU.

tiny-cuda-nn:

tiny-cuda-nn comes with a PyTorch extension that allows using the fast MLPs and input encodings from within a Python context. These bindings can be significantly faster than full Python implementations; in particular for the multi-resolution hash encoding.