AI

NVIDIA CUDA

What is CUDA?

CUDA (Compute Unified Device Architecture) is a parallel computing platform and application programming interface (API) model created by NVIDIA. It allows software developers to use a CUDA-enabled graphics processing unit (GPU) for general-purpose processing.

Install NVIDIA Driver

Ubuntu

  1. Update your server to ensure all packages are up to date:
$ sudo apt update && sudo apt upgrade -y
  1. Verify Hardware Compatibility
$ lspci | grep -i nvidia
  1. Add NVIDIA Repository
$ sudo add-apt-repository ppa:graphics-drivers/ppa
$ sudo apt update
  1. Install Build Tools
$ sudo apt install build-essential dkms ubuntu-drivers-common
  1. Install NVIDIA Drivers

Find the recommended driver from the list of available drivers:

$ ubuntu-drivers devices

Install the recommended driver (replace <driver-version> with the specific version):

$ sudo apt install nvidia-driver-<driver-version>
  1. Reboot your system:
$ sudo reboot
  1. Verify Driver Installation
$ nvidia-smi

Alternately, you can install the latest driver from NVIDIA official downloads:

  1. https://www.nvidia.com/en-us/drivers

Choose your RTX 5080/5090, select Linux 64-bit, and download the .run file.

  1. Shut down your graphical session.

You can't install NVIDIA drivers while the display manager is running. Switch to SSH in, then stop the display manager.

$ sudo systemctl isolate multi-user.target
$ sudo systemctl stop gdm
  1. Install

Now install the user-space components of the driver.

$ sudo sh NVIDIA-Linux-x86_64-570.169.run -m=kernel-open

If you prefer to install kernel module manually, you can clone and build the open GPU kernel modules.

$ git clone https://github.com/NVIDIA/open-gpu-kernel-modules.git
$ cd open-gpu-kernel-modules
$ make modules -j$(nproc)
$ sudo make modules_install -j$(nproc)
$ sudo sh NVIDIA-Linux-x86_64-570.169.run --no-kernel-modules

What to do when prompted:

  • Select MIT/GPL open-source kernel modules
  • Say yes to 32-bit support if you plan on using apps like Steam or Wine
  • Allow it to write X configs if you use X11
  1. Reboot

Install CUDA Toolkit

First, run nvcc --version to verify that the CUDA compiler is installed:

$ nvcc --version

CUDA 13.0 + Ubuntu 24.04 + Jetson ARM64

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb
$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
$ sudo apt update
$ sudo apt -y install cuda-toolkit-13-0

CUDA 13.0 + Ubuntu 24.04 + X86_64

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
$ sudo apt update
$ sudo apt -y install cuda-toolkit-13-0

CUDA 12.8 + Ubuntu 24.04

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
$ sudo apt update
$ sudo apt -y install cuda-toolkit-12-8

CUDA 12.8 + Ubuntu 22.04

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
$ sudo apt update
$ sudo apt -y install cuda-toolkit-12-8

CUDA 12.6 + Ubuntu 24.04

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
$ sudo apt update
$ sudo apt -y install cuda-toolkit-12-6

CUDA 12.4 + Ubuntu 22.04

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
$ sudo apt update
$ sudo apt -y install cuda-toolkit-12-4

After running the above command, you should update your .bashrc file to include the CUDA paths. For example you installed CUDA 12.4:

$ echo "export PATH=/usr/local/cuda-<CUDA_VERSION_HERE>/bin:$PATH" >> ~/.bashrc
$ echo "export LD_LIBRARY_PATH=/usr/local/cuda-<CUDA_VERSION_HERE>/lib64:$LD_LIBRARY_PATH" >> ~/.bashrc
$ source ~/.bashrc

Run nvcc --version again to verify that the CUDA compiler is installed:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

Performance Monitor

It's recommended to install nvidia-smi and nvtop for monitoring GPU performance.

nvidia-smi

$ watch -n 1 nvidia-smi
  • -n 1: Refresh every 1 second.

nvtop

Alternatively, you can use nvtop for a more detailed view of GPU performance:

$ sudo apt install nvtop

tegrastats

This utility comes with the Jetson system and reports memory usage, processor usage, GPU usage (“GR3D”), etc.

Example usage:

$ sudo tegrastats --readall

jtop

nvtop relies on NVML (NVIDIA Management Library) used for discrete GPUs (e.g., desktop/server GeForce/Quadro) to get memory-usage info. On Jetson (Tegra/embedded), NVML may not be supported or GPU memory usage is shared and not exposed the same way.

Many users report “No GPU to monitor” when running nvtop on a Jetson board.

🔍 Recommendation for your use case (Jetson Nano / AGX Thor)

$ sudo apt update
$ sudo apt install python3-pip python3-setuptools -y
$ sudo pip3 install --break-system-packages -U jetson-stats
#or
$ sudo pip3 install --break-system-packages git+https://github.com/rbonghi/jetson_stats.git
$ sudo reboot
#or just logout
$ jtop

If you

$ sudo systemctl restart jtop.service
Previous
Bytebot
Next
DataStax