Nvidia
Useful resources:
- https://docs.kinetica.com/7.1/install/nvidia_rhel/
- https://developer.nvidia.com/blog/streamlining-nvidia-driver-deployment-on-rhel-8-with-modularity-streams/
- Good Nvidia driver installation guide: https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf
How to install NVIDIA driver on RHEL8 for Specific GPU model (using .run file)
The following steps uses NVIDIA's .run file. For steps on how to install nvidia-driver via dnf/command line, please refer to this link
Another helpful link: https://www.if-not-true-then-false.com/2021/install-nvidia-drivers-on-centos-rhel-rocky-linux/
Notes:
• The following steps have been tested on Dell Precision 3480 (RTX A500 Laptop GPU) and Dell Precision 7670 (RTX A2000 8GB Laptop GPU).
Steps: 1. First connect to the system via SSH/PuTTy and check the NVIDIA graphics card model:
$ lspci | grep -i nvidia
In this example output, the NVIDIA graphics card is RTX A2000 8GB Laptop GPU.
2. Search the driver from Nvidia's website and download it locally
You can use wget to download the .run file directly from Nvidia’s website to the system. Tip: To get the correct URL link, click “Download” and then right-click on “Agree & Download”.
$ wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.90.07/NVIDIA-Linux-x86_64-550.90.07.run
3. Install prerequisites:
$ dzdo yum install gcc kernel-devel libglvnd-devel elfutils-libelf-devel
4. Then perform the following steps to disable the nouveau driver: a. Edit the /etc/default/grub file and ensure that the options rd.driver.blacklist=nouveau nouveau.modeset=0 is at the end of the GRUB_CMD_LINUX line:
$ dzdo vi /etc/default/grub
Example:
GRUB_CMDLINE_LINUX="resume=/dev/mapper/cmw--rhel-swap rd.lvm.lv=cmw-rhel/root rd.lvm.lv=cmw-rhel/swap rhgb quiet audit=1 audit_backlog_limit=8192 pti=on page_poison=1 slub_debug=P fips=1 boot=UUID=1125ad64-b4b3-4995-928c-8f8a1fa2c48b rd.driver.blacklist=nouveau nouveau.modeset=0"
b. Save the file and exit. c. Next, rebuild the GRUB configuration file:
$ dzdo grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
d. Create the disable-nouveau.conf file under the /etc/modprobe.d/ directory:
$ dzdo vi /etc/modprobe.d/disable-nouveau.conf
And then insert these separate lines:
blacklist nouveau options nouveau modeset=0
Example:
$ dzdo cat /etc/modprobe.d/disable-nouveau.conf blacklist nouveau options nouveau modeset=0
e. Modify the permissions and then regenerate the initramfs file by running dracut. Then reboot.
$ dzdo chmod 644 /etc/modprobe.d/disable-nouveau.conf $ dzdo dracut -f $ dzdo reboot
5. Log back in via SSH and then run the following commands:
$ dzdo init 3 $ chmod +x NVIDIA-Linux-x86_64-<version.number>.run $ dzdo mount -o remount,exec /tmp $ dzdo yum remove nvidia-driver
6. Then run the installer and answer the prompts:
$ dzdo ./NVIDIA-Linux-x86_64-<version.number>.run
Wait for graphical installer then Answer 5 prompts/ 4 questions:
- Install 32-bit compatibility Libraries – Yes
- Register with DKMS – No
- Initramfs rebuild – Yes
- Update X Configuration File – Yes
- Installation Complete – Click OK
7. Then confirm the latest driver version was installed successfully (check Driver Version):
$ nvidia-smi
8. Reboot the system and confirm that its booting up and working as expected. Once logged back in, re-run nvidia-smi to confirm.
$ dzdo reboot $ nvidia-smi (or) $ nvidia-smi -q | grep -i “driver version”
9. Clean up: remove the .run installer once completed successfully.
$ dzdo rm NVIDIA-Linux-x86_64-<version.number>.run
How to uninstall the NVIDIA driver after installing it using the .run file
sudo ./NVIDIA-Linux-x86_64-<version.number>.run --uninstall
How to check Nvidia driver version and other information
$ nvidia-smi Tue Jun 4 13:09:18 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A2000 8GB Lap... Off | 00000000:01:00.0 Off | N/A | | N/A 49C P0 25W / 60W | 1MiB / 8192MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
Alternatively, you can run the following command which will give you the same information (or more) in a non-table format:
$ nvidia-smi -q
==============NVSMI LOG==============
Timestamp : Thu Jun 27 14:55:53 2024
Driver Version : 555.42.02
CUDA Version : 12.5
Attached GPUs : 1
GPU 00000000:02:00.0
Product Name : NVIDIA RTX A500 Laptop GPU
Product Brand : NVIDIA RTX
Product Architecture : Ampere
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Addressing Mode : None
MIG Mode
Current : N/A
Pending : N/A
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-fabe1493-c99a-410b-5941-abeb61f58c80
Minor Number : 0
VBIOS Version : 94.07.7C.00.0B
or
$ nvidia-smi -q | grep -i driver
Driver Version : 555.42.02
Driver Model
NVIDIA DOCUMENTATION ON HOW TO DISABLE NOUVEAU
https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/nouveau.html https://docs.nvidia.com/ai-enterprise/deployment-guide-bare-metal/0.1.0/nouveau.html
GSP Firmware
GSP firmware being enabled has been causing issues with some Dell models that contain specific GPU hardware.
What is GSP?
Some GPUs include a GPU System Processor (GSP) which can be used to offload GPU initialization and management tasks. This processor is driven by firmware files distributed with the driver. The GSP firmware is used by default on GPUs which support it.
Offloading tasks which were traditionally performed by the driver on the CPU can improve performance due to lower latency access to GPU hardware internals.
Firmware files gsp_*.bin are installed in /lib/firmware/nvidia/560.28.03/. Each GSP firmware file is named after a GPU architecture (for example, gsp_tu10x.bin is named after Turing) and supports GPUs from one or more architectures.
Disabling GSP Mode
The driver can be forced to disable use of GSP firmware by setting the kernel module parameter nvidia.NVreg_EnableGpuFirmware=0.
The nvidia-smi utility can be used to query the current use of GSP firmware. It will display a valid version if GSP firmware is enabled, or “N/A” if disabled
$ nvidia-smi -q | grep -iE 'driver version|gsp'
Driver Version : 555.42.06
GSP Firmware Version : N/A
Enabling GSP Mode
The GSP firmware will be used by default for all Turing and later GPUs. The driver can be explicitly configured to use the GSP firmware by setting the kernel module parameter nvidia.NVreg_EnableGpuFirmware=1.
https://download.nvidia.com/XFree86/Linux-x86_64/560.28.03/README/gsp.html
Issues with the 560 Driver Not Able to Disable GSP
Nvidia then released the 560 driver, which also comes with the GSP enabled. However, starting with the 560 driver, Nvidia decided to make the open kernel module as the default installation instead of the proprietary version. From my research, I found that we were not able to disable the GSP firmware when using the open kernel module version of the driver, because it ignores the NVreg_EnableGpuFirmware=0 kernel parameter.
To resolve this, you must uninstall the open kernel module and reinstall the proprietary version in order for the kernel parameter to work and successfully disable the GSP firmware. To install the proprietary version, use the command sudo dnf module install -y nvidia-driver:latest-dkms so that the GSP firmware gets disabled during the image process.
Here are some links as a reference:
- The updated nvidia_driver.sh script (See lines 30-60 for the fix): http://va3cngl01a.cacicorenet.com/linux_image/rhel8/-/blob/main/Dev_wk_files/nvidia_driver.sh
- From this link: https://download.nvidia.com/XFree86/Linux-x86_64/560.28.03/README/kernel_open.html it says "Because the two flavors of kernel modules are mutually exclusive, one or the other must be chosen at install time. By default, installation will choose which flavor of kernel modules to install, based on the GPUs detected in the system. If a pre-Turing GPU is detected, installation will default to the proprietary flavor of kernel modules. Otherwise, installation will default to the open flavor of kernel modules."
- Per https://www.reddit.com/r/linux_gaming/comments/1cp4heq/news_starting_with_nvidia_560_the_open_source/ it says "Starting in the release 560 series, it will be recommended to use the open flavor of NVIDIA Linux Kernel Modules wherever possible (Turing or later GPUs, or Ada or later when using GPU virtualization).
- If installing from the .run file, installation will detect what GPUs are present and default to installing the open kernel modules if all NVIDIA GPUs in the system can be driven by the open kernel modules. Distribution-specific repackaging of the NVIDIA driver may require additional steps, specific to that packaging, to choose the open flavor.
- In the release 560 series, it will still be possible to configure the .run file to install the proprietary flavor of kernel modules, with the --kernel-module-type=proprietary command line option. However, in the future, some GPUs may only be supported with the open flavor."