CUDA Notes

1 Pre-installation of CUDA software

1.1 Check for a CUDA-enabled card

Sources

  1. http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#axzz3qFKp3T3Z
  2. https://developer.nvidia.com/cuda-gpus

To check for a CUDA card in Linux, use the command:

lspci | grep -i nvidia

On my laptop, I get the following output:

01:00.0 VGA compatible controller: NVIDIA Corporation GF108M [GeForce GT 555M] (rev a1)

I checked that the GeForce GT 555m was CUDA-capable at NVIDIA's website listing all CUDA GPU's (see link above). While it's certainly not a new GPU card, it is CUDA-enabled with Compute Capability 2.1.


2 Installing CUDA Toolkit in Debian with a 'normal' GPU

For normal graphics cards like one may find on a desktop computer, most of the information about installing the Nvidia drivers may be found at the Debian wiki: https://wiki.debian.org/NvidiaGraphicsDrivers

For me, the directions on the Wiki have never worked without some slight fixes. Be sure to:

  1. Install the nvidia-driver from jessie-backports.
  2. Install nvidia-xconfig and run it with sudo to create the Xorg configuration script.
  3. Here's the key part: make sure that Debian's nouveau driver is actually blacklisted. I have found that the install scripts used by nvidia-xconfig typically add comment characters to the lines inside the blacklist files in /etc/modprobe.d/, which results in nouveau not actually being blacklisted.

To fix the above problem, I add the following to the file /etc/modprobe.d/nvidia-blacklists-nouveau.conf:

blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off

This should successfully prevent the system from using the nouveau driver (after a reboot, of course).


2.1 Installing CUDA and OpenCL

Once the Nvidia drivers are working, you will probably want to install CUDA and OpenCL. Install the packages nvidia-cuda-toolkit and nvidia-opencl-common.


3 Installation of the CUDA Toolkit in Debian with an Optimus chip

My first attempts at installing the CUDA toolkit on Debian Jessie met in failure, likely because libraries were not placed in their proper positions. I was able to install everything and compile the CUDA samples, but the compiled samples would always complain that there was an insufficient driver for the run time environment and not work. More specifically, they raised the somewhat common CUDA error 35: CUDA driver version is insufficient for CUDA runtime version.

I eventually found that installing the bumblebee-nvidia primus package, followed by nvidia-cuda-toolkit and nvidia-nsight produced a full working environment with a version 6.0 toolkit and IDE, minus the CUDA samples. To get the samples, I downloaded the CUDA toolkit version 6.0 .run file from Nvidia and installed only the samples. They were compiled with the gcc version required for the 6.0 toolkit, namely gcc 4.7.3.

Everything listed below did not work in the end, so just install the packages above if you're on a laptop with an Optimus card. I'm still waiting for feedback on the nvidia dev forums to see whether I can install the later toolkits.


3.1 Installation of the CUDA driver

Sources

  1. https://wiki.debian.org/NvidiaGraphicsDrivers
  2. https://wiki.debian.org/Bumblebee
  3. http://linuxconfig.org/nvidia-geforce-driver-installation-on-debian-jessie-linux-8-64bit
  4. https://packages.debian.org/jessie/devel/nvidia-cuda-toolkit

I mostly followed the directions at the first link above. However, it took me a few days of failure to realize that the Nvidia card on my laptop was using Nvidia Optimus, and thus required a specialized set of drivers known as bumblebee. Before I realized this, I would install the nvidia-driver package and boot to a black screen with a single, blinking cursor. This is the result of creating an xorg.conf file inside /etc/X11/, as cautioned on the Debian Wiki.

Luckily, the installation of the proper Nvidia drivers proceeded smoothly in Debian Jessie once I realized that everything is passed through my Intel video card and therefore requires bumblebee.

The first indication that you might require bumblbee comes from probing for VGA devices using PCI buses.

lspci | grep VGA

I get the following result:

00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
01:00.0 VGA compatible controller: NVIDIA Corporation GF108M [GeForce GT 555M] (rev ff)

The result on the top line suggests that I need bumblebee because there is an Intel integrated graphics card in my laptop. You can also see what card is used to render graphics on your computer by using the glxheads command. (I'm not sure what package it's part of.) This will display a rotating figure of a triangle and output some useful information on the renderer to your shell. Doing this produces the following on my device:

kmdouglass@kmd-laptop1:~$ glxheads
glxheads: exercise multiple GLX connections (any key = exit)
Usage:
  glxheads xdisplayname ...
Example:
  glxheads :0 mars:0 venus:1
Name: :0.0
  Display:     0xe6d010
  Window:      0x3c00002
  Context:     0xe7baa0
  GL_VERSION:  3.0 Mesa 10.3.2
  GL_VENDOR:   Intel Open Source Technology Center
  GL_RENDERER: Mesa DRI Intel(R) Sandybridge Mobile

As you can see, GL is being rendered by the Intel chip, not the nvidia chip. Next, I installed bumblebee using

sudo apt-get install bumblebee-nvidia primus

After a system restart, I tested that the driver installed correctly using the glxgears example:

sudo optirun glxgears -info

This opens a window of three gears, green, blue and red, that are turning. The optirun command is provided by bumblebee and more-or-less tells the operating system to use the nvidia card, not the Intel chip. Trying glxheads with optirun produces

kmdouglass@kmd-laptop1:~$ optirun glxheads
glxheads: exercise multiple GLX connections (any key = exit)
Usage:
  glxheads xdisplayname ...
Example:
  glxheads :0 mars:0 venus:1
Name: :0.0
  Display:     0x1d5d060
  Window:      0x3c00002
  Context:     0x1e586b0
  GL_VERSION:  4.4.0 NVIDIA 340.65
  GL_VENDOR:   NVIDIA Corporation
  GL_RENDERER: GeForce GT 555M/PCIe/SSE2

You can also verify that the driver exists by checking for the nvidia drivers in /proc/driver/ using optirun (this won't work if you do not use optirun).

kmdouglass@kmd-laptop1:~$ sudo optirun cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  340.65  Tue Dec  2 09:50:34 PST 2014
GCC version:  gcc version 4.8.4 (Debian 4.8.4-1)

3.2 Installation of the toolkit from Nvidia

Sources

  1. http://docs.roguewave.com/totalview/8.14.1/html/index.html#page/User_Guides/totalviewug-about-cuda.31.4.html
  2. http://askubuntu.com/questions/131506/how-can-i-get-nvidia-cuda-or-opencl-working-on-a-laptop-with-nvidia-discrete-car
  3. http://stackoverflow.com/questions/13125714/how-to-get-the-nvidia-driver-version-from-the-command-line
  4. http://developer.download.nvidia.com/compute/cuda/6_5/rel/docs/CUDA_Getting_Started_Linux.pdf

I have read that the CUDA samples are not included with the Debian nvidia-cuda-toolkit. Since these samples are a good learning resource, I have decided that I cannot live without them ;) So, I decided to install the toolkit from Nvidia instead of the Debian package manager.

First, I had to find what toolkit version worked with the CUDA driver version that was installed from the Debian package repositories (see the previous section Installation of the CUDA driver). (To my knowledge, the nvidia-smi package won't work if you use Bumblebee because nvidia-smi will not detect a working graphics card.)

kmdouglass@kmd-laptop1:~$ sudo aptitude versions nvidia-driver
Package nvidia-driver:                        
i A 340.65-2                                      stable                    995 

Package nvidia-driver-bin:
i A 340.65-2                                      stable                    995

I verified this by using optirun to check the driver version. (I really need to better understand Bumblebee and optirun.)

kmdouglass@kmd-laptop1:~$ optirun cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  340.65  Tue Dec  2 09:50:34 PST 2014
GCC version:  gcc version 4.8.4 (Debian 4.8.4-1)

So it looks like I am using driver version 340.65. The minimum version to use the SDK Tool Chain 6.5 is 340.21, so I believe I can download and use the 6.5 toolkit. I downloaded the cuda_6.5.14_linux_64.run file from https://developer.nvidia.com/cuda-toolkit-65 and followed the instructions on this forum: http://askubuntu.com/questions/131506/how-can-i-get-nvidia-cuda-or-opencl-working-on-a-laptop-with-nvidia-discrete-car.

UPDATE: The following text that is crossed out will result in you not being able to compile CUDA files since gcc 4.9 and up will absolutely not work with the CUDA 6.5 toolkit. Instead of overriding the installation checks, I compiled a gcc version that worked with the 6.5 toolkit, namely gcc 4.8.2. I then used update-alternatives to set the current system compiler to this version and reran the install.

After making the .run file executable, I had to add an –override flag to the installation command because my version of gcc (4.9.2) was supposedly not supported.

I accepted all the defaults (excep that I chose not to install the included nvidia driver). The install ran fine with gcc 4.8.2, but I received a notification in the final install message about these missing library files:

Missing recommended library: libX11.so
Missing recommended library: libXi.so
Missing recommended library: libXmu.so
Missing recommended library: libGL.so

This happened even though I had the relevant *-dev packages installed. Fortunately, I remembered having similar issues in the past and that they were caused by Debian moving many important .so files to /usr/lib/x86_64-linux-gnu; many programs still expect to find libraries in /usr/lib. To fix this, I created symlinks in /usr/lib/ to the corresponding files:

sudo ln -s x86_64-linux-gnu/libX11.so libX11.so
sudo ln -s x86_64-linux-gnu/libXi.so libXi.so
sudo ln -s x86_64-linux-gnu/libXmu.so libXmu.so
sudo ln -s x86_64-linux-gnu/libGL.so libGL.so

I uninstalled and reinstalled the toolkit, finally receiving the following output:

===========
= Summary =
===========

Driver:   Not Selected
Toolkit:  Installed in /usr/local/cuda-6.5
Samples:  Installed in /home/kmdouglass

Please make sure that
 -   PATH includes /usr/local/cuda-6.5/bin
 -   LD_LIBRARY_PATH includes /usr/local/cuda-6.5/lib64, or, add /usr/local/cuda-6.5/lib64 to /etc/ld.so.conf and run ldconfig as root

To uninstall the CUDA Toolkit, run the uninstall script in /usr/local/cuda-6.5/bin
To uninstall the NVIDIA Driver, run nvidia-uninstall

Please see CUDA_Getting_Started_Guide_For_Linux.pdf in /usr/local/cuda-6.5/doc/pdf for detailed information on setting up CUDA.

***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 340.00 is required for CUDA 6.5 functionality to work.
To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:
    sudo <CudaInstaller>.run -silent -driver

Logfile is /tmp/cuda_install_7306.log

I next updated my PATH environment variable and added /usr/local/cuda-6.5/lib64 to ld.so.conf and ran ldconfig as directed.


3.2.1 Switching to another gcc version

Sources

  1. http://lektiondestages.blogspot.ch/2013/05/installing-and-switching-gccg-versions.html
  2. http://www.tellurian.com.au/whitepapers/multiplegcc.php

Looking inside /usr/bin I noticed that I had two versions of gcc installed, 4.8 and 4.9. I found that I could easily switch between them by using update-alternatives. I configured the alternatives with the following commands:

sudo update-alternatives --remove-all gcc
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 60
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 40

Next, I switched to version 4.8 using

sudo update-alternatives --config gcc

Checking the version on the command line to ensure that 4.8 was selected:

kmdouglass@kmd-laptop1:~/src$ gcc --version
gcc (Debian 4.8.4-1) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Unfortunately, this 4.8.4 is not supported by CUDA either, so I attempted to install my own isolated version of gcc 4.8.2 instead. First, I downloaded 4.8.2 from a mirror, unzipped it, and created a build directory. I switched back to gcc 4.9 using update-alternatives as above, and then I ran the configure script.

cd ~/src
tar -xvzf gcc-4.8.2.tar.gz
mkdir gobj
cd gobj
../gcc-4.8.2/configure --prefix=/usr/local/gcc/4.8.2 --enable-multiarch

(I am not sure whether the –enable-multiarch flag was necessary.) I had to install the libmpfr-dev and libmpc-dev packages to pass the configuration checks. Once the configuration was succesful, I started the build.

make bootstrap

During the build I encountered a few errors. One is that I didn't have g++ installed (this was a surprise). After installing it, I received an error about a missing cdefs.h header file. This is related to missing 32 bit libraries as discussed here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=761468. The solution is to install the gcc-multilib package. Be sure that zip (yes, really) is installed because the compilation raised an error after 2 hours because zip was missing. After installing these packages, rerun the configure script, and then rerun make bootstrap.

When the installation finishes (mine took about 3 hours), install the compiler to the directory that was set in the configure script:

sudo make install

Don't forget to use update-alternatives and switch gcc to the version you want.

sudo update-alternatives --remove gcc
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 70 --slave /usr/bin/g++ g++ /usr/bin/g++-4.9 
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 20 
sudo update-alternatives --install /usr/bin/gcc gcc /usr/local/gcc/4.8.2/bin/gcc 10 --slave /usr/bin/g++ g++ /usr/local/gcc/4.8.2/bin/g++

3.3 Compiling the samples

If everything was installed correctly, you can try compiling the samples:

cd ~/NVIDIA_CUDA-6.5_Samples
make

They compiled, but ultimately they would not run. For example, if I ran deviceQuery, I would receive an error stating CUDA driver version is insufficient for CUDA runtime version, even though my driver was newer than the one provided by the CUDA 6.5 toolkit. The same happened if I ran the compiled programs with optirun or primusrun.


4 Uninstalling the Nvidia drivers

After uninstalling the drivers, it's important to remove the nouveau blacklist if it was added to modprobe.d. These blacklist files are saved in /etc/modprobe.d. You may also need to reinstall the xserver and reconfigure it:

sudo apt-get install --reinstall xserver-xorg-video-nouveau
sudo dpkg-reconfigure xserver-xorg
sudo shutdown -r now

More information may be found here.