LLVM Compiler/Installation

Installing Pre-build Binaries
on Ubuntu
 * sudo apt-get install clang llvm

Installing from github master
Hardware/Operating system
 * Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
 * Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card https://developer.nvidia.com/cuda-gpus.
 * Linux (ubuntu 18.04)

Prerequisites
sudo apt update sudo apt install build-essential sudo apt install cmake sudo apt install -y libelf-dev libffi-dev sudo apt install -y pkg-config

Install CUDA 10.2, using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /" sudo apt-get update sudo apt-get -y install cuda

After installation, export two environment variables, vim ~/.bashrc to add two lines
 * export PATH=$PATH:/usr/local/cuda-10.2/bin/
 * export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/

Get source packages
Assuming you are in your home directory

cd

git clone https://github.com/llvm/llvm-project.git

Build using GCC
mkdir build

cd build/

cmake -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp"   \ -DCMAKE_BUILD_TYPE=Release  \ -DLLVM_TARGETS_TO_BUILD="X86;NVPTX"    \ -DCMAKE_INSTALL_PREFIX=$(pwd)/../llvm  \ -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_37   \ -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=35,37,50,52,60,61,70,75 \ -DCMAKE_C_COMPILER=gcc   \ -DCMAKE_CXX_COMPILER=g++  \ -G "Unix Makefiles" ../llvm-project/llvm

time make -j time make -j install

Explanation for configuration options
 * -DCMAKE_C_COMPILER=gcc  // the C compiler used to compile clang/llvm,  GCC
 * -DCMAKE_CXX_COMPILER=g++ // the C++ compiler used to compile clang/llvm: G++
 * -DLLVM_TARGETS_TO_BUILD=X86;PowerPC;NVPTX;AMDGPU // explicitly specify target devices to support, Intel ,Nvidia, IBM, and AMD CPUs or GPUs
 * -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_70 // default GPU computing capability version to support, https://developer.nvidia.com/cuda-gpus lists such information.
 * -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 // all GPU computing capability versions to build in libomptarget

Other optional options
 * -DGCC_INSTALL_PREFIX=${GCC_PATH}
 * -DCMAKE_C_COMPILER=${GCC_PATH}/bin/gcc
 * -DCMAKE_CXX_COMPILER=${GCC_PATH}/bin/g++
 * -DCMAKE_Fortran_COMPILER=${GCC_PATH}/bin/gfortran
 * -DCUDA_PATH=      // this option should be automatically set if cuda is in your search path
 * -DCUDA_TOOLKIT_ROOT_DIR= // this option should be automatically set if cuda is in your search path
 * -DOPENMP_ENABLE_LIBOMPTARGET=ON // this should be on by default
 * -DLIBOMP_FORTRAN_MODULES=ON
 * -DBUILD_SHARED_LIBS=OFF // turn off shared libs

More examples
 * cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp" -DCMAKE_INSTALL_PREFIX=/Users/abc/llvm-research/inst-10.0.1 -DCMAKE_BUILD_TYPE=Debug ../llvm/
 * ninja -j8
 * ninja install

Rebuild using Clang
add the path to the installed clang, vim ~/.bashrc export PATH=~/llvm/bin:$PATH export LD_LIBRARY_PATH=~/llvm/lib:$LD_LIBRARY_PATH

cd build-openmp cd build-openmp/

cmake -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp" \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" \ -DCMAKE_INSTALL_PREFIX=$(pwd)/../llvm \ -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_37  \ -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=35,37,50,52,60,61,70,75   \ -DCMAKE_C_COMPILER=clang \ -DCMAKE_CXX_COMPILER=clang++  \ -G "Unix Makefiles" ../llvm-project/llvm

make -j make -j install

Optional options, explictly turn on bitcode lib, and the compiler/linker to build it
 * -DLIBOMPTARGET_NVPTX_ENABLE_BCLIB=true
 * -DLIBOMPTARGET_NVPTX_CUDA_COMPILER=${PREFIX}/bin/clang
 * -DLIBOMPTARGET_NVPTX_BC_LINKER=${PREFIX}/bin/llvm-link

Installing from releases
Hardware/Operating system
 * Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
 * Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card here.
 * Linux (ubuntu 18.04)

Prerequisites
sudo apt update sudo apt install build-essential sudo apt install cmake sudo apt install -y libelf-dev libffi-dev sudo apt install -y pkg-config

Install CUDA 10.2, using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork

Assuming you current path is
 * /home/ubuntu/omp5-gpu-llvm

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /" sudo apt-get update sudo apt-get -y install cuda

After installation, export two environment variables, vim ~/.bashrc to add two lines
 * export PATH=$PATH:/usr/local/cuda-10.2/bin/
 * export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/

Get source packages
Three steps to download, untar, and put them into the right locations. wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/llvm-10.0.0.src.tar.xz wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/clang-10.0.0.src.tar.xz wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/openmp-10.0.0.src.tar.xz wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/compiler-rt-10.0.0.src.tar.xz

tar xf llvm-10.0.0.src.tar.xz tar xf clang-10.0.0.src.tar.xz tar xf openmp-10.0.0.src.tar.xz tar xf compiler-rt-10.0.0.src.tar.xz

mv clang-10.0.0.src llvm-10.0.0.src/tools/clang mv openmp-10.0.0.src llvm-10.0.0.src/projects/openmp mv compiler-rt-10.0.0.src llvm-10.0.0.src/projects/compiler-rt

In the end, the directory layout should look like
 * llvm-10.0.0.src
 * tools/clang
 * projects/openmp
 * projects/compiler-rt

Build the Compiler with OpenMP offloading support
You need to know the Compute Capability version of your GPU. https://developer.nvidia.com/cuda-gpus lists such information. For example, some typical GPUs and their CC versions are:
 * Tesla K80	3.7, sm_37
 * Tesla P100	6.0, sm_37
 * Tesla V100	7.0, sm_37

mkdir build cd build

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$(pwd)/../install \ -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_37 \ -DCMAKE_C_COMPILER=gcc   \ -DCMAKE_CXX_COMPILER=g++  \ -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 ../llvm-10.0.0.src
 * 1) this step is to generate a make file using cmake. picking gcc/g++ as the compiler


 * 1) the screen output of the step above should show the following info:
 * 2) -- Found LIBOMPTARGET_DEP_CUDA_DRIVER: /usr/lib/x86_64-linux-gnu/libcuda.so
 * 3) -- LIBOMPTARGET: Building offloading runtime library libomptarget.
 * 4) -- LIBOMPTARGET: Building CUDA offloading plugin.
 * 5) -- LIBOMPTARGET: Building x86_64 offloading plugin.

make -j6

make install -j6 After the installation, you should expand your PATH and LD_LIBRARY_PATH again
 * export PATH=$PATH:/home/ubuntu/omp5-gpu-llvm/install/bin
 * export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ubuntu/omp5-gpu-llvm/install/lib

Rebuild the OpenMP runtime libraries
You should use the freshly installed clang to rebuild the OpenMP runtime library

cd /home/ubuntu/omp5-gpu-llvm mkdir build-openmp cd build-openmp

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$(pwd)/../install \ -DCMAKE_C_COMPILER=$(pwd)/../install/bin/clang \ -DCMAKE_CXX_COMPILER=$(pwd)/../install/bin/clang++ \ -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 \ ../llvm-10.0.0.src/projects/openmp

make -j6 make install -j6

Test the installation
save the following code into a file named ongpu.c


 * 1) include 
 * 2) include 

int main { int runningOnGPU = 0; /* Test if GPU is available using OpenMP4.5 */ {   if (omp_is_initial_device == 0) runningOnGPU = 1; } /* If still running on CPU, GPU must not be available */ if (runningOnGPU) printf("### Able to use the GPU! ### \n"); else printf("### Unable to use the GPU, using CPU! ###\n");
 * 1) pragma omp target map(from:runningOnGPU)

return 0; }

Compile and run it clang -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda ongpu.c

./a.out


 * 1) Able to use the GPU! ###

error while loading libomp.so
./a.out: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory solution
 * export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/pathTo/installed-llvm/lib