Instructor guide

Why we teach this lesson

Hardly any system today runs without GPU acceleration, so knowing how to program GPUs is essential. Only the native programming model — CUDA for NVIDIA GPUs — gives access to every feature of the hardware and lets you make full use of it. The fundamentals are not limited to NVIDIA GPUs: they also apply to other GPUs and transfer easily.

Intended learning outcomes

After completing this module, learners will be able to:

  • Explain the accelerator programming model and identify which parts of a program run on the CPU vs the GPU

  • Write a CUDA kernel using the __global__ specifier and launch it from host code using the triple-chevron syntax in C/C++ and the equivalent syntax in CUDA Fortran (<<<...>>>)

  • Map thread and block indices to data elements using the built-in threadIdx / blockIdx / blockDim / gridDim variables, and apply the grid-stride loop pattern to handle arbitrary data sizes

  • Apply the function execution space specifiers (__global__, __device__, __host__ __device__) correctly and respect the restrictions that apply to device code

  • Detect and handle CUDA errors using error-checking macros, and understand the asynchronous nature of kernel-launch error reporting

Timing

Episode

Lecture

Exercises

Total

Software setup

10 min

0 min

10 min

1. CUDA Programming Model

45 min

15 min

60 min

2. Kernel Rules and Error Handling

30 min

15 min

45 min

Total

85 min

30 min

115 min

These numbers are estimates: the material has so far been taught from slides, in a different order and without notebooks, which may speed things up. The estimates may be on the low side; re-measure them at the first delivery.

Hardware requirements

An NVIDIA GPU with compute capability 7.5 or newer is recommended; compute capability 7.0 still works but its support is being phased out. Use CUDA Toolkit 12.x: CUDA 13 drops support for these older GPUs, so 12.x is both the floor and, for CC 7.0 hardware, the ceiling. CUDA Fortran requires the NVIDIA HPC SDK (nvfortran). Providing GPU access for a course run (workstations, a training cluster or scheduler-allocated nodes) is the responsibility of the site delivering the course; the number of participants is limited by the number of available GPUs and supervisors. Profiling on compute capability 7.0 is being phased out as well: Nsight Compute 2025.3 and Nsight Systems 2025.4 dropped Volta support, so to profile on such a device use the Nsight versions bundled with HPC SDK 25.7 or with the CUDA 12.x toolchain of later HPC SDKs (profiling is not needed for this module).

Participants need a machine with an NVIDIA GPU — either their own workstation or a node of a cluster — to compile and run the exercises; the software-setup episode tells them so in its first paragraph.

Learner personas

Learners need sufficient knowledge of C/C++ or Fortran. If the exercises are done on the command line, basic shell skills and familiarity with a text editor are also recommended; an IDE that can call nvcc/nvfortran works equally well. Some knowledge about parallel programming is a plus.

How to teach each episode

The lists below are derived from the material of each episode and from the questions that came up in earlier deliveries. Every teaching episode opens with a short introduction, a questions box and an objectives box; the questions work well as a two-minute warm-up before the lecture part.

Software setup (10 min)

  • Emphasise: the exercises are compiled and run by the participants themselves on a machine with an NVIDIA GPU; nvcc for .cu, nvfortran for .cuf, and which flag does what.

  • Run: the checklist at the end of the episode, before starting episode 1.

  • Demo live: compile and run one program on the GPU machine the participants will use, so that they see the module/scheduler commands of that system once.

  • Common questions: which module to load or container to start, and which queue of the batch system to use — all of this is site-specific; refer to the documentation of the system the course runs on.

Episode 1: CUDA Programming Model (45 min + 15 min)

  • Emphasise: the accelerator programming model (host orchestrates, device computes, both run asynchronously), the execution configuration and the built-in index variables, and that threads per block should be a multiple of 32.

  • Run: the type-along (the first CUDA program in both languages), the discussion Why not one enormous block? after the SM diagram, and the kernel-launch exercise.

  • Demo live: the four steps of the vector-addition slideshow, ending with the grid-stride loop.

  • Common questions: what grid, blocks and threads per block are, why this structure exists, and what good defaults are (block size; 256 threads per block in the examples).

Episode 2: Kernel Rules and Error Handling (30 min + 15 min)

  • Emphasise: the three function execution space specifiers and their Fortran equivalents, that kernel arguments must point to device-accessible memory, and that a <<<>>> launch returns no error code.

  • Run: the device-function-call exercise.

  • Demo live: a kernel that dereferences a host pointer, so that the error (“an illegal memory access was encountered” or “unspecified launch failure”) is reported by a later cudaMemcpy or cudaDeviceSynchronize (see the demo in the episode); then how cudaVerify(cudaDeviceSynchronize()) placed directly after the launch pins it to that kernel, and how cudaVerify(cudaGetLastError()) after the launch catches configuration errors such as too many threads per block.

  • Common questions: which C++/Fortran features work in a kernel (use the two lists in the episode), and why an error shows up at a different line than the one that caused it.

Additional teaching recommendations

Preparing exercises

The day before the course:

  • Test that all exercises and their solutions run.

  • Check the software environment, e.g. that cross-compilation works as intended. Profiling tools are not needed for this module.

  • Make sure all materials (module pages, exercise templates and solutions) are reachable by the participants.

Exercise logistics

Learners download the exercise templates from the episode pages, edit them in an editor or IDE of their choice and compile them with nvcc (CUDA C/C++) or nvfortran (CUDA Fortran) on a machine with an NVIDIA GPU; solutions are offered for download next to each template. The two exercises of this module (kernel launch, device function call) are self-contained single files and do not need any additional header. The error-checking macro cudaVerify shown in episode 2 lives in cuda_utils.h; the header is offered for download in episode 2 (it lives in both exercises/templates/ and exercises/solutions/) and has to sit in the same directory as the .cu file that includes it. The CUDA Memory Management module uses it in every C template and solution.

Delivery formats

The module can be delivered on-site or online. The lecture parts are given from the notebook-based episode pages, the exercises are done by the participants on the GPU machine, in a shell or an IDE. Since the episodes are Jupyter notebooks, a planned extension adds compile/run cells so that the episodes can also be worked through interactively on a GPU-equipped JupyterHub; those cells will be excluded from the documentation build and carry pre-recorded output.

Other practical aspects

  • The module is delivered instructor-led with hands-on exercises, online or on-site.

  • The number of participants is limited by the number of available GPUs and supervisors.

  • See the hardware requirements above for the GPU and toolkit versions to provide.

Interesting questions you might get

Most questions concern the grid structure: what grid, blocks and threads per block are, why this structure exists, and — very often — what good defaults are, for the block size but also for other tuning points.

Typical pitfalls

The instructor’s experience is that learners run into three kinds of problems:

  • An incomplete or incorrect understanding of the SIMT model leads to poor performance, incorrect results or crashes.

  • Incorrect synchronisation between host and device.

  • Segmentation faults caused by incorrect memory handling.

The first two are the ones that bite in this module (memory handling is the subject of the CUDA Memory Management module). Where they show up, episode by episode:

  • Episode 1, SIMT model: The number of threads per block should be a multiple of 32 (the warp size); otherwise performance typically drops significantly. The order of execution of blocks is not guaranteed. Exceeding the limits of the execution configuration (1024 threads per block) means the kernel does not execute at all.

  • Episode 1, SIMT model: Counting starts at 0 in C/C++ and 1 in Fortran; this matters especially for library calls (cuBLAS, cuSPARSE, etc.). In Fortran every unused dim3 entry has to be set to 1.

  • Episode 1, synchronisation: Kernels are executed asynchronously with respect to the host; the host continues after the launch, so results are only valid after explicit synchronisation (e.g. cudaDeviceSynchronize()).

  • Episode 2, SIMT model: Not everything can be used in kernels — for example VLAs, system calls, file I/O, RTTI, long double, and in Fortran allocatable arrays, pointers, recursive/pure/elemental procedures, optional arguments and the save attribute.

  • Episode 2, memory handling: Pointers or passed-by-reference variables handed to a kernel must point to device-accessible memory; otherwise the kernel faults with an illegal-memory-access error.

  • Episode 2, synchronisation: Kernel launches with <<<>>> return no error code, and errors from asynchronous calls surface on the next blocking call; an “unspecified launch failure” reported by cudaMemcpy usually means a prior kernel crashed, most commonly due to an invalid memory access.