CUDA Programming Model¶
This episode introduces the CUDA programming model. It starts with the terminology — host, device, kernel, grid, thread block, warp — and a complete first CUDA program in C/C++ and Fortran. It then shows how device code is written as a kernel and launched from host code with the triple-chevron syntax, and what the execution configuration of grids and thread blocks means.
The second half explains why the grid of thread blocks exists and how it maps to the streaming multiprocessors of the GPU, how the SIMT model lets every thread pick its own share of the whole data set through the built-in index variables, and how a loop is moved to the GPU step by step — from one block to a grid-stride loop, with vector addition as the guiding example. It closes with the differences between CUDA Fortran and CUDA C/C++ and a kernel-launch exercise.
Questions
How does a GPU run thousands of threads at once, and how is that different from threads on a CPU?
What actually happens at the moment a kernel is launched?
If every thread runs the same code, how does each one know which data to work on?
Objectives
Understand the SIMT (Single Instruction Multiple Thread) execution model
Know the key terminology: host, device, kernel, grid, thread block, warp
Write and launch a simple CUDA kernel in C/C++ and Fortran
Understand the execution configuration (grid and block dimensions)
Use built-in variables to distribute work across threads
Instructor note
45 min teaching
15 min exercises
Terminology¶
Before diving into code, let us establish the key terms:
Host — The entire system the GPU is built into, including CPU, main memory, network interface card, etc.
Device — The GPU, including device memory. Often used interchangeably with “GPU”.
System/main memory — Main memory of the host (CPU RAM).
Device/global memory — Main memory of the device (GPU VRAM).
Host code — Code that executes on the host (CPU).
Device code — Code that executes on the device (GPU).
(Streaming) multiprocessor (SM) — A subdivision of the GPU, comparable to a CPU core. Sometimes referred to as a “GPU core”.
A first CUDA program¶
A CUDA program is usually organised as follows:
A regular host program that controls overall program flow, manages GPU memory allocations, transfers data between host and device, and launches kernels on the GPU.
Device code containing functions (called kernels if callable from host code) that are suitable for execution on the GPU.
This is the accelerator programming model: the host orchestrates the overall program flow, and the device executes the compute-heavy work that the host tells it to do. Host and device can work independently on their work queues, so synchronisation is necessary.
Here is a complete CUDA program in C and Fortran side by side:
Type-Along
1#include <stdio.h>
2#include <stdlib.h>
3#include <cuda_runtime.h>
4
5#define BLOCKSIZE 256
6#define D 100
7
8__global__ void init(double* X, double* Y) {
9 int i = blockIdx.x * blockDim.x + threadIdx.x;
10 if (i < D) {
11 X[i] = i;
12 Y[i] = 2 * i;
13 }
14}
15
16int main(void) {
17 dim3 gridDimensions((BLOCKSIZE - 1 + D) / BLOCKSIZE);
18 dim3 blockDimensions(BLOCKSIZE);
19
20 double *X_d, *Y_d;
21 cudaMallocManaged((void**)&X_d, D * sizeof(double));
22 cudaMallocManaged((void**)&Y_d, D * sizeof(double));
23
24 init<<<gridDimensions, blockDimensions>>>(X_d, Y_d);
25 cudaDeviceSynchronize();
26 cudaFree(X_d);
27 cudaFree(Y_d);
28}
1attributes(global) subroutine init(x, y)
2 double precision, device :: x(:), y(:)
3 integer :: i
4 i = threadIdx%x + (blockIdx%x - 1) * blockDim%x
5 if (i <= size(x)) then
6 x(i) = i - 1
7 y(i) = 2 * (i - 1)
8 end if
9end subroutine init
10
11program main
12 USE cudafor
13 IMPLICIT NONE
14 INTEGER :: D = 100, blocksize = 256, istat
15 DOUBLE PRECISION, managed, ALLOCATABLE, DIMENSION(:) :: X_d, Y_d
16
17 type(dim3) :: gridDimensions, blockDimensions
18 gridDimensions = dim3((blocksize - 1 + D) / blocksize, 1, 1)
19 blockDimensions = dim3(blocksize, 1, 1)
20
21 ALLOCATE(X_d(D), Y_d(D))
22 call init<<<gridDimensions, blockDimensions>>>(X_d, Y_d)
23 istat = cudaDeviceSynchronize()
24 deallocate(X_d, Y_d)
25end program main
How to create device code — kernels¶
To create code that runs on the GPU:
Include the CUDA runtime API with
#include <cuda_runtime.h>(C) or load the moduleUSE cudafor(Fortran).Mark functions for GPU execution with
__global__(C) orattributes(global)(Fortran). Such a function is called a kernel.Variables declared in a kernel are private to each executing thread and stored in registers by default.
Kernels cannot have return values.
In Fortran, kernels must be defined inside modules.
Note
If you compile with nvcc, the header cuda_runtime.h is included automatically for all .cu files, so the explicit #include is technically optional there. Keep it anyway: other compilers and IDE tooling do not add it for you.
#include <cuda_runtime.h>
#define D 100
__global__ void init(double* X, double* Y) {
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < D) {
X[i] = i;
Y[i] = 2 * i;
}
}
module kernel_mod
attributes(global) subroutine init(x, y)
double precision, device :: x(:), y(:)
integer :: i
i = threadIdx%x + (blockIdx%x - 1) * blockDim%x
if (i <= size(x)) then
x(i) = i - 1
y(i) = 2 * (i - 1)
end if
end subroutine init
end module kernel_mod
How to call device code — kernel launch¶
Kernels are launched from host code using the triple chevron syntax:
init<<<gridDimensions, blockDimensions>>>(X_d, Y_d);
call init<<<gridDimensions, blockDimensions>>>(X_d, Y_d)
The pair (gridDimensions, blockDimensions) is called the execution configuration.
Important
Kernels are executed asynchronously with respect to the host. The host continues executing after launching a kernel without waiting for it to finish. Explicit synchronisation (e.g., cudaDeviceSynchronize()) may be necessary.
Execution configuration — grids and thread blocks¶
The execution configuration arguments gridDimensions and blockDimensions are either unsigned integers or of type dim3. An integer num_threads is automatically converted to dim3(num_threads, 1, 1) in C.
Caution
In Fortran, you must set every unused entry of a dim3 to 1 yourself, e.g., type(dim3) :: blockDimensions = dim3(32, 1, 16).
The total number of threads launched is:
This execution model is called Single Instruction Multiple Thread (SIMT) — a single kernel launch starts many threads. NVIDIA GPUs can launch thousands of threads in parallel (they are massively parallel).
Why grids of thread blocks?¶
The GPU hardware is organised into streaming multiprocessors (SMs). Each SM can execute one or more thread blocks simultaneously. The grid/block structure maps naturally to this hardware:
Thread blocks are assigned to SMs.
Within an SM, threads are grouped into warps of 32 threads that execute in lock-step.

Why not one enormous block?
A thread block is assigned to a single SM, and a GPU has many SMs. So what happens to the rest of the GPU if you launch your whole problem as one block of 1024 threads?
Now take the opposite extreme — one thread per block. Every block still has to be scheduled onto an SM, and threads only cooperate within a block. What does that cost you?
SIMT — how does it work?¶
Variables declared in a kernel are private to each thread and stored in registers by default. All threads execute (almost) the same instructions but on different data.
Each thread can determine which portion of data to work on using the built-in variables:
C/C++ |
Fortran |
Description |
|---|---|---|
|
|
Thread index within the block in x dimension |
|
|
Block index within the grid in x dimension |
|
|
Number of threads per block in x dimension |
|
|
Number of blocks in the grid in x dimension |
(Similarly for .y and .z dimensions.)
Counting starts from 0 in C/C++ and 1 in Fortran.
A typical use case is to split a large loop (where each iteration is independent) into chunks distributed over all threads.
Tip
In many cases, a chunk size of 1 per thread works well, i.e. one thread executes one loop iteration. The vector-addition example below starts from exactly this case and then generalises it with a loop.
Key rules for thread blocks:
Threads within a block can communicate with each other (via shared memory), but not with threads from different blocks.
The three dimensions (x, y, z) are mostly for convenience, not necessarily for performance. For example,
dim3(8,2,2),dim3(16,1,2), anddim3(4,4,2)all have 32 threads.
Caution
The number of threads per block should be a multiple of 32 (the warp size). Otherwise, performance will typically drop significantly.
The order of execution of blocks is not guaranteed. Blocks can be scheduled in any order and may be executed in parallel or sequentially.
Vector addition — from one block to a grid-stride loop¶
Guiding example: add two vectors, \(c \leftarrow c + a\) with \(c, a \in \mathbb{R}^d\). On the CPU this is a single loop over the elements. To move it to the GPU we have to take care of three things:
Create a kernel of the algorithm (the vector addition).
Make data available on the GPU.
Consider the needed synchronisation — recall the asynchronous kernel execution.
The following four steps show how the loop is distributed over threads and thread blocks using the built-in variables. Click the step buttons to advance.
Step 1 — threads only¶
Simplest case: the vector is not longer than 1024 elements and we use exactly one thread per element of the vector, in a single block.
// c = c+a
__global__ void compute_kernel(float* c, float* a, size_t vector_size){
int index = threadIdx.x;
if( index < vector_size){
c[index] += a[index];
}
}
int main(void){
// other code
int vector_size = ...; // <=1024
compute_kernel<<<1,vector_size>>>(vector_c, vector_a, vector_size);
// other code
}
! c = c+a
module vector_add_mod
contains
attributes(global) subroutine compute_kernel(c, a, vector_size)
integer, value :: vector_size
real, dimension(:), device :: c, a
integer :: index
index = threadIdx%x
if (index <= vector_size) then
c(index) = c(index) + a(index)
end if
end subroutine compute_kernel
end module vector_add_mod
program main
use cudafor
use vector_add_mod
! other code
! vector_size <= 1024
call compute_kernel<<<1,vector_size>>>(vector_c, vector_a, vector_size)
! other code
end program main
Step 2 — loop with threads¶
In general we can do the addition using a loop in each thread if we cannot (or do not want to) generate as many threads as the vector is long. The thread block size is fixed to 256 threads.
// c = c+a
__global__ void compute_kernel(float* c, float* a, size_t vector_size){
for (int index = threadIdx.x; index<vector_size;
index+=blockDim.x){
c[index] += a[index];
}
}
int main(void){
// other code
int vector_size = ...; // arbitrary
// thread block size fixed to 256 threads
compute_kernel<<<1, 256>>>(vector_c, vector_a, vector_size);
// other code
}
! c = c+a
module vector_add_mod
contains
attributes(global) subroutine compute_kernel(c, a, vector_size)
integer, value :: vector_size
real, dimension(:), device :: c, a
integer :: index
do index = threadIdx%x, vector_size, blockDim%x
c(index) = c(index) + a(index)
end do
end subroutine compute_kernel
end module vector_add_mod
program main
use cudafor
use vector_add_mod
! other code
! vector_size arbitrary; thread block size fixed to 256 threads
call compute_kernel<<<1, 256>>>(vector_c, vector_a, vector_size)
! other code
end program main
Step 3 — blocks and threads¶
We use multiple thread blocks (each thread block with blockDim.x = 256 threads). If the vector is not too long, we can just cover it with sufficiently many blocks.
// c = c+a
__global__ void compute_kernel(float* c, float* a, size_t vector_size){
int index = threadIdx.x + blockDim.x*blockIdx.x;
if( index < vector_size){
c[index] += a[index];
}
}
int main(void){
// other code
int vector_size = ...; // arbitrary
// thread block size fixed to 256 threads
compute_kernel<<<(vector_size+256-1)/256, 256>>>(vector_c, vector_a, vector_size);
// other code
}
! c = c+a
module vector_add_mod
contains
attributes(global) subroutine compute_kernel(c, a, vector_size)
integer, value :: vector_size
real, dimension(:), device :: c, a
integer :: index
index = threadIdx%x + blockDim%x*(blockIdx%x-1)
if (index <= vector_size) then
c(index) = c(index) + a(index)
end if
end subroutine compute_kernel
end module vector_add_mod
program main
use cudafor
use vector_add_mod
! other code
! vector_size arbitrary; thread block size fixed to 256 threads
call compute_kernel<<<(vector_size+256-1)/256, 256>>>(vector_c, vector_a, vector_size)
! other code
end program main
Step 4 — looping with blocks and threads¶
We use multiple thread blocks (each block with blockDim.x = 256 threads) and loop over the elements in each thread. The loop stride is the total number of threads in the grid, blockDim.x * gridDim.x; this pattern is called a grid-stride loop and works for any vector length with a fixed execution configuration.
// c = c+a
__global__ void compute_kernel(float* c, float* a, size_t vector_size){
for (int index = threadIdx.x + blockDim.x*blockIdx.x;
index<vector_size; index+=blockDim.x*gridDim.x){
c[index] += a[index];
}
}
int main(void){
// other code
int vector_size = ...; // arbitrary
// thread block size fixed to 256 threads, number of blocks to 1024
compute_kernel<<<1024, 256>>>(vector_c, vector_a, vector_size);
// other code
}
! c = c+a
module vector_add_mod
contains
attributes(global) subroutine compute_kernel(c, a, vector_size)
integer, value :: vector_size
real, dimension(:), device :: c, a
integer :: index
do index = threadIdx%x + blockDim%x*(blockIdx%x-1), vector_size, blockDim%x*gridDim%x
c(index) = c(index) + a(index)
end do
end subroutine compute_kernel
end module vector_add_mod
program main
use cudafor
use vector_add_mod
! other code
! vector_size arbitrary; thread block size fixed to 256 threads, number of blocks to 1024
call compute_kernel<<<1024, 256>>>(vector_c, vector_a, vector_size)
! other code
end program main
Differences between Fortran and C/C++¶
CUDA Fortran is based on CUDA for C/C++. Most functions that exist for C/C++ also exist for Fortran. Key differences:
In Fortran, there is sometimes the option to use native Fortran arguments (e.g., arrays) or
TYPE(C_PTR)/TYPE(C_DEVPTR)pointers.When native Fortran arguments are used, counts are in elements. When
TYPE(C_PTR)/TYPE(C_DEVPTR)is used, counts are typically in bytes.
Attention
Counting starts at 0 in C/C++ and 1 in Fortran. Always pay attention, especially for library calls (cuBLAS, cuSPARSE, etc.).
Exercise: Kernel launch¶
Kernel Launch
The following program counts the number of threads that execute and records information from specific threads. Complete the TODO items:
Add the kernel specifier (
__global__in C/C++ /attributes(global)in CUDA Fortran) to the function.Define the execution configuration (grid and block dimensions).
Complete the
ifconditions: the thread with the lowest sum of all indices should execute the first block, and the thread with the highest sum should execute the second block.
Hint: remember that block and grid dimensions can be read from blockDim.x, gridDim.x, etc. (in CUDA Fortran: blockDim%x, gridDim%x).
Downloadable templates: C/C++ — CUDA Fortran
Warning
There are limits on the number of threads per block (at most 1024 in total, on every current GPU) and on the number of blocks in a grid (2^31 − 1 in the x direction, 65535 in y and z). If you exceed them, the launch fails with a configuration error and the kernel does not run — the next episode shows how to detect this.
#include <cstdio>
#include "cuda_runtime.h"
__managed__ int counter{0};
__managed__ dim3 blockDims_;
__managed__ dim3 gridDims_;
__managed__ int lowest_index_sum{-1};
__managed__ int highest_index_sum{-1};
// CUDA Kernel //
// TODO: Add the kernel specifier for execution on GPU
void count_threads(int* counter) {
// count number of threads
atomicAdd(counter,1);
//++counter;
// TODO: only the thread with lowest possible sum of thread and block indeces should execute the block
if(______){
blockDims_ = blockDim;
gridDims_ = gridDim;
lowest_index_sum = threadIdx.x+threadIdx.y+threadIdx.z+blockIdx.x+blockIdx.y+blockIdx.z;
}
// TODO: only the thread with highest possible sum of thread and block indeces should execute the block
if(_____){
highest_index_sum = threadIdx.x+threadIdx.y+threadIdx.z+blockIdx.x+blockIdx.y+blockIdx.z;
}
}
// CPU code //
int main() {
// TODO: specify the execution configuration
count_threads<<<________>>>(&counter);
// We need to wait until the kernel finishes.
cudaDeviceSynchronize();
if(counter==0){
printf("Kernel did not work correctly.\n");
}
printf("Program started %d = %dx%d thread(s) grouped in %d thread blocks "
"(grid layout was x=%d,y=%d,z=%d) "
"with %d threads per block "
"(block had dimensions x=%d,y=%d,z=%d).\n",
counter, gridDims_.x*gridDims_.y*gridDims_.z, blockDims_.x*blockDims_.y*blockDims_.z,
gridDims_.x*gridDims_.y*gridDims_.z, gridDims_.x,gridDims_.y,gridDims_.z,
blockDims_.x*blockDims_.y*blockDims_.z, blockDims_.x,blockDims_.y,blockDims_.z);
printf("This information was written by thread with sum of indices equals %d.\n", lowest_index_sum);
printf("Thread with highest sum of indices has sum equals %d.\n", highest_index_sum);
// exit //
return 0;
}
Compile with nvcc -o exe exercise_kernel-launch.cu and run with ./exe. Test different grid and block layouts and check that the results match your expectations.
! Module containing kernel
module counting_kernel
use cudafor
implicit none
contains
! CUDA Kernel
! TODO: Add the kernel specifier for execution on GPU
subroutine count_threads(counter, blockDims_, gridDims_, lowest_index_sum, highest_index_sum)
type(dim3), managed :: blockDims_
type(dim3), managed :: gridDims_
integer, managed :: counter,lowest_index_sum,highest_index_sum
integer :: old_val
! count number of threads
old_val = atomicAdd(counter,1)
! counter = counter +1
! TODO: only the thread with lowest possible sum of thread and block indeces should execute the block
if (_____) then
blockDims_ = blockDim
gridDims_ = gridDim
lowest_index_sum = threadIdx%x+threadIdx%y+threadIdx%z+blockIdx%x+blockIdx%y+blockIdx%z
end if
! TODO: only the thread with highest possible sum of thread and block indeces should execute the block
if(_____) then
highest_index_sum = threadIdx%x+threadIdx%y+threadIdx%z+blockIdx%x+blockIdx%y+blockIdx%z
end if
end subroutine
end module counting_kernel
! CPU code
program main
use cudafor
use counting_kernel
implicit none
! TODO: specify the execution configuration
___ :: grid_definition, thread_block_definition
integer :: istat
integer, managed :: counter,lowest_index_sum,highest_index_sum
! used to store the actual configuration when running
type(dim3), managed :: blockDims_
type(dim3), managed :: gridDims_
counter = 0
lowest_index_sum = -1
highest_index_sum = -1
! TODO: specify the execution configuration
grid_definition = ___
thread_block_definition = ___
call count_threads<<<grid_definition,thread_block_definition>>>(counter, blockDims_, gridDims_, lowest_index_sum, highest_index_sum)
istat = cudaGetLastError()
if (istat /= cudasuccess) then
print *, cudaGetErrorString(istat)
end if
! We need to wait until the kernel finishes.
istat = cudaDeviceSynchronize()
if (istat /= cudasuccess) then
print *, cudaGetErrorString(istat)
end if
if (counter==0) then
print *, "Kernel did not work correctly.\n"
else
1 format (A,I0,A,I0,A,I0,A,I0,A,A,I0,A,I0,A,I0,A,A,I0,A,I0,A,I0,A,I0,A)
print 1, "Program started ", counter, " = ", gridDims_%x*gridDims_%y*gridDims_%z, "x", &
blockDims_%x*blockDims_%y*blockDims_%z," thread(s) grouped in ",gridDims_%x*gridDims_%y*gridDims_%z," thread blocks ", &
"(grid layout was x=",gridDims_%x,",y=",gridDims_%y,",z=",gridDims_%z,") ", &
"with ",blockDims_%x*blockDims_%y*blockDims_%z," threads per block (block had dimensions x=",blockDims_%x,",y=",blockDims_%y,",z=",blockDims_%z,")."
end if
print '(A,I0)', "This information was written by thread with sum of indices equals ", lowest_index_sum
print '(A,I0)', "Thread with highest sum of indices has sum equals ", highest_index_sum
! exit
end program main
Compile with nvfortran -o exe exercise_kernel-launch.cuf and run with ./exe. Test different grid and block layouts and check that the results match your expectations.
Solution
__global__ void count_threads(int* counter) {
atomicAdd(counter, 1);
// Lowest sum: all indices are 0
if (threadIdx.x + threadIdx.y + threadIdx.z
+ blockIdx.x + blockIdx.y + blockIdx.z == 0) {
blockDims_ = blockDim;
gridDims_ = gridDim;
lowest_index_sum = 0;
}
// Highest sum: all indices at their maximum
if (threadIdx.x == blockDim.x - 1 && threadIdx.y == blockDim.y - 1
&& threadIdx.z == blockDim.z - 1
&& blockIdx.x == gridDim.x - 1 && blockIdx.y == gridDim.y - 1
&& blockIdx.z == gridDim.z - 1) {
highest_index_sum = threadIdx.x + threadIdx.y + threadIdx.z
+ blockIdx.x + blockIdx.y + blockIdx.z;
}
}
int main() {
count_threads<<<dim3(2, 2, 1), dim3(32, 4, 1)>>>(&counter);
cudaDeviceSynchronize();
// ...
}
Full solution: exercises/solutions/exercise_kernel-launch.cu.
attributes(global) subroutine count_threads(counter, blockDims_, gridDims_, lowest_index_sum, highest_index_sum)
type(dim3), managed :: blockDims_, gridDims_
integer, managed :: counter, lowest_index_sum, highest_index_sum
integer :: old_val
old_val = atomicAdd(counter,1)
! In CUDA Fortran, thread/block indices are 1-based — lowest sum is all-1s
if (threadIdx%x==1 .and. threadIdx%y==1 .and. threadIdx%z==1 &
.and. blockIdx%x==1 .and. blockIdx%y==1 .and. blockIdx%z==1) then
blockDims_ = blockDim
gridDims_ = gridDim
lowest_index_sum = threadIdx%x+threadIdx%y+threadIdx%z+blockIdx%x+blockIdx%y+blockIdx%z
end if
! Highest sum — all indices at their maximum
if (threadIdx%x==blockDim%x .and. threadIdx%y==blockDim%y .and. threadIdx%z==blockDim%z &
.and. blockIdx%x==gridDim%x .and. blockIdx%y==gridDim%y .and. blockIdx%z==gridDim%z) then
highest_index_sum = threadIdx%x+threadIdx%y+threadIdx%z+blockIdx%x+blockIdx%y+blockIdx%z
end if
end subroutine
! execution configuration in the program block
type(dim3) :: grid_definition, thread_block_definition
grid_definition = dim3(16,16,16)
thread_block_definition = dim3(2,16,16)
call count_threads<<<grid_definition,thread_block_definition>>>( &
counter, blockDims_, gridDims_, lowest_index_sum, highest_index_sum)
Full solution: exercises/solutions/exercise_kernel-launch.cuf.
Keypoints
A CUDA program consists of host code (CPU) and device code (GPU kernels)
Kernels are launched with the triple chevron syntax
<<<gridDim, blockDim>>>The SIMT model executes the same kernel across many threads, each identified by built-in index variables
Thread blocks are assigned to streaming multiprocessors; threads within a block can communicate
The number of threads per block should be a multiple of 32 (the warp size)