Kernel Rules and Error Handling¶
This episode covers the rules for writing kernels and device functions. It revisits the three function execution space specifiers — __global__, __device__ and __host__ — and their CUDA Fortran equivalents, the rules for argument passing, and what a kernel may and may not do in C/C++ and in Fortran, including which language features are available in device code and which are not.
The second part is about error handling. C++ exceptions are not supported in device code, so CUDA uses error codes: the episode explains how the runtime API reports them, why errors from asynchronous calls and kernel launches only surface at a later blocking call, and how an error-checking macro such as cudaVerify makes checking every call practical. It closes with an exercise on calling a helper function from both host and device code.
Questions
Which parts of C++ or Fortran can you actually use inside a kernel — and which will the compiler reject?
If a kernel launch returns no error code, how do you find out that it failed?
Why can an error caused by one call surface at a completely different line?
Objectives
Understand the distinction between
__global__,__device__, and__host__functionsKnow the rules and restrictions for kernels and device functions
Know what language features are available in device code
Implement proper error handling for CUDA API calls and kernel launches
Instructor note
30 min teaching
15 min exercises
Kernels and device functions revisited¶
CUDA provides three function execution space specifiers that control where a function can be called from and where it executes:
Specifier |
Called from |
Executes on |
Notes |
|---|---|---|---|
|
Host (or device with dynamic parallelism) |
Device |
Kernel — requires execution configuration |
|
Device |
Device |
Device function — can be called from kernels or device functions |
|
Host |
Host |
Regular host function (default if no specifier) |
|
Both |
Either |
Compiled for both host and device |
In Fortran, the equivalents are attributes(global), attributes(device), attributes(host), and attributes(host, device).
Key rules for kernels and device functions:
Kernels (
__global__) cannot have return values and need an execution configuration to be called.Functions with
__host__ __device__are compiled for both execution spaces. The compiler generates code for both host and device. Exception:constexpr/constevalfunctions may, with restrictions, be called across execution spaces.Only device functions may be called from kernels or other device functions.
Variables passed by value to kernels are stored in constant memory and limited to 4 KB (or 32,764 bytes starting with Volta and CUDA 12.1). They are automatically copied — no explicit allocation or transfer needed.
Variables declared in a kernel are private to each thread and stored in registers (or so called local memory if the compiler chooses).
Warning
Pointers or passed-by-reference variables must point to device-accessible memory (device memory, managed (unified) memory, or mapped pinned host memory). Otherwise the kernel faults with an illegal-memory-access error — see the error-handling section below.
Argument passing¶
Argument passing works as usual in each language:
C/C++: Arguments are passed by value by default. Use pointers for pass-by-reference.
Fortran: Arguments are passed by reference by default. Use the
valueattribute for pass-by-value.
Tip
In CUDA Fortran, declare scalar kernel arguments such as an array length with the value attribute (integer, value :: vector_size, as in the vector-addition kernels of the previous episode); arrays are passed by reference as usual.
Rules for kernels (C/C++)¶
Kernels (
__global__functions) do not support recursion.Non-trivially copyable or non-trivially-destructible argument types are allowed when launched from host code, but the processing does not follow the standard C++ model (see the programming guide for details).
Kernels cannot have a variable number of arguments.
Kernel parameters should be passed by value or by pointer, not by reference (
T&): the launch copies the raw bytes of each argument, and a reference is only an address that must again point to device-accessible memory. Pass by reference can work for unified memory, see NVIDIA’s Unified Memory. Older versions of the Programming Guide state that__global__parameters cannot be pass-by-reference; the current one no longer does.
Rules for kernels (Fortran)¶
Return values of functions or
intent(out)/intent(inout)variables cannot have thevalueattribute.A subroutine or function with the
deviceorglobalattribute may not be recursive, pure, or elemental.Device and global subroutines/functions must appear within a Fortran module, and device functions may only be called from device subprograms in the same module.
They may not contain another subprogram.
They may not be contained in another subroutine or function.
What can be used in kernels¶
Each hardware generation supports a specific set of features, called compute capability or CC for short. They are mostly inclusive, so newer devices support in most cases features of older devices with lower compute capability.
Available since early CUDA¶
All mathematical operators
Control flow constructs (
if,for,while,case,goto)Single-precision transcendental mathematical functions (see the mathematical standard library functions section of the Programming Guide [3])
Calls to device functions/subroutines
Pointers (C), structs/classes (C), statically-sized arrays (C, Fortran)
Assumed-shape arrays as kernel arguments (Fortran)
CUDA-specific built-in functions
C++ templates and function overloading
Available on newer devices (CC ≥ 2.0, CUDA ≥ 4)¶
Double-precision floating point and DP mathematical functions
__float128mathematical functions (CC 10+, if host compiler supports__float128or_Float128, no dedicated hardware)Recursive function calls (C)
Allocatable arrays, pointers
Function pointers
C++ polymorphic classes
printf(C), limitedprintsupport (Fortran)malloc,new(no Fortran equivalent yet)R-value references
Static variables declared in device functions
What cannot be used in kernels¶
Warning
C99/C++ dynamically-sized
[]-arrays (VLAs)System calls, file I/O, general memory management (
fopen,fprintf,system, etc.)Run Time Type Information (RTTI), e.g., no
dynamic_castArguments with virtual base classes
long doublefloatsAllocatable arrays, pointers,
valuewithintent(out)orintent(inout)(Fortran)Recursive, pure, elemental function calls, optional function arguments (Fortran)
saveattribute (Fortran)
Error handling¶
C++ exception handling is not supported in device code. Instead, CUDA uses error codes:
All runtime API functions return a
cudaError_t(C) orinteger(F) error code.If no error occurred, the returned value is
cudaSuccess.cudaGetErrorString(error)returns a human-readable description of theerror.cudaGetLastError()returns the last error and resets it tocudaSuccess.cudaPeekAtLastError()returns the last error without resetting it.
__host__ __device__ const char* cudaGetErrorName(cudaError_t error)
__host__ __device__ cudaError_t cudaGetLastError(void)
__host__ __device__ cudaError_t cudaPeekAtLastError(void)
function cudaGetErrorString(errcode)
integer, intent(in) :: errcode
character*(*) :: cudaGetErrorString
integer function cudaGetLastError()
integer function cudaPeekAtLastError()
Important caveats¶
Caution
Asynchronous calls cannot return errors while the code runs (except for launch errors at entry).
Kernel launches using the
<<<>>>syntax do not return any error code at all.Errors from asynchronous functions or kernel launches typically surface on the next blocking call (e.g.,
cudaDeviceSynchronizeorcudaMemcpy). This meanscudaMemcpymay report an “unspecified launch failure” caused by a prior kernel. The launch itself only enqueues the kernel and returns at once; when the kernel later faults, the runtime records a sticky error which the next API call — often acudaMemcpyfar away in the code — returns.A common error is “unspecified launch failure” (on current GPUs and drivers more often reported as “an illegal memory access was encountered”), which usually means the kernel crashed — most commonly due to invalid memory access (array bounds violation, accessing host memory from device code, etc.).
Error-checking macro¶
In practice, wrapping CUDA API calls in an error-checking macro is the most convenient approach. Here is a simple macro used in this course:
#ifndef NDEBUG
#define cudaVerify(x) do { \
cudaError_t __cu_result = x; \
if (__cu_result != cudaSuccess) { \
fprintf(stderr, \
"%s:%i: error: cuda function call failed:\n" \
" %s;\nmessage: %s\n", \
__FILE__, __LINE__, \
#x, cudaGetErrorString(__cu_result)); \
exit(1); \
} \
} while(0)
#endif
Usage:
#include "cuda_utils.h"
cudaVerify(cudaMalloc(&ptr, size));
cudaVerify(cudaMemcpy(dst, src, size, cudaMemcpyHostToDevice));
// For kernel launches, use cudaGetLastError after the launch:
myKernel<<<grid, block>>>(args);
cudaVerify(cudaGetLastError());
cudaVerify(cudaDeviceSynchronize());
The complete header — it also contains a cudaVerifyKernel variant and a simple timer — is cuda_utils.h; place it in the same directory as the .cu file that includes it (once per working directory). The two exercises of this module do not need it, the CUDA Memory Management module uses it in every C/C++ template and solution.
The same approach works in Fortran using a preprocessor.
An error that surfaces at a later cudaMemcpy
The kernel below dereferences a host pointer. The launch itself succeeds — it only enqueues the kernel — and nothing checks the kernel afterwards, so the fault is reported by the first API call that has to wait for the device: the cudaMemcpy several lines further down.
1#include <stdio.h>
2#include "cuda_utils.h"
3
4__global__ void square(const double* in, double* out) {
5 out[0] = in[0] * in[0];
6}
7
8int main(void) {
9 double h_in = 3.0, h_out = 0.0, *d_out;
10 cudaVerify(cudaMalloc((void**)&d_out, sizeof(double)));
11
12 square<<<1, 1>>>(&h_in, d_out); // bug: &h_in is a host address
13
14 // ... other host work, nothing checks the kernel ...
15 cudaVerify(cudaMemcpy(&h_out, d_out, sizeof(double), cudaMemcpyDeviceToHost));
16 printf("%f\n", h_out);
17 return 0;
18}
demo.cu:15: error: cuda function call failed:
cudaMemcpy(&h_out, d_out, sizeof(double), cudaMemcpyDeviceToHost);
message: an illegal memory access was encountered
The message points at line 15, although the bug is in line 12 (older drivers report the same fault as “unspecified launch failure”). Now check the launch directly:
square<<<1, 1>>>(&h_in, d_out);
cudaVerify(cudaGetLastError()); // configuration errors, reported at once
cudaVerify(cudaDeviceSynchronize()); // faults while the kernel runs, pinned to this launch
The error is now reported by the cudaDeviceSynchronize() directly after the launch, which identifies the failing kernel. cudaGetLastError() alone would not catch it: the launch was valid, the kernel crashed later. To see the other kind of error, launch with <<<1, 2048>>> — too many threads per block — and cudaGetLastError() reports “invalid configuration argument” immediately.
Tip
A kernel launch returns no error code of its own. Check it with cudaVerify(cudaGetLastError()) directly after the launch, and with cudaVerify(cudaDeviceSynchronize()) afterwards to catch errors that occur while the kernel runs, as in the usage example above.
Exercise: Device function call¶
Device Function Call
The program below computes the element-wise square root of a vector on both CPU and GPU. The square-root helper is currently only callable from host code.
Tasks:
Make the square-root helper callable from both host and device code (in C/C++: add
__host__ __device__; in CUDA Fortran: addattributes(host, device)).Define the execution configuration for the kernel.
Downloadable templates: C/C++ — CUDA Fortran
#include <cstdio>
#include <cstdlib>
// Define vector length and allocate memory
const int vec_size = 1'000'000;
__managed__ double vector[vec_size];
__managed__ dim3 blockDims_;
__managed__ dim3 gridDims_;
__managed__ double root_dev;
// Forward declaration for use in kernel
// TODO: Make this function callable in device code.
double sqrt_of(double, double=1e-8);
// CUDA Kernel //
__global__ void sqrt_kernel(double* __restrict__ vector, int vec_size) {
int index = threadIdx.x+blockDim.x*blockIdx.x;
if(index < vec_size){
// sqrt_of must be callable from device code
vector[index] = sqrt_of(vector[index]);
}
}
// function will be callable from host and device code
// TODO: Make this function callable in host and device code.
double sqrt_of(double x, double precision){
// Approximate the square root of x up to precision
double approximation=x/2;
double prev_val;
double error = 1.0;
if(x==0.0){
return 0.0;
}
if(x<0){
return -1.0;
}
while(error>precision){
prev_val = approximation;
approximation = 0.5*(approximation + x/approximation);
// error estimate (upper bound)
error = (prev_val*approximation-x)*(prev_val*approximation-x)/(2*prev_val*approximation*approximation);
}
return approximation;
}
// CPU code //
int main() {
double* vector_cpu = (double*) malloc(vec_size*sizeof(double));
// TODO: Define the execution configuration for the kernel sqrt_kernel
const ___ num_threads = ____;
const ___ num_blocks = ____;
// fill vector
for(int i=0; i<vec_size; ++i){
vector[i]=2*i;
}
// compute elementwise square root (approximately) (CPU)
for(int index = 0; index<10; ++index){
vector_cpu[index] = vector[index];
vector_cpu[index] = sqrt_of(vector_cpu[index]);
}
// compute elementwise square root (approximately) (GPU)
sqrt_kernel<<<num_blocks,num_threads>>>(vector, vec_size);
cudaDeviceSynchronize();
// Print first ten roots
for(int i = 0; i<10; ++i){
double error = fabs(vector[i]- vector_cpu[i]);
printf("Sqrt(%d): %f (GPU), %f (CPU), abs. difference error %f\n", 2*i, vector[i], vector_cpu[i], error);
if(error>1e-5){
fprintf(stderr, "Error larger than 1e-5, result wrong.");
return EXIT_FAILURE;
}
}
// exit //
return EXIT_SUCCESS;
}
module numeric_kernels
use cudafor
contains
! CUDA Kernel
attributes(global) subroutine sqrt_kernel(vector, vec_size)
integer, value :: vec_size
real(kind=8), managed, intent(inout) :: vector(0:vec_size-1)
integer :: ind
ind = threadIdx%x-1+blockDim%x*(blockIdx%x-1);
if (ind < vec_size) then
vector(ind) = sqrtOf(vector(ind))
end if
end subroutine
! function will be callable from host and device code
! TODO: Make this function callable in host and device code.
real(kind=8) function sqrtOf(x, prec_para) result (approximation)
! Approximate the square root of x up to precision
real(kind=8), intent(in) :: x
real(kind=8), intent(in), optional :: prec_para
real(kind=8) :: prev_val, error, prec
if(.not. present(prec_para)) then
prec = 1e-8
else
prec = prec_para
end if
approximation = x/2.0
if (x==0.0) then
return
end if
if (x<0) then
approximation = -1.0
return
end if
error = 1.0
do while (error > prec)
prev_val = approximation
approximation = 0.5*(approximation + x/approximation)
! error estimate (upper bound)
error = (prev_val*approximation-x)*(prev_val*approximation-x)/(2*prev_val*approximation*approximation)
end do
end function
end module
! CPU code
program main
use cudafor
use numeric_kernels
! length of vector
integer, parameter :: vec_size = 1000000
real(kind=8), managed :: vector(0:vec_size-1)
real(kind=8) :: vector_cpu(0:vec_size-1)
! TODO: Define the execution configuration for the kernel sqrt_kernel
___, parameter :: num_threads = ___
___, parameter :: num_blocks = ___
integer :: i, istat
real(kind=8) :: error
! fill vector
do i=0, vec_size-1
vector(i)=(2*i);
end do
vector_cpu = vector
! compute elementwise square root (approximately) (CPU)
do i=0,10-1
vector_cpu(i) = sqrtOf(vector_cpu(i))
end do
! compute elementwise square root (approximately) (GPU)
call sqrt_kernel<<<num_blocks,num_threads>>>(vector, vec_size)
istat = cudaDeviceSynchronize()
! Print first ten roots
do i=0,10-1
error = abs(vector(i)-vector_cpu(i))
print *, "Sqrt(",2*i,"): ",vector(i), "(GPU)", vector_cpu(i), "(CPU), abs. difference error", error
if (error > 1e-5) then
stop "Error larger than 1e-5"
end if
end do
! exit
end program main
Solution
Add __host__ __device__ to the function declaration and forward declaration:
__host__ __device__ double sqrt_of(double, double=1e-8); // forward declaration
// ...
__host__ __device__ double sqrt_of(double x, double precision){
// ... (function body unchanged)
}
const int num_threads = 128;
const int num_blocks = (vec_size + num_threads - 1) / num_threads;
Full solution: exercises/solutions/exercise_function-call.cu.
Add attributes(host, device) to the function declaration:
attributes(host, device) real(kind=8) function sqrtOf(x, prec_para) result (approximation)
! ... (function body unchanged)
end function
! execution configuration in the program block
integer, parameter :: num_threads = 128
integer, parameter :: num_blocks = (vec_size + num_threads - 1) / num_threads
Full solution: exercises/solutions/exercise_function-call.cuf.
Keypoints
__global__marks a kernel (called from host, runs on device),__device__marks a device function,__host__ __device__compiles for bothKernels have restrictions: no return values, no recursion, no variable arguments
Most standard mathematical operations work on the GPU; system calls and I/O do not
Always check CUDA error codes — use a macro like
cudaVerifyto wrap API callsKernel launch errors surface asynchronously on the next blocking call