Kernel Rules and Error Handling

This episode covers the rules for writing kernels and device functions. It revisits the three function execution space specifiers — __global__, __device__ and __host__ — and their CUDA Fortran equivalents, the rules for argument passing, and what a kernel may and may not do in C/C++ and in Fortran, including which language features are available in device code and which are not.

The second part is about error handling. C++ exceptions are not supported in device code, so CUDA uses error codes: the episode explains how the runtime API reports them, why errors from asynchronous calls and kernel launches only surface at a later blocking call, and how an error-checking macro such as cudaVerify makes checking every call practical. It closes with an exercise on calling a helper function from both host and device code.

Questions

  • Which parts of C++ or Fortran can you actually use inside a kernel — and which will the compiler reject?

  • If a kernel launch returns no error code, how do you find out that it failed?

  • Why can an error caused by one call surface at a completely different line?

Objectives

  • Understand the distinction between __global__, __device__, and __host__ functions

  • Know the rules and restrictions for kernels and device functions

  • Know what language features are available in device code

  • Implement proper error handling for CUDA API calls and kernel launches

Instructor note

  • 30 min teaching

  • 15 min exercises

Kernels and device functions revisited

CUDA provides three function execution space specifiers that control where a function can be called from and where it executes:

Specifier

Called from

Executes on

Notes

__global__

Host (or device with dynamic parallelism)

Device

Kernel — requires execution configuration <<<>>>

__device__

Device

Device

Device function — can be called from kernels or device functions

__host__

Host

Host

Regular host function (default if no specifier)

__host__ __device__

Both

Either

Compiled for both host and device

In Fortran, the equivalents are attributes(global), attributes(device), attributes(host), and attributes(host, device).

Key rules for kernels and device functions:

  • Kernels (__global__) cannot have return values and need an execution configuration to be called.

  • Functions with __host__ __device__ are compiled for both execution spaces. The compiler generates code for both host and device. Exception: constexpr/consteval functions may, with restrictions, be called across execution spaces.

  • Only device functions may be called from kernels or other device functions.

  • Variables passed by value to kernels are stored in constant memory and limited to 4 KB (or 32,764 bytes starting with Volta and CUDA 12.1). They are automatically copied — no explicit allocation or transfer needed.

  • Variables declared in a kernel are private to each thread and stored in registers (or so called local memory if the compiler chooses).

Warning

Pointers or passed-by-reference variables must point to device-accessible memory (device memory, managed (unified) memory, or mapped pinned host memory). Otherwise the kernel faults with an illegal-memory-access error — see the error-handling section below.

Argument passing

Argument passing works as usual in each language:

  • C/C++: Arguments are passed by value by default. Use pointers for pass-by-reference.

  • Fortran: Arguments are passed by reference by default. Use the value attribute for pass-by-value.

Tip

In CUDA Fortran, declare scalar kernel arguments such as an array length with the value attribute (integer, value :: vector_size, as in the vector-addition kernels of the previous episode); arrays are passed by reference as usual.

Rules for kernels (C/C++)

  • Kernels (__global__ functions) do not support recursion.

  • Non-trivially copyable or non-trivially-destructible argument types are allowed when launched from host code, but the processing does not follow the standard C++ model (see the programming guide for details).

  • Kernels cannot have a variable number of arguments.

  • Kernel parameters should be passed by value or by pointer, not by reference (T&): the launch copies the raw bytes of each argument, and a reference is only an address that must again point to device-accessible memory. Pass by reference can work for unified memory, see NVIDIA’s Unified Memory. Older versions of the Programming Guide state that __global__ parameters cannot be pass-by-reference; the current one no longer does.

Rules for kernels (Fortran)

  • Return values of functions or intent(out)/intent(inout) variables cannot have the value attribute.

  • A subroutine or function with the device or global attribute may not be recursive, pure, or elemental.

  • Device and global subroutines/functions must appear within a Fortran module, and device functions may only be called from device subprograms in the same module.

  • They may not contain another subprogram.

  • They may not be contained in another subroutine or function.

What can be used in kernels

Each hardware generation supports a specific set of features, called compute capability or CC for short. They are mostly inclusive, so newer devices support in most cases features of older devices with lower compute capability.

Available since early CUDA

  • All mathematical operators

  • Control flow constructs (if, for, while, case, goto)

  • Single-precision transcendental mathematical functions (see the mathematical standard library functions section of the Programming Guide [3])

  • Calls to device functions/subroutines

  • Pointers (C), structs/classes (C), statically-sized arrays (C, Fortran)

  • Assumed-shape arrays as kernel arguments (Fortran)

  • CUDA-specific built-in functions

  • C++ templates and function overloading

Available on newer devices (CC ≥ 2.0, CUDA ≥ 4)

  • Double-precision floating point and DP mathematical functions

  • __float128 mathematical functions (CC 10+, if host compiler supports __float128 or _Float128, no dedicated hardware)

  • Recursive function calls (C)

  • Allocatable arrays, pointers

  • Function pointers

  • C++ polymorphic classes

  • printf (C), limited print support (Fortran)

  • malloc, new (no Fortran equivalent yet)

  • R-value references

  • Static variables declared in device functions

What cannot be used in kernels

Warning

  • C99/C++ dynamically-sized []-arrays (VLAs)

  • System calls, file I/O, general memory management (fopen, fprintf, system, etc.)

  • Run Time Type Information (RTTI), e.g., no dynamic_cast

  • Arguments with virtual base classes

  • long double floats

  • Allocatable arrays, pointers, value with intent(out) or intent(inout) (Fortran)

  • Recursive, pure, elemental function calls, optional function arguments (Fortran)

  • save attribute (Fortran)

Error handling

C++ exception handling is not supported in device code. Instead, CUDA uses error codes:

  • All runtime API functions return a cudaError_t (C) or integer (F) error code.

  • If no error occurred, the returned value is cudaSuccess.

  • cudaGetErrorString(error) returns a human-readable description of the error.

  • cudaGetLastError() returns the last error and resets it to cudaSuccess.

  • cudaPeekAtLastError() returns the last error without resetting it.

__host__ __device__ const char* cudaGetErrorName(cudaError_t error)
__host__ __device__ cudaError_t cudaGetLastError(void)
__host__ __device__ cudaError_t cudaPeekAtLastError(void)

Important caveats

Caution

  • Asynchronous calls cannot return errors while the code runs (except for launch errors at entry).

  • Kernel launches using the <<<>>> syntax do not return any error code at all.

  • Errors from asynchronous functions or kernel launches typically surface on the next blocking call (e.g., cudaDeviceSynchronize or cudaMemcpy). This means cudaMemcpy may report an “unspecified launch failure” caused by a prior kernel. The launch itself only enqueues the kernel and returns at once; when the kernel later faults, the runtime records a sticky error which the next API call — often a cudaMemcpy far away in the code — returns.

  • A common error is “unspecified launch failure” (on current GPUs and drivers more often reported as “an illegal memory access was encountered”), which usually means the kernel crashed — most commonly due to invalid memory access (array bounds violation, accessing host memory from device code, etc.).

Error-checking macro

In practice, wrapping CUDA API calls in an error-checking macro is the most convenient approach. Here is a simple macro used in this course:

cuda_utils.h
#ifndef NDEBUG
#define cudaVerify(x) do {                                   \
    cudaError_t __cu_result = x;                             \
    if (__cu_result != cudaSuccess) {                         \
        fprintf(stderr,                                      \
                "%s:%i: error: cuda function call failed:\n"  \
                "  %s;\nmessage: %s\n",                      \
                __FILE__, __LINE__,                           \
                #x, cudaGetErrorString(__cu_result));          \
        exit(1);                                              \
    }                                                         \
} while(0)
#endif

Usage:

#include "cuda_utils.h"

cudaVerify(cudaMalloc(&ptr, size));
cudaVerify(cudaMemcpy(dst, src, size, cudaMemcpyHostToDevice));

// For kernel launches, use cudaGetLastError after the launch:
myKernel<<<grid, block>>>(args);
cudaVerify(cudaGetLastError());
cudaVerify(cudaDeviceSynchronize());

The complete header — it also contains a cudaVerifyKernel variant and a simple timer — is cuda_utils.h; place it in the same directory as the .cu file that includes it (once per working directory). The two exercises of this module do not need it, the CUDA Memory Management module uses it in every C/C++ template and solution.

The same approach works in Fortran using a preprocessor.

An error that surfaces at a later cudaMemcpy

The kernel below dereferences a host pointer. The launch itself succeeds — it only enqueues the kernel — and nothing checks the kernel afterwards, so the fault is reported by the first API call that has to wait for the device: the cudaMemcpy several lines further down.

demo.cu
 1#include <stdio.h>
 2#include "cuda_utils.h"
 3
 4__global__ void square(const double* in, double* out) {
 5    out[0] = in[0] * in[0];
 6}
 7
 8int main(void) {
 9    double h_in = 3.0, h_out = 0.0, *d_out;
10    cudaVerify(cudaMalloc((void**)&d_out, sizeof(double)));
11
12    square<<<1, 1>>>(&h_in, d_out);   // bug: &h_in is a host address
13
14    // ... other host work, nothing checks the kernel ...
15    cudaVerify(cudaMemcpy(&h_out, d_out, sizeof(double), cudaMemcpyDeviceToHost));
16    printf("%f\n", h_out);
17    return 0;
18}
Output
demo.cu:15: error: cuda function call failed:
  cudaMemcpy(&h_out, d_out, sizeof(double), cudaMemcpyDeviceToHost);
message: an illegal memory access was encountered

The message points at line 15, although the bug is in line 12 (older drivers report the same fault as “unspecified launch failure”). Now check the launch directly:

square<<<1, 1>>>(&h_in, d_out);
cudaVerify(cudaGetLastError());        // configuration errors, reported at once
cudaVerify(cudaDeviceSynchronize());   // faults while the kernel runs, pinned to this launch

The error is now reported by the cudaDeviceSynchronize() directly after the launch, which identifies the failing kernel. cudaGetLastError() alone would not catch it: the launch was valid, the kernel crashed later. To see the other kind of error, launch with <<<1, 2048>>> — too many threads per block — and cudaGetLastError() reports “invalid configuration argument” immediately.

Tip

A kernel launch returns no error code of its own. Check it with cudaVerify(cudaGetLastError()) directly after the launch, and with cudaVerify(cudaDeviceSynchronize()) afterwards to catch errors that occur while the kernel runs, as in the usage example above.

Exercise: Device function call

Device Function Call

The program below computes the element-wise square root of a vector on both CPU and GPU. The square-root helper is currently only callable from host code.

Tasks:

  1. Make the square-root helper callable from both host and device code (in C/C++: add __host__ __device__; in CUDA Fortran: add attributes(host, device)).

  2. Define the execution configuration for the kernel.

Downloadable templates: C/C++ — CUDA Fortran

#include <cstdio>
#include <cstdlib>

// Define vector length and allocate memory
const int vec_size = 1'000'000;
__managed__ double vector[vec_size];
__managed__ dim3 blockDims_;
__managed__ dim3 gridDims_;
__managed__ double root_dev;

// Forward declaration for use in kernel
// TODO: Make this function callable in device code.
double sqrt_of(double, double=1e-8);

// CUDA Kernel //
__global__ void sqrt_kernel(double* __restrict__ vector, int vec_size) {
  int index = threadIdx.x+blockDim.x*blockIdx.x;
  if(index < vec_size){
    // sqrt_of must be callable from device code
    vector[index] = sqrt_of(vector[index]);
  }
}

// function will be callable from host and device code
// TODO: Make this function callable in host and device code.
double sqrt_of(double x, double precision){
  // Approximate the square root of x up to precision
  double approximation=x/2;
  double prev_val;
  double error = 1.0;
  if(x==0.0){
    return 0.0;
  }
  if(x<0){
    return -1.0;
  }
  while(error>precision){
    prev_val = approximation;
    approximation = 0.5*(approximation + x/approximation);
    // error estimate (upper bound)
    error = (prev_val*approximation-x)*(prev_val*approximation-x)/(2*prev_val*approximation*approximation);
  }
  return approximation;
}

// CPU code //
int main() {
  double* vector_cpu = (double*) malloc(vec_size*sizeof(double));
  // TODO: Define the execution configuration for the kernel sqrt_kernel
  const ___ num_threads = ____;
  const ___ num_blocks  = ____;

  // fill vector
  for(int i=0; i<vec_size; ++i){
    vector[i]=2*i;
  }
  // compute elementwise square root (approximately) (CPU)
  for(int index = 0; index<10; ++index){
    vector_cpu[index] = vector[index];
    vector_cpu[index] = sqrt_of(vector_cpu[index]);
  }
  // compute elementwise square root (approximately) (GPU)
  sqrt_kernel<<<num_blocks,num_threads>>>(vector, vec_size);
  cudaDeviceSynchronize();
  // Print first ten roots
  for(int i = 0; i<10; ++i){
    double error = fabs(vector[i]- vector_cpu[i]);
    printf("Sqrt(%d): %f (GPU), %f (CPU), abs. difference error %f\n", 2*i, vector[i], vector_cpu[i], error);
    if(error>1e-5){
      fprintf(stderr, "Error larger than 1e-5, result wrong.");
      return EXIT_FAILURE;
    }
  }
  // exit //
  return EXIT_SUCCESS;
}

Keypoints

  • __global__ marks a kernel (called from host, runs on device), __device__ marks a device function, __host__ __device__ compiles for both

  • Kernels have restrictions: no return values, no recursion, no variable arguments

  • Most standard mathematical operations work on the GPU; system calls and I/O do not

  • Always check CUDA error codes — use a macro like cudaVerify to wrap API calls

  • Kernel launch errors surface asynchronously on the next blocking call

See also