CUDA GPU Workloads in Rust with cuda-oxide

Description

This project explores CUDA programming in Rust using NVIDIA’s cuda-oxide. It shows how core GPU programming concepts - thread indexing, kernel launches, atomics, streams, asynchronous transfers, matrix multiplication, activation functions, softmax, and profiling - can be implemented from Rust while still fitting into a normal Cargo-based project structure.

The repository is organized as standalone examples exposed through root-level Makefile targets, with additional Triton and PyTorch scripts for framework-level comparisons.

Source code is available on GitHub.

Open Table of contents

Environment
Implemented GPU workloads
Verification and testing
Remote development workflow
What was tested

Environment

The project targets a Linux machine with an Ampere-or-newer NVIDIA GPU (sm_80+), NVIDIA driver, CUDA Toolkit, compute-sanitizer, LLVM/Clang, Rust nightly, and cargo-oxide.

The Makefile pins the main tooling inputs:

cuda-oxide revision: 76a1d8840e5360207943805d036d8e299bda7728
Rust toolchain: nightly-2026-04-03
LLVM/Clang: 21

On a Debian/Ubuntu GPU host, the setup target installs the required host tools, Rust toolchain, and cargo-oxide, then runs the environment validation:

make setup

For an already configured GPU host:

make doctor

make doctor checks that nvidia-smi, nvcc, compute-sanitizer, cargo, and cargo-oxide are available, then runs cargo oxide doctor from the first Rust CUDA project.

Implemented GPU workloads

The Rust side contains 15 standalone CUDA examples:

Workload	What was implemented and checked
`vector-add`	Adds two ten-million-element vectors on the GPU and checks every result against CPU addition.
`indexing`	Launches a three-dimensional CUDA grid, flattens thread IDs, and verifies all flattened IDs are unique and correct.
`multidimensional-vector-add`	Runs equivalent 1D and 3D launch geometries over `100 x 100 x 1000` elements and verifies both outputs against the same CPU reference.
`atomics`	Uses `DeviceAtomicU32::fetch_add` across `1_000_000` GPU threads and verifies that every returned value from `0..1_000_000` appears exactly once.
`streams`	Uploads independent inputs on separate streams, adds an explicit dependency, and verifies ordered vector addition.
`advanced-streams`	Uses pinned host memory, events, stream ordering, a host callback, and result download validation for `(input * 2) + 1`.
`naive-matmul`	Computes one matrix output cell per GPU thread and checks every cell against CPU matrix multiplication.
`tiled-matmul`	Loads matrix tiles into shared memory, synchronizes with barriers, and checks every output cell against the CPU result.
`profiling-matmul`	Repeats verified matrix multiplication kernels inside a CUDA-event timing window while keeping transfers outside the measured interval.
`profiling`	Builds a profiler-friendly repeated matmul workload with warmup, CUDA event timing, and device-side profiling markers.
`unrolling`	Compares loop-based and explicitly expanded GPU kernels over ten million elements and verifies both against a CPU reference.
`convolution`	Implements same-padded 2D convolution over NCHW tensors and checks the flattened output against expected values.
`tanh`	Computes bounded tanh on the GPU using device arithmetic and checks every output against host `f32::tanh()` within tolerance.
`softmax`	Implements numerically stable row-wise softmax and verifies every row against a stable CPU softmax reference.
`polynomial-activation`	Computes `y = x * x + x + 1` and checks exact `f32` equality for deterministic binary-fraction inputs.

Verification and testing

Local checks

Local checks do not require a GPU:

make check

This target runs formatting and metadata checks for every Rust example:

cargo fmt --all -- --check
cargo metadata --locked --format-version 1 --no-deps

It also byte-compiles the Python scripts and runs the dependency-free stable softmax reference:

python3 -m py_compile python/references/softmax_reference.py python/triton/vector_add.py python/triton/softmax.py python/pytorch/tanh.py
python3 python/references/softmax_reference.py

The Python softmax reference validates a known input and checks that every deterministic output row is normalized.

GPU run targets

Each Rust workload can be built and run from the repository root:

make list
make run-vector-add
make run-tiled-matmul
make run-softmax

All Rust GPU workloads can be executed in order:

make run-all

For numerical examples, a successful run means deterministic input was used, the GPU result was compared with the expected CPU result, and the binary reported PASSED.

GPU sanitizer targets

Every Rust workload has a matching test target:

make test-vector-add
make test-softmax
make test-all

The common test path first runs the workload, then checks the produced release binary with NVIDIA Compute Sanitizer:

compute-sanitizer --tool memcheck ./target/release/<binary>

Additional sanitizer checks are wired into selected examples:

tiled-matmul also runs compute-sanitizer --tool racecheck.
profiling also runs compute-sanitizer --tool initcheck.
atomics includes a separate make demonstrate-atomics-race target for the intentionally incorrect non-atomic counter update, including a racecheck run.

Python reference and framework comparisons

The repository also includes Python comparisons:

make run-python-reference
make run-python-gpu

The CPU-only reference computes stable softmax with the Python standard library. The GPU-backed Python examples require a CUDA-enabled Python environment:

python/triton/vector_add.py compares Triton vector addition with PyTorch CUDA and uses a default input size that exercises masked tail accesses.
python/triton/softmax.py compares row-wise Triton softmax with torch.softmax, including large shifted logits that require max subtraction for numerical stability.
python/pytorch/tanh.py compares a stable custom tanh expression with torch.tanh using rtol=1e-6 and atol=1e-6.

Benchmark targets are available, but they run only after correctness checks pass:

make benchmark-python-triton
make benchmark-python-pytorch

Remote development workflow

The project supports editing locally while running CUDA workloads on a remote Linux GPU host. After exporting the remote connection variables, the root Makefile syncs the repository while excluding generated build output and local editor files:

export REMOTE_HOST="<GPU_HOST>"
export REMOTE_USER="<USER>"
export REMOTE_DIR="~/cuda-course-rust"
# export REMOTE_PORT="<SSH_PORT>"
# export SSH_KEY="$HOME/.ssh/id_ed25519"

make sync

After syncing, the same run and test targets can be executed on the GPU host.

What was tested

The project tests correctness at several layers:

Deterministic CPU references are used for vector addition, launch geometry, matrix multiplication, convolution, tanh, softmax, and polynomial activation.
Every Rust GPU workload has a run-* path that validates output before reporting success.
Every Rust GPU workload has a test-* path that adds compute-sanitizer --tool memcheck.
Shared-memory matrix multiplication gets an extra race check.
The profiling workload gets an extra initialization check.
Atomic behavior is validated by checking the complete set of fetch_add return values.
Python comparison scripts validate Triton and PyTorch CUDA behavior before optional benchmarking.

That gives the repository two useful properties: the examples can be run one by one while developing a specific kernel, and the full suite can be exercised with make run-all or make test-all on a configured NVIDIA GPU machine.