How NDSR upscales Ocarina of Time on Switch 2

It ships as one file

NDSR lives in the game's data, not in the console's driver. Inside romfs/Lib/agl/agl_resource.sarc is agl_ndsr_vr_light.pck: a 548 KB bundle with the GPU programs, their launch settings, a lookup table for SMAA and seven sets of network weights. The game copies the whole file into GPU memory and runs the programs where they land.

Hover a block to see what it holds.

Each set of weights is a separate model for one upscale ratio. Every model writes a 48×48 block of output pixels; what changes is how many input pixels it reads to make it. When the game's dynamic resolution lands between two ratios, the shader resamples to cover the gap.

Four passes per frame

The rendered frame goes through four compute passes. The first three are SMAA, Nintendo's middleware build SMAA_For_NDSR 2.3.2, rewritten to pack edges into bit masks. The fourth is the network.

Find edges. Compare each pixel with its neighbours; where the difference passes 0.1, set a bit. Each 4×4 block of pixels becomes one texel of bits.
Work out blend weights. Measure each edge's length straight from the bit masks, then look up how much to blend in the area table below.
Blend. Mix each edge pixel with its neighbour, take the square root and store the frame in a compact 32-bit format.
Upscale. The network, described in the next section.

Frame before SMAA — The captured frame before and after the first three passes.

Frame after SMAA — The captured frame before and after the first three passes.

SMAA area lookup table — The 192×128 area table stored in the file. The tiles on the left cover straight edges, the smaller ones on the right diagonal edges. Red and green are the two blend weights.

Inside the network

The network only looks at brightness. Each GPU workgroup takes a 40×40 tile of the frame plus a 3-pixel border, converts it to luma, and cuts it into sixteen overlapping 16×16 windows, 10 pixels apart. Every window goes through the same six stages:

Y ← ReLU(A · Yᵀ · B + C · Yᵀ · D)

A, B, C and D are learned 16×16 matrices, different for each stage. Multiplying on the left mixes along one image axis and on the right along the other, so each stage is a full layer over all 256 values for the price of four small matrix multiplies, the size the tensor cores take in one instruction. The last stage skips the ReLU. Its top-left 12×12 values become this window's share of a 48×48 detail map: 10 input pixels in, 12 detail values out.

Pick a tile in the frame, then a window in the patch.

40×40 input tiles. One GPU workgroup handles each.

The tile's 46×46 luma patch and its 16 windows.

The 48×48 detail map. The chosen window's 12×12 block is outlined.

The chosen window going in, then after each stage. Amber is positive, teal negative, each tile scaled to its own largest value.

All 24 weight matrices of the 1.2× model, one row per stage. Hover a stage above to highlight its row.

From detail map to pixels

The network never sees colour. For each output pixel, the shader samples the frame with a plain bilinear filter, adds the detail value to red, green and blue alike, and squares the result to undo the square root from the blend pass. A small sharpen finishes it: 1.25 times the pixel's luma, minus 0.3125 times each of its four neighbours.

The tile you picked above, at each step, enlarged.

Try it on your own image

Your image is treated as a rendered frame: converted to linear light, square-root encoded, then upscaled. The network was trained on game renders, so expect the most from sharp, aliased edges. Large images take a few seconds.

Choose an image Output size No image chosen.

How we know this is the real thing

The network on this page is rebuilt from the program at offset 0x1be00 in agl_ndsr_vr_light.pck. The game ran that program from GPU address 0x11491c8e00: the file's base address plus that offset.
Its 1,456 instructions were disassembled with NVIDIA's nvdisasm and run instruction by instruction in a small emulator, using the constants the game passed on a captured frame.
On that frame, the model behind this page matches the image Lichii produced by running the game's own shader exactly for 78% of pixels, with an average difference of 0.0005. The rest differ in the last bit of half-precision rounding.
The weights are read straight from the file. Nothing here was retrained or tuned.

Worked out while improving Lichii, a Switch 2 emulator. Ocarina of Time, its screenshots and the NDSR model belong to Nintendo; they appear here to explain how the upscaler works. This page isn't affiliated with or endorsed by Nintendo or NVIDIA.