vertex-skip: The Case of Luigi's Extra Triangle

While testing Mario Golf: Toadstool Tour in Gecko, my GameCube and Wii emulator, I noticed a thin triangle stretching from Luigi’s hand across the character selection menu.

Look between Luigi’s hand and Wario’s portrait:

The same character selection frame after the fix, without the stray triangle.

Mario Golf before the fix: a thin diagonal triangle stretches from Luigi's hand across the menu.

The cause was a position index meaning skip this vertex, which Gecko treated as an ordinary array entry and turned into an extra point and triangle.

A bit of geometry

The GameCube’s graphics processor, GX, fills triangles built from vertices: corners with positions and attributes such as color.

In a triangle strip, the first three vertices make a triangle and each additional vertex makes another triangle with the previous two corners. Number a rectangle’s corners like this:

0 ------ 2
|        |
|        |
1 ------ 3

Sending 0, 1, 2, 3 makes triangles 0, 1, 2 and 2, 1, 3, which share the diagonal between corners 1 and 2. GX uses the order of a triangle’s vertices to tell its front from its back: going around them clockwise faces the opposite way to going counterclockwise. Swapping the first two vertices of every other triangle keeps this winding order consistent, so both face the same way.

Games can send coordinates directly or use indices into a position array, where index 2 means “use position number 2” and saves us from sending those coordinates again.

GX supports both 8-bit and 16-bit position indices:

Position format	Reserved index	Meaning
8-bit	`0xff` (255)	Skip this vertex
16-bit	`0xffff` (65’535)	Skip this vertex

The skip rule applies to the indexed position field, while other fields use ff according to their own formats.

Nintendo’s US6700586B1 patent documents these reserved values.

Skipping a vertex

If we send:

0, 1, SKIP, 2, 3

the retained vertices are simply 0, 1, 2, 3, giving us the same rectangle because the skipped record doesn’t occupy a corner in the strip.

Select “Array entry” below to fetch the reserved index as point P and show the triangles touching it in red.

Interpretation 0xff location

0 · 1 · 0xff (SKIP) · 2 · 3

Move 0xff to the end: fetching P adds one red triangle to the blue rectangle, just like Luigi’s spike.

With 0xff in the middle, the demo’s restart option splits the strip into two pairs of vertices, neither of which can form a triangle. The dashed outline is just a guide.

Back to Luigi

The affected Mario Golf strip contained 11 valid vertices followed by a skip record, but Gecko fetched a position for that final record anyway and interpreted unrelated memory as:

[-27.166016, 15.875, 0.0]

Transforming that point and connecting it to the previous two corners produced the spike.

The fix removes skipped records before vertex decoding and updates the vertex count for each draw. Keeping draws separate prevents a removed record from accidentally joining one draw’s vertices to the next.

Those corrected boundaries also feed the depth plane tracking used by z-freeze, since both features need to agree about which triangles exist.

The bytes still exist

Skipping a vertex doesn’t remove its bytes from the incoming command stream.

For example, an 8-bit position index followed by an RGBA color takes five bytes, even when the vertex is skipped. Here are three consecutive records:

After reading the skip index FF, the parser must pass over all four color bytes or it will mistake E6 for the next position index and lose its place. The alpha byte FF just means fully opaque.

A 16-bit position index takes two bytes, making the same record six bytes long, while other formats can add more attributes. The active format determines the position field and record boundaries.

The simplified logic looks like this:

for each vertex record in the draw:
    record = consume the complete record using the current format

    if its indexed position is the reserved all ones value:
        continue

    decode the record and append its vertex

assemble the draw's triangles from the retained vertices

Checking the behavior

My test draws the same rectangle with direct positions, 8-bit indices and 16-bit indices, inserting a skip between vertices 1 and 2 in both indexed versions. All three should match:

The test stores a displaced position at the reserved array entries and makes the skipped records red, so an incorrect lookup produces obvious stray geometry. Build it with make vertex-skip and run build/vertex-skip.dol.

Gecko’s implementation lives in filter_skipped_vertices. All that trouble over a vertex that wasn’t supposed to exist :)

Thank you

A big thank you to Zayd for taking the time to proofread this post. He’s the creator of beanwii and also makes some really cool videos about emulation on his YouTube channel. If you enjoyed this post, go check them out!