Wow, this is growing long...
If it is female, then genitive would be „der Tessellation“ (the number of l apparently varies depending on who you ask), which is why I would say Tessellationshader (Shader der Tessellation), except if Tessellation were plural.
That's not how it works ;)
For example, Information is feminine, correct construct i.e. "Informationsverarbeitung" (with an s), but "Verarbeitung der Information". Just like "Tesselations-shader" (Shader der Tesselation).
Am I missing something here?
I'll go eat lunch now, later we could discuss this further off-list maybe ;)
Best, Julian