I will do more testing, but if that's the case, perhaps we would want a way to tell the shader runner when a shader should compiler but not it is not required to be executed properly.
That's broadly what 4d17758657 was for. It was specifically meant for cases where testing syntax was interesting but testing execution wasn't, but it can work here too.