Juan Lang wrote:
So, in my opinion, the general principle is this: if the behavior is very different on different Windows versions, and if the behavior on some Windows version is to crash, it seems reasonable to remove the test, as: 1) we want our tests to reflect Windows behavior that applications count on, and different behavior (especially crashing) often implies the applications don't count on a particular behavior, and 2) having regression tests crash is undesirable, as many other valid tests will not be run on a platform.
I don't think you can fully argument with this nowadays. There are many applications out there that specifically state to not support pre W2K or sometimes even XP and often refuse to even install on older systems. So in this case they could be perfectly happy with passing a NULL value to that function and since they never have been tested on an older system the developer wouldn't even know that there exists a potential problem.
So I would think it's useful to test that the Wine API does not crash on a NULL parameter since newer Windows versions won't do that either.
This doesn't mean that we can't make Wine not crash given the bogus input, it just means we may not be able to have a regression test for this behavior. And, of course, there may be specific instances in which we wish to test potentially crashy behavior, but I'd argue that they better be really important to override points 1) and 2).
This case seems fairly simple to me. As long as Wine strives to provide support for Win9x and NT4 it needs to make efforts to support both the newer and the older behaviour. For the API itself this means allowing NULL values to be passed without crashing and for the tests it means skipping that particular test for those versions that do crash on it. And while a version test seems the easiest I can also understand why the tests should not have such artificial dependencies especially since these tests would have to distinguish between running on Windows or Wine set to emulate Win9x.
But here it seems fairly easy to work around this problem. The fact that certain Snmp APIs are not implemented or possibly return an ERROR_UNIMPLEMENTED status on those platforms should be enough to determine if this test should be skipped.
And coming back to this test: I still think it's easier just to remove the test than to come up with a hacky way of avoiding it on a crashy system.
No argument with the being easier part. But not writing tests at all would be also easier, but I do not think that is a valid argument to not do them. Typically to do a good test costs more time than an actual implementation of a function and Wine development could be sped up by not writing tests. But it would only implement more functionality with a lot more errors (or should that be less bugs :-).
Rolf Kalbermatter