2009/4/13 James McKenzie jjmckenzie51@earthlink.net:
Ben Klein wrote:
2009/4/13 chris ahrendt celticht32@aol.com:
Vincent Povirk wrote:
On Sun, Apr 12, 2009 at 5:24 PM, Ben Klein shacklein@gmail.com wrote:
2009/4/13 Vincent Povirk madewokherd+8cd9@gmail.com:
But the description doesn't say "invalid conditions". It says "serious errors in Wine". That's something that should never happen in tests, as it would imply that the state of the libraries we're testing (and thus the test) is invalid.
How about "serious errors in Wine, or in the test cases, sometimes deliberate errors in the tests"? As Vitaliy points out, some tests deliberately do invalid things to make sure they fail. We can't ONLY test on things that succeed!
I'm not against changing the description. If it's OK to have err messages for cases that we know fail (but don't crash or prevent the library from continuing to function), then the Developer's Guide is wrong and should be changed. I also don't care how long it takes to make the change, as long as there's a consensus that it's the guide that's wrong and not the reality of what's in Wine.
This is EXACTLY the point I am trying to get to.. if they are normal and not ERRORS but warnings then they should be thus and noted in the developers guide. Right now Wine isn't even following its own guidelines in this case.
No. Not warnings. Errors. They are errors. There is no way to distinguish an error caused by a real application from an error caused by a Wine test.
If the situation is an error and it is expected, the test should handle this, like:
ok (some_test_here_that_should_fail, "The test that should fail, did/n")
I'm guessing that most of the tests that should fail, do. I don't know if there is a failure like there is an ok.
AFAIK, this is how expected failures in tests are handled. I saw a few recent tests submitted and/or committed that do it like this.
If you don't like it, run all your tests with WINEDEBUG=-all.
And that will prove nothing. Tests should be run with debugging on. You are really being sarcastic, right?
As to the discussion, I will add my .02 Eurocent here:
Fixme: Code that is not implemented that should be. Warning: Code that encountered an unexpected or new condition that is not normal. Data should not be corrupted by this event. Error: Code encountered a condition that will result in corruption of data. It appears that we have 'error creep' and that is not good from a user point of view and it is really irritating from a support point of view.
During testing an error could be either a Warning or an Error. Tests should not be run against non-existent Wine code, but should against existing Windows code. The situation with testing is that unexpected or improper behavior of code should be an error. There is no such thing as a warning during a test run. Either the test passes, which means that Wine is acting the same as a certain version of Windows, or it is not.
There is no way for the Wine components that are producing the errors to distinguish between a test run and a real application. Again, the tests are triggering these errors not in the test applications but in other Wine components. In some (possibly all) cases that this happens, this is expected and correct (because it is impossible to distinguish between a genuine error and a test error).
Now, the problem is that we are sending cryptic messages to end users, most of which they can and should ignore. Err messages piling up on their terminal windows should be a cause for concern. If we know that a condition does not cause data corruption, then we should not be marking it as an error, but maybe a warning or if the code is non-existent or improper, a fixme.
End users shouldn't be running the tests without understanding how they work.
Can we start to clean up some of the most obvious "it is not an error but the message says it is" so that we can calm down users who encounter them?
The ERRs are being produced by components of Wine outside the test cases. It's highly likely for those ERRs to change in later versions of Wine. If you want to maintain a complete list of where ERRs will be triggered by test cases and include a message for each one that says "It's OK, you can ignore this ERR", then I'm sure you're welcome to try it.
2009/4/13 James McKenzie jjmckenzie51@earthlink.net:
Vitaliy Margolen wrote:
You wrong. As I said, tests are _allowed_ and some really _supposed to_ test invalid cases. Where Wine 100% correct about complaining. But the goal of each test is either succeed or fail. If it succeeds then there is no bug.
Conversely, if a good test starts to fail or error messages appear where there were none, we have a problem that needs to be reported so it can be fixed (this happened with Wine 1.1.18).
There is a BIG difference between "test caused an ERR message" and "test failed". They are handled differently in the test runs.
Again there are not failures in the tests your ran! There is nothing to fix here. Move along don't waste everyone time for something YOU DO NOT LIKE! Or send patches.
The problem is that error messages should not appear where there are no errors. They should be either warnings or fixmes. If the test passes but a bunch of error messages appear during the test run, there is a coding problem that needs to be fixed. This is why we are answering user questions with 'The error message can be ignored, there is nothing wrong with the execution of your program.' This makes it appear that the developers don't know what they are doing and that makes the overall project appear bad.
So deliberately invalid cases in the tests should never cause ERRs, even though the same invalid cases produced by genuine applications should? Good luck implementing that.
Conversation is over. DO NOT WASTE everyone's time!!!
We are not 'wasting time.' This issue needs to be addressed by the development community so that errors, warnings and fixmes are properly reported.
They are. It's just that *some* people, who shall remain nameless here, are misinterpreting them. These are test cases, and test cases have to test on both positive and negative cases, expecting success and failure respectively. If the positive cases are generating ERRs, then that's indicative of code in Wine that needs cleaning up, but if the negative cases do something *deliberately invalid* and generate ERRs, then there's nothing to worry about.
We can and maybe should get more granular when we work with code. Maybe a fixme for a known but unimplemented function and a warning for an incomplete function.
Shouldn't this be the other way around? Complete non-implementation (or stubiness) is more likely to cause problems for applications than partially implemented functions. At the moment, both of these cases are FIXMEs (stub and partial stub respectively).
Maybe changing the wording on an error message that really is an error in code but will not cause a program to crash. I've seen error messages that really should be fixmes and the other way around where a fixme causes some programs to terminate with a screen full of debugging code.
ERRs and FIXMEs are classified by developers for developers, not for end-users. It's impractical to limit all ERRs to things that crash programs, and all FIXMEs to things that don't crash. In some cases, I'm sure, the app would detect an unexpected value caused by either a problem that triggers a FIXME or ERR and work around the problem.
At least I have the time to waste at the moment :P