new d3d9/device.ok test always fails here, but not a regression?
Hi,
Bugs in newly added code cannot be regressions.
Similarly, if we add a dll and some app starts using it and breaks, technically that's not a regression even though the behavior got worse, because it has always been broken, it just wasn't exercised before.
What does this distinction bring us? The old definition, "if it worked in version N and doesn't in N+1, then it is a regression" was straightforward and easy to understand. There was little to debate, no room left for confusion. Every such regression could be associated with the SHA1 of a commit. The new definition can get arbitrarily complex. Consider bug #19496. Bug resolution likely involves moving from mciwave to mciavi. As mciavi is not equally advanced, this move will likely cause bugs to appear in other apps. But we wouldn't call them regressions "because it has always been broken"?!? That makes no sense to users. What benefit does it have for developers? Having one's name appear less often in source.winehq.org/regressions when doing The Right Thing (TRT)? Regards, Jörg Höhle
Joerg-Cyril.Hoehle(a)t-systems.com writes:
Hi,
Bugs in newly added code cannot be regressions.
Similarly, if we add a dll and some app starts using it and breaks, technically that's not a regression even though the behavior got worse, because it has always been broken, it just wasn't exercised before.
What does this distinction bring us? The old definition, "if it worked in version N and doesn't in N+1, then it is a regression" was straightforward and easy to understand. There was little to debate, no room left for confusion. Every such regression could be associated with the SHA1 of a commit.
The new definition can get arbitrarily complex. Consider bug #19496. Bug resolution likely involves moving from mciwave to mciavi. As mciavi is not equally advanced, this move will likely cause bugs to appear in other apps. But we wouldn't call them regressions "because it has always been broken"?!? That makes no sense to users.
What benefit does it have for developers? Having one's name appear less often in source.winehq.org/regressions when doing The Right Thing (TRT)?
Yes, it reduces the noise to let you concentrate on the real regressions, i.e. the ones where working code got broken, and where you can get useful information from the previous state. Having a new piece of code tagged a regression is just noise, there's nothing you can do with that information. All it tells you is that the bug in that new code wasn't present when the code didn't exist (doh). The primary goal of Bugzilla should be to present information in a way that makes it easier for developers to fix bugs. A simple "my app got broken" flag may be easier to understand for users, but it's less useful. -- Alexandre Julliard julliard(a)winehq.org
On Tue, Jan 3, 2012 at 13:01, Alexandre Julliard <julliard(a)winehq.org> wrote:
Joerg-Cyril.Hoehle(a)t-systems.com writes:
What benefit does it have for developers? Having one's name appear less often in source.winehq.org/regressions when doing The Right Thing (TRT)?
Yes, it reduces the noise to let you concentrate on the real regressions, i.e. the ones where working code got broken, and where you can get useful information from the previous state.
Having a new piece of code tagged a regression is just noise, there's nothing you can do with that information. All it tells you is that the bug in that new code wasn't present when the code didn't exist (doh).
The primary goal of Bugzilla should be to present information in a way that makes it easier for developers to fix bugs. A simple "my app got broken" flag may be easier to understand for users, but it's less useful.
Might be difficult to explain to "plain" users though. Some already struggle performing a regression test (most won't even bother), let alone understanding the fine details of what a regression really represents. http://wiki.winehq.org/RegressionTesting states "Sometimes, committing patches in Wine to fix bugs and introduce new features causes new problems. This is called a regression" As users (and not only devs) are allowed to submit bug reports, and they/most don't have a clue whether an app did work/doesn't anymore is due to a new feature or not, maybe we have some additional field, visible by default only to people with "advanced" permissions in bugzilla, that indicates it's really a regression or not (we shouldn't discourage users to provide a commit ID which can be useful anyway) ? This would allow devs to concentrate to the real regressions, as you explained. Frédéric
2012/1/3 Frédéric Delanoy <frederic.delanoy(a)gmail.com>:
On Tue, Jan 3, 2012 at 13:01, Alexandre Julliard <julliard(a)winehq.org> wrote:
Joerg-Cyril.Hoehle(a)t-systems.com writes:
What benefit does it have for developers? Having one's name appear less often in source.winehq.org/regressions when doing The Right Thing (TRT)?
Yes, it reduces the noise to let you concentrate on the real regressions, i.e. the ones where working code got broken, and where you can get useful information from the previous state.
Having a new piece of code tagged a regression is just noise, there's nothing you can do with that information. All it tells you is that the bug in that new code wasn't present when the code didn't exist (doh).
The primary goal of Bugzilla should be to present information in a way that makes it easier for developers to fix bugs. A simple "my app got broken" flag may be easier to understand for users, but it's less useful.
Might be difficult to explain to "plain" users though. Some already struggle performing a regression test (most won't even bother), let alone understanding the fine details of what a regression really represents.
Most 'plain' users aren't filing bugs. Bugzilla is primarily for developers, not user support.
http://wiki.winehq.org/RegressionTesting states "Sometimes, committing patches in Wine to fix bugs and introduce new features causes new problems. This is called a regression"
As users (and not only devs) are allowed to submit bug reports, and they/most don't have a clue whether an app did work/doesn't anymore is due to a new feature or not, maybe we have some additional field, visible by default only to people with "advanced" permissions in bugzilla, that indicates it's really a regression or not (we shouldn't discourage users to provide a commit ID which can be useful anyway) ?
This would allow devs to concentrate to the real regressions, as you explained.
That's already done...The regression keyword indicates that the bug is a regression (or at least though to be one), and the Regression SHA1SUM field points out the exact commit (and feeds the hall of shame page). -- -Austin
2012/1/3 Frédéric Delanoy <frederic.delanoy(a)gmail.com>:
Might be difficult to explain to "plain" users though. Some already struggle performing a regression test (most won't even bother), let alone understanding the fine details of what a regression really represents.
The original thread starter excluded, I don't usually have much of a problem explaining things to users. You'll always have a couple of exceptions, but as a general principle there's probably value in the idea that people will act about as stupid as you treat them. I'm also not sure the data really backs up the idea that most people don't bother with a regression test. On http://source.winehq.org/regressions it's about 256 with commit ID vs 70 without, and the ones without probably tend to stay open longer. Granted, sometimes someone else does the regression test instead of the original reporter, but I don't see that as a big problem.
I had hoped this was one of those threads that would go away if I ignored it. Oh well. I actually think there's value in handling anything that breaks an application as a regression. Despite claims to the contrary, we like users to test releases and find regressions sooner rather than later. If applications are broken for extended periods of time that only encourages users / potential testers to stick with outdated releases, and it doesn't really matter there if something broke because a patch itself was bad or if that patch just exposed some other broken code. (As an aside, the latter seems much more common.) Unfortunately doing things like that requires a certain amount of good judgment about what is reasonable to mark as a regression and what isn't, as opposed to rigidly applying a set of rules. For example, if adding a new dll exposes the fact that we need a HLSL compiler, that isn't really going to benefit from being marked as regression, it's just something that practically everyone already knows, and just takes a certain amount of effort. On the other hand, if it's something that can easily be fixed by e.g. adding a stub, or just fixing a bug somewhere else, I think that makes sense. (Although if it's easy enough and you're a developer anyway, you might as well just fix it yourself.) Evidently this is hard for people. Fundamentally, keywords are a tool for developers to solve bugs more efficiently. They're explicitly not some kind of "management" tool, or a larger megaphone for users to shout "LOOK HERE, MY BUGS ARE IMPORTANT!!1". (Incidentally, there are actually people you can pay to prioritize your bugs, but even those will tell you to go away if you try to be too much of a dick about it.) Similarly, joking about the "Hall of Shame" is all good fun, but the main purpose of that page is to give people a good way of keeping track of their regressions, since IMO plain bugzilla isn't very good at it. Now, about the original bug this thread was about. I obviously run the D3D tests myself almost daily before sending patches. I also run those regularly (but less often) on a couple of other machines with different configurations, and on Windows. I watch test.winehq.org from time to time for new failures in the D3D tests. (On that subject, it would probably be nice if we could see a history of changes in failures for specific tags, to see how consistent those failures are.) The test.winehq.org data shows that it fails for some people, but passes for most. It consistently passes on Windows. It consistently passes on all my boxes. This all means the following things: - The only way that bug report was going to tell me something I didn't already know was if I didn't watch test.winehq.org (wtf) and either didn't run the tests myself (wtf) or they didn't fail for me (in which case I wouldn't be able to reproduce the bug with the information in the report). - Given that it consistently passes on Windows, and given the nature of the test, it's probably more likely that there's an issue with e.g. event delivery somewhere than with the test itself, or even anything D3D related. - The most useful thing someone who can consistently reproduce that bug can do is to try to debug it, asking for help where needed. Failing that, figuring out what makes it consistently reproducible would be a good second. - There's no possible way that adding the regression keyword to that bug is going to help anything.
participants (5)
-
Alexandre Julliard -
Austin English -
Frédéric Delanoy -
Henri Verbeet -
Joerg-Cyril.Hoehle＠t-systems.com