[Bug 48209] New: Ignore random parts of test failure messages
https://bugs.winehq.org/show_bug.cgi?id=48209 Bug ID: 48209 Summary: Ignore random parts of test failure messages Product: Wine-Testbot Version: unspecified Hardware: x86 OS: Linux Status: NEW Severity: normal Priority: P2 Component: unknown Assignee: wine-bugs(a)winehq.org Reporter: fgouget(a)codeweavers.com Distribution: --- The TestBot needs to detect if a patch introduces NEW test failures. To do so it compares the failure messages with those from reference WineTest runs. But in some cases part of the message changes from one run to the next, causing the failure to look new. Of course in all cases the ideal solution is to fix the test so it does not fail. Barring that there are also cases where the varying value is not really useful (e.g. handle or pointer values) and in that case it should just be removed from the message. But there may be cases where knowing the value is still useful. So one solution would be to tag such values so the TestBot knows it can ignore them when comparing failure messages. For instance they could be enclosed in double braces, curly braces, or other. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48209 François Gouget <fgouget(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48209 --- Comment #1 from François Gouget <fgouget(a)codeweavers.com> --- For reference: * https://www.winehq.org/pipermail/wine-devel/2020-March/163169.html Contains a list of tests with always new failures. * https://www.winehq.org/pipermail/wine-devel/2019-December/157168.html This patchset did not get accepted. * https://www.winehq.org/pipermail/wine-devel/2020-March/163133.html Thread discussing this issue. * https://www.winehq.org/pipermail/wine-devel/2020-March/163383.html A proposed blacklisting solution with safeguards. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48209 François Gouget <fgouget(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |DUPLICATE Depends on| |48912 Status|NEW |RESOLVED --- Comment #2 from François Gouget <fgouget(a)codeweavers.com> --- Bug 48912 contains the proposal set forth in the march blacklisting email. As such it's an alternative approach to dealing with "random parts of test failure messages" and this bug should be considered a duplicate of it. *** This bug has been marked as a duplicate of bug 48912 *** -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48209 Gijs Vermeulen <gijsvrm(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED --- Comment #3 from Gijs Vermeulen <gijsvrm(a)gmail.com> --- Closing duplicate. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48209 Bug 48209 depends on bug 48912, which changed state. Bug 48912 Summary: Allow blacklisting unreliable and always new failures https://bugs.winehq.org/show_bug.cgi?id=48912 What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
participants (1)
-
WineHQ Bugzilla