[Bug 48209] New: Ignore random parts of test failure messages

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48209

Bug ID: 48209
           Summary: Ignore random parts of test failure messages
           Product: Wine-Testbot
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

The TestBot needs to detect if a patch introduces NEW test failures. To do so
it compares the failure messages with those from reference WineTest runs. But
in some cases part of the message changes from one run to the next, causing the
failure to look new.

Of course in all cases the ideal solution is to fix the test so it does not
fail.

Barring that there are also cases where the varying value is not really useful
(e.g. handle or pointer values) and in that case it should just be removed from
the message.

But there may be cases where knowing the value is still useful. So one solution
would be to tag such values so the TestBot knows it can ignore them when
comparing failure messages. For instance they could be enclosed in double
braces, curly braces, or other.