On Fri, 27 Mar 2020, Zebediah Figura wrote: [...]
(2) Remi's comment inspired another option, though it's definitely more work for François: lookup the line number in the relevant source file and compare the source line.
That's pretty hard for three reasons: * The test reports are analyzed on the TestBot server which does not have access to the Wine source (giving it access to a Wine checkout is on the todo list for other reasons). * What it needs is the patched source code whereas what it would have access to is the unpatched source. * On the server multiple processes may need access to the patched source code at the same time.
Also a number of tests loop over the test cases so that they all involve the same lines but the failures are still distinct. For instance (from the set of "new" failures in a recent "32 bit WineTest: base VMs" job):
d3d11.c:2729: Test failed: Test 22: Got unexpected hr 0x8007000e (format 0x9). d3d11.c:2729: Test failed: Test 23: Got unexpected hr 0x8007000e (format 0x9). d3d11.c:2729: Test failed: Test 24: Got unexpected hr 0x8007000e (format 0xf). d3d11.c:2729: Test failed: Test 25: Got unexpected hr 0x8007000e (format 0x13).
These all come from the same line but it would be wrong to say that a failure on iteration 22 is equal to the one on iteration 25.