On 3/27/2020 11:53 AM, Francois Gouget wrote:
The issue is that some test failures include values that change on every run which prevents the TestBot from knowing whether they are new or not. In turn this means the TestBot systematically blames the author of whichever patch is being tested for these failures.
So you're wanting to add a sort of comment syntax so that TestBot can ignore parts of failure messages referring to transient data.
For instance one could enclose the irrelevant parts in double-parentheses [2] as follows:
Test failed: hwnd ((0028050C)) message 0738
For this case in particular, I don't believe it's useful to output the hwnd at all, unless that's a useful piece of information in diagnosing the test failure.
Test failed: WaitCommEvent used ((1594)) ms for waiting Test failed: Unexpected date value ((24/03/2020)), got ((2020/03/24)) [3]
I agree that the delimiter must be chosen carefully. I would prefer something like TESTBOT_IGNORE(...) which would never show up in a failure message otherwise, and would even have some possibility of being discovered by grepping the source code.
-Jefferson