[Bug 32003] New: Properly account for WineTestBot errors

http://bugs.winehq.org/show_bug.cgi?id=32003

Bug #: 32003
           Summary: Properly account for WineTestBot errors
           Product: Wine-Testbot
           Version: unspecified
          Platform: x86
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: unknown
        AssignedTo: [e-mail]
        ReportedBy: [e-mail]
    Classification: Unclassified

Sometimes a test cannot be run on a Windows VM for a reason internal to
WineTestBot rather than due to the patch author. This can be due to a
WineTestBot bug or due to an administrator error. For instance:
 * A Linux VM that got declared as a Windows one.
 * TestAgentd was not started on the VM.
 * The VM is not reachable because it has the wrong IP address or some such.

In those cases WineTestBot entirely reports something like this in the log:
 * Can't copy exe to VM (Bad IP address)
 * Child process died unexpectedly (Linux VM used as a Windows one)

And yet on the summary page it shows 'completed - 0 failures'. Instead the
summary page should either show something like 'completed with errors - 0
failures' or 'completed - 1 failures' so one knows something is wrong.

[Bug 32003] Properly account for WineTestBot errors

http://bugs.winehq.org/show_bug.cgi?id=32003

François Gouget  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
         AssignedTo|[e-mail]        |[e-mail]

--- Comment #1 from François Gouget  2012-10-24 17:24:32 CDT ---
The logic for UpdateStatus() and Check(), where it is duplicated, is broken: if
a task failed its step is marked as failed but the job won't be marked as
failed unless the step was a build step. Presumably this was done this way to
avoid having test failures cause a job to be marked as failed. However a task
status only reflects whether the task ran as expected. So if there are test
failures the TestFailures field will be non-zero and the Status 'completed'.

I will send a patch to fix this and deduplicate the code.

[Bug 32003] Properly account for WineTestBot errors

http://bugs.winehq.org/show_bug.cgi?id=32003

--- Comment #2 from François Gouget  2013-01-30 10:38:20 CST ---
This is mostly done:

commit 4c266ab9e904222186b0fae0c28b190b80d22ce1
Author: Francois Gouget 
Date:   Fri Nov 2 00:11:00 2012 +0100

testbot: Fix the Job and Step status update.

A job that has a non-build step fail should still be marked as
    failed. That's because when there are test failures the TestFailures
    field gets set to a non-zero value but the task's Status is still set
    to completed. So a failed Step really reflects a bot error.  Also
    simplify the code and avoid duplicating the UpdateStatus() logic in
    the Check() method.

However now if a patch fails to apply or compile the job is marked as failed
which may not be appropriate.

[Bug 32003] Properly account for WineTestBot errors

http://bugs.winehq.org/show_bug.cgi?id=32003

François Gouget  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #3 from François Gouget  2013-04-12 10:45:42 CDT ---
The remaining issue got fixed by the following commit:

commit fab277d195f56acd2e5e6078045077df215fdfa5
Author: Francois Gouget 
Date:   Fri Mar 29 14:02:16 2013 +0100

testbot: Add new status values to distinguish bad patches, build errors and
TestBot errors.

Also rename the 'failed' status to 'boterror' to make its meaning
    clearer and avoid confusion with test failures.

[Bug 32003] Properly account for WineTestBot errors

http://bugs.winehq.org/show_bug.cgi?id=32003

André H.  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED
                 CC|                            |[e-mail]

--- Comment #4 from André H.  2013-04-13 16:33:12 CDT ---
closing