Question on Conformance Test

[James McKenzie]

All:

I am writing a conformance test for the richedit function 
EM_FINDWORDBREAK.  I realize that WindowsNT 4.0 is our base 
configuration and most of the values process the same.  However, a 
couple of values process differently for Windows NT 4.0/Windows2000, 
WindowsXP/Windows2003 and Windows Vista onward.  What would be the BEST 
method of annotating this:

Per the development guide:

ok ( GetLastError() == WINXP_ERROR || GetLastError() == WINNT40_ERROR, ...);

or:

ok ( GetLastError() == WINXP_ERROR || Broken (GetLastError() == WINNT40_ERROR));

In other words, should I avoid the use of Broken() in this case, as the returned value is correct?

Thank you.

James McKenzie

[Austin English]

On Tue, Apr 26, 2011 at 22:50, James McKenzie  wrote:
> All:
>
> I am writing a conformance test for the richedit function EM_FINDWORDBREAK.
>  I realize that WindowsNT 4.0 is our base configuration and most of the
> values process the same.  However, a couple of values process differently
> for Windows NT 4.0/Windows2000, WindowsXP/Windows2003 and Windows Vista
> onward.  What would be the BEST method of annotating this:
>
> Per the development guide:
>
> ok ( GetLastError() == WINXP_ERROR || GetLastError() == WINNT40_ERROR, ...);
>
> or:
>
> ok ( GetLastError() == WINXP_ERROR || Broken (GetLastError() ==
> WINNT40_ERROR));
>
> In other words, should I avoid the use of Broken() in this case, as the
> returned value is correct?

Normal practice is to try to match the XP+ behavior, and mark NT as
broken if it differs.

[Stefan Dösinger]

On Wednesday 27 April 2011 18:05:59 Austin English wrote:
> Normal practice is to try to match the XP+ behavior, and mark NT as
> broken if it differs.

In d3d we prefer results that some applications depend on. E.g. Windows XP has 
pretty relaxed or no error checking on IDirect3DVertexBuffer9::Lock, even 
though msdn says otherwise. Windows Vista added error checking as documented 
in the msdn. There are a few games out there that depend on the lack of error 
checking and crash on Windows Vista(or crashed at the time). We decided to 
replicate the behavior that kept the games working in Wine and mark the 
stricter check as broken()

If we don't know of any app that depends on the behavior we replicate the one 
that is nicer from an API design point of view and mark the others with 
broken()

If a result is clearly wrong(violates the docs, and doesn't make any sense) I 
mark it as broken if we don't know any "mainstream" app that depends on 
it(e.g. if the test was written to understand how something works). 
Alternatively we may just get rid of the test. If there's a mainstream app(ie, 
something you can buy in a shop, but not some random techdemo, tutorial or 
demoscene stuff) that depends on the behavior and this app doesn't work on this 
driver/Windows version I let the test fail. Sorry, your driver's broken, do 
not pass go, do not collect $200, do not pass the tests. This mostly applies 
to old hardware like my Radeon Mobility 9000, non-gaming hardware(Intel GPUs 
to a certain extend, rare stuff like Matrox cards) or outdated drivers.