[Bug 15716] New: Wine testcases should not use the public internet, as it causes flakiness
http://bugs.winehq.org/show_bug.cgi?id=15716
Summary: Wine testcases should not use the public internet, as it causes flakiness Product: Wine Version: 1.1.6 Platform: Other OS/Version: other Status: NEW Severity: normal Priority: P2 Component: testcases AssignedTo: wine-bugs@winehq.org ReportedBy: dank@kegel.com
Just about every day, patchwatcher rejects a patch incorrectly because some conformance test is relying on the Internet to be there. Sadly, the Internet is not 100% reliable, and therefore must not be used by any test suite that needs to be 100% reliable.
We must expunge all use of the internet from our test suite, in favor of mock servers running locally.
One way to start would be to add a configure-time option to cause all hostname resolution to return 127.0.0.1. Then people who needed ultrarepeatable conformance test results (like patchwatcher) could use that option and never have flaky results due to the public internet.
http://bugs.winehq.org/show_bug.cgi?id=15716
--- Comment #1 from Dan Kegel dank@kegel.com 2008-10-28 09:13:58 --- Also, the current practice of checking early in the test to see if the net is available, and skipping the test if it's not, is not sufficient, because the net could go down after this check. We need our tests to be thoroughly insulated from the public internet; a bandaid like that is fine for development, but insufficient for production testing.
http://bugs.winehq.org/show_bug.cgi?id=15716
--- Comment #2 from Austin English austinenglish@gmail.com 2008-10-28 12:05:18 --- (In reply to comment #1)
Also, the current practice of checking early in the test to see if the net is available, and skipping the test if it's not, is not sufficient, because the net could go down after this check. We need our tests to be thoroughly insulated from the public internet; a bandaid like that is fine for development, but insufficient for production testing.
Why not install Apache on the patchwatcher box and edit /etc/hosts to point to that local server?
http://bugs.winehq.org/show_bug.cgi?id=15716
--- Comment #3 from Dan Kegel dank@kegel.com 2008-10-28 12:12:07 --- This isn't really patchwatcher specific. What we need is an easy script that anybody can run that starts all the needed services. As new tests are added that need new services, the developers who add those tests should add the setup for the needed services to the server setup script.
Alternately, we could have each test set up its own mock server. (That's harder, but is fairly common practice in many projects.)
http://bugs.winehq.org/show_bug.cgi?id=15716
--- Comment #4 from Dan Kegel dank@kegel.com 2008-10-28 12:13:51 --- Ideally, the script should run well on either cygwin or ubuntu. That would let people who run the winetest executable on windows give good results even if their net connection is flaky, and raise the quality of the results reported to test.winehq.org.
http://bugs.winehq.org/show_bug.cgi?id=15716
Austin English austinenglish@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |austinenglish@gmail.com Keywords| |source, testcase
http://bugs.winehq.org/show_bug.cgi?id=15716
Jeff Zaroyko jeffz@jeffz.name changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |jeffz@jeffz.name
--- Comment #5 from Jeff Zaroyko jeffz@jeffz.name 2008-10-28 20:35:36 --- I might mention here that I submit results to test.winehq.org while running winetest without an internet connection on a win98 machine and now also w2k8.
http://bugs.winehq.org/show_bug.cgi?id=15716
--- Comment #6 from Dan Kegel dank@kegel.com 2008-10-28 21:54:56 --- Flaky connections can produce lots more failures than absent ones. This bug is about protecting us from flaky connections, so it's neither here nor there how flaky tests are when there is no internet connection.
http://bugs.winehq.org/show_bug.cgi?id=15716
Saulius K. saulius2@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |saulius2@gmail.com
http://bugs.winehq.org/show_bug.cgi?id=15716
butraxz@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |butraxz@gmail.com
--- Comment #7 from butraxz@gmail.com 2012-05-18 05:59:25 CDT --- This bug has not been updated for four years. Is this still an issue i current (1.5.4) or newer wine ? You may also close this as abandoned if you feel that that this is issue is no longer relevant to you.
https://bugs.winehq.org/show_bug.cgi?id=15716
roger@mailinator.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |roger@mailinator.com
--- Comment #8 from roger@mailinator.com --- Last response Dan Kegel 2008-10-28 21:54:56 CDT
what do you want to do about this issue?
https://bugs.winehq.org/show_bug.cgi?id=15716
François Gouget fgouget@codeweavers.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |fgouget@codeweavers.com
--- Comment #9 from François Gouget fgouget@codeweavers.com --- Jacek can probably provide more details but my understanding is that we have SSL tests that require connecting to a server having a valid (and possibly various kinds of invalid) SSL certificates. That's all pretty hard to do with a local server.
So I don't think this is possible. I'm also not convinced that the reliability of the Internet connection is a significant issue. It seems to be fine for the WineTestBot for instance (at least from the tests side).
-
wine-bugs＠winehq.org