Babystep: Testing framework
The attached revision to Alexandres patch modifies the Wine makefiles such that a Linux developer can: make test (to confirm all regression tests) make clean-test (to clear the tests and try again) It also creates three sample tests in programs/winetest/samples which hopefully demonstrate to a test author how to write a test. (The *.test files are human readable). It has provisions for Perl and Winelib/C tests, as well as allowing for the arbitrary execution of any other command (which should allow any other test tools to be plugged in). It has many flaws, but is intended as a possible jumpstart to get this process rolling. Flaws: 1. It has no provision for use on Windows. IMO the right way to fix this is to go through the effort of making Wine's ./configure script be intelligent enough to build 'Winelib' programs, so that in Windows a 'make test' should just work. 2. I really don't like that a C/Winelib test requires its own directory (see samples/sample3). However, AFAICT, that was the only way to create a simple and clean build environment for the Winelib app. 3. No doco. Thoughts? Comments? Jer diff --exclude=CVS -rNu oldwine/programs/winetest/Makefile.in wine/programs/winetest/Makefile.in --- oldwine/programs/winetest/Makefile.in Sat Nov 24 11:07:14 2001 +++ wine/programs/winetest/Makefile.in Wed Jan 9 09:20:50 2002 @@ -12,6 +12,8 @@ EXTRA_OBJS = wine.o +SUBDIRS = samples + @MAKE_PROG_RULES@ diff --exclude=CVS -rNu oldwine/programs/winetest/runtest wine/programs/winetest/runtest --- oldwine/programs/winetest/runtest Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/runtest Wed Jan 9 10:22:42 2002 @@ -0,0 +1,70 @@ +#!/bin/sh +#---------------------------------------------------------------------------- +# runtest <wine dir> <test name> <output file name> +# Run a regression test +# +# This script sources a '.test' file which must set one of the following: +# test_invoke (a command to invoke) +# test_perl_script (a Perl test script to be run with winetest) +# and can set one of the following: +# test_reference=<name of reference file with required diff> +# test_pattern=<pattern required in output> +#---------------------------------------------------------------------------- +topobjdir="$1" +shift +perldir="$topobjdir/programs/winetest" +LD_LIBRARY_PATH="$topobjdir/dlls:$topobjdir:$LD_LIBRARY_PATH" +export LD_LIBRARY_PATH +WINESERVER="$topobjdir/server/wineserver" +export WINESERVER + +testfile="$1" +testfiledir=`dirname $testfile` +shift + +outfile="$1" + + +export test_invoke= +export test_perl_script= +export test_reference= +export test_pattern= + +source $testfile + + +if [ "$test_perl_script" ] ; then + $perldir/winetest -debugmsg -all -- -I $perldir $testfiledir/"$test_perl_script" >$outfile +elif [ "$test_invoke" ] ; then + $testfiledir/$test_invoke >$outfile +else + echo "Error - neither test_invoke nor test_perl_script set in .test file" + rm -f $outfile + exit 1 +fi +return_status=$? + +if [ $return_status -ne 0 ] ; then + echo "Test $@ failed - return status $return_status" + rm -f $outfile + exit $return_status +fi + +if [ "$test_reference" ] ; then + diff -q $outfile $testfiledir/"$test_reference" >/dev/null 2>&1 + if [ $? -ne 0 ] ; then + echo "Test $@ failed - output failed to match reference" + rm -f $outfile + exit $? + fi +fi + +if [ "$test_pattern" ] ; then + grep -q "$test_pattern" $outfile + if [ $? -ne 0 ] ; then + echo "Test $@ failed - pattern $test_pattern not found" + rm -f $outfile + exit $? + fi +fi + diff --exclude=CVS -rNu oldwine/programs/winetest/samples/Makefile.in wine/programs/winetest/samples/Makefile.in --- oldwine/programs/winetest/samples/Makefile.in Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/Makefile.in Wed Jan 9 10:23:08 2002 @@ -0,0 +1,14 @@ +TOPSRCDIR = @top_srcdir@ +TOPOBJDIR = ../../.. +SRCDIR = @srcdir@ +VPATH = @srcdir@ +MODULE = samples + +SUBDIRS = sample3 + +TESTS = sample1.test \ + sample2.test \ + sample3.test + +(a)MAKE_RULES@ + diff --exclude=CVS -rNu oldwine/programs/winetest/samples/sample1.ref wine/programs/winetest/samples/sample1.ref --- oldwine/programs/winetest/samples/sample1.ref Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/sample1.ref Tue Jan 8 22:51:51 2002 @@ -0,0 +1 @@ +Sample Test 1 Output diff --exclude=CVS -rNu oldwine/programs/winetest/samples/sample1.sh wine/programs/winetest/samples/sample1.sh --- oldwine/programs/winetest/samples/sample1.sh Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/sample1.sh Tue Jan 8 22:51:32 2002 @@ -0,0 +1,2 @@ +#!/bin/bash +echo Sample Test 1 Output diff --exclude=CVS -rNu oldwine/programs/winetest/samples/sample1.test wine/programs/winetest/samples/sample1.test --- oldwine/programs/winetest/samples/sample1.test Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/sample1.test Tue Jan 8 23:29:30 2002 @@ -0,0 +1,15 @@ +#!/bin/sh +#---------------------------------------------------------------------------- +# Wine Regression Test Driver File +# +# Name: +# sample1 +# +# Test Purpose: +# To show how to write a simple test that runs a shell script +# +#---------------------------------------------------------------------------- +#test_perl_script=<name of Perl script to be used> +test_invoke=sample1.sh +test_reference=sample1.ref +test_pattern='Sample Test' diff --exclude=CVS -rNu oldwine/programs/winetest/samples/sample2.pl wine/programs/winetest/samples/sample2.pl --- oldwine/programs/winetest/samples/sample2.pl Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/sample2.pl Tue Jan 8 22:59:07 2002 @@ -0,0 +1,28 @@ +# +# Test script for the winetest program +# + +use wine; + +$wine::debug = 0; + +################################################################ +# Declarations for functions we use in this script + +wine::declare( "kernel32", + GetCurrentProcessId => "int" +); + +################################################################ +# Test some simple function calls + +# Test string arguments +$pid = GetCurrentProcessId(); +print "pid is *$pid\n"; + +if ($pid == 0) +{ + exit 1 +}; + +exit 0; diff --exclude=CVS -rNu oldwine/programs/winetest/samples/sample2.test wine/programs/winetest/samples/sample2.test --- oldwine/programs/winetest/samples/sample2.test Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/sample2.test Tue Jan 8 22:54:53 2002 @@ -0,0 +1,16 @@ +#!/bin/sh +#---------------------------------------------------------------------------- +# Wine Regression Test Driver File +# +# Name: +# sample2 +# +# Test Purpose: +# Show how to use the Perl winetest prgoram to invoke a simple +# Windows API and check results +# +#---------------------------------------------------------------------------- +test_perl_script=sample2.pl +#test_invoke=<cmd to invoke> +#test_reference= +#test_pattern= diff --exclude=CVS -rNu oldwine/programs/winetest/samples/sample3/Makefile.in wine/programs/winetest/samples/sample3/Makefile.in --- oldwine/programs/winetest/samples/sample3/Makefile.in Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/sample3/Makefile.in Wed Jan 9 10:21:36 2002 @@ -0,0 +1,12 @@ +TOPSRCDIR = @top_srcdir@ +TOPOBJDIR = ../../../.. +SRCDIR = @srcdir@ +VPATH = @srcdir@ +MODULE = sample3 + +C_SRCS = \ + sample3.c + +(a)MAKE_PROG_RULES@ + +### Dependencies: diff --exclude=CVS -rNu oldwine/programs/winetest/samples/sample3/sample3.c wine/programs/winetest/samples/sample3/sample3.c --- oldwine/programs/winetest/samples/sample3/sample3.c Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/sample3/sample3.c Wed Jan 9 09:14:36 2002 @@ -0,0 +1,16 @@ +#include <windows.h> + + +void main(void) +{ + HANDLE pid; + + pid = GetCurrentProcessId(); + + printf("pid is 0x%x\n", pid); + + if (pid) + exit(0); + + exit(1); +} diff --exclude=CVS -rNu oldwine/programs/winetest/samples/sample3/sample3.spec wine/programs/winetest/samples/sample3/sample3.spec --- oldwine/programs/winetest/samples/sample3/sample3.spec Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/sample3/sample3.spec Wed Jan 9 09:15:25 2002 @@ -0,0 +1,16 @@ +name sample3 +type win32 +mode cuiexe +init main + +import advapi32.dll +import comdlg32.dll +import gdi32.dll +import kernel32.dll +import ntdll.dll +import odbc32.dll +import ole32.dll +import oleaut32.dll +import shell32.dll +import user32.dll +import winspool.drv diff --exclude=CVS -rNu oldwine/programs/winetest/samples/sample3.test wine/programs/winetest/samples/sample3.test --- oldwine/programs/winetest/samples/sample3.test Wed Dec 31 18:00:00 1969 +++ wine/programs/winetest/samples/sample3.test Wed Jan 9 10:20:24 2002 @@ -0,0 +1,16 @@ +#!/bin/sh +#---------------------------------------------------------------------------- +# Wine Regression Test Driver File +# +# Name: +# sample3 +# +# Test Purpose: +# Show how to use a C Winelib to invoke a simple +# Windows API and check results +# +#---------------------------------------------------------------------------- +#test_perl_script=<perl script> +test_invoke=sample3/sample3 +#test_reference= +#test_pattern= --- oldwine/Make.rules.in Fri Dec 14 17:14:22 2001 +++ wine/Make.rules.in Wed Jan 9 09:10:16 2002 @@ -58,6 +58,8 @@ ALLLINTFLAGS = $(LINTFLAGS) $(DEFS) $(OPTIONS) $(DIVINCL) WINAPI_CHECK = $(TOPSRCDIR)/tools/winapi_check/winapi_check WINEBUILD = $(TOPOBJDIR)/tools/winebuild/winebuild +WINETEST = $(TOPOBJDIR)/programs/winetest/winetest +RUNTEST = $(TOPOBJDIR)/programs/winetest/runtest MAKEDEP = $(TOPOBJDIR)/tools/makedep WRC = $(TOPOBJDIR)/tools/wrc/wrc WMC = $(TOPOBJDIR)/tools/wmc/wmc @@ -95,7 +97,7 @@ # Implicit rules -.SUFFIXES: .mc .rc .mc.rc .res .spec .spec.c .glue.c +.SUFFIXES: .mc .rc .mc.rc .res .spec .spec.c .glue.c .test .test.out .c.o: $(CC) -c $(ALLCFLAGS) -o $@ $< @@ -121,6 +123,9 @@ .c.ln: $(LINT) -c $(ALLLINTFLAGS) $< || ( $(RM) $@ && exit 1 ) +.test.test.out: + $(RUNTEST) $(TOPOBJDIR) $< $@ + .PHONY: all install uninstall clean distclean depend dummy # 'all' target first in case the enclosing Makefile didn't define any target @@ -216,7 +221,7 @@ -cd `dirname $@` && $(RM) $(CLEAN_FILES) clean:: $(SUBDIRS:%=%/__clean__) $(EXTRASUBDIRS:%=%/__clean__) - $(RM) $(CLEAN_FILES) $(GEN_C_SRCS) $(GEN_ASM_SRCS) $(RC_SRCS:.rc=.res) $(RC_SRCS16:.rc=.res) $(MC_SRCS:.mc=.mc.rc) $(PROGRAMS) + $(RM) $(CLEAN_FILES) $(GEN_C_SRCS) $(GEN_ASM_SRCS) $(RC_SRCS:.rc=.res) $(RC_SRCS16:.rc=.res) $(MC_SRCS:.mc=.mc.rc) $(TESTS:%=%.out) $(PROGRAMS) # Rules for installing @@ -226,6 +231,22 @@ $(SUBDIRS:%=%/__uninstall__): dummy cd `dirname $@` && $(MAKE) uninstall +# Rules for testing + +$(SUBDIRS:%=%/__clean-test__): dummy + @cd `dirname $@` && $(MAKE) clean-test + +$(SUBDIRS:%=%/__test__): dummy + @cd `dirname $@` && $(MAKE) test + +test:: $(SUBDIRS:%=%/__test__) $(WINETEST) $(TESTS:%=%.out) + +clean-test:: $(SUBDIRS:%=%/__clean-test__) + $(RM) $(TESTS:%=%.out) + +$(WINETEST): + cd $(TOPOBJDIR)/programs/winetest && $(MAKE) winetest + # Misc. rules $(SPEC_SRCS:.spec=.spec.c): $(WINEBUILD) --- oldwine/Makefile.in Sun Jan 6 12:38:45 2002 +++ wine/Makefile.in Wed Jan 9 09:10:16 2002 @@ -130,6 +130,9 @@ @cd dlls && $(MAKE) checklink @cd debugger && $(MAKE) checklink +test:: + @cd dlls && $(MAKE) test + TAGS etags: etags `find $(TOPSRCDIR) -name '*.[chS]' -print | grep -v dbgmain` --- oldwine/configure Mon Jan 7 15:00:27 2002 +++ wine/configure Wed Jan 9 10:20:42 2002 @@ -6250,16 +6250,15 @@ #line 6251 "configure" #include "confdefs.h" #include <stdio.h> -#include <sys/types.h> -main() +int main() { FILE *f=fopen("conftestval", "w"); - if (!f) exit(1); + if (!f) return(1); fprintf(f, "%d\n", sizeof(long long)); - exit(0); + return(0); } EOF -if { (eval echo configure:6263: \"$ac_link\") 1>&5; (eval $ac_link) 2>&5; } && test -s conftest${ac_exeext} && (./conftest; exit) 2>/dev/null +if { (eval echo configure:6262: \"$ac_link\") 1>&5; (eval $ac_link) 2>&5; } && test -s conftest${ac_exeext} && (./conftest; exit) 2>/dev/null then ac_cv_sizeof_long_long=`cat conftestval` else @@ -6280,12 +6279,12 @@ echo $ac_n "checking whether linux/input.h is for real""... $ac_c" 1>&6 -echo "configure:6284: checking whether linux/input.h is for real" >&5 +echo "configure:6283: checking whether linux/input.h is for real" >&5 if eval "test \"`echo '$''{'wine_cv_linux_input_h'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6289 "configure" +#line 6288 "configure" #include "confdefs.h" #include <linux/input.h> @@ -6298,7 +6297,7 @@ ; return 0; } EOF -if { (eval echo configure:6302: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6301: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* wine_cv_linux_input_h=yes else @@ -6322,12 +6321,12 @@ echo $ac_n "checking whether we can use re-entrant gethostbyname_r Linux style""... $ac_c" 1>&6 -echo "configure:6326: checking whether we can use re-entrant gethostbyname_r Linux style" >&5 +echo "configure:6325: checking whether we can use re-entrant gethostbyname_r Linux style" >&5 if eval "test \"`echo '$''{'wine_cv_linux_gethostbyname_r_6'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6331 "configure" +#line 6330 "configure" #include "confdefs.h" #include <netdb.h> @@ -6348,7 +6347,7 @@ ; return 0; } EOF -if { (eval echo configure:6352: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6351: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* wine_cv_linux_gethostbyname_r_6=yes else @@ -6374,12 +6373,12 @@ if test "$ac_cv_header_linux_joystick_h" = "yes" then echo $ac_n "checking whether linux/joystick.h uses the Linux 2.2+ API""... $ac_c" 1>&6 -echo "configure:6378: checking whether linux/joystick.h uses the Linux 2.2+ API" >&5 +echo "configure:6377: checking whether linux/joystick.h uses the Linux 2.2+ API" >&5 if eval "test \"`echo '$''{'wine_cv_linux_joystick_22_api'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6383 "configure" +#line 6382 "configure" #include "confdefs.h" #include <sys/ioctl.h> @@ -6394,7 +6393,7 @@ /*empty*/ ; return 0; } EOF -if { (eval echo configure:6398: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6397: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* wine_cv_linux_joystick_22_api=yes else @@ -6421,12 +6420,12 @@ if test "$ac_cv_header_sys_vfs_h" = "yes" then echo $ac_n "checking whether sys/vfs.h defines statfs""... $ac_c" 1>&6 -echo "configure:6425: checking whether sys/vfs.h defines statfs" >&5 +echo "configure:6424: checking whether sys/vfs.h defines statfs" >&5 if eval "test \"`echo '$''{'wine_cv_sys_vfs_has_statfs'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6430 "configure" +#line 6429 "configure" #include "confdefs.h" #include <sys/types.h> @@ -6443,7 +6442,7 @@ ; return 0; } EOF -if { (eval echo configure:6447: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6446: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* wine_cv_sys_vfs_has_statfs=yes else @@ -6470,12 +6469,12 @@ if test "$ac_cv_header_sys_statfs_h" = "yes" then echo $ac_n "checking whether sys/statfs.h defines statfs""... $ac_c" 1>&6 -echo "configure:6474: checking whether sys/statfs.h defines statfs" >&5 +echo "configure:6473: checking whether sys/statfs.h defines statfs" >&5 if eval "test \"`echo '$''{'wine_cv_sys_statfs_has_statfs'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6479 "configure" +#line 6478 "configure" #include "confdefs.h" #include <sys/types.h> @@ -6490,7 +6489,7 @@ ; return 0; } EOF -if { (eval echo configure:6494: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6493: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* wine_cv_sys_statfs_has_statfs=yes else @@ -6517,12 +6516,12 @@ if test "$ac_cv_header_sys_mount_h" = "yes" then echo $ac_n "checking whether sys/mount.h defines statfs""... $ac_c" 1>&6 -echo "configure:6521: checking whether sys/mount.h defines statfs" >&5 +echo "configure:6520: checking whether sys/mount.h defines statfs" >&5 if eval "test \"`echo '$''{'wine_cv_sys_mount_has_statfs'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6526 "configure" +#line 6525 "configure" #include "confdefs.h" #include <sys/types.h> @@ -6537,7 +6536,7 @@ ; return 0; } EOF -if { (eval echo configure:6541: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6540: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* wine_cv_sys_mount_has_statfs=yes else @@ -6563,7 +6562,7 @@ echo $ac_n "checking for statfs.f_bfree""... $ac_c" 1>&6 -echo "configure:6567: checking for statfs.f_bfree" >&5 +echo "configure:6566: checking for statfs.f_bfree" >&5 if eval "test \"`echo '$''{'wine_cv_statfs_bfree'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else @@ -6572,7 +6571,7 @@ wine_cv_statfs_bfree=no else cat > conftest.$ac_ext <<EOF -#line 6576 "configure" +#line 6575 "configure" #include "confdefs.h" #include <sys/types.h> @@ -6599,7 +6598,7 @@ ; return 0; } EOF -if { (eval echo configure:6603: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6602: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* wine_cv_statfs_bfree=yes else @@ -6623,7 +6622,7 @@ fi echo $ac_n "checking for statfs.f_bavail""... $ac_c" 1>&6 -echo "configure:6627: checking for statfs.f_bavail" >&5 +echo "configure:6626: checking for statfs.f_bavail" >&5 if eval "test \"`echo '$''{'wine_cv_statfs_bavail'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else @@ -6632,7 +6631,7 @@ wine_cv_statfs_bavail=no else cat > conftest.$ac_ext <<EOF -#line 6636 "configure" +#line 6635 "configure" #include "confdefs.h" #include <sys/types.h> @@ -6659,7 +6658,7 @@ ; return 0; } EOF -if { (eval echo configure:6663: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6662: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* wine_cv_statfs_bavail=yes else @@ -6684,12 +6683,12 @@ echo $ac_n "checking for msg_accrights in struct msghdr""... $ac_c" 1>&6 -echo "configure:6688: checking for msg_accrights in struct msghdr" >&5 +echo "configure:6687: checking for msg_accrights in struct msghdr" >&5 if eval "test \"`echo '$''{'ac_cv_c_msg_accrights'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6693 "configure" +#line 6692 "configure" #include "confdefs.h" #include <sys/types.h> #include <sys/socket.h> @@ -6697,7 +6696,7 @@ struct msghdr hdr; hdr.msg_accrights=0 ; return 0; } EOF -if { (eval echo configure:6701: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6700: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* ac_cv_c_msg_accrights="yes" else @@ -6720,12 +6719,12 @@ echo $ac_n "checking for sa_len in struct sockaddr""... $ac_c" 1>&6 -echo "configure:6724: checking for sa_len in struct sockaddr" >&5 +echo "configure:6723: checking for sa_len in struct sockaddr" >&5 if eval "test \"`echo '$''{'ac_cv_c_sockaddr_sa_len'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6729 "configure" +#line 6728 "configure" #include "confdefs.h" #include <sys/types.h> #include <sys/socket.h> @@ -6734,7 +6733,7 @@ static struct sockaddr addr; addr.sa_len = 1 ; return 0; } EOF -if { (eval echo configure:6738: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6737: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* ac_cv_c_sockaddr_sa_len="yes" else @@ -6757,12 +6756,12 @@ echo $ac_n "checking for sun_len in struct sockaddr_un""... $ac_c" 1>&6 -echo "configure:6761: checking for sun_len in struct sockaddr_un" >&5 +echo "configure:6760: checking for sun_len in struct sockaddr_un" >&5 if eval "test \"`echo '$''{'ac_cv_c_sockaddr_sun_len'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6766 "configure" +#line 6765 "configure" #include "confdefs.h" #include <sys/types.h> #include <sys/socket.h> @@ -6771,7 +6770,7 @@ static struct sockaddr_un addr; addr.sun_len = 1 ; return 0; } EOF -if { (eval echo configure:6775: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then +if { (eval echo configure:6774: \"$ac_compile\") 1>&5; (eval $ac_compile) 2>&5; }; then rm -rf conftest* ac_cv_c_sockaddr_sun_len="yes" else @@ -6794,12 +6793,12 @@ echo $ac_n "checking whether we need to define __i386__""... $ac_c" 1>&6 -echo "configure:6798: checking whether we need to define __i386__" >&5 +echo "configure:6797: checking whether we need to define __i386__" >&5 if eval "test \"`echo '$''{'ac_cv_cpp_def_i386'+set}'`\" = set"; then echo $ac_n "(cached) $ac_c" 1>&6 else cat > conftest.$ac_ext <<EOF -#line 6803 "configure" +#line 6802 "configure" #include "confdefs.h" #if (defined(i386) || defined(__i386)) && !defined(__i386__) yes @@ -7075,6 +7074,8 @@ programs/wineconsole/Makefile programs/winemine/Makefile programs/winetest/Makefile +programs/winetest/samples/Makefile +programs/winetest/samples/sample3/Makefile programs/winhelp/Makefile programs/winver/Makefile relay32/Makefile @@ -7338,6 +7339,8 @@ programs/wineconsole/Makefile programs/winemine/Makefile programs/winetest/Makefile +programs/winetest/samples/Makefile +programs/winetest/samples/sample3/Makefile programs/winhelp/Makefile programs/winver/Makefile relay32/Makefile --- oldwine/configure.in Mon Jan 7 15:00:27 2002 +++ wine/configure.in Wed Jan 9 10:20:39 2002 @@ -1328,6 +1328,8 @@ programs/wineconsole/Makefile programs/winemine/Makefile programs/winetest/Makefile +programs/winetest/samples/Makefile +programs/winetest/samples/sample3/Makefile programs/winhelp/Makefile programs/winver/Makefile relay32/Makefile --- oldwine/dlls/Makedll.rules.in Mon Sep 17 15:09:08 2001 +++ wine/dlls/Makedll.rules.in Wed Jan 9 09:10:16 2002 @@ -36,6 +36,10 @@ checklink:: lib$(MODULE).$(LIBEXT) $(CC) -o checklink $(TOPSRCDIR)/library/checklink.c -L. -l$(MODULE) $(ALL_LIBS) && $(RM) checklink +# Rules for testing + +$(TESTS:%=tests/%.out): lib$(MODULE).$(LIBEXT) + # Rules for debug channels debug_channels: dummy
Jeremy White <jwhite(a)codeweavers.com> writes:
It has provisions for Perl and Winelib/C tests, as well as allowing for the arbitrary execution of any other command (which should allow any other test tools to be plugged in).
I can't say I like your *.test files; I think this should be taken care of by the makefile directly. Also it seems the consensus is that we should simply check the exit status of the test, not compare the test output.
Flaws: 1. It has no provision for use on Windows. IMO the right way to fix this is to go through the effort of making Wine's ./configure script be intelligent enough to build 'Winelib' programs, so that in Windows a 'make test' should just work.
I'm not sure about that; I think it would be preferable to have a simple, self-contained environment. Running configure would require a lot of stuff to be installed on the Windows side.
2. I really don't like that a C/Winelib test requires its own directory (see samples/sample3). However, AFAICT, that was the only way to create a simple and clean build environment for the Winelib app.
That's the problem with C tests. You cannot have one executable for each test, at least not with Winelib, so we need a way to build multiple tests inside a single executable. Not very hard, but there's a bit of work involved. -- Alexandre Julliard julliard(a)winehq.com
I can't say I like your *.test files; I think this should be taken care of by the makefile directly.
I started down this road, but then I ended up needing PERL_TESTS and INVOKE_TESTS in the Makefile, and IMHO, that was uglier than the *.test file. This may simply be a lack of Makefile moxy on my part; the point where I threw up my hands in disgust was when I was requiring my Winelib/C tests to be compiled into a 'foo.itest' so that I could do a .itest.test.out dependency rule.
Also it seems the consensus is that we should simply check the exit status of the test, not compare the test output.
Sorry, this wasn't clear. That is the default behavior. I preserved the pattern and reference checking as an optional addition.
Flaws: 1. It has no provision for use on Windows. IMO the right way to fix this is to go through the effort of making Wine's ./configure script be intelligent enough to build 'Winelib' programs, so that in Windows a 'make test' should just work.
I'm not sure about that; I think it would be preferable to have a simple, self-contained environment. Running configure would require a lot of stuff to be installed on the Windows side.
Well, I was sufficiently impressed with the Cygwin install to think it wouldn't be that bad; if nothing else, I think we should try to have both.
2. I really don't like that a C/Winelib test requires its own directory (see samples/sample3). However, AFAICT, that was the only way to create a simple and clean build environment for the Winelib app.
That's the problem with C tests. You cannot have one executable for each test, at least not with Winelib, so we need a way to build multiple tests inside a single executable. Not very hard, but there's a bit of work involved.
I would argue, that except perhaps for your objection to my *.test format, that the framework as I have proposed would be a good start. We can refine the Winelib stuff later. It seems to me that the alternate is to spend another five weeks debating possible frameworks, and ending with none implemented. Jer
Jeremy White <jwhite(a)codeweavers.com> writes:
Well, I was sufficiently impressed with the Cygwin install to think it wouldn't be that bad; if nothing else, I think we should try to have both.
If we can support Cygwin without major trouble of course we should do it. But if the idea is to attract Windows developers, offering them a Unix environment may not be the best choice.
I would argue, that except perhaps for your objection to my *.test format, that the framework as I have proposed would be a good start. We can refine the Winelib stuff later.
It seems to me that the alternate is to spend another five weeks debating possible frameworks, and ending with none implemented.
It's not really a question of debating frameworks, it's more a question of getting the building process into a useable form. I'm not going to create dozens of directories and spec files in the CVS tree pending a better solution. But I'll work on getting at least the Perl stuff into the tree, and then we can work on refining the C infrastructure. -- Alexandre Julliard julliard(a)winehq.com
--- Alexandre Julliard <julliard(a)winehq.com> wrote:
I'm not going to create dozens of directories and spec files in the CVS tree pending a better solution
As I mentioned earlier I'm not very experienced with this stuff. Excuse me for asking newbie questions: 1) why do we have to create a separate directory for each winelib application? 2) can we pack .spec information to one big file and then generate .spec files from it? Andriy Palamarchuk __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
On Wed, 9 Jan 2002, Jeremy White wrote: [...]
Also it seems the consensus is that we should simply check the exit status of the test, not compare the test output.
Sorry, this wasn't clear. That is the default behavior. I preserved the pattern and reference checking as an optional addition.
I think we should definitely be able to check the test output against a reference version. [...]
2. I really don't like that a C/Winelib test requires its own directory (see samples/sample3). However, AFAICT, that was the only way to create a simple and clean build environment for the Winelib app.
You don't need to. You can do things like winemaker does and then you can put as many executables and libraries as you like in a single directory. But you may not be able to reuse the standard @MAKE_PROG_RULES(a).
That's the problem with C tests. You cannot have one executable for each test, at least not with Winelib, so we need a way to build multiple tests inside a single executable. Not very hard, but there's a bit of work involved.
Why can't you have one executable per test? Or is there some test/check confusion here? One check per executable, or perl file for that matter would be crazy. -- Francois Gouget fgouget(a)free.fr http://fgouget.free.fr/ La terre est une bêta...
Francois Gouget <fgouget(a)free.fr> writes:
I think we should definitely be able to check the test output against a reference version.
The idea is that there is no test output (except for debugging); everything is checked right inside the test.
Why can't you have one executable per test? Or is there some test/check confusion here? One check per executable, or perl file for that matter would be crazy.
Because a Winelib executable is too heavyweight; you need a spec file, multiple object files, a lot of disk space, etc. You cannot realistically have hundreds of them without creating a horrible mess. -- Alexandre Julliard julliard(a)winehq.com
On 9 Jan 2002, Alexandre Julliard wrote:
Francois Gouget <fgouget(a)free.fr> writes:
I think we should definitely be able to check the test output against a reference version.
The idea is that there is no test output (except for debugging); everything is checked right inside the test.
So the test has to do the diff itself. And no calling 'diff' because it's not portable :-(
Why can't you have one executable per test? Or is there some test/check confusion here? One check per executable, or perl file for that matter would be crazy.
Because a Winelib executable is too heavyweight; you need a spec file, multiple object files, a lot of disk space, etc. You cannot realistically have hundreds of them without creating a horrible mess.
A C test should just be a .spec file and a .c file. If they are all handled the same way it's actually quite simple. In 'Programming Windows 98' I have 148 executables/libraries and 171 C files. It's quite manageable. -- Francois Gouget fgouget(a)free.fr http://fgouget.free.fr/ Good judgment comes from experience, and experience comes from bad judgment -- Barry LePatner
Francois Gouget <fgouget(a)free.fr> writes:
So the test has to do the diff itself. And no calling 'diff' because it's not portable :-(
But you shouldn't need diff at all. You just have to write the test slightly differently to have checks instead of printfs.
A C test should just be a .spec file and a .c file. If they are all handled the same way it's actually quite simple. In 'Programming Windows 98' I have 148 executables/libraries and 171 C files. It's quite manageable.
I'm not sure about that; if you run make in libtest/ you end up with 70 files and more than 7Mb of disk space, for just 10 nearly empty programs. And the makefile is a big mess too. -- Alexandre Julliard julliard(a)winehq.com
Francois Gouget <fgouget(a)free.fr> writes:
So the test has to do the diff itself. And no calling 'diff' because it's not portable :-(
But you shouldn't need diff at all. You just have to write the test slightly differently to have checks instead of printfs.
The normal behaviour of the tests is that they are oblidged to provide an 0 exit code on success; I think that is an excellent standard. However, having the diff feature allows us to more rapdily adapt existing programs to become tests. Since it's done (and it's trivial code), I don't see the harm in leaving in the feature. We can hide it/discourage in in the (as yet unwritten) doco if you like.
A C test should just be a .spec file and a .c file. If they are all handled the same way it's actually quite simple. In 'Programming Windows 98' I have 148 executables/libraries and 171 C files. It's quite manageable.
I'm not sure about that; if you run make in libtest/ you end up with 70 files and more than 7Mb of disk space, for just 10 nearly empty programs. And the makefile is a big mess too.
The entire samples directory that I submitted consisted of 2,705 bytes. For all CVS checkouts and 'normal' Wine use, there will be an insignificant penalty. And yes, in the framework I submitted, if you choose to run 'make test', it chews disk space (and the Winelib tests are fairly slow to launch). But the Makefile.in is, IMO, clean and tight, and easy to replicate. And we don't *have* to use Winelib/C tests - the Perl tests work fine. Further, if we had 70 regression tests, that would be exactly 70 more than we have now! We can fix the big bloatedness later. Finally, the more I think about my structure of using the '.test' files, the more I believe it is the right solution, because I think it gives us great long term flexibility. For example, say we have 70 Winelib/C tests that are big and bloated and we want to fix that. We could define a new make target, say a '.test.so'. We could modify winetest to be able to load a .test.so file and call main() with the right parameters and suddenly all of our bloat is gone. We'd just need a rule to build a .test.so file from a .test.c file, and then we'd need to tweak our foo.test file to add a new command 'so_test=foo.test.so' instead of 'invoke=foo' and we're done. We could probably even eliminate the .spec file (only for console apps, obviously). And another thing, what if I have an app that creates a graphical image and then uses Gimp or some other scripting process to take screen shots and then compares them to a known reference? In a makefile only solution, it'd be a pain to add a target to call some fancy script to perform a regression test. This case is already handled by the .test file. And don't forget the Julian fries! The very best feature of the framework! <g> Jer
Jeremy White <jwhite(a)codeweavers.com> writes:
The normal behaviour of the tests is that they are oblidged to provide an 0 exit code on success; I think that is an excellent standard. However, having the diff feature allows us to more rapdily adapt existing programs to become tests. Since it's done (and it's trivial code), I don't see the harm in leaving in the feature. We can hide it/discourage in in the (as yet unwritten) doco if you like.
The problem is that it's hard to support both types of tests from the makefile, so it would force tests without output to generate dummy output and reference files. I think it's better to make the default be no output and have tests that need output diffs do the extra work.
The entire samples directory that I submitted consisted of 2,705 bytes. For all CVS checkouts and 'normal' Wine use, there will be an insignificant penalty.
"Normal" Wine use should definitely include running regression tests. I don't want to be the only one to run them.
For example, say we have 70 Winelib/C tests that are big and bloated and we want to fix that. We could define a new make target, say a '.test.so'. We could modify winetest to be able to load a .test.so file and call main() with the right parameters and suddenly all of our bloat is gone. We'd just need a rule to build a .test.so file from a .test.c file, and then we'd need to tweak our foo.test file to add a new command 'so_test=foo.test.so' instead of 'invoke=foo' and we're done. We could probably even eliminate the .spec file (only for console apps, obviously).
But without your .test file it's even easier to do, all you have to do is to hack the makefile. Plus you can get the dependencies right (how can make guess what file your .test is referencing?)
And another thing, what if I have an app that creates a graphical image and then uses Gimp or some other scripting process to take screen shots and then compares them to a known reference? In a makefile only solution, it'd be a pain to add a target to call some fancy script to perform a regression test. This case is already handled by the .test file.
You can do that from a Perl test too, but I don't think this should be encouraged. Tests should not depend on external programs any more than strictly necessary, otherwise people will soon need a complete Linux distribution on their Windows machine to run our tests. -- Alexandre Julliard julliard(a)winehq.com
Alexandre Julliard wrote:
Jeremy White <jwhite(a)codeweavers.com> writes:
The normal behaviour of the tests is that they are oblidged to provide an 0 exit code on success; I think that is an excellent standard. However, having the diff feature allows us to more rapdily adapt existing programs to become tests. Since it's done (and it's trivial code), I don't see the harm in leaving in the feature. We can hide it/discourage in in the (as yet unwritten) doco if you like.
The problem is that it's hard to support both types of tests from the makefile, so it would force tests without output to generate dummy output and reference files. I think it's better to make the default be no output and have tests that need output diffs do the extra work.
Um...the way I constructed it, there is no additional effort required in the Makefile whatsoever. Now, if you get rid of my .test file, yes, then, doing this all in the Makefile is a pain in the neck.
"Normal" Wine use should definitely include running regression tests. I don't want to be the only one to run them.
Point taken; but I still believe that this is an issue not relevant to my proposed patch, but is a problem with using C tests as a whole. A solution to this problem will be just as appropriate with or without my patch.
But without your .test file it's even easier to do, all you have to do is to hack the makefile. Plus you can get the dependencies right (how can make guess what file your .test is referencing?)
If the goal is to make it easy for a newcomer to add a test, then requiring them to hack the Makefile is going to be a barrier. Asking them to copy and cut/paste a 'samples/foo.test' file will not be. I do concede the point that my .test file hides dependencies, and a Makefile only approach can do that better.
You can do that from a Perl test too, but I don't think this should be encouraged. Tests should not depend on external programs any more than strictly necessary, otherwise people will soon need a complete Linux distribution on their Windows machine to run our tests.
I threw that out as an example, not necessarily a serious case. I think the bottom line question is whether or not the test scripts should be built such that we depend on a shell script (runtest) and a configuration file for each test (foo.test), or whether we rely completely on the Makefiles to hold the test configuration information. I think either choice is valid, but let me lay down the gauntlet: replace my 'winetest/samples' directory with a Make based equivalent that allows for the same sorts of tests, and makes it similarly obvious to a newcomer how to add a new test. You are better with Make than I am; you may persuade me readily with a patch. Jer p.s. If you're just rejecting the .test files because they have long horizontal comment lines, we could negotiate that...<g>
Jeremy White <jwhite(a)codeweavers.com> writes:
Um...the way I constructed it, there is no additional effort required in the Makefile whatsoever. Now, if you get rid of my .test file, yes, then, doing this all in the Makefile is a pain in the neck.
There is no effort unless you want to get the dependencies right; then you are in for a lot of trouble. With your proposal the only dependency is on the .test file, which will most likely never change over the lifetime of a test. OTOH the Perl/C source will change frequently, but make test will never notice it and won't re-run the test. That defeats the whole purpose of using make.
If the goal is to make it easy for a newcomer to add a test, then requiring them to hack the Makefile is going to be a barrier. Asking them to copy and cut/paste a 'samples/foo.test' file will not be.
Sorry if I wasn't clear. My comment was about your example of adding a new category of tests, like support for .so tests. That's when you need some serious hacking. If all you do is add a Perl or C test, you only have to add a line in the Makefile, just like with your solution.
I think either choice is valid, but let me lay down the gauntlet: replace my 'winetest/samples' directory with a Make based equivalent that allows for the same sorts of tests, and makes it similarly obvious to a newcomer how to add a new test. You are better with Make than I am; you may persuade me readily with a patch.
It would look pretty much the same. Just replace this: TESTS = sample1.test sample2.test sample3.test in your makefile by: PLTESTS = sample2.pl CTESTS = sample3.c and delete the .test files (there is no SHTESTS for shell tests, but I think that's a feature if we want portability). Creating a new test involves simply adding its name on the right line; no .test to write, no .spec file, no new Makefile. (if you'd like to try it I have committed the rules for PLTESTS so this should work now; CTESTS will still need a bit more work) -- Alexandre Julliard julliard(a)winehq.com
--- Jeremy White <jwhite(a)codeweavers.com> wrote:
However, having the diff feature allows us to more rapdily adapt existing programs to become tests. Since it's done (and it's trivial code), I don't see the harm in leaving in the feature. We can hide it/discourage in in the (as yet unwritten) doco if you like.
Some disadvantages of the diff approach was discussed before. I just realized the problem which will make using this approach practically impossible. The problem - variations of output as result of: 1) using different Windows versions. 2) using TODO tests. The problem becomes even worse if more than one Win32 implementation project (e.g. ODIN) starts to use the test, because list of TODOs is project-specific 3) I'm thinking about introducing WONTFIX feature. This is when we choose, e.g. WinNT behavior and don't implement Win95 behaviour. These WONTFIX calls are also project-specific. Now, imagin combinations of these :-) Explicit check, on other hand, nicely comments all these conditions in one place - in code. Andriy Palamarchuk __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
On Thu, 10 Jan 2002, Andriy Palamarchuk wrote:
--- Jeremy White <jwhite(a)codeweavers.com> wrote:
However, having the diff feature allows us to more rapdily adapt existing programs to become tests. Since it's done (and it's trivial code), I don't see the harm in leaving in the feature. We can hide it/discourage in in the (as yet unwritten) doco if you like.
Some disadvantages of the diff approach was discussed before. I just realized the problem which will make using this approach practically impossible. The problem - variations of output as result of: 1) using different Windows versions.
.ref -> deneric reference .win95 -> win95 reference .win98 -> win98 reference etc. Okay, if the difference is between nt and win 95/98/me then it may get a bit hairy.
2) using TODO tests. The problem becomes even worse if more than one Win32 implementation project (e.g. ODIN) starts to use the test, because list of TODOs is project-specific
.ref.diff files. If the diff between the .out and the relevant .ref file matches that diff file, then all is good. [...]
Now, imagin combinations of these :-)
WONTFIX will not be very practical. But are they really needed? I would say that TODOs should be enough.
Explicit check, on other hand, nicely comments all these conditions in one place - in code.
Except that in some cases they will make the checks pretty complex. -- Francois Gouget fgouget(a)free.fr http://fgouget.free.fr/ In theory, theory and practice are the same, but in practice they're different.
Francois Gouget <fgouget(a)free.fr> writes:
.ref.diff files. If the diff between the .out and the relevant .ref file matches that diff file, then all is good.
So for a test that has both version differences and TODO tests (quite frequent IMO), you need: .ref .ref.diff .win95 .win95.diff .win98 .win98.diff etc... That's clearly not practical, especially since you need to update all these files everytime you change something to the test. -- Alexandre Julliard julliard(a)winehq.com
--- Francois Gouget <fgouget(a)free.fr> wrote:
2) using TODO tests. The problem becomes even worse if more than one Win32 implementation project (e.g. ODIN) starts to use the test, because list of TODOs is project-specific
.ref.diff files. If the diff between the .out and the relevant .ref file matches that diff file, then all is good.
like: .win95.wine .nt4.wine .win95.odin .nt4.odin ?
Now, imagin combinations of these :-)
WONTFIX will not be very practical. But are they really needed? I would say that TODOs should be enough.
WONTFIX is not TODO, it is DONE :-) I don't want to use TODO for this, because it will show up as unfinished work.
Explicit check, on other hand, nicely comments all these conditions in one place - in code.
Except that in some cases they will make the checks pretty complex.
Arguable. IMO to retrieve the same information from diffs - you need to visit: test1.c test1.win95 test1.win98 test1.nt4 test1.w2k test1.wine test1.win95.wine - TODOs test1.win98.wine - TODOs test1.nt4.wine - TODOs test1.w2k.wine - TODOs In many of these files you need to filter hundreds of lines of code to correspond the check and its output. This holds true even for checks which work in the same way on different Windows platforms, have no TODO and WONTFIX attributes, e.g. for 99.9% checks. How much lines the explicit check with all these attributes will take? 2 additional lines in the worst case, nothing - in 99.9% of cases. Andriy Palamarchuk __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
participants (4)
-
Alexandre Julliard -
Andriy Palamarchuk -
Francois Gouget -
Jeremy White