"Dimitrie O. Paun" dpaun@rogers.com writes:
Speaking of the test results, I've noticed the following problems:
- Some errors reported in the summary don't get reported in the differences.
Good catch, fixed (*)
- The differences tables are inconsistent.
How can you say that?! Do you think WineHQ can't reliably run my program? :) You may find my logic wrong, though. The differences show, well, the differences. Lines are pruned iff - all the reports are the same (char by char) AND - no run failed AND - there were no errors (* from now). It doesn't work as well as I expected because lots of successful tests produce variable output. I thought about fixing them, but I could as well change the pruning policy with much less work. Shall I simply drop the first condition?
we also shouldn't make the '0' in the summary a link, since it has nowhere to take us.
Those links don't target specific lines, so they could as well stay, but I've got no objection against nuking them.
- We are running multiple test _per_ build, but only one is curretly reported. Currently, it says:
Main summary for build 200406171000 where '200406171000' is a link to the test. But since we have multiple downloads, it should be: Main summary for build 200406171000: [0] [1] [2] [3] (or somesuch), where [x] is a link to the download.
This is a more complicated issue which I haven't dared to touch yet. I planned to handle this as different builds, so there is no machinery in place for this variability. Neither for the download link, nor for the results. The Tag could be overloaded for this purpose, and the download links should go into the differences header for easy access, where they could also serve as a quick indication of the build type. Then we need short names or icons for the builds.
- Each of the testers run and submit 4 sets of tests for a build, but only one is reflected in the results. This is related to the above point, and needs fixing.
4? I though it was 2: Kevin's and Paul's. Or do they both plan to build with MinGW and MSVC? Anyway, the reports should be present with the same tag, successively numbered. The annoying thing is that people started to number their tags, which leads to strange-looking headers. Oh well.
It would be possible to subdivide the columns under the tags by builds. We would need a short build id for this at a well-defined place, like in a new field in the report or maybe in the first line of the Build info: block.