[GSoC] Project proposal : Winetest Scripting Interface

[Yash Yadav]

Hi,
I am Yash Yadav and this is my first proposal for the GSoC idea 
"Winetest Scripting Interface".
Please excuse the delay and me doing this near the deadline.

*Winetest Scripting Interface***

*Abstract***

Wine(Wine is not an emulator) depends on multiple different third party projects which are mostly drivers or low-level system programs that allow wine to convert and forward windows calls into POSIX-compliant environment. Example supporter projects : openal-soft, gecko, mesa, etc.

Having dependency on multiple third party projects naturally brings in the trouble of keeping track of what the other projects are doing. To make things easier, wine has testsuite that could be run by the third party developers (graphics drivers devs for example) to catch problems their own tests do not see.

The problem is that with existing testsuite, the reports are not specific enough to count on, as they only report 
“success” or 
“failures” on completion/termination. This leaves developers to debug by manually going through the output of the tests in terminal, which is not really feasible as ignorable failure reports would be crowding all the output.

*Why This Project***

Being an avid user of wine(Mostly for games though) from the day I shifted to linux, I’ve always wanted to contribute to the project since wine has space for infinite contribution until it reaches the state where everything of windows would run on a non microsoft kernel. Thanks to GSoC and a project I love, I can go up there taking baby steps from now.

This project has a good scope of improving wine development process by helping other projects that wine runs on. For a beginner contributor like me, this seems to be a better start at getting familiar with wine codebase compared to just jumping into the core.

*Project** Goals***

The project revolves around development of script(s) that enhance the functionality of the existing wine testsuite.

The idea is to write a script that runs wine tests keeps a track of reports of tests over different patches and filter out new failures compared to previous reports.

The script(s) would be written with non-wine developers in mind. Designed to be run by the developers of other projects that wine depends on.

The script(s) would :

1.Report more sensible information than just reporting 
“Failure” or “Success” or leaving the testers with lines and lines of test results that they should be ignoring. (This could be done by backtracking a little from the failure point.)

2.Filter out the new failures only that show up with new code changes.

3.Know what all errors are ignorable and need not crowd up the reports.

4.Allow running tests specific to the section. (Sound testing wont run graphics tests)

*Implementation***

Target is to write the script in python 3 since I’m more fluent in python as compared to other scripting languages. If python is not allowed, I’ll 
go with Bash shell.

If database is used at a later point, I’m planning to go with /sqllite3 
/or /Pandas/ python library if csv(comma separated values) is considered.

The project implementation is expressed in the* Timeline* section.

*Timeline***

*Pre GSoC : **Community Bonding | April 24 - May 13***

Get acquainted with the code base of wine and the test suite. Also, discuss with the team of what exactly needs to be done and who all can be helpful besides the mentor.

*Week 1 | May 14 - May 20***

Understand the relevant parts of the testbot.winehq.org code base and try to figure out how the existing structure works on windows. Try out the testsuite.

*Week 2 | May 21 - May27***

Try out the existing testsuite for different versions of the supporting projects(the ones that wine depends on) and record patterns in the output from the testsuite. Design the workflow for the script(s). Essentially figuring out to the minute details, i.e. the dataset and the information that’s important from the test reports.

*Week 3 - 4 | May 28 - June 10***

Code the initial version of the script that would run the “make install” in selected directories depending upon the type of test 
that is 
called in for. Then add support to catch data from the output of the individual tests.

*Week 5 | June 11 - June 15***

Phase 1 evaluation.

*Week 5 - 8 | June 16 - July 8***

Extend script to filter out the raw data from the reports to gather important information, (the points relative to the failure). This may require backtracking from a failure point or just the failure report for the particular test thats running. (This is to be decided after running the tests manually and understanding the patterns, week 2 work).

Result out useful data from the test reports. Also, add a script to check if the report is new or is repeated. This would require use of database 
or csv(comma separated values) unless suggested otherwise.

*Week 9 | July 9 - July 13***

Phase 2 evaluation.

*Week 9 - 11 | July 14 - July 29***

Filter out the edge cases and test the script itself for undesired output or working. Work on the back-end (preferably the database that would store the results). Add another script to obtain the data as required. (Date wise/patchset/commit wise). 
See if it’s possible to integrate the data fetching script into the testbot website to display the reports 
/timeline/.

*Week 12 - 13 | July 30 - August 14***

Fix bugs, clean the code. Write documentation for the script(s). Final evaluation.

*The API***

The script(s) would

* Keep the test reports under version control system(preferably GIT). The way new failures or bugfixes can be catched.

* Only show the relevant data to the failure if a testcase fails. (May require backtracking from the failure point).

* The data relevant to the failure would also help in keeping track of previous failures and keeping a timeline of tests.

*About Me***

Name : Yash Yadav

Email : [e-mail]_ 
 | [e-mail]

Telephone : +91 - 810-427-1666

Github : https://github.com/OhYash

Telegram : @OhYash

IRC : ohyash on freenode IRC

Country of Residence : India

Timezone : UTC + 05:30

I am an undergraduate student in my 3^rd 
  year of Bachelors in technology in Lovely Professional University, in Punjab, India.

My language of choice is C but I frequently choose between python and C depending upon the type task I do. I have been using git for the past one year.

My semester would end on 21^st 
  May and yes, I would be having exams within that period. I would have some placement preparatory classes in June, but If I am selected, I would be able to give about 5-6 hours everyday in weekdays and more(8-10) on weekends.