Hello,
I recently noticed that we don't currently get Coverity reports for files in the test suite. I emailed Amine Khaldi, who does the Coverity reports, and asked if we could start getting reports on the tests. Amine made it sound like this had been discussed before and due to the large volume of reports, there was not interest in getting reports for anything other than the actual Wine code.
Is there enough interest now to start looking at the test suite reports? Would it hurt anything to be able to see that information?
-Alex
On 26.02.2017 20:53, Alex Henrie wrote:
Hello,
I recently noticed that we don't currently get Coverity reports for files in the test suite. I emailed Amine Khaldi, who does the Coverity reports, and asked if we could start getting reports on the tests. Amine made it sound like this had been discussed before and due to the large volume of reports, there was not interest in getting reports for anything other than the actual Wine code.
Is there enough interest now to start looking at the test suite reports? Would it hurt anything to be able to see that information?
-Alex
Yes, we discussed this on irc at some point, by we I mean Amine and myself, and that's probably what he meant. I don't think it makes sense to scan tests yet, I'd expect noise rate to be overwhelming, resulting in thousands on reports of little to no value, like missing return value checks. We still have hundreds of potentially valuable ones for implementation part, that not a lot of people are looking at.
So for the last question, I'm worried that it would hurt in a way that reports we potentially care about will get buried. Of course I could be wrong, we could simply try once and see what happens. I'm not sure actually what happens to reports when next scan does not include failed files, you'll probably lose all comments, report status updates etc.
If you have time to clean some of existing ones, that would be greatly appreciated.
On Sun, Feb 26, 2017 at 12:09 PM, Nikolay Sivov bunglehead@gmail.com wrote:
On 26.02.2017 20:53, Alex Henrie wrote:
Hello,
I recently noticed that we don't currently get Coverity reports for files in the test suite. I emailed Amine Khaldi, who does the Coverity reports, and asked if we could start getting reports on the tests. Amine made it sound like this had been discussed before and due to the large volume of reports, there was not interest in getting reports for anything other than the actual Wine code.
Is there enough interest now to start looking at the test suite reports? Would it hurt anything to be able to see that information?
-Alex
Yes, we discussed this on irc at some point, by we I mean Amine and myself, and that's probably what he meant. I don't think it makes sense to scan tests yet, I'd expect noise rate to be overwhelming, resulting in thousands on reports of little to no value, like missing return value checks. We still have hundreds of potentially valuable ones for implementation part, that not a lot of people are looking at.
So for the last question, I'm worried that it would hurt in a way that reports we potentially care about will get buried. Of course I could be wrong, we could simply try once and see what happens. I'm not sure actually what happens to reports when next scan does not include failed files, you'll probably lose all comments, report status updates etc.
If you have time to clean some of existing ones, that would be greatly appreciated.
FWIW, coverity supports putting issues into different components based on filename. It would be trivial to set up two different components for wine, e.g., 'wine' and 'wine-tests', then have wine/*/*/tests/* go to wine-tests, and wine/* to wine.
For those not familiar with coverity administration, this would not require two separate builds, the results are separated when results are displayed on the server.
On 26.02.2017 21:32, Austin English wrote:
On Sun, Feb 26, 2017 at 12:09 PM, Nikolay Sivov bunglehead@gmail.com wrote:
On 26.02.2017 20:53, Alex Henrie wrote:
Hello,
I recently noticed that we don't currently get Coverity reports for files in the test suite. I emailed Amine Khaldi, who does the Coverity reports, and asked if we could start getting reports on the tests. Amine made it sound like this had been discussed before and due to the large volume of reports, there was not interest in getting reports for anything other than the actual Wine code.
Is there enough interest now to start looking at the test suite reports? Would it hurt anything to be able to see that information?
-Alex
Yes, we discussed this on irc at some point, by we I mean Amine and myself, and that's probably what he meant. I don't think it makes sense to scan tests yet, I'd expect noise rate to be overwhelming, resulting in thousands on reports of little to no value, like missing return value checks. We still have hundreds of potentially valuable ones for implementation part, that not a lot of people are looking at.
So for the last question, I'm worried that it would hurt in a way that reports we potentially care about will get buried. Of course I could be wrong, we could simply try once and see what happens. I'm not sure actually what happens to reports when next scan does not include failed files, you'll probably lose all comments, report status updates etc.
If you have time to clean some of existing ones, that would be greatly appreciated.
FWIW, coverity supports putting issues into different components based on filename. It would be trivial to set up two different components for wine, e.g., 'wine' and 'wine-tests', then have wine/*/*/tests/* go to wine-tests, and wine/* to wine.
For those not familiar with coverity administration, this would not require two separate builds, the results are separated when results are displayed on the server.
If it's possible to make them appear separately, so tests don't affect our main reports, I see no harm in scanning tests too.
Nikolay Sivov bunglehead@gmail.com writes:
On 26.02.2017 21:32, Austin English wrote:
On Sun, Feb 26, 2017 at 12:09 PM, Nikolay Sivov bunglehead@gmail.com wrote: FWIW, coverity supports putting issues into different components based on filename. It would be trivial to set up two different components for wine, e.g., 'wine' and 'wine-tests', then have wine/*/*/tests/* go to wine-tests, and wine/* to wine.
For those not familiar with coverity administration, this would not require two separate builds, the results are separated when results are displayed on the server.
If it's possible to make them appear separately, so tests don't affect our main reports, I see no harm in scanning tests too.
There's probably no harm in getting a scan once, but I'm concerned that it would make people start "fixing" things that don't matter.
On 26/02/2017 21:53, Alexandre Julliard wrote:
Nikolay Sivov bunglehead@gmail.com writes:
On 26.02.2017 21:32, Austin English wrote:
On Sun, Feb 26, 2017 at 12:09 PM, Nikolay Sivov bunglehead@gmail.com wrote: FWIW, coverity supports putting issues into different components based on filename. It would be trivial to set up two different components for wine, e.g., 'wine' and 'wine-tests', then have wine/*/*/tests/* go to wine-tests, and wine/* to wine.
For those not familiar with coverity administration, this would not require two separate builds, the results are separated when results are displayed on the server.
If it's possible to make them appear separately, so tests don't affect our main reports, I see no harm in scanning tests too.
There's probably no harm in getting a scan once, but I'm concerned that it would make people start "fixing" things that don't matter.
I have created a "wine-tests" component using .*/tests/.* and a "wine" component for the remaining. Please let me know if you need anything else.
Regards, Amine.
2017-02-26 15:11 GMT-07:00 Amine Khaldi amine.khaldi@reactos.org:
I have created a "wine-tests" component using .*/tests/.* and a "wine" component for the remaining. Please let me know if you need anything else.
Cool, thanks. I'm hoping to use this to debug and resolve some of the bug reports we have about running the tests with Valgrind.
2017-02-26 13:53 GMT-07:00 Alexandre Julliard julliard@winehq.org:
There's probably no harm in getting a scan once, but I'm concerned that it would make people start "fixing" things that don't matter.
There are a limited number of developers with access to Coverity, so I don't think we'll get too many people trying to fix things that aren't actually a problem. If it becomes a problem, we should be able to limit access or go back to hiding the test file reports.
-Alex
On Sun, Feb 26, 2017 at 5:54 PM, Alex Henrie alexhenrie24@gmail.com wrote:
2017-02-26 15:11 GMT-07:00 Amine Khaldi amine.khaldi@reactos.org:
I have created a "wine-tests" component using .*/tests/.* and a "wine" component for the remaining. Please let me know if you need anything else.
Cool, thanks. I'm hoping to use this to debug and resolve some of the bug reports we have about running the tests with Valgrind.
Woo!
On 26 February 2017 at 19:09, Nikolay Sivov bunglehead@gmail.com wrote:
Yes, we discussed this on irc at some point, by we I mean Amine and myself, and that's probably what he meant. I don't think it makes sense to scan tests yet, I'd expect noise rate to be overwhelming, resulting in thousands on reports of little to no value, like missing return value checks. We still have hundreds of potentially valuable ones for implementation part, that not a lot of people are looking at.
I'm sure I knew at some point 10 years ago, but how does one get Coverity access these days?
2017-02-26 21:44 GMT-07:00 Henri Verbeet hverbeet@gmail.com:
I'm sure I knew at some point 10 years ago, but how does one get Coverity access these days?
Someone with access logs in, sends you an invite, and Amine approves it. I just sent you an invite.
-Alex
2017-02-26 21:55 GMT-07:00 Alex Henrie alexhenrie24@gmail.com:
2017-02-26 21:44 GMT-07:00 Henri Verbeet hverbeet@gmail.com:
I'm sure I knew at some point 10 years ago, but how does one get Coverity access these days?
Someone with access logs in, sends you an invite, and Amine approves it. I just sent you an invite.
Alternatively, you can go to https://scan.coverity.com/projects/wine click "Add me to project", and wait for Amine's approval.
-Alex
-
Alex Henrie -
Alexandre Julliard -
Amine Khaldi -
Austin English -
Henri Verbeet -
Nikolay Sivov