[Bug 40350] New: MinGW binary crashes, but msvc6's one works fine in function when returning aggregate value

https://bugs.winehq.org/show_bug.cgi?id=40350

Bug ID: 40350
           Summary: MinGW binary crashes, but msvc6's one works fine in
                    function when returning aggregate value
           Product: Wine
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: -unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

This is very strange bug.
Application (https://github.com/SDLash3D/xash3d) contains function that returns
aggregate value.
It may be built with MinGW or MSVC.
Function is called from msvc code. It works fine in windows with both msvc and
MinGW libraries (that is very strange as gcc has different way of returning
aggregates), but it does not work in wine (causes stack corruption).

Structure and function code

struct pmtrace_s
{
    qboolean    allsolid;        // if true, plane is not valid
    qboolean    startsolid;    // if true, the initial point was in a solid
area
    qboolean    inopen, inwater;  // End point is in empty space or in water
    float    fraction;        // time completed, 1.0 = didn't hit anything
    vec3_t    endpos;        // final position
    pmplane_t    plane;        // surface normal at impact
    int    ent;        // entity at impact
    vec3_t    deltavelocity;    // Change in player's velocity caused by
impact.  
                // Only run on server.
    int    hitgroup;
};

static pmtrace_t pfnPlayerTrace(float *start, float *end, int traceFlags, int
ignore_pe)
{
    return PM_PlayerTraceExt( svgame.pmove, start, end, traceFlags,
svgame.pmove->numphysent, svgame.pmove->physents, ignore_pe, NULL );
}

Also, application has DLL loader for linux (based on mplayer code)

It has some workaround for this function.
When dll loader used, application export this repacement to API:
#ifdef DLL_LOADER
static pmtrace_t *pfnPlayerTrace_w32(pmtrace_t *trace, float *start, float
*end, int traceFlags, int ignore_pe)
{
    *trace = PM_PlayerTraceExt( svgame.pmove, start, end, traceFlags,
svgame.pmove->numphysent, svgame.pmove->physents, ignore_pe, NULL );
    return trace;
}
#endif
It works correctly when called from msvc code on linux.
File with function, causing problems:
https://github.com/SDLash3D/xash3d/blob/master/engine/server/sv_pmove.c
MSVC binaries:
https://transfer.sh/8wRoD/xash3d-win.7z
MinGW binary:
https://transfer.sh/par30/xash3d-mingw-16-03-21-12-57.7z
Linux binary:
https://transfer.sh/pmKos/xash3d-linux-16-03-21-12-57.7z
The main question: Why it works correctly in windows and how to workaround this
in wine?

[Bug 40350] MinGW binary crashes, but msvc6's one works fine in function when returning aggregate value

https://bugs.winehq.org/show_bug.cgi?id=40350

--- Comment #1 from Henri Verbeet  ---
(In reply to Dmitry from comment #0)
> The main question: Why it works correctly in windows and how to workaround
> this in wine?

Well, the issue is the difference in calling convention between gcc and msvc
that you mention. My guess would be that you're just getting lucky with the
stack layout with MinGW on Windows. You can tweak the function prototype to
make it more or less work, which is what the "#ifdef DLL_LOADER" bit does
(although I don't think it's quite right either, PM_PlayerTraceExt() has the
same issue). It would be much better to just avoid returning aggregates.

[Bug 40350] MinGW binary crashes, but msvc6's one works fine in function when returning aggregate value

https://bugs.winehq.org/show_bug.cgi?id=40350

--- Comment #2 from Dmitry  ---
Yes, PM_PlayerTraceExt has same workaround.
So it is just luck that it works on windows?
I'll check with this workaround in mingw later, i need to make sure that is
will work with both mingw sides too, or leave it as is (and maybe add dynamic
option for enabling workaround).
I know that returning aggregate is bad idea, but it is not our solution. It was
originally made by ValveSoftware in halflife code many years ago and used in
big amount of proprietary code.

[Bug 40350] MinGW binary crashes, but msvc6's one works fine in function when returning aggregate value

https://bugs.winehq.org/show_bug.cgi?id=40350

--- Comment #3 from Henri Verbeet  ---
(In reply to Dmitry from comment #2)
> So it is just luck that it works on windows?

Quite possibly. You'd have to look at the code that gets generated for the
caller to be sure, but my guess would be that it ends up overwriting some value
that isn't really used.