[Bug 5657] EVE Online reports VM Size as 0 while on Windows it reports the memory usage (NtQueryInformationProcess ProcessVmCounters info class: actual value for 'PagefileUsage' required)

http://bugs.winehq.org/show_bug.cgi?id=5657

Anastasius Focht  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                URL|                            |http://community.eveonline.
                   |                            |com/support/download/
                 CC|                            |[e-mail]
          Component|-unknown                    |ntdll
            Version|unspecified                 |0.9.17.
            Summary|EVE Online reports VM Size  |EVE Online reports VM Size
                   |as 0 while on Windows it    |as 0 while on Windows it
                   |reports the memory usage    |reports the memory usage
                   |                            |(NtQueryInformationProcess
                   |                            |ProcessVmCounters info
                   |                            |class: actual value for
                   |                            |'PagefileUsage' required)

--- Comment #13 from Anastasius Focht  2013-10-27 16:02:50 CDT ---
Hello folks,

is seems the culprit is NtQueryInformationProcess -> ProcessVmCounters not
returning useful information.

There are some tools to reverse EVE Online client code (compiled stackless
python and encrypted).

You need:

http://python.org/ftp/python/2.7.5/python-2.7.5.msi (Python for win32)

https://github.com/wibiti/evedec (decrypts Eve Online python files and passes
them to uncompyle2 to decompile)

https://github.com/wibiti/uncompyle2 (Python 2.7 byte-code decompiler, written
in Python 2.7)

Unfortunately 'evedec' makes use of parallel processing lib, requiring pipe
message mode (bug 17195 and bug 17273) so you have to work around with
unofficial patchset(s) floating around or rewriting the tool to use single
processing mode (without Queue).

Decompiling eve-8.32.622857 client and looking for code that updates the memory
stat page gives a hit here:

carbon/client/script/util/monitor.py:

--- snip ---
...
    def UpdateUI(self):
        wnd = self.GetWnd()
        if not wnd or wnd.destroyed:
            return
        hd, li = blue.pyos.ProbeStuff()
        info = Row(list(hd), li)
        virtualMem = info.pagefileUsage / 1024
        if self.lastVM is None:
            self.lastVM = virtualMem
        delta = virtualMem - self.lastVM
        self.totalVMDelta += delta
        self.lastVM = virtualMem
        delta = delta or self.lastVMDelta
        self.lastVMDelta = delta
        dc = ['<color=0xff00ff00>', '<color=0xffff0000>'][delta > 0]
        tdc = ['<color=0xff00ff00>', '<color=0xffff0000>'][self.totalVMDelta >
0]
        try:
            dev = trinity.device
            iml = ''
            if blue.logInMemory.isActive:
                iml = 'In-memory logging <color=0xff00ff00>active</color> (%s /
%s)' % (LOGTYPE_MAPPING.get(blue.logInMemory.threshold, ('Unknown', ''))[0],
blue.logInMemory.capacity)
            fps = 'Fps: %.1f - %s' % (blue.os.fps, iml)
            if wnd.sr.fpsText.text != fps:
                wnd.sr.fpsText.text = fps
            vm = 'VM Size/D/TD: %sK / %s%sK<color=0xffffffff> /
%s%sK<color=0xffb0b0b0> - totalVidMem: %sM' % (util.FmtAmt(virtualMem),
             dc,
             util.FmtAmt(delta),
             tdc,
             util.FmtAmt(self.totalVMDelta),
             self.totalVidMem)
            if wnd.sr.vmText.text != vm:
                wnd.sr.vmText.text = vm
            inUse = util.FmtAmt(blue.motherLode.memUsage / 1024)
            total = util.FmtAmt(blue.motherLode.maxMemUsage / 1024)
            num = blue.motherLode.size()
            vm = 'Resource Cache Usage: %sK / %sK - %s objects' % (inUse,
total, num)
            if wnd.sr.cacheText.text != vm:
                wnd.sr.cacheText.text = vm
            spaceMgr = sm.StartService('space')
            mo = 'Lazy Queue: %s' % getattr(spaceMgr, 'lazyLoadQueueCount', 0)
            if session.role & service.ROLEMASK_ELEVATEDPLAYER:
                mo += ' - Preload Queue: %s' % getattr(spaceMgr,
'preloadQueueCount', 22)
            if wnd.sr.queueText.text != mo:
                wnd.sr.queueText.text = mo
        except Exception as e:
            print 'e', e
            self.uitimer = None
            sys.exc_clear()
...
--- snip ---

The value in question is 'info.pagefileUsage' (the others are just interval
deltas).

carbon/common/lib/win32.py:

--- snip ---
...
class PROCESS_MEMORY_COUNTERS_EX(Structure):
    _fields_ = [('cb', DWORD),
     ('PageFaultCount', DWORD),
     ('PeakWorkingSetSize', SIZE_T),
     ('WorkingSetSize', SIZE_T),
     ('QuotaPeakPagedPoolUsage', SIZE_T),
     ('QuotaPagedPoolUsage', SIZE_T),
     ('QuotaPeakNonPagedPoolUsage', SIZE_T),
     ('QuotaNonPagedPoolUsage', SIZE_T),
     ('PagefileUsage', SIZE_T),
     ('PeakPagefileUsage', SIZE_T),
     ('PrivateUsage', SIZE_T)]
...
def GetProcessMemoryInfo():
    f = windll.Kernel32.GetCurrentProcess
    f.retval = HANDLE
    h = f()
    counters = PROCESS_MEMORY_COUNTERS_EX()
    windll.Psapi.GetProcessMemoryInfo(HANDLE(h), byref(counters),
sizeof(counters))
    return StructToKeyval(counters)
...
--- snip ---

Other code making use of that value:

carbon/common/script/net/ExceptionWrapperGPCS.py:

--- snip ---
ram = blue.win32.GetProcessMemoryInfo()['PagefileUsage'] / 1024 / 1024
cpuLoad = self.machoNet.GetCPULoad()
m = blue.win32.GlobalMemoryStatus()
memLeft = m['AvailPhys'] / 1024 / 1024
txt = 'System Information: '
txt += 'Total CPU load: %s%%' % int(cpuLoad)
txt += ' | Process memory in use: %s MB' % ram
txt += ' | Physical memory left: %s MB\n' % memLeft
--- snip ---

Psapi.GetProcessMemoryInfo -> kernel32.K32GetProcessMemoryInfo ->
NtQueryInformationProcess(process, ProcessVmCounters, ...)

I found the following articles useful how to estimate/substitute some values:

http://stackoverflow.com/questions/372484/how-do-i-programmatically-check-me...

http://nadeausoftware.com/articles/2012/07/c_c_tip_how_get_process_resident_...

http://locklessinc.com/articles/memory_usage/

As already mentioned, only 'PagefileUsage' is actually required for EVE.
The best substitute for 'PagefileUsage' might be 'VmSize' from either
'/proc/<pid>/stat' (VmSize) or '/proc/<pid>/statm' (VmSize*page_size).

I added a small Linux implementation to ProcessVmCounters getter to fill
'pvmi.PagefileUsage' member with current VmSize and it was enough to have EVE
client display proper memory usage (current + deltas).

$ wine --version
wine-1.7.5-100-g3ab406d

Regards