http://bugs.winehq.org/show_bug.cgi?id=11585
--- Comment #1 from Lei Zhang thestig@google.com 2008-02-14 10:26:27 --- The kernel controls the size of the cache, Wine can't "leak" to it.
Can you periodically run:
ps -o user,cmd,rss,vsize -C appname
where appname is program you're trying to run, and see if memory usage is actually going up?
Also, please upgrade to the latest version of Wine if possible.