Re: Speeding up wineserver syncronization objects with shared memory

[Alexandre Julliard]

David Howells <dhowells(a)cambridge.redhat.com> writes:
> My main gripe is the slow speed of access to files... Every Read/WriteFile
> goes to the wineserver to convert the handle into a file descriptor and to
> check for locking. The FD is then passed back over a UNIX domain socket, used
> once and then closed.

Note that we are no longer doing that in the latest versions; the file
descriptor is only transferred once, and all further requests are done
on a pipe which is faster than a socket.
> Gavriel State <gav(a)transgaming.com> wrote:
> > In my experience Alexandre far prefers incremental change to kind of
> > approach you're taking.  Using that kind of approach will improve the
> > chances that your code will make it into Wine at some point.
>
> Hmmm... It's difficult to determine how to do it incrementally without making
> for even more work, but I think I know what you mean.

The kernel module itself may be hard to do incrementally, but you
should really consider reusing the existing server API so that your
module can be plugged in easily. For instance your module entry points
should be the same as the server requests, and use the same request
structures.
> Context switching is the main element of it. Going to the wineserver and back
> again just for a ReadFile() call or a Wait*() function incurs a fairly serious
> penalty (particularly on an X86, I think). Plus there's no requirement for the
> kernel to pass the remains of your timeslice to the wineserver and back
> again.

I still think that it should be possible to improve that by a small
kernel hack. It will never be as fast as doing everything in the
kernel of course, but it may just be fast enough to avoid the need to
reimplement the whole server.
> Look at the VM size of something like MS Word, and think of not having to
> allocate buffers to store all those DLLs, thus eating a massive chunk out of
> your machines total VM.

Have you measured how many dirty pages you can avoid with your change?
It seems to me that in most cases, when the dll is loaded at its
preferred address, the number of pages made dirty by the fixups should
be quite small anyway.

-- 
Alexandre Julliard
julliard(a)winehq.com

Speeding up wineserver syncronization objects with shared memory

[David Howells]

> Note that we are no longer doing that in the latest versions; the file
> descriptor is only transferred once,

Fair enough... I see that ZwClose/NtClose isn't actually a problem (since
unlike most other Zw* calls, it can't affect other processes).

Oh... I see how you're doing it... sending the handle->fd translate request to
the server, which sends a response saying you've got it cached; then using
dup() locally to emulate the old behaviour; and then closing the fd.

So this saves you the cost of the fd transfer net packet. Though you still
have to do the two context switches, which is my main contention.
> and all further requests are done on a pipe which is faster than a socket.

True, but I'd have thought that the context switches involved are still a cost
you can't get rid of so easily. Out of interest, how do you plan on doing the
locking stuff for Read/WriteFile? Cache it locally? It is unfortunate, but you
can't really make use of UNIX file locking, since this is mostly advisory and
as such doesn't actively stop read/write calls.
> The kernel module itself may be hard to do incrementally, but you
> should really consider reusing the existing server API so that your
> module can be plugged in easily. For instance your module entry points
> should be the same as the server requests, and use the same request
> structures.

What? Miss the opportunity to implement "int 0x2e" directly? *grin*

Seriously, though, whilst this'd be a lot easier in many ways (and it would
allow you to avoid the context-switch penalties), you wouldn't be able to take
full advantage of the available support in the kernel, which is more capable
than the standard UNIX userspace API suggests.

It'd still have to paste handles to fds for most file operation calls, and
you'd still have the PE Images soaking up a fair amount of memory.

If this is what you want, then it might be better done as a network protocol
module that just pretends to be a wineserver, and supports the same
read/write/sendmsg/recvmsg interface. (It'd have to be a network protocol to
be able to get sendmsg/recvmsg calls):

	int serv = socket(AF_WINE,SOCK_STREAM,0);
	short addr = AF_WINE;
	connect(serv,(struct sockaddr*)&addr,sizeof(addr));
> I still think that it should be possible to improve that by a small
> kernel hack. It will never be as fast as doing everything in the
> kernel of course, but it may just be fast enough to avoid the need to
> reimplement the whole server.

If you want to suggest exactly what you'd like to see as a hack...
> Have you measured how many dirty pages you can avoid with your change?
> It seems to me that in most cases, when the dll is loaded at its
> preferred address, the number of pages made dirty by the fixups should
> be quite small anyway.

As far as I've observed (I've got Win2000 available), most Windows DLL's have
512-byte (sector) alignment internally, _not_ 4096-byte (page) alignment for
the sections. This means that the separate sections can't be mmap'd (or else
they'd lose their required relative relationships):

    VIRTUAL_mmap()
    {
	...
	/* mmap() failed; if this is because the file offset is not    */
	/* page-aligned (EINVAL), or because the underlying filesystem */
	/* does not support mmap() (ENOEXEC,ENODEV), we do it by hand. */
	...
    }

This appears to happen a lot. And then _all_ the pages in that section are
dirty, irrespective of whether fixups are done or not.

Also, since DLLs and EXEs are not compiled as PIC (the MSDEV compiler not
having such an option as far as I can recall), the fixup tables usually seem
to apply to just about every page in the code section.

I'll have to write a small program to collect some statistics:-) 

As for the DLL being loaded at it's preferred address, the kernel module jumps
around the fixup stuff, and doesn't even consider trying to perform it.

Plus pages that have been altered by the fixup code are actually marked
_clean_ by the VM subsystem, and can thus be simply discarded when physical
memory needs to be reclaimed.

David

Speeding up wineserver syncronization objects with shared memory

[David Howells]

> I'll have to write a small program to collect some statistics:-)

Okay, I did a quick and dirty program to do that. It looked through the
following products that I've got installed:

	Win2000 sp1
	MSDEV 98
	MS Office 2000

Using the following command:

      	find  /fs/win2000/WINNT \
              "/fs/win2000/Program Files/Microsoft Visual Studio" \
	      "/fs/win2000/Program Files/Microsoft Office" -type f |
	./scandll

And came up with the following statistics:

	files accessed          : 3985
	valid PE images         : 1752
	- libraries             : 1280
	  - badly VM aligned    : 20
	  - standard image base : 42
	  - specific image base : 1218
	  - file align by sector: 976
	  - file align by page  : 284
	- executables           : 472
	  - badly VM aligned    : 133
	  - standard image base : 70
	  - specific image base : 269
	  - file align by sector: 287
	  - file align by page  : 52

I've attached the program I used.

As you can see, most DLLs actually have their image base set manually to avoid
conflicts. Also a lot of EXEs have it set too... presumably so that they can
also be used as DLLs.

However! the majority of DLLs and EXEs are sector-aligned (512) in the files,
not page-aligned (4096).

David


/* scandll.c: scan DLLs and EXEs
 *
 * Copyright (c) 2001   David Howells (dhowells(a)redhat.com).
 */
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <linux/types.h>

typedef __u8				BOOL, BYTE;
typedef __u16				WORD;
typedef __u32				DWORD, UINT;

#define PAGE_SIZE 4096

#define _H_229A4E44_C7CC_11D4_8F92_0000C0005742 /* skip section.h */
#include "pefile.h"

IMAGE_SECTION_HEADER sechdr;
IMAGE_DOS_HEADER imghdr;
IMAGE_NT_HEADERS nthdr;

int files;
int valid_image;
int dlls;
int exes;
int bad_vm_align[2];
int file_align_sector[2];
int file_align_page[2];
int normal_image_base[2];
int specific_image_base[2];

/*****************************************************************************/
/*
 *
 */
int main(int argc, char **argv)
{
	char fname[1024], *cp;
	int fd = -1, tmp, isexe;

	while (close(fd),
	       fd = -1,
	       fgets(fname,sizeof(fname),stdin)
		)
	{
		cp = strpbrk(fname,"\r\n");
		if (cp)
			*cp = 0;

		fd = open(fname,O_RDONLY);
		if (!fd) {
			if (errno==EISDIR)
				continue;
			perror("open");
			return 1;
		}

		/* get the DOS image header */
		tmp = read(fd,&imghdr,sizeof(imghdr));
		if (tmp<0) {
			if (errno==EISDIR)
				continue;
			perror("read doshdr");
			return 1;
		}
		if (tmp<sizeof(imghdr))
			continue;

		files++;

		if (imghdr.e_magic != IMAGE_DOS_SIGNATURE)
			continue;

		/* get the NT image header */
		lseek(fd,imghdr.e_lfanew,SEEK_SET);
		tmp = read(fd,&nthdr,sizeof(nthdr));
		if (tmp<0) {
			perror("read nthdr");
			return 1;
		}
		if (tmp<sizeof(nthdr))
			continue;

		if (nthdr.Signature != IMAGE_NT_SIGNATURE ||
		    nthdr.FileHeader.Machine != IMAGE_FILE_MACHINE_I386
		    )
			continue;

		if (!(nthdr.FileHeader.Characteristics &
		      IMAGE_FILE_EXECUTABLE_IMAGE))
			continue;

		if (nthdr.OptionalHeader.Magic != IMAGE_NT_OPTIONAL_HDR_MAGIC)
			continue;

		/* okay... it's a valid PE image */
		valid_image++;

		/* is it an EXE file? */
		if (nthdr.FileHeader.Characteristics & IMAGE_FILE_DLL) {
			dlls++;
			isexe = 0;
		}
		else {
			exes++;
			isexe = 1;
		}

		/* check the VM alignment */
		if (!nthdr.OptionalHeader.SectionAlignment ||
		    nthdr.OptionalHeader.SectionAlignment&(PAGE_SIZE-1)
		    ) {
			bad_vm_align[isexe]++;
			continue;
		}

		/* check the file alignment */
		if (!nthdr.OptionalHeader.FileAlignment ||
		    nthdr.OptionalHeader.FileAlignment&(PAGE_SIZE-1)
		    ) {
			file_align_sector[isexe]++;
		}
		else {
			file_align_page[isexe]++;
		}

		/* consider the image base */
		printf("%08x %s\n",nthdr.OptionalHeader.ImageBase,fname);
		if (nthdr.OptionalHeader.ImageBase==0x10000000 ||
		    nthdr.OptionalHeader.ImageBase==0x00400000
			) {
			normal_image_base[isexe]++;
		}
		else {
			specific_image_base[isexe]++;
		}
	}

	printf("files accessed          : %d\n",files);
	printf("valid PE images         : %d\n",valid_image);
	printf("- libraries             : %d\n",dlls);
	printf("  - badly VM aligned    : %d\n",bad_vm_align[0]);
	printf("  - standard image base : %d\n",normal_image_base[0]);
	printf("  - specific image base : %d\n",specific_image_base[0]);
	printf("  - file align by sector: %d\n",file_align_sector[0]);
	printf("  - file align by page  : %d\n",file_align_page[0]);
	printf("- executables           : %d\n",exes);
	printf("  - badly VM aligned    : %d\n",bad_vm_align[1]);
	printf("  - standard image base : %d\n",normal_image_base[1]);
	printf("  - specific image base : %d\n",specific_image_base[1]);
	printf("  - file align by sector: %d\n",file_align_sector[1]);
	printf("  - file align by page  : %d\n",file_align_page[1]);

	return 0;
} /* end main() */
\n",exes);
	printf("  - badly VM aligned    : %d\n",bad_vm_align[1]);
	printf("  - standard image base : %d\n",normal_image_base[1]);
	printf("  - specific image base : %d\n",specific_image_base[1]);
	printf("  - file align by sector: %d\n",file_align_sector[1]);
	printf("  - file align by page  : %d\n",file_align_page[1]);

	return 0;
} /* end main() */