[PATCH v12 0/2] MR6904: ntdll.dll: Update NtQueryDirectoryFile to align with current Windows behaviour
This PR updates the behaviour of `NtQueryDirectoryFile`, bringing it in line with current Windows behaviour. The need for this update was discovered when attempting to build the Unreal Engine with MSVC under Wine. In certain cases conditional include statements do not behave as expected, due to MSVC depending on undocumented behaviour of `NtQueryDirectoryFile`.
We ran tests on multiple versions of Windows, and discovered that the behaviour has changed since the original Wine implementation, but the documentation has not. The source code for our test tool, and a set of results can be found [here](https://github.com/TensorWorks/NtQueryDirectoryFile-Test). As of Windows 8, calling `NtQueryDirectoryFile` with a re-used handle, a new mask, and setting the `RestartScan` flag to True, causes the cached results to be erased and a new scan to be performed with the updated mask. Currently, Wine performs as did earlier versions of Windows, where the changed mask is ignored, and the cache is reused. This can cause `NtQueryDirectoryFile` under Wine to falsely report that files exist, when they do not.
This PR corrects this behaviour, invalidating the cache when required. Implementing this exposed further undocumented behaviour of `NtQueryDirectoryFile`, where a search for a non-existent file will return either `STATUS_NO_MORE_FILES` or `STATUS_NO_SUCH_FILE`, depending on whether or not the handle had been previously used regardless of the value of `RestartScan`. This was reflected in a `winetest` which allowed for the response to be either `STATUS_SUCCESS` or `STATUS_NO_MORE_FILES`.
This patch also adds unit tests for the new behaviour of `NtQueryDirectoryFile`. These tests pass when running `winetest` under Windows, and under Wine with these changes in place, but they will fail under older versions of Wine. If run under older versions of Windows the test suite will detect that this functionality is not supported, and will not run the updated tests.
-- v12: ntdll.dll: Update NtQueryDirectoryFile to align with current Windows behaviour ntdll: Test updated NtQueryDirectoryFile mask reset behaviour
From: Eugene McArdle eugene@tensorworks.com.au
--- dlls/ntdll/tests/directory.c | 237 +++++++++++++++++++++++++++++++++++ 1 file changed, 237 insertions(+)
diff --git a/dlls/ntdll/tests/directory.c b/dlls/ntdll/tests/directory.c index 52ff9936560..cd7c21ceeba 100644 --- a/dlls/ntdll/tests/directory.c +++ b/dlls/ntdll/tests/directory.c @@ -964,6 +964,242 @@ done: pRtlFreeUnicodeString(&ntdirname); }
+static void set_up_mask_test(const char *testdir) +{ + BOOL ret; + char buf[MAX_PATH + 5]; + HANDLE h; + + ret = CreateDirectoryA(testdir, NULL); + ok( ret, "couldn't create dir '%s', error %ld\n", testdir, GetLastError() ); + + sprintf(buf, "%s\%s", testdir, "a-file.h"); + h = CreateFileA(buf, GENERIC_READ|GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, + FILE_ATTRIBUTE_NORMAL, 0); + ok( h != INVALID_HANDLE_VALUE, "failed to create temp file '%s'\n", buf ); + CloseHandle(h); + + sprintf(buf, "%s\%s", testdir, "another-file.h"); + h = CreateFileA(buf, GENERIC_READ|GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, + FILE_ATTRIBUTE_NORMAL, 0); + ok( h != INVALID_HANDLE_VALUE, "failed to create temp file '%s'\n", buf ); + CloseHandle(h); +} + +static void tear_down_mask_test(const char *testdir) +{ + int ret; + char buf[MAX_PATH]; + + sprintf(buf, "%s\%s", testdir, "a-file.h"); + ret = DeleteFileA(buf); + ok( ret || (GetLastError() == ERROR_FILE_NOT_FOUND) || (GetLastError() == ERROR_PATH_NOT_FOUND), + "Failed to rm %s, error %ld\n", buf, GetLastError() ); + + sprintf(buf, "%s\%s", testdir, "another-file.h"); + ret = DeleteFileA(buf); + ok( ret || (GetLastError() == ERROR_FILE_NOT_FOUND) || (GetLastError() == ERROR_PATH_NOT_FOUND), + "Failed to rm %s, error %ld\n", buf, GetLastError() ); + + RemoveDirectoryA(testdir); +} + + +/* Performs a query with NtQueryDirectoryFile() */ +static BOOL test_NtQueryDirectoryFile_mask(HANDLE handle, BOOL restart_scan, UNICODE_STRING *mask, + NTSTATUS expected_status, UNICODE_STRING *match, BOOL validate_only) +{ + NTSTATUS status; + IO_STATUS_BLOCK io; + UINT data_size = sizeof(FILE_DIRECTORY_INFORMATION) + (MAX_PATH * sizeof(wchar_t)); + BYTE data[8192]; + FILE_DIRECTORY_INFORMATION *dir_info; + WCHAR *name; + ULONG name_len; + WCHAR *mask_value = {0}; + ULONG mask_len = 0; + WCHAR *match_value = {0}; + ULONG match_len = 0; + + if (mask) + { + mask_len = mask->Length / sizeof(WCHAR); + mask_value = mask->Buffer; + } + + /* Perform the query */ + status = pNtQueryDirectoryFile( handle, NULL, NULL, NULL, &io, data, data_size, + FileDirectoryInformation, TRUE, mask, restart_scan ); + + if (validate_only && status != expected_status) return FALSE; + + todo_wine_if(status != expected_status) + ok( status == expected_status, "unexpected status : 0x%lx Test settings: file mask: '%s', restart: %d, expected status: 0x%lx\n", + status, wine_dbgstr_wn(mask_value, mask_len), restart_scan, expected_status ); + + /* Verify the results if required */ + if (status == 0 && match != NULL) + { + dir_info = (FILE_DIRECTORY_INFORMATION *)data; + name = dir_info->FileName; + name_len = dir_info->FileNameLength / sizeof(WCHAR); + match_len = match->Length / sizeof(WCHAR); + match_value = match->Buffer; + + todo_wine_if(name_len != match_len) + ok( name_len == match_len, "unexpected filename length %lu, expected %lu\n", name_len, match_len ); + todo_wine_if(name_len != match_len) + ok( !memcmp(name, match_value, match_len * sizeof(WCHAR)), "unexpected filename %s, expected %s\n", + wine_dbgstr_wn(name, name_len), wine_dbgstr_wn(match_value, match_len) ); + } + return TRUE; +} + +static void test_NtQueryDirectoryFile_change_mask(void) +{ + NTSTATUS status; + HANDLE dirh; + HANDLE dirh_test_multiple_handles; + HANDLE dirh_test_fresh_null; + HANDLE dirh_test_fresh_empty; + char testdir[MAX_PATH]; + OBJECT_ATTRIBUTES attr; + IO_STATUS_BLOCK io; + UNICODE_STRING ntdirname; + WCHAR testdir_w[MAX_PATH]; + + UNICODE_STRING atestfile; + UNICODE_STRING anothertestfile; + UNICODE_STRING notatestfile; + UNICODE_STRING testmask; + UNICODE_STRING emptymask; + + pRtlInitUnicodeString(&atestfile, L"a-file.h"); + pRtlInitUnicodeString(&anothertestfile, L"another-file.h"); + pRtlInitUnicodeString(¬atestfile, L"not-a-file.h"); + pRtlInitUnicodeString(&testmask, L"*.h"); + pRtlInitUnicodeString(&emptymask, L""); + + /* Clean up from prior aborted run, if any, then set up test files */ + ok( GetTempPathA(MAX_PATH, testdir), "couldn't get temp dir\n" ); + strcat(testdir, "mask.tmp"); + tear_down_mask_test(testdir); + set_up_mask_test(testdir); + + pRtlMultiByteToUnicodeN(testdir_w, sizeof(testdir_w), NULL, testdir, strlen(testdir) + 1); + if (!pRtlDosPathNameToNtPathName_U(testdir_w, &ntdirname, NULL, NULL)) + { + ok( 0, "RtlDosPathNametoNtPathName_U failed\n" ); + goto done; + } + InitializeObjectAttributes(&attr, &ntdirname, OBJ_CASE_INSENSITIVE, 0, NULL); + + /* Open a handle for our test directory */ + status = pNtOpenFile(&dirh, SYNCHRONIZE | FILE_LIST_DIRECTORY, &attr, &io, FILE_SHARE_READ, + FILE_SYNCHRONOUS_IO_NONALERT | FILE_OPEN_FOR_BACKUP_INTENT | FILE_DIRECTORY_FILE); + ok( status == STATUS_SUCCESS, "failed to open dir '%s', ret 0x%lx, error %ld\n", testdir, status, GetLastError() ); + if (status != STATUS_SUCCESS) + { + skip("can't test if we can't open the directory\n"); + goto done; + } + + /* Verify that NtQueryDirectoryFile mask reset behaviour introduced in Windows 8 is supported */ + test_NtQueryDirectoryFile_mask(dirh, TRUE, &atestfile, STATUS_SUCCESS, &atestfile, TRUE); + if (!winetest_platform_is_wine && !test_NtQueryDirectoryFile_mask(dirh, TRUE, ¬atestfile, STATUS_NO_MORE_FILES, NULL, TRUE)) + { + skip("Win8+ NtQueryDirectoryFile mask reset behaviour not supported\n"); + goto done; + } + + /* Test that caches for two handles to a single directory do not interfere with each other*/ + /* Open a handle for our test directory to test multiple handles */ + status = pNtOpenFile(&dirh_test_multiple_handles, SYNCHRONIZE | FILE_LIST_DIRECTORY, &attr, &io, FILE_SHARE_READ, + FILE_SYNCHRONOUS_IO_NONALERT | FILE_OPEN_FOR_BACKUP_INTENT | FILE_DIRECTORY_FILE); + ok( status == STATUS_SUCCESS, "failed to open second handle to dir '%s', ret 0x%lx, error %ld\n", testdir, status, GetLastError() ); + + /* Search for a non-existent file, putting it into a state with an empty cache */ + /* This also confirms that using a non-existent mask with a fresh handle returns a different status*/ + test_NtQueryDirectoryFile_mask(dirh_test_multiple_handles, TRUE, ¬atestfile, STATUS_NO_SUCH_FILE, ¬atestfile, FALSE); + /* Confirm that handle is in a state where it fails to find atestfile */ + test_NtQueryDirectoryFile_mask(dirh_test_multiple_handles, FALSE, &atestfile, STATUS_NO_MORE_FILES, &atestfile, FALSE); + /* Confirm that another handle is able to find atestfile */ + test_NtQueryDirectoryFile_mask(dirh, TRUE, &atestfile, STATUS_SUCCESS, &atestfile, FALSE); + /* Close handle used for testing multiple handles */ + pNtClose(&dirh_test_multiple_handles); + + /* All searches for `notatestfile` are expected to fail */ + /* Tests should also fail if the scan is not reset, and the mask changes to an incompatible one */ + test_NtQueryDirectoryFile_mask(dirh, TRUE, &atestfile, STATUS_SUCCESS, &atestfile, FALSE); + test_NtQueryDirectoryFile_mask(dirh, TRUE, &anothertestfile, STATUS_SUCCESS, &anothertestfile, FALSE); + test_NtQueryDirectoryFile_mask(dirh, TRUE, ¬atestfile, STATUS_NO_MORE_FILES, NULL, FALSE); + + test_NtQueryDirectoryFile_mask(dirh, TRUE, &atestfile, STATUS_SUCCESS, &atestfile, FALSE); + test_NtQueryDirectoryFile_mask(dirh, FALSE, ¬atestfile, STATUS_NO_MORE_FILES, NULL, FALSE); + + test_NtQueryDirectoryFile_mask(dirh, TRUE, &atestfile, STATUS_SUCCESS, &atestfile, FALSE); + test_NtQueryDirectoryFile_mask(dirh, FALSE, &anothertestfile, STATUS_NO_MORE_FILES, &anothertestfile, FALSE); + + /* Test mask that matches multiple files, do not check results against the mask */ + test_NtQueryDirectoryFile_mask(dirh, TRUE, &testmask, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, FALSE, ¬atestfile, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, TRUE, ¬atestfile, STATUS_NO_MORE_FILES, NULL, FALSE); + + /* Test NULL mask with a previously used handle that last returned an error */ + /* Ensure dirh is in a failure state */ + test_NtQueryDirectoryFile_mask(dirh, TRUE, ¬atestfile, STATUS_NO_MORE_FILES, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, TRUE, NULL, STATUS_NO_MORE_FILES, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, FALSE, NULL, STATUS_NO_MORE_FILES, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, TRUE, NULL, STATUS_NO_MORE_FILES, NULL, FALSE); + + /* Test NULL mask with a previously used handle that last returned STATUS_SUCCESS */ + /* Ensure dirh is in a success state */ + test_NtQueryDirectoryFile_mask(dirh, TRUE, &testmask, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, TRUE, NULL, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, FALSE, NULL, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, TRUE, NULL, STATUS_SUCCESS, NULL, FALSE); + + /* Test empty mask with a previously used handle that last returned an error */ + /* Ensure dirh is in a failure state */ + test_NtQueryDirectoryFile_mask(dirh, TRUE, ¬atestfile, STATUS_NO_MORE_FILES, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, TRUE, &emptymask, STATUS_NO_MORE_FILES, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, FALSE, &emptymask, STATUS_NO_MORE_FILES, NULL, FALSE); + + /* Test empty mask with a previously used handle that last returned STATUS_SUCCESS */ + /* Ensure dirh is in a success state */ + test_NtQueryDirectoryFile_mask(dirh, TRUE, &testmask, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, TRUE, &emptymask, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh, FALSE, &emptymask, STATUS_SUCCESS, NULL, FALSE); + + /* Open a handle for our test directory to test null masks with a fresh handle */ + status = pNtOpenFile(&dirh_test_fresh_null, SYNCHRONIZE | FILE_LIST_DIRECTORY, &attr, &io, FILE_SHARE_READ, + FILE_SYNCHRONOUS_IO_NONALERT | FILE_OPEN_FOR_BACKUP_INTENT | FILE_DIRECTORY_FILE); + ok( status == STATUS_SUCCESS, "failed to open handle to dir '%s' for testing NULL masks, ret 0x%lx, error %ld\n", testdir, status, GetLastError() ); + + /* Test NULL mask with a fresh handle */ + test_NtQueryDirectoryFile_mask(dirh_test_fresh_null, TRUE, NULL, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh_test_fresh_null, FALSE, NULL, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh_test_fresh_null, TRUE, NULL, STATUS_SUCCESS, NULL, FALSE); + /* Release fresh handle for null mask tests */ + pNtClose(&dirh_test_fresh_null); + + /* Open a handle for our test directory to test empty masks with a fresh handle */ + status = pNtOpenFile(&dirh_test_fresh_empty, SYNCHRONIZE | FILE_LIST_DIRECTORY, &attr, &io, FILE_SHARE_READ, + FILE_SYNCHRONOUS_IO_NONALERT | FILE_OPEN_FOR_BACKUP_INTENT | FILE_DIRECTORY_FILE); + ok( status == STATUS_SUCCESS, "failed to open handle to dir '%s' for testing empty masks, ret 0x%lx, error %ld\n", testdir, status, GetLastError() ); + + /* Test empty mask with a fresh handle */ + test_NtQueryDirectoryFile_mask(dirh_test_fresh_empty, TRUE, &emptymask, STATUS_SUCCESS, NULL, FALSE); + test_NtQueryDirectoryFile_mask(dirh_test_fresh_empty, FALSE, &emptymask, STATUS_SUCCESS, NULL, FALSE); + /* Release fresh handle for empty mask tests */ + pNtClose(&dirh_test_fresh_empty); + + /* Cleanup */ +done: + tear_down_mask_test(testdir); + pNtClose(&dirh); +} + static NTSTATUS get_file_id( FILE_INTERNAL_INFORMATION *info, const WCHAR *root, const WCHAR *name ) { OBJECT_ATTRIBUTES attr; @@ -1157,5 +1393,6 @@ START_TEST(directory) test_directory_sort( sysdir ); test_NtQueryDirectoryFile(); test_NtQueryDirectoryFile_case(); + test_NtQueryDirectoryFile_change_mask(); test_redirection(); }
From: Eugene McArdle eugene@tensorworks.com.au
--- dlls/ntdll/tests/directory.c | 3 -- dlls/ntdll/unix/file.c | 69 ++++++++++++++++++++++++++++++++---- 2 files changed, 62 insertions(+), 10 deletions(-)
diff --git a/dlls/ntdll/tests/directory.c b/dlls/ntdll/tests/directory.c index cd7c21ceeba..f60b1f217f0 100644 --- a/dlls/ntdll/tests/directory.c +++ b/dlls/ntdll/tests/directory.c @@ -1033,7 +1033,6 @@ static BOOL test_NtQueryDirectoryFile_mask(HANDLE handle, BOOL restart_scan, UNI
if (validate_only && status != expected_status) return FALSE;
- todo_wine_if(status != expected_status) ok( status == expected_status, "unexpected status : 0x%lx Test settings: file mask: '%s', restart: %d, expected status: 0x%lx\n", status, wine_dbgstr_wn(mask_value, mask_len), restart_scan, expected_status );
@@ -1046,9 +1045,7 @@ static BOOL test_NtQueryDirectoryFile_mask(HANDLE handle, BOOL restart_scan, UNI match_len = match->Length / sizeof(WCHAR); match_value = match->Buffer;
- todo_wine_if(name_len != match_len) ok( name_len == match_len, "unexpected filename length %lu, expected %lu\n", name_len, match_len ); - todo_wine_if(name_len != match_len) ok( !memcmp(name, match_value, match_len * sizeof(WCHAR)), "unexpected filename %s, expected %s\n", wine_dbgstr_wn(name, name_len), wine_dbgstr_wn(match_value, match_len) ); } diff --git a/dlls/ntdll/unix/file.c b/dlls/ntdll/unix/file.c index 8bc69557057..c0b9b6cd4d2 100644 --- a/dlls/ntdll/unix/file.c +++ b/dlls/ntdll/unix/file.c @@ -225,6 +225,7 @@ struct dir_data struct file_identity id; /* directory file identity */ struct dir_data_names *names; /* directory file names */ struct dir_data_buffer *buffer; /* head of data buffers list */ + UNICODE_STRING mask; /* the mask used when creating the cache entry */ };
static const unsigned int dir_data_buffer_initial_size = 4096; @@ -565,6 +566,7 @@ static void free_dir_data( struct dir_data *data ) free( buffer ); } free( data->names ); + if (data->mask.Buffer) free( data->mask.Buffer ); free( data ); }
@@ -2607,6 +2609,14 @@ static NTSTATUS init_cached_dir_data( struct dir_data **data_ret, int fd, const return status; }
+ /* if a mask was specified then copy it into the cache entry */ + if (mask) + { + data->mask.Length = data->mask.MaximumLength = mask->Length; + if (!(data->mask.Buffer = calloc( 1, mask->Length ))) return STATUS_NO_MEMORY; + memcpy(data->mask.Buffer, mask->Buffer, mask->Length); + } + /* sort filenames, but not "." and ".." */ i = 0; if (i < data->count && !strcmp( data->names[i].unix_name, "." )) i++; @@ -2629,17 +2639,34 @@ static NTSTATUS init_cached_dir_data( struct dir_data **data_ret, int fd, const }
+/*********************************************************************** + * ustring_equal + * + * Simplified version of RtlEqualUnicodeString that performs only case-sensitive comparisons. + */ +static BOOLEAN ustring_equal( const UNICODE_STRING *a, const UNICODE_STRING *b ) +{ + USHORT length_a = (a ? a->Length : 0); + USHORT length_b = (b ? b->Length : 0); + if (length_a != length_b) return FALSE; + + if (length_a == 0 && length_b == 0) return TRUE; + return (!memcmp(a->Buffer, b->Buffer, a->Length)); +} + + /*********************************************************************** * get_cached_dir_data * * Retrieve the cached directory data, or initialize it if necessary. */ static unsigned int get_cached_dir_data( HANDLE handle, struct dir_data **data_ret, int fd, - const UNICODE_STRING *mask ) + const UNICODE_STRING *mask, BOOLEAN restart_scan ) { unsigned int i; int entry = -1, free_entries[16]; unsigned int status; + BOOLEAN fresh_handle;
SERVER_START_REQ( get_directory_cache_entry ) { @@ -2676,9 +2703,33 @@ static unsigned int get_cached_dir_data( HANDLE handle, struct dir_data **data_r dir_data_cache_size = size; }
- if (!dir_data_cache[entry]) status = init_cached_dir_data( &dir_data_cache[entry], fd, mask ); + /* If we have an existing cache entry then set the flag to indicate that the handle is not fresh */ + fresh_handle = (dir_data_cache[entry] == NULL); + + /* If we have an existing cache entry, but restart_scan is true and a new non-empty mask + was specified then we need to invalidate the existing cache entry and create a new one + */ + if (dir_data_cache[entry] && restart_scan && mask && + !ustring_equal(&dir_data_cache[entry]->mask, mask)) + { + TRACE( "invalidating existing cache entry for handle %p, old mask: "%s", new mask: "%s"\n", + handle, debugstr_us(&(dir_data_cache[entry]->mask)), debugstr_us(mask)); + free_dir_data( dir_data_cache[entry] ); + dir_data_cache[entry] = NULL; + } + + if (!dir_data_cache[entry]) + { + status = init_cached_dir_data( &dir_data_cache[entry], fd, mask ); + /* Return STATUS_NO_MORE_FILES if a mask did not match a file and the handle had previously been used */ + if (status == STATUS_NO_SUCH_FILE && !fresh_handle) + { + status = STATUS_NO_MORE_FILES; + } + }
*data_ret = dir_data_cache[entry]; + if (restart_scan) (*data_ret)->pos = 0; return status; }
@@ -2741,18 +2792,18 @@ NTSTATUS WINAPI NtQueryDirectoryFile( HANDLE handle, HANDLE event, PIO_APC_ROUTI }
io->Information = 0; + if (mask && mask->Length == 0) mask = NULL;
mutex_lock( &dir_mutex );
cwd = open( ".", O_RDONLY ); if (fchdir( fd ) != -1) { - if (!(status = get_cached_dir_data( handle, &data, fd, mask ))) + status = get_cached_dir_data( handle, &data, fd, mask, restart_scan ); + if (!status) { union file_directory_info *last_info = NULL;
- if (restart_scan) data->pos = 0; - while (!status && data->pos < data->count) { status = get_dir_data_entry( data, buffer, io, length, info_class, &last_info ); @@ -2762,13 +2813,17 @@ NTSTATUS WINAPI NtQueryDirectoryFile( HANDLE handle, HANDLE event, PIO_APC_ROUTI
if (!last_info) status = STATUS_NO_MORE_FILES; else if (status == STATUS_MORE_ENTRIES) status = STATUS_SUCCESS; - - io->Status = status; } if (cwd == -1 || fchdir( cwd ) == -1) chdir( "/" ); } else status = errno_to_status( errno );
+ /* io->Status should only update if status isn't STATUS_NO_SUCH_FILE */ + if (status != STATUS_NO_SUCH_FILE) + { + io->Status = status; + } + mutex_unlock( &dir_mutex );
if (needs_close) close( fd );
On Wed Feb 5 08:05:10 2025 +0000, eric pouech wrote:
Adam, agreeing on these remarks. Real life impact would however vary from usage of rescan (between restart keeping current mask vs use a different mask). Out of curiosity, do you have some figures to share here from your current test cases? My thoughts behind the question deals in fact with the data structures used... I wondered if the mask should be part of the cache, or part of an iterator on top of the directory view (meaning that the cached directory view shouldn't be filtered by the mask). That would cleanly separate iterator operation (restart, set a different mask) from flushing the cache entry. But that would have other impacts. So I'm not considering it for this MR. So ok to keep the mask inside the cache entry.
Apologies for the delay in getting the performance numbers back to you, the flooding and resulting network outages that impacted our office slowed things down a bit.
Eugene ran some tests with MSVC building the Unreal Engine's `ShaderCompileWorker` target (which contains around 100 C++ source files), and found that on average the build times increased by about 3 seconds from a baseline of around 150 seconds, representing approximately a 2% overhead when unconditionally flushing the cache regardless of whether the mask has changed. It's not a massive difference for a small build target like this one, but given that we expect it to scale with the number of translation units being compiled, it would likely represent a more significant issue for larger compilation jobs.
I've rebased the PR on the latest master, and even though we're still seeing failing tests in the pipeline, none of those are caused by code in this PR. I think we're good to merge this in :thumbsup:
On Thu Feb 6 03:41:18 2025 +0000, Adam Rehn wrote:
Apologies for the delay in getting the performance numbers back to you, the flooding and resulting network outages that impacted our office slowed things down a bit. Eugene ran some tests with MSVC building the Unreal Engine's `ShaderCompileWorker` target (which contains around 100 C++ source files), and found that on average the build times increased by about 3 seconds from a baseline of around 150 seconds, representing approximately a 2% overhead when unconditionally flushing the cache regardless of whether the mask has changed. It's not a massive difference for a small build target like this one, but given that we expect it to scale with the number of translation units being compiled, it would likely represent a more significant issue for larger compilation jobs.
Thanks Adam & Eugene for taking the time to document this. So yes that's worth the effort of storing the mask for cases that stress the directory cache (compilation being ofc one of them).
-
Adam Rehn (＠adamrehn) -
eric pouech (＠epo)
-
Eugene McArdle
-
Eugene McArdle (＠eugenemcardle)