[PATCH v2 0/4] MR9510: Draft: win32u: Implement timeline semaphore rewinding.

[Rémi Bernon]

Pretty much the same kind of thing Proton does, this is only for non-shared semaphores for now.

This creates timeline events backed by host timeline semaphores, for every wait and signal operation on a client timeline semaphore. Events are host timeline semaphores with monotonically increasing values. One event is a `(host semaphore, value)` unique tuple, and their value is incremented every time they get signaled. They can be reused right away as a different event, with the new value. Signaled events are queued to a per-device event list for reuse, as we cannot safely destroy them [^1] and as creating them might be costly. 

A thread is spawned for every device that uses timeline semaphores, monitoring the events and semaphore value changes. The semaphore wrapper keeps the current client semaphore value, which is read directly by `vkGetSemaphoreCounterValue`.

CPU and GPU waits on the client semaphore are swapped with a wait on a timeline event, and GPU signals with a signal on a timeline event. CPU signals simply update the client semaphore value on the wrapper, signaling the timeline thread to check and notify any waiter. The timeline thread waits on signal events, coming from the GPU, and on a per-device semaphore for CPU notifications (wait/signal list updates, CPU-side signal). It will then signal any pending wait for a client semaphore value that has been reached, and removes timed out waits.

---

For shared semaphores my idea is to use the client timeline semaphore host handle itself as a signal to notify other device threads of semaphore value changes, as the host semaphore is what is exported and imported in other devices. The timeline threads would wait on that semaphore too in addition to the signal events and thread notification semaphore.

The main issue with shared semaphores actually comes from sharing the client semaphore values, so that they can be read from each device timeline threads as well as from other processes. There's two scenarios to support, one that is for in-process sharing which could perhaps keep the information locally, but we also need to support cross-process sharing so I think there's no other way than to involve wineserver.

My idea then is to move shared semaphore client value to wineserver, which will cost a request on every signal and wait (and reading the value for waits could later be moved to a shared memory object). This will also allow us to implement `D3DKMTWaitForSynchronizationObjectFromCpu` which might be useful for D3D12 fence implementation as it'll allow us to translate a timeline semaphore wait to an asynchronous NT event signal.

[^1]: Vulkan spec indicates that semaphores may only be destroyed *after* every operation that uses them has fully executed, and it's unclear whether signaling or waiting a semaphore is enough as an indicator for full execution or whether we would need to wait on a submit fence.

--
  v2: win32u: Implement timeline semaphore rewinding support.
      win32u: Make a copy of the timeline semaphore value arrays.
      win32u: Spawn a device thread for timeline semaphores.
      win32u: Hook more vulkan timeline semaphore functions.
 
https://gitlab.winehq.org/wine/wine/-/merge_requests/9510

[PATCH v2 1/4] win32u: Hook more vulkan timeline semaphore functions.

[Rémi Bernon]

From: Rémi Bernon <rbernon(a)codeweavers.com>

---
 dlls/win32u/vulkan.c            | 174 ++++++++++++++
 dlls/winevulkan/make_vulkan     |   8 +
 dlls/winevulkan/vulkan_thunks.c | 406 +++++++++++---------------------
 include/wine/vulkan_driver.h    |   7 +
 4 files changed, 327 insertions(+), 268 deletions(-)

diff --git a/dlls/win32u/vulkan.c b/dlls/win32u/vulkan.c
index 6255f5b46c0..fd34ce8b968 100644
--- a/dlls/win32u/vulkan.c
+++ b/dlls/win32u/vulkan.c
@@ -2091,6 +2091,79 @@ failed:
     return VK_ERROR_UNKNOWN;
 }
 
+static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count, const VkBindSparseInfo *binds, VkFence client_fence )
+{
+    struct vulkan_fence *fence = client_fence ? vulkan_fence_from_handle( client_fence ) : NULL;
+    struct vulkan_queue *queue = vulkan_queue_from_handle( client_queue );
+    struct vulkan_device *device = queue->device;
+
+    TRACE( "queue %p, count %u, binds %p, fence %p\n", queue, count, binds, fence );
+
+    for (uint32_t i = 0; i < count; i++)
+    {
+        VkBindSparseInfo *bind = (VkBindSparseInfo *)binds + i; /* cast away const, chain has been copied in the thunks */
+        VkBaseOutStructure **next, *prev = (VkBaseOutStructure *)bind;
+
+        for (const VkSparseBufferMemoryBindInfo *buffer = bind->pBufferBinds, *end = buffer + bind->bufferBindCount; buffer < end; buffer++)
+        {
+            for (uint32_t j = 0; j < buffer->bindCount; j++)
+            {
+                VkSparseMemoryBind *binds = (VkSparseMemoryBind *)buffer->pBinds; /* cast away const, chain has been copied in the thunks */
+                struct vulkan_device_memory *memory = vulkan_device_memory_from_handle( binds[j].memory );
+                if (memory) binds[j].memory = memory->host.device_memory;
+            }
+        }
+
+        for (const VkSparseImageOpaqueMemoryBindInfo *opaque = bind->pImageOpaqueBinds, *end = opaque + bind->imageOpaqueBindCount; opaque < end; opaque++)
+        {
+            for (uint32_t j = 0; j < opaque->bindCount; j++)
+            {
+                VkSparseMemoryBind *binds = (VkSparseMemoryBind *)opaque->pBinds; /* cast away const, chain has been copied in the thunks */
+                struct vulkan_device_memory *memory = vulkan_device_memory_from_handle( binds[j].memory );
+                if (memory) binds[j].memory = memory->host.device_memory;
+            }
+        }
+
+        for (const VkSparseImageMemoryBindInfo *image = bind->pImageBinds, *end = image + bind->imageBindCount; image < end; image++)
+        {
+            for (uint32_t j = 0; j < image->bindCount; j++)
+            {
+                VkSparseImageMemoryBind *binds = (VkSparseImageMemoryBind *)image->pBinds; /* cast away const, chain has been copied in the thunks */
+                struct vulkan_device_memory *memory = vulkan_device_memory_from_handle( binds[j].memory );
+                if (memory) binds[j].memory = memory->host.device_memory;
+            }
+        }
+
+        for (next = &prev->pNext; *next; prev = *next, next = &(*next)->pNext)
+        {
+            switch ((*next)->sType)
+            {
+            case VK_STRUCTURE_TYPE_DEVICE_GROUP_BIND_SPARSE_INFO: break;
+            case VK_STRUCTURE_TYPE_FRAME_BOUNDARY_EXT: break;
+            case VK_STRUCTURE_TYPE_FRAME_BOUNDARY_TENSORS_ARM: break;
+            case VK_STRUCTURE_TYPE_TIMELINE_SEMAPHORE_SUBMIT_INFO: break;
+            default: FIXME( "Unhandled sType %u.\n", (*next)->sType ); break;
+            }
+        }
+
+        for (uint32_t j = 0; j < bind->waitSemaphoreCount; j++)
+        {
+            VkSemaphore *semaphores = (VkSemaphore *)bind->pWaitSemaphores; /* cast away const, it has been copied in the thunks */
+            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
+            semaphores[j] = semaphore->host.semaphore;
+        }
+
+        for (uint32_t j = 0; j < bind->signalSemaphoreCount; j++)
+        {
+            VkSemaphore *semaphores = (VkSemaphore *)bind->pSignalSemaphores; /* cast away const, it has been copied in the thunks */
+            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
+            semaphores[j] = semaphore->host.semaphore;
+        }
+    }
+
+    return device->p_vkQueueBindSparse( queue->host.queue, count, binds, fence ? fence->host.fence : 0 );
+}
+
 static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, const VkSubmitInfo *submits, VkFence client_fence )
 {
     struct vulkan_fence *fence = client_fence ? vulkan_fence_from_handle( client_fence ) : NULL;
@@ -2464,6 +2537,100 @@ static void win32u_vkDestroySemaphore( VkDevice client_device, VkSemaphore clien
     free( semaphore );
 }
 
+static VkResult get_semaphore_counter_value( struct vulkan_device *device, struct semaphore *semaphore, uint64_t *value,
+                                             PFN_vkGetSemaphoreCounterValue p_vkGetSemaphoreCounterValue )
+{
+    return p_vkGetSemaphoreCounterValue( device->host.device, semaphore->obj.host.semaphore, value );
+}
+
+static VkResult win32u_vkGetSemaphoreCounterValue( VkDevice client_device, VkSemaphore client_semaphore, uint64_t *value )
+{
+    struct vulkan_device *device = vulkan_device_from_handle( client_device );
+    struct semaphore *semaphore = semaphore_from_handle( client_semaphore );
+    TRACE( "device %p, semaphore %p, value %p\n", device, semaphore, value );
+    return get_semaphore_counter_value( device, semaphore, value, device->p_vkGetSemaphoreCounterValue );
+}
+
+static VkResult win32u_vkGetSemaphoreCounterValueKHR( VkDevice client_device, VkSemaphore client_semaphore, uint64_t *value )
+{
+    struct vulkan_device *device = vulkan_device_from_handle( client_device );
+    struct semaphore *semaphore = semaphore_from_handle( client_semaphore );
+    TRACE( "device %p, semaphore %p, value %p\n", device, semaphore, value );
+    return get_semaphore_counter_value( device, semaphore, value, device->p_vkGetSemaphoreCounterValueKHR );
+}
+
+static VkResult signal_semaphore( struct vulkan_device *device, const VkSemaphoreSignalInfo *signal_info,
+                                  PFN_vkSignalSemaphore p_vkSignalSemaphore )
+{
+    struct semaphore *semaphore = semaphore_from_handle( signal_info->semaphore );
+    VkSemaphoreSignalInfo info = *signal_info;
+
+    if (info.pNext) FIXME( "pNext not implemented\n" );
+    info.pNext = NULL;
+
+    info.semaphore = semaphore->obj.host.semaphore;
+    return p_vkSignalSemaphore( device->host.device, &info );
+}
+
+static VkResult win32u_vkSignalSemaphore( VkDevice client_device, const VkSemaphoreSignalInfo *signal_info )
+{
+    struct vulkan_device *device = vulkan_device_from_handle( client_device );
+    TRACE( "device %p, signal_info %p\n", device, signal_info );
+    return signal_semaphore( device, signal_info, device->p_vkSignalSemaphore );
+}
+
+static VkResult win32u_vkSignalSemaphoreKHR( VkDevice client_device, const VkSemaphoreSignalInfo *signal_info )
+{
+    struct vulkan_device *device = vulkan_device_from_handle( client_device );
+    TRACE( "device %p, signal_info %p\n", device, signal_info );
+    return signal_semaphore( device, signal_info, device->p_vkSignalSemaphoreKHR );
+}
+
+static VkResult wait_semaphores( struct vulkan_device *device, const VkSemaphoreWaitInfo *wait_info, uint64_t timeout,
+                                 PFN_vkWaitSemaphores p_vkWaitSemaphores )
+{
+    VkResult res = VK_ERROR_OUT_OF_HOST_MEMORY;
+    VkSemaphoreWaitInfo info = *wait_info;
+    struct mempool pool = {0};
+    VkSemaphore *semaphores;
+    uint64_t *values;
+
+    if (!(semaphores = mem_alloc( &pool, info.semaphoreCount * sizeof(semaphores) ))) return VK_ERROR_OUT_OF_HOST_MEMORY;
+    info.pSemaphores = memcpy( semaphores, info.pSemaphores, info.semaphoreCount * sizeof(semaphores) );
+
+    if (!(values = mem_alloc( &pool, info.semaphoreCount * sizeof(values) ))) goto failed;
+    info.pValues = memcpy( values, info.pValues, info.semaphoreCount * sizeof(values) );
+
+    if (info.pNext) FIXME( "pNext not implemented\n" );
+    info.pNext = NULL;
+
+    for (uint32_t i = 0; i < info.semaphoreCount; i++)
+    {
+        struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[i] );
+        semaphores[i] = semaphore->host.semaphore;
+    }
+
+    res = p_vkWaitSemaphores( device->host.device, &info, timeout );
+
+failed:
+    mem_free( &pool );
+    return res;
+}
+
+static VkResult win32u_vkWaitSemaphores( VkDevice client_device, const VkSemaphoreWaitInfo *wait_info, uint64_t timeout )
+{
+    struct vulkan_device *device = vulkan_device_from_handle( client_device );
+    TRACE( "device %p, wait_info %p, timeout %jd\n", device, wait_info, (intmax_t)timeout );
+    return wait_semaphores( device, wait_info, timeout, device->p_vkWaitSemaphores );
+}
+
+static VkResult win32u_vkWaitSemaphoresKHR( VkDevice client_device, const VkSemaphoreWaitInfo *wait_info, uint64_t timeout )
+{
+    struct vulkan_device *device = vulkan_device_from_handle( client_device );
+    TRACE( "device %p, wait_info %p, timeout %jd\n", device, wait_info, (intmax_t)timeout );
+    return wait_semaphores( device, wait_info, timeout, device->p_vkWaitSemaphoresKHR );
+}
+
 static VkResult win32u_vkGetSemaphoreWin32HandleKHR( VkDevice client_device, const VkSemaphoreGetWin32HandleInfoKHR *handle_info, HANDLE *handle )
 {
     struct vulkan_device *device = vulkan_device_from_handle( client_device );
@@ -2855,17 +3022,24 @@ static struct vulkan_funcs vulkan_funcs =
     .p_vkGetPhysicalDeviceSurfaceFormats2KHR = win32u_vkGetPhysicalDeviceSurfaceFormats2KHR,
     .p_vkGetPhysicalDeviceSurfaceFormatsKHR = win32u_vkGetPhysicalDeviceSurfaceFormatsKHR,
     .p_vkGetPhysicalDeviceWin32PresentationSupportKHR = win32u_vkGetPhysicalDeviceWin32PresentationSupportKHR,
+    .p_vkGetSemaphoreCounterValue = win32u_vkGetSemaphoreCounterValue,
+    .p_vkGetSemaphoreCounterValueKHR = win32u_vkGetSemaphoreCounterValueKHR,
     .p_vkGetSemaphoreWin32HandleKHR = win32u_vkGetSemaphoreWin32HandleKHR,
     .p_vkImportFenceWin32HandleKHR = win32u_vkImportFenceWin32HandleKHR,
     .p_vkImportSemaphoreWin32HandleKHR = win32u_vkImportSemaphoreWin32HandleKHR,
     .p_vkMapMemory = win32u_vkMapMemory,
     .p_vkMapMemory2KHR = win32u_vkMapMemory2KHR,
+    .p_vkQueueBindSparse = win32u_vkQueueBindSparse,
     .p_vkQueuePresentKHR = win32u_vkQueuePresentKHR,
     .p_vkQueueSubmit = win32u_vkQueueSubmit,
     .p_vkQueueSubmit2 = win32u_vkQueueSubmit2,
     .p_vkQueueSubmit2KHR = win32u_vkQueueSubmit2KHR,
+    .p_vkSignalSemaphore = win32u_vkSignalSemaphore,
+    .p_vkSignalSemaphoreKHR = win32u_vkSignalSemaphoreKHR,
     .p_vkUnmapMemory = win32u_vkUnmapMemory,
     .p_vkUnmapMemory2KHR = win32u_vkUnmapMemory2KHR,
+    .p_vkWaitSemaphores = win32u_vkWaitSemaphores,
+    .p_vkWaitSemaphoresKHR = win32u_vkWaitSemaphoresKHR,
 };
 
 static VkResult nulldrv_vulkan_surface_create( HWND hwnd, const struct vulkan_instance *instance, VkSurfaceKHR *surface,
diff --git a/dlls/winevulkan/make_vulkan b/dlls/winevulkan/make_vulkan
index ba8e8695fd1..7ce993e88ab 100755
--- a/dlls/winevulkan/make_vulkan
+++ b/dlls/winevulkan/make_vulkan
@@ -227,17 +227,24 @@ USER_DRIVER_FUNCS = {
     "vkGetPhysicalDeviceSurfaceFormats2KHR",
     "vkGetPhysicalDeviceSurfaceFormatsKHR",
     "vkGetPhysicalDeviceWin32PresentationSupportKHR",
+    "vkGetSemaphoreCounterValue",
+    "vkGetSemaphoreCounterValueKHR",
     "vkGetSemaphoreWin32HandleKHR",
     "vkImportFenceWin32HandleKHR",
     "vkImportSemaphoreWin32HandleKHR",
     "vkMapMemory",
     "vkMapMemory2KHR",
+    "vkQueueBindSparse",
     "vkQueuePresentKHR",
     "vkQueueSubmit",
     "vkQueueSubmit2",
     "vkQueueSubmit2KHR",
+    "vkSignalSemaphore",
+    "vkSignalSemaphoreKHR",
     "vkUnmapMemory",
     "vkUnmapMemory2KHR",
+    "vkWaitSemaphores",
+    "vkWaitSemaphoresKHR",
 }
 
 # functions for which the unix thunk is manually implemented
@@ -307,6 +314,7 @@ STRUCT_CHAIN_CONVERSIONS = {
     "VkFenceCreateInfo": {},
     "VkSubmitInfo": {},
     "VkSubmitInfo2": {},
+    "VkBindSparseInfo": {},
 
     # Ignore to not confuse host loader.
     "VkDeviceCreateInfo": {"strip": ["VK_STRUCTURE_TYPE_LOADER_DEVICE_CREATE_INFO"]},
diff --git a/dlls/winevulkan/vulkan_thunks.c b/dlls/winevulkan/vulkan_thunks.c
index 0497c19add2..3a92999edc4 100644
--- a/dlls/winevulkan/vulkan_thunks.c
+++ b/dlls/winevulkan/vulkan_thunks.c
@@ -40336,7 +40336,7 @@ static void convert_VkMemoryMapInfo_win32_to_unwrapped_host(const VkMemoryMapInf
 }
 
 #ifdef _WIN64
-static const VkSemaphore *convert_VkSemaphore_array_win64_to_host(struct conversion_context *ctx, const VkSemaphore *in, uint32_t count)
+static const VkSemaphore *convert_VkSemaphore_array_win64_to_unwrapped_host(struct conversion_context *ctx, const VkSemaphore *in, uint32_t count)
 {
     VkSemaphore *out;
     unsigned int i;
@@ -40346,128 +40346,7 @@ static const VkSemaphore *convert_VkSemaphore_array_win64_to_host(struct convers
     out = conversion_context_alloc(ctx, count * sizeof(*out));
     for (i = 0; i < count; i++)
     {
-        out[i] = vulkan_semaphore_from_handle(in[i])->host.semaphore;
-    }
-
-    return out;
-}
-#endif /* _WIN64 */
-
-#ifdef _WIN64
-static void convert_VkSparseMemoryBind_win64_to_host(const VkSparseMemoryBind *in, VkSparseMemoryBind *out)
-{
-    if (!in) return;
-
-    out->resourceOffset = in->resourceOffset;
-    out->size = in->size;
-    out->memory = in->memory ? vulkan_device_memory_from_handle(in->memory)->host.device_memory : 0;
-    out->memoryOffset = in->memoryOffset;
-    out->flags = in->flags;
-}
-#endif /* _WIN64 */
-
-#ifdef _WIN64
-static const VkSparseMemoryBind *convert_VkSparseMemoryBind_array_win64_to_host(struct conversion_context *ctx, const VkSparseMemoryBind *in, uint32_t count)
-{
-    VkSparseMemoryBind *out;
-    unsigned int i;
-
-    if (!in || !count) return NULL;
-
-    out = conversion_context_alloc(ctx, count * sizeof(*out));
-    for (i = 0; i < count; i++)
-    {
-        convert_VkSparseMemoryBind_win64_to_host(&in[i], &out[i]);
-    }
-
-    return out;
-}
-#endif /* _WIN64 */
-
-#ifdef _WIN64
-static void convert_VkSparseBufferMemoryBindInfo_win64_to_host(struct conversion_context *ctx, const VkSparseBufferMemoryBindInfo *in, VkSparseBufferMemoryBindInfo *out)
-{
-    if (!in) return;
-
-    out->buffer = in->buffer;
-    out->bindCount = in->bindCount;
-    out->pBinds = convert_VkSparseMemoryBind_array_win64_to_host(ctx, in->pBinds, in->bindCount);
-}
-#endif /* _WIN64 */
-
-#ifdef _WIN64
-static const VkSparseBufferMemoryBindInfo *convert_VkSparseBufferMemoryBindInfo_array_win64_to_host(struct conversion_context *ctx, const VkSparseBufferMemoryBindInfo *in, uint32_t count)
-{
-    VkSparseBufferMemoryBindInfo *out;
-    unsigned int i;
-
-    if (!in || !count) return NULL;
-
-    out = conversion_context_alloc(ctx, count * sizeof(*out));
-    for (i = 0; i < count; i++)
-    {
-        convert_VkSparseBufferMemoryBindInfo_win64_to_host(ctx, &in[i], &out[i]);
-    }
-
-    return out;
-}
-#endif /* _WIN64 */
-
-#ifdef _WIN64
-static void convert_VkSparseImageOpaqueMemoryBindInfo_win64_to_host(struct conversion_context *ctx, const VkSparseImageOpaqueMemoryBindInfo *in, VkSparseImageOpaqueMemoryBindInfo *out)
-{
-    if (!in) return;
-
-    out->image = in->image;
-    out->bindCount = in->bindCount;
-    out->pBinds = convert_VkSparseMemoryBind_array_win64_to_host(ctx, in->pBinds, in->bindCount);
-}
-#endif /* _WIN64 */
-
-#ifdef _WIN64
-static const VkSparseImageOpaqueMemoryBindInfo *convert_VkSparseImageOpaqueMemoryBindInfo_array_win64_to_host(struct conversion_context *ctx, const VkSparseImageOpaqueMemoryBindInfo *in, uint32_t count)
-{
-    VkSparseImageOpaqueMemoryBindInfo *out;
-    unsigned int i;
-
-    if (!in || !count) return NULL;
-
-    out = conversion_context_alloc(ctx, count * sizeof(*out));
-    for (i = 0; i < count; i++)
-    {
-        convert_VkSparseImageOpaqueMemoryBindInfo_win64_to_host(ctx, &in[i], &out[i]);
-    }
-
-    return out;
-}
-#endif /* _WIN64 */
-
-#ifdef _WIN64
-static void convert_VkSparseImageMemoryBind_win64_to_host(const VkSparseImageMemoryBind *in, VkSparseImageMemoryBind *out)
-{
-    if (!in) return;
-
-    out->subresource = in->subresource;
-    out->offset = in->offset;
-    out->extent = in->extent;
-    out->memory = in->memory ? vulkan_device_memory_from_handle(in->memory)->host.device_memory : 0;
-    out->memoryOffset = in->memoryOffset;
-    out->flags = in->flags;
-}
-#endif /* _WIN64 */
-
-#ifdef _WIN64
-static const VkSparseImageMemoryBind *convert_VkSparseImageMemoryBind_array_win64_to_host(struct conversion_context *ctx, const VkSparseImageMemoryBind *in, uint32_t count)
-{
-    VkSparseImageMemoryBind *out;
-    unsigned int i;
-
-    if (!in || !count) return NULL;
-
-    out = conversion_context_alloc(ctx, count * sizeof(*out));
-    for (i = 0; i < count; i++)
-    {
-        convert_VkSparseImageMemoryBind_win64_to_host(&in[i], &out[i]);
+        out[i] = in[i];
     }
 
     return out;
@@ -40475,56 +40354,97 @@ static const VkSparseImageMemoryBind *convert_VkSparseImageMemoryBind_array_win6
 #endif /* _WIN64 */
 
 #ifdef _WIN64
-static void convert_VkSparseImageMemoryBindInfo_win64_to_host(struct conversion_context *ctx, const VkSparseImageMemoryBindInfo *in, VkSparseImageMemoryBindInfo *out)
-{
-    if (!in) return;
-
-    out->image = in->image;
-    out->bindCount = in->bindCount;
-    out->pBinds = convert_VkSparseImageMemoryBind_array_win64_to_host(ctx, in->pBinds, in->bindCount);
-}
-#endif /* _WIN64 */
-
-#ifdef _WIN64
-static const VkSparseImageMemoryBindInfo *convert_VkSparseImageMemoryBindInfo_array_win64_to_host(struct conversion_context *ctx, const VkSparseImageMemoryBindInfo *in, uint32_t count)
+static void convert_VkBindSparseInfo_win64_to_unwrapped_host(struct conversion_context *ctx, const VkBindSparseInfo *in, VkBindSparseInfo *out)
 {
-    VkSparseImageMemoryBindInfo *out;
-    unsigned int i;
-
-    if (!in || !count) return NULL;
-
-    out = conversion_context_alloc(ctx, count * sizeof(*out));
-    for (i = 0; i < count; i++)
-    {
-        convert_VkSparseImageMemoryBindInfo_win64_to_host(ctx, &in[i], &out[i]);
-    }
-
-    return out;
-}
-#endif /* _WIN64 */
+    const VkBaseInStructure *in_header;
+    VkBaseOutStructure *out_header = (void *)out;
 
-#ifdef _WIN64
-static void convert_VkBindSparseInfo_win64_to_host(struct conversion_context *ctx, const VkBindSparseInfo *in, VkBindSparseInfo *out)
-{
     if (!in) return;
 
     out->sType = in->sType;
-    out->pNext = in->pNext;
+    out->pNext = NULL;
     out->waitSemaphoreCount = in->waitSemaphoreCount;
-    out->pWaitSemaphores = convert_VkSemaphore_array_win64_to_host(ctx, in->pWaitSemaphores, in->waitSemaphoreCount);
+    out->pWaitSemaphores = convert_VkSemaphore_array_win64_to_unwrapped_host(ctx, in->pWaitSemaphores, in->waitSemaphoreCount);
     out->bufferBindCount = in->bufferBindCount;
-    out->pBufferBinds = convert_VkSparseBufferMemoryBindInfo_array_win64_to_host(ctx, in->pBufferBinds, in->bufferBindCount);
+    out->pBufferBinds = in->pBufferBinds;
     out->imageOpaqueBindCount = in->imageOpaqueBindCount;
-    out->pImageOpaqueBinds = convert_VkSparseImageOpaqueMemoryBindInfo_array_win64_to_host(ctx, in->pImageOpaqueBinds, in->imageOpaqueBindCount);
+    out->pImageOpaqueBinds = in->pImageOpaqueBinds;
     out->imageBindCount = in->imageBindCount;
-    out->pImageBinds = convert_VkSparseImageMemoryBindInfo_array_win64_to_host(ctx, in->pImageBinds, in->imageBindCount);
+    out->pImageBinds = in->pImageBinds;
     out->signalSemaphoreCount = in->signalSemaphoreCount;
-    out->pSignalSemaphores = convert_VkSemaphore_array_win64_to_host(ctx, in->pSignalSemaphores, in->signalSemaphoreCount);
+    out->pSignalSemaphores = convert_VkSemaphore_array_win64_to_unwrapped_host(ctx, in->pSignalSemaphores, in->signalSemaphoreCount);
+
+    for (in_header = (void *)in->pNext; in_header; in_header = (void *)in_header->pNext)
+    {
+        switch (in_header->sType)
+        {
+        case VK_STRUCTURE_TYPE_DEVICE_GROUP_BIND_SPARSE_INFO:
+        {
+            VkDeviceGroupBindSparseInfo *out_ext = conversion_context_alloc(ctx, sizeof(*out_ext));
+            const VkDeviceGroupBindSparseInfo *in_ext = (const VkDeviceGroupBindSparseInfo *)in_header;
+            out_ext->sType = VK_STRUCTURE_TYPE_DEVICE_GROUP_BIND_SPARSE_INFO;
+            out_ext->pNext = NULL;
+            out_ext->resourceDeviceIndex = in_ext->resourceDeviceIndex;
+            out_ext->memoryDeviceIndex = in_ext->memoryDeviceIndex;
+            out_header->pNext = (void *)out_ext;
+            out_header = (void *)out_ext;
+            break;
+        }
+        case VK_STRUCTURE_TYPE_FRAME_BOUNDARY_EXT:
+        {
+            VkFrameBoundaryEXT *out_ext = conversion_context_alloc(ctx, sizeof(*out_ext));
+            const VkFrameBoundaryEXT *in_ext = (const VkFrameBoundaryEXT *)in_header;
+            out_ext->sType = VK_STRUCTURE_TYPE_FRAME_BOUNDARY_EXT;
+            out_ext->pNext = NULL;
+            out_ext->flags = in_ext->flags;
+            out_ext->frameID = in_ext->frameID;
+            out_ext->imageCount = in_ext->imageCount;
+            out_ext->pImages = in_ext->pImages;
+            out_ext->bufferCount = in_ext->bufferCount;
+            out_ext->pBuffers = in_ext->pBuffers;
+            out_ext->tagName = in_ext->tagName;
+            out_ext->tagSize = in_ext->tagSize;
+            out_ext->pTag = in_ext->pTag;
+            out_header->pNext = (void *)out_ext;
+            out_header = (void *)out_ext;
+            break;
+        }
+        case VK_STRUCTURE_TYPE_FRAME_BOUNDARY_TENSORS_ARM:
+        {
+            VkFrameBoundaryTensorsARM *out_ext = conversion_context_alloc(ctx, sizeof(*out_ext));
+            const VkFrameBoundaryTensorsARM *in_ext = (const VkFrameBoundaryTensorsARM *)in_header;
+            out_ext->sType = VK_STRUCTURE_TYPE_FRAME_BOUNDARY_TENSORS_ARM;
+            out_ext->pNext = NULL;
+            out_ext->tensorCount = in_ext->tensorCount;
+            out_ext->pTensors = in_ext->pTensors;
+            out_header->pNext = (void *)out_ext;
+            out_header = (void *)out_ext;
+            break;
+        }
+        case VK_STRUCTURE_TYPE_TIMELINE_SEMAPHORE_SUBMIT_INFO:
+        {
+            VkTimelineSemaphoreSubmitInfo *out_ext = conversion_context_alloc(ctx, sizeof(*out_ext));
+            const VkTimelineSemaphoreSubmitInfo *in_ext = (const VkTimelineSemaphoreSubmitInfo *)in_header;
+            out_ext->sType = VK_STRUCTURE_TYPE_TIMELINE_SEMAPHORE_SUBMIT_INFO;
+            out_ext->pNext = NULL;
+            out_ext->waitSemaphoreValueCount = in_ext->waitSemaphoreValueCount;
+            out_ext->pWaitSemaphoreValues = in_ext->pWaitSemaphoreValues;
+            out_ext->signalSemaphoreValueCount = in_ext->signalSemaphoreValueCount;
+            out_ext->pSignalSemaphoreValues = in_ext->pSignalSemaphoreValues;
+            out_header->pNext = (void *)out_ext;
+            out_header = (void *)out_ext;
+            break;
+        }
+        default:
+            FIXME("Unhandled sType %u.\n", in_header->sType);
+            break;
+        }
+    }
 }
 #endif /* _WIN64 */
 
 #ifdef _WIN64
-static const VkBindSparseInfo *convert_VkBindSparseInfo_array_win64_to_host(struct conversion_context *ctx, const VkBindSparseInfo *in, uint32_t count)
+static const VkBindSparseInfo *convert_VkBindSparseInfo_array_win64_to_unwrapped_host(struct conversion_context *ctx, const VkBindSparseInfo *in, uint32_t count)
 {
     VkBindSparseInfo *out;
     unsigned int i;
@@ -40534,14 +40454,14 @@ static const VkBindSparseInfo *convert_VkBindSparseInfo_array_win64_to_host(stru
     out = conversion_context_alloc(ctx, count * sizeof(*out));
     for (i = 0; i < count; i++)
     {
-        convert_VkBindSparseInfo_win64_to_host(ctx, &in[i], &out[i]);
+        convert_VkBindSparseInfo_win64_to_unwrapped_host(ctx, &in[i], &out[i]);
     }
 
     return out;
 }
 #endif /* _WIN64 */
 
-static const VkSemaphore *convert_VkSemaphore_array_win32_to_host(struct conversion_context *ctx, const VkSemaphore *in, uint32_t count)
+static const VkSemaphore *convert_VkSemaphore_array_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSemaphore *in, uint32_t count)
 {
     VkSemaphore *out;
     unsigned int i;
@@ -40551,24 +40471,24 @@ static const VkSemaphore *convert_VkSemaphore_array_win32_to_host(struct convers
     out = conversion_context_alloc(ctx, count * sizeof(*out));
     for (i = 0; i < count; i++)
     {
-        out[i] = vulkan_semaphore_from_handle(in[i])->host.semaphore;
+        out[i] = in[i];
     }
 
     return out;
 }
 
-static void convert_VkSparseMemoryBind_win32_to_host(const VkSparseMemoryBind32 *in, VkSparseMemoryBind *out)
+static void convert_VkSparseMemoryBind_win32_to_unwrapped_host(const VkSparseMemoryBind32 *in, VkSparseMemoryBind *out)
 {
     if (!in) return;
 
     out->resourceOffset = in->resourceOffset;
     out->size = in->size;
-    out->memory = in->memory ? vulkan_device_memory_from_handle(in->memory)->host.device_memory : 0;
+    out->memory = in->memory;
     out->memoryOffset = in->memoryOffset;
     out->flags = in->flags;
 }
 
-static const VkSparseMemoryBind *convert_VkSparseMemoryBind_array_win32_to_host(struct conversion_context *ctx, const VkSparseMemoryBind32 *in, uint32_t count)
+static const VkSparseMemoryBind *convert_VkSparseMemoryBind_array_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSparseMemoryBind32 *in, uint32_t count)
 {
     VkSparseMemoryBind *out;
     unsigned int i;
@@ -40578,22 +40498,22 @@ static const VkSparseMemoryBind *convert_VkSparseMemoryBind_array_win32_to_host(
     out = conversion_context_alloc(ctx, count * sizeof(*out));
     for (i = 0; i < count; i++)
     {
-        convert_VkSparseMemoryBind_win32_to_host(&in[i], &out[i]);
+        convert_VkSparseMemoryBind_win32_to_unwrapped_host(&in[i], &out[i]);
     }
 
     return out;
 }
 
-static void convert_VkSparseBufferMemoryBindInfo_win32_to_host(struct conversion_context *ctx, const VkSparseBufferMemoryBindInfo32 *in, VkSparseBufferMemoryBindInfo *out)
+static void convert_VkSparseBufferMemoryBindInfo_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSparseBufferMemoryBindInfo32 *in, VkSparseBufferMemoryBindInfo *out)
 {
     if (!in) return;
 
     out->buffer = in->buffer;
     out->bindCount = in->bindCount;
-    out->pBinds = convert_VkSparseMemoryBind_array_win32_to_host(ctx, (const VkSparseMemoryBind32 *)UlongToPtr(in->pBinds), in->bindCount);
+    out->pBinds = convert_VkSparseMemoryBind_array_win32_to_unwrapped_host(ctx, (const VkSparseMemoryBind32 *)UlongToPtr(in->pBinds), in->bindCount);
 }
 
-static const VkSparseBufferMemoryBindInfo *convert_VkSparseBufferMemoryBindInfo_array_win32_to_host(struct conversion_context *ctx, const VkSparseBufferMemoryBindInfo32 *in, uint32_t count)
+static const VkSparseBufferMemoryBindInfo *convert_VkSparseBufferMemoryBindInfo_array_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSparseBufferMemoryBindInfo32 *in, uint32_t count)
 {
     VkSparseBufferMemoryBindInfo *out;
     unsigned int i;
@@ -40603,22 +40523,22 @@ static const VkSparseBufferMemoryBindInfo *convert_VkSparseBufferMemoryBindInfo_
     out = conversion_context_alloc(ctx, count * sizeof(*out));
     for (i = 0; i < count; i++)
     {
-        convert_VkSparseBufferMemoryBindInfo_win32_to_host(ctx, &in[i], &out[i]);
+        convert_VkSparseBufferMemoryBindInfo_win32_to_unwrapped_host(ctx, &in[i], &out[i]);
     }
 
     return out;
 }
 
-static void convert_VkSparseImageOpaqueMemoryBindInfo_win32_to_host(struct conversion_context *ctx, const VkSparseImageOpaqueMemoryBindInfo32 *in, VkSparseImageOpaqueMemoryBindInfo *out)
+static void convert_VkSparseImageOpaqueMemoryBindInfo_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSparseImageOpaqueMemoryBindInfo32 *in, VkSparseImageOpaqueMemoryBindInfo *out)
 {
     if (!in) return;
 
     out->image = in->image;
     out->bindCount = in->bindCount;
-    out->pBinds = convert_VkSparseMemoryBind_array_win32_to_host(ctx, (const VkSparseMemoryBind32 *)UlongToPtr(in->pBinds), in->bindCount);
+    out->pBinds = convert_VkSparseMemoryBind_array_win32_to_unwrapped_host(ctx, (const VkSparseMemoryBind32 *)UlongToPtr(in->pBinds), in->bindCount);
 }
 
-static const VkSparseImageOpaqueMemoryBindInfo *convert_VkSparseImageOpaqueMemoryBindInfo_array_win32_to_host(struct conversion_context *ctx, const VkSparseImageOpaqueMemoryBindInfo32 *in, uint32_t count)
+static const VkSparseImageOpaqueMemoryBindInfo *convert_VkSparseImageOpaqueMemoryBindInfo_array_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSparseImageOpaqueMemoryBindInfo32 *in, uint32_t count)
 {
     VkSparseImageOpaqueMemoryBindInfo *out;
     unsigned int i;
@@ -40628,25 +40548,25 @@ static const VkSparseImageOpaqueMemoryBindInfo *convert_VkSparseImageOpaqueMemor
     out = conversion_context_alloc(ctx, count * sizeof(*out));
     for (i = 0; i < count; i++)
     {
-        convert_VkSparseImageOpaqueMemoryBindInfo_win32_to_host(ctx, &in[i], &out[i]);
+        convert_VkSparseImageOpaqueMemoryBindInfo_win32_to_unwrapped_host(ctx, &in[i], &out[i]);
     }
 
     return out;
 }
 
-static void convert_VkSparseImageMemoryBind_win32_to_host(const VkSparseImageMemoryBind32 *in, VkSparseImageMemoryBind *out)
+static void convert_VkSparseImageMemoryBind_win32_to_unwrapped_host(const VkSparseImageMemoryBind32 *in, VkSparseImageMemoryBind *out)
 {
     if (!in) return;
 
     out->subresource = in->subresource;
     out->offset = in->offset;
     out->extent = in->extent;
-    out->memory = in->memory ? vulkan_device_memory_from_handle(in->memory)->host.device_memory : 0;
+    out->memory = in->memory;
     out->memoryOffset = in->memoryOffset;
     out->flags = in->flags;
 }
 
-static const VkSparseImageMemoryBind *convert_VkSparseImageMemoryBind_array_win32_to_host(struct conversion_context *ctx, const VkSparseImageMemoryBind32 *in, uint32_t count)
+static const VkSparseImageMemoryBind *convert_VkSparseImageMemoryBind_array_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSparseImageMemoryBind32 *in, uint32_t count)
 {
     VkSparseImageMemoryBind *out;
     unsigned int i;
@@ -40656,22 +40576,22 @@ static const VkSparseImageMemoryBind *convert_VkSparseImageMemoryBind_array_win3
     out = conversion_context_alloc(ctx, count * sizeof(*out));
     for (i = 0; i < count; i++)
     {
-        convert_VkSparseImageMemoryBind_win32_to_host(&in[i], &out[i]);
+        convert_VkSparseImageMemoryBind_win32_to_unwrapped_host(&in[i], &out[i]);
     }
 
     return out;
 }
 
-static void convert_VkSparseImageMemoryBindInfo_win32_to_host(struct conversion_context *ctx, const VkSparseImageMemoryBindInfo32 *in, VkSparseImageMemoryBindInfo *out)
+static void convert_VkSparseImageMemoryBindInfo_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSparseImageMemoryBindInfo32 *in, VkSparseImageMemoryBindInfo *out)
 {
     if (!in) return;
 
     out->image = in->image;
     out->bindCount = in->bindCount;
-    out->pBinds = convert_VkSparseImageMemoryBind_array_win32_to_host(ctx, (const VkSparseImageMemoryBind32 *)UlongToPtr(in->pBinds), in->bindCount);
+    out->pBinds = convert_VkSparseImageMemoryBind_array_win32_to_unwrapped_host(ctx, (const VkSparseImageMemoryBind32 *)UlongToPtr(in->pBinds), in->bindCount);
 }
 
-static const VkSparseImageMemoryBindInfo *convert_VkSparseImageMemoryBindInfo_array_win32_to_host(struct conversion_context *ctx, const VkSparseImageMemoryBindInfo32 *in, uint32_t count)
+static const VkSparseImageMemoryBindInfo *convert_VkSparseImageMemoryBindInfo_array_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSparseImageMemoryBindInfo32 *in, uint32_t count)
 {
     VkSparseImageMemoryBindInfo *out;
     unsigned int i;
@@ -40681,13 +40601,13 @@ static const VkSparseImageMemoryBindInfo *convert_VkSparseImageMemoryBindInfo_ar
     out = conversion_context_alloc(ctx, count * sizeof(*out));
     for (i = 0; i < count; i++)
     {
-        convert_VkSparseImageMemoryBindInfo_win32_to_host(ctx, &in[i], &out[i]);
+        convert_VkSparseImageMemoryBindInfo_win32_to_unwrapped_host(ctx, &in[i], &out[i]);
     }
 
     return out;
 }
 
-static void convert_VkBindSparseInfo_win32_to_host(struct conversion_context *ctx, const VkBindSparseInfo32 *in, VkBindSparseInfo *out)
+static void convert_VkBindSparseInfo_win32_to_unwrapped_host(struct conversion_context *ctx, const VkBindSparseInfo32 *in, VkBindSparseInfo *out)
 {
     const VkBaseInStructure32 *in_header;
     VkBaseOutStructure *out_header = (void *)out;
@@ -40697,15 +40617,15 @@ static void convert_VkBindSparseInfo_win32_to_host(struct conversion_context *ct
     out->sType = in->sType;
     out->pNext = NULL;
     out->waitSemaphoreCount = in->waitSemaphoreCount;
-    out->pWaitSemaphores = convert_VkSemaphore_array_win32_to_host(ctx, (const VkSemaphore *)UlongToPtr(in->pWaitSemaphores), in->waitSemaphoreCount);
+    out->pWaitSemaphores = convert_VkSemaphore_array_win32_to_unwrapped_host(ctx, (const VkSemaphore *)UlongToPtr(in->pWaitSemaphores), in->waitSemaphoreCount);
     out->bufferBindCount = in->bufferBindCount;
-    out->pBufferBinds = convert_VkSparseBufferMemoryBindInfo_array_win32_to_host(ctx, (const VkSparseBufferMemoryBindInfo32 *)UlongToPtr(in->pBufferBinds), in->bufferBindCount);
+    out->pBufferBinds = convert_VkSparseBufferMemoryBindInfo_array_win32_to_unwrapped_host(ctx, (const VkSparseBufferMemoryBindInfo32 *)UlongToPtr(in->pBufferBinds), in->bufferBindCount);
     out->imageOpaqueBindCount = in->imageOpaqueBindCount;
-    out->pImageOpaqueBinds = convert_VkSparseImageOpaqueMemoryBindInfo_array_win32_to_host(ctx, (const VkSparseImageOpaqueMemoryBindInfo32 *)UlongToPtr(in->pImageOpaqueBinds), in->imageOpaqueBindCount);
+    out->pImageOpaqueBinds = convert_VkSparseImageOpaqueMemoryBindInfo_array_win32_to_unwrapped_host(ctx, (const VkSparseImageOpaqueMemoryBindInfo32 *)UlongToPtr(in->pImageOpaqueBinds), in->imageOpaqueBindCount);
     out->imageBindCount = in->imageBindCount;
-    out->pImageBinds = convert_VkSparseImageMemoryBindInfo_array_win32_to_host(ctx, (const VkSparseImageMemoryBindInfo32 *)UlongToPtr(in->pImageBinds), in->imageBindCount);
+    out->pImageBinds = convert_VkSparseImageMemoryBindInfo_array_win32_to_unwrapped_host(ctx, (const VkSparseImageMemoryBindInfo32 *)UlongToPtr(in->pImageBinds), in->imageBindCount);
     out->signalSemaphoreCount = in->signalSemaphoreCount;
-    out->pSignalSemaphores = convert_VkSemaphore_array_win32_to_host(ctx, (const VkSemaphore *)UlongToPtr(in->pSignalSemaphores), in->signalSemaphoreCount);
+    out->pSignalSemaphores = convert_VkSemaphore_array_win32_to_unwrapped_host(ctx, (const VkSemaphore *)UlongToPtr(in->pSignalSemaphores), in->signalSemaphoreCount);
 
     for (in_header = UlongToPtr(in->pNext); in_header; in_header = UlongToPtr(in_header->pNext))
     {
@@ -40775,7 +40695,7 @@ static void convert_VkBindSparseInfo_win32_to_host(struct conversion_context *ct
     }
 }
 
-static const VkBindSparseInfo *convert_VkBindSparseInfo_array_win32_to_host(struct conversion_context *ctx, const VkBindSparseInfo32 *in, uint32_t count)
+static const VkBindSparseInfo *convert_VkBindSparseInfo_array_win32_to_unwrapped_host(struct conversion_context *ctx, const VkBindSparseInfo32 *in, uint32_t count)
 {
     VkBindSparseInfo *out;
     unsigned int i;
@@ -40785,7 +40705,7 @@ static const VkBindSparseInfo *convert_VkBindSparseInfo_array_win32_to_host(stru
     out = conversion_context_alloc(ctx, count * sizeof(*out));
     for (i = 0; i < count; i++)
     {
-        convert_VkBindSparseInfo_win32_to_host(ctx, &in[i], &out[i]);
+        convert_VkBindSparseInfo_win32_to_unwrapped_host(ctx, &in[i], &out[i]);
     }
 
     return out;
@@ -40802,24 +40722,6 @@ static void convert_VkOutOfBandQueueTypeInfoNV_win32_to_host(const VkOutOfBandQu
         FIXME("Unexpected pNext\n");
 }
 
-#ifdef _WIN64
-static const VkSemaphore *convert_VkSemaphore_array_win64_to_unwrapped_host(struct conversion_context *ctx, const VkSemaphore *in, uint32_t count)
-{
-    VkSemaphore *out;
-    unsigned int i;
-
-    if (!in || !count) return NULL;
-
-    out = conversion_context_alloc(ctx, count * sizeof(*out));
-    for (i = 0; i < count; i++)
-    {
-        out[i] = in[i];
-    }
-
-    return out;
-}
-#endif /* _WIN64 */
-
 #ifdef _WIN64
 static const VkSwapchainKHR *convert_VkSwapchainKHR_array_win64_to_unwrapped_host(struct conversion_context *ctx, const VkSwapchainKHR *in, uint32_t count)
 {
@@ -40989,22 +40891,6 @@ static void convert_VkPresentInfoKHR_win64_to_unwrapped_host(struct conversion_c
 }
 #endif /* _WIN64 */
 
-static const VkSemaphore *convert_VkSemaphore_array_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSemaphore *in, uint32_t count)
-{
-    VkSemaphore *out;
-    unsigned int i;
-
-    if (!in || !count) return NULL;
-
-    out = conversion_context_alloc(ctx, count * sizeof(*out));
-    for (i = 0; i < count; i++)
-    {
-        out[i] = in[i];
-    }
-
-    return out;
-}
-
 static const VkSwapchainKHR *convert_VkSwapchainKHR_array_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSwapchainKHR *in, uint32_t count)
 {
     VkSwapchainKHR *out;
@@ -42257,25 +42143,13 @@ static void convert_VkLatencySleepModeInfoNV_win32_to_host(const VkLatencySleepM
         FIXME("Unexpected pNext\n");
 }
 
-#ifdef _WIN64
-static void convert_VkSemaphoreSignalInfo_win64_to_host(const VkSemaphoreSignalInfo *in, VkSemaphoreSignalInfo *out)
-{
-    if (!in) return;
-
-    out->sType = in->sType;
-    out->pNext = in->pNext;
-    out->semaphore = vulkan_semaphore_from_handle(in->semaphore)->host.semaphore;
-    out->value = in->value;
-}
-#endif /* _WIN64 */
-
-static void convert_VkSemaphoreSignalInfo_win32_to_host(const VkSemaphoreSignalInfo32 *in, VkSemaphoreSignalInfo *out)
+static void convert_VkSemaphoreSignalInfo_win32_to_unwrapped_host(const VkSemaphoreSignalInfo32 *in, VkSemaphoreSignalInfo *out)
 {
     if (!in) return;
 
     out->sType = in->sType;
     out->pNext = NULL;
-    out->semaphore = vulkan_semaphore_from_handle(in->semaphore)->host.semaphore;
+    out->semaphore = in->semaphore;
     out->value = in->value;
     if (in->pNext)
         FIXME("Unexpected pNext\n");
@@ -42614,7 +42488,7 @@ static void convert_VkPresentWait2InfoKHR_win32_to_host(const VkPresentWait2Info
 }
 
 #ifdef _WIN64
-static void convert_VkSemaphoreWaitInfo_win64_to_host(struct conversion_context *ctx, const VkSemaphoreWaitInfo *in, VkSemaphoreWaitInfo *out)
+static void convert_VkSemaphoreWaitInfo_win64_to_unwrapped_host(struct conversion_context *ctx, const VkSemaphoreWaitInfo *in, VkSemaphoreWaitInfo *out)
 {
     if (!in) return;
 
@@ -42622,12 +42496,12 @@ static void convert_VkSemaphoreWaitInfo_win64_to_host(struct conversion_context
     out->pNext = in->pNext;
     out->flags = in->flags;
     out->semaphoreCount = in->semaphoreCount;
-    out->pSemaphores = convert_VkSemaphore_array_win64_to_host(ctx, in->pSemaphores, in->semaphoreCount);
+    out->pSemaphores = convert_VkSemaphore_array_win64_to_unwrapped_host(ctx, in->pSemaphores, in->semaphoreCount);
     out->pValues = in->pValues;
 }
 #endif /* _WIN64 */
 
-static void convert_VkSemaphoreWaitInfo_win32_to_host(struct conversion_context *ctx, const VkSemaphoreWaitInfo32 *in, VkSemaphoreWaitInfo *out)
+static void convert_VkSemaphoreWaitInfo_win32_to_unwrapped_host(struct conversion_context *ctx, const VkSemaphoreWaitInfo32 *in, VkSemaphoreWaitInfo *out)
 {
     if (!in) return;
 
@@ -42635,7 +42509,7 @@ static void convert_VkSemaphoreWaitInfo_win32_to_host(struct conversion_context
     out->pNext = NULL;
     out->flags = in->flags;
     out->semaphoreCount = in->semaphoreCount;
-    out->pSemaphores = convert_VkSemaphore_array_win32_to_host(ctx, (const VkSemaphore *)UlongToPtr(in->pSemaphores), in->semaphoreCount);
+    out->pSemaphores = convert_VkSemaphore_array_win32_to_unwrapped_host(ctx, (const VkSemaphore *)UlongToPtr(in->pSemaphores), in->semaphoreCount);
     out->pValues = UlongToPtr(in->pValues);
     if (in->pNext)
         FIXME("Unexpected pNext\n");
@@ -59536,7 +59410,7 @@ static NTSTATUS thunk64_vkGetSemaphoreCounterValue(void *args)
 
     TRACE("%p, 0x%s, %p\n", params->device, wine_dbgstr_longlong(params->semaphore), params->pValue);
 
-    params->result = vulkan_device_from_handle(params->device)->p_vkGetSemaphoreCounterValue(vulkan_device_from_handle(params->device)->host.device, vulkan_semaphore_from_handle(params->semaphore)->host.semaphore, params->pValue);
+    params->result = vk_funcs->p_vkGetSemaphoreCounterValue(params->device, params->semaphore, params->pValue);
     return STATUS_SUCCESS;
 }
 #endif /* _WIN64 */
@@ -59553,7 +59427,7 @@ static NTSTATUS thunk32_vkGetSemaphoreCounterValue(void *args)
 
     TRACE("%#x, 0x%s, %#x\n", params->device, wine_dbgstr_longlong(params->semaphore), params->pValue);
 
-    params->result = vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->p_vkGetSemaphoreCounterValue(vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->host.device, vulkan_semaphore_from_handle(params->semaphore)->host.semaphore, (uint64_t *)UlongToPtr(params->pValue));
+    params->result = vk_funcs->p_vkGetSemaphoreCounterValue((VkDevice)UlongToPtr(params->device), params->semaphore, (uint64_t *)UlongToPtr(params->pValue));
     return STATUS_SUCCESS;
 }
 
@@ -59564,7 +59438,7 @@ static NTSTATUS thunk64_vkGetSemaphoreCounterValueKHR(void *args)
 
     TRACE("%p, 0x%s, %p\n", params->device, wine_dbgstr_longlong(params->semaphore), params->pValue);
 
-    params->result = vulkan_device_from_handle(params->device)->p_vkGetSemaphoreCounterValueKHR(vulkan_device_from_handle(params->device)->host.device, vulkan_semaphore_from_handle(params->semaphore)->host.semaphore, params->pValue);
+    params->result = vk_funcs->p_vkGetSemaphoreCounterValueKHR(params->device, params->semaphore, params->pValue);
     return STATUS_SUCCESS;
 }
 #endif /* _WIN64 */
@@ -59581,7 +59455,7 @@ static NTSTATUS thunk32_vkGetSemaphoreCounterValueKHR(void *args)
 
     TRACE("%#x, 0x%s, %#x\n", params->device, wine_dbgstr_longlong(params->semaphore), params->pValue);
 
-    params->result = vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->p_vkGetSemaphoreCounterValueKHR(vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->host.device, vulkan_semaphore_from_handle(params->semaphore)->host.semaphore, (uint64_t *)UlongToPtr(params->pValue));
+    params->result = vk_funcs->p_vkGetSemaphoreCounterValueKHR((VkDevice)UlongToPtr(params->device), params->semaphore, (uint64_t *)UlongToPtr(params->pValue));
     return STATUS_SUCCESS;
 }
 
@@ -60301,8 +60175,8 @@ static NTSTATUS thunk64_vkQueueBindSparse(void *args)
     TRACE("%p, %u, %p, 0x%s\n", params->queue, params->bindInfoCount, params->pBindInfo, wine_dbgstr_longlong(params->fence));
 
     init_conversion_context(ctx);
-    pBindInfo_host = convert_VkBindSparseInfo_array_win64_to_host(ctx, params->pBindInfo, params->bindInfoCount);
-    params->result = vulkan_queue_from_handle(params->queue)->device->p_vkQueueBindSparse(vulkan_queue_from_handle(params->queue)->host.queue, params->bindInfoCount, pBindInfo_host, params->fence ? vulkan_fence_from_handle(params->fence)->host.fence : 0);
+    pBindInfo_host = convert_VkBindSparseInfo_array_win64_to_unwrapped_host(ctx, params->pBindInfo, params->bindInfoCount);
+    params->result = vk_funcs->p_vkQueueBindSparse(params->queue, params->bindInfoCount, pBindInfo_host, params->fence);
     free_conversion_context(ctx);
     return STATUS_SUCCESS;
 }
@@ -60325,8 +60199,8 @@ static NTSTATUS thunk32_vkQueueBindSparse(void *args)
     TRACE("%#x, %u, %#x, 0x%s\n", params->queue, params->bindInfoCount, params->pBindInfo, wine_dbgstr_longlong(params->fence));
 
     init_conversion_context(ctx);
-    pBindInfo_host = convert_VkBindSparseInfo_array_win32_to_host(ctx, (const VkBindSparseInfo32 *)UlongToPtr(params->pBindInfo), params->bindInfoCount);
-    params->result = vulkan_queue_from_handle((VkQueue)UlongToPtr(params->queue))->device->p_vkQueueBindSparse(vulkan_queue_from_handle((VkQueue)UlongToPtr(params->queue))->host.queue, params->bindInfoCount, pBindInfo_host, params->fence ? vulkan_fence_from_handle(params->fence)->host.fence : 0);
+    pBindInfo_host = convert_VkBindSparseInfo_array_win32_to_unwrapped_host(ctx, (const VkBindSparseInfo32 *)UlongToPtr(params->pBindInfo), params->bindInfoCount);
+    params->result = vk_funcs->p_vkQueueBindSparse((VkQueue)UlongToPtr(params->queue), params->bindInfoCount, pBindInfo_host, params->fence);
     free_conversion_context(ctx);
     return STATUS_SUCCESS;
 }
@@ -61258,12 +61132,10 @@ static NTSTATUS thunk32_vkSetPrivateDataEXT(void *args)
 static NTSTATUS thunk64_vkSignalSemaphore(void *args)
 {
     struct vkSignalSemaphore_params *params = args;
-    VkSemaphoreSignalInfo pSignalInfo_host;
 
     TRACE("%p, %p\n", params->device, params->pSignalInfo);
 
-    convert_VkSemaphoreSignalInfo_win64_to_host(params->pSignalInfo, &pSignalInfo_host);
-    params->result = vulkan_device_from_handle(params->device)->p_vkSignalSemaphore(vulkan_device_from_handle(params->device)->host.device, &pSignalInfo_host);
+    params->result = vk_funcs->p_vkSignalSemaphore(params->device, params->pSignalInfo);
     return STATUS_SUCCESS;
 }
 #endif /* _WIN64 */
@@ -61280,8 +61152,8 @@ static NTSTATUS thunk32_vkSignalSemaphore(void *args)
 
     TRACE("%#x, %#x\n", params->device, params->pSignalInfo);
 
-    convert_VkSemaphoreSignalInfo_win32_to_host((const VkSemaphoreSignalInfo32 *)UlongToPtr(params->pSignalInfo), &pSignalInfo_host);
-    params->result = vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->p_vkSignalSemaphore(vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->host.device, &pSignalInfo_host);
+    convert_VkSemaphoreSignalInfo_win32_to_unwrapped_host((const VkSemaphoreSignalInfo32 *)UlongToPtr(params->pSignalInfo), &pSignalInfo_host);
+    params->result = vk_funcs->p_vkSignalSemaphore((VkDevice)UlongToPtr(params->device), &pSignalInfo_host);
     return STATUS_SUCCESS;
 }
 
@@ -61289,12 +61161,10 @@ static NTSTATUS thunk32_vkSignalSemaphore(void *args)
 static NTSTATUS thunk64_vkSignalSemaphoreKHR(void *args)
 {
     struct vkSignalSemaphoreKHR_params *params = args;
-    VkSemaphoreSignalInfo pSignalInfo_host;
 
     TRACE("%p, %p\n", params->device, params->pSignalInfo);
 
-    convert_VkSemaphoreSignalInfo_win64_to_host(params->pSignalInfo, &pSignalInfo_host);
-    params->result = vulkan_device_from_handle(params->device)->p_vkSignalSemaphoreKHR(vulkan_device_from_handle(params->device)->host.device, &pSignalInfo_host);
+    params->result = vk_funcs->p_vkSignalSemaphoreKHR(params->device, params->pSignalInfo);
     return STATUS_SUCCESS;
 }
 #endif /* _WIN64 */
@@ -61311,8 +61181,8 @@ static NTSTATUS thunk32_vkSignalSemaphoreKHR(void *args)
 
     TRACE("%#x, %#x\n", params->device, params->pSignalInfo);
 
-    convert_VkSemaphoreSignalInfo_win32_to_host((const VkSemaphoreSignalInfo32 *)UlongToPtr(params->pSignalInfo), &pSignalInfo_host);
-    params->result = vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->p_vkSignalSemaphoreKHR(vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->host.device, &pSignalInfo_host);
+    convert_VkSemaphoreSignalInfo_win32_to_unwrapped_host((const VkSemaphoreSignalInfo32 *)UlongToPtr(params->pSignalInfo), &pSignalInfo_host);
+    params->result = vk_funcs->p_vkSignalSemaphoreKHR((VkDevice)UlongToPtr(params->device), &pSignalInfo_host);
     return STATUS_SUCCESS;
 }
 
@@ -61884,8 +61754,8 @@ static NTSTATUS thunk64_vkWaitSemaphores(void *args)
     TRACE("%p, %p, 0x%s\n", params->device, params->pWaitInfo, wine_dbgstr_longlong(params->timeout));
 
     init_conversion_context(ctx);
-    convert_VkSemaphoreWaitInfo_win64_to_host(ctx, params->pWaitInfo, &pWaitInfo_host);
-    params->result = vulkan_device_from_handle(params->device)->p_vkWaitSemaphores(vulkan_device_from_handle(params->device)->host.device, &pWaitInfo_host, params->timeout);
+    convert_VkSemaphoreWaitInfo_win64_to_unwrapped_host(ctx, params->pWaitInfo, &pWaitInfo_host);
+    params->result = vk_funcs->p_vkWaitSemaphores(params->device, &pWaitInfo_host, params->timeout);
     free_conversion_context(ctx);
     return STATUS_SUCCESS;
 }
@@ -61907,8 +61777,8 @@ static NTSTATUS thunk32_vkWaitSemaphores(void *args)
     TRACE("%#x, %#x, 0x%s\n", params->device, params->pWaitInfo, wine_dbgstr_longlong(params->timeout));
 
     init_conversion_context(ctx);
-    convert_VkSemaphoreWaitInfo_win32_to_host(ctx, (const VkSemaphoreWaitInfo32 *)UlongToPtr(params->pWaitInfo), &pWaitInfo_host);
-    params->result = vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->p_vkWaitSemaphores(vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->host.device, &pWaitInfo_host, params->timeout);
+    convert_VkSemaphoreWaitInfo_win32_to_unwrapped_host(ctx, (const VkSemaphoreWaitInfo32 *)UlongToPtr(params->pWaitInfo), &pWaitInfo_host);
+    params->result = vk_funcs->p_vkWaitSemaphores((VkDevice)UlongToPtr(params->device), &pWaitInfo_host, params->timeout);
     free_conversion_context(ctx);
     return STATUS_SUCCESS;
 }
@@ -61924,8 +61794,8 @@ static NTSTATUS thunk64_vkWaitSemaphoresKHR(void *args)
     TRACE("%p, %p, 0x%s\n", params->device, params->pWaitInfo, wine_dbgstr_longlong(params->timeout));
 
     init_conversion_context(ctx);
-    convert_VkSemaphoreWaitInfo_win64_to_host(ctx, params->pWaitInfo, &pWaitInfo_host);
-    params->result = vulkan_device_from_handle(params->device)->p_vkWaitSemaphoresKHR(vulkan_device_from_handle(params->device)->host.device, &pWaitInfo_host, params->timeout);
+    convert_VkSemaphoreWaitInfo_win64_to_unwrapped_host(ctx, params->pWaitInfo, &pWaitInfo_host);
+    params->result = vk_funcs->p_vkWaitSemaphoresKHR(params->device, &pWaitInfo_host, params->timeout);
     free_conversion_context(ctx);
     return STATUS_SUCCESS;
 }
@@ -61947,8 +61817,8 @@ static NTSTATUS thunk32_vkWaitSemaphoresKHR(void *args)
     TRACE("%#x, %#x, 0x%s\n", params->device, params->pWaitInfo, wine_dbgstr_longlong(params->timeout));
 
     init_conversion_context(ctx);
-    convert_VkSemaphoreWaitInfo_win32_to_host(ctx, (const VkSemaphoreWaitInfo32 *)UlongToPtr(params->pWaitInfo), &pWaitInfo_host);
-    params->result = vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->p_vkWaitSemaphoresKHR(vulkan_device_from_handle((VkDevice)UlongToPtr(params->device))->host.device, &pWaitInfo_host, params->timeout);
+    convert_VkSemaphoreWaitInfo_win32_to_unwrapped_host(ctx, (const VkSemaphoreWaitInfo32 *)UlongToPtr(params->pWaitInfo), &pWaitInfo_host);
+    params->result = vk_funcs->p_vkWaitSemaphoresKHR((VkDevice)UlongToPtr(params->device), &pWaitInfo_host, params->timeout);
     free_conversion_context(ctx);
     return STATUS_SUCCESS;
 }
diff --git a/include/wine/vulkan_driver.h b/include/wine/vulkan_driver.h
index d2e2db3e360..6e37a462ef2 100644
--- a/include/wine/vulkan_driver.h
+++ b/include/wine/vulkan_driver.h
@@ -340,17 +340,24 @@ struct vulkan_funcs
     PFN_vkGetPhysicalDeviceSurfaceFormats2KHR p_vkGetPhysicalDeviceSurfaceFormats2KHR;
     PFN_vkGetPhysicalDeviceSurfaceFormatsKHR p_vkGetPhysicalDeviceSurfaceFormatsKHR;
     PFN_vkGetPhysicalDeviceWin32PresentationSupportKHR p_vkGetPhysicalDeviceWin32PresentationSupportKHR;
+    PFN_vkGetSemaphoreCounterValue p_vkGetSemaphoreCounterValue;
+    PFN_vkGetSemaphoreCounterValueKHR p_vkGetSemaphoreCounterValueKHR;
     PFN_vkGetSemaphoreWin32HandleKHR p_vkGetSemaphoreWin32HandleKHR;
     PFN_vkImportFenceWin32HandleKHR p_vkImportFenceWin32HandleKHR;
     PFN_vkImportSemaphoreWin32HandleKHR p_vkImportSemaphoreWin32HandleKHR;
     PFN_vkMapMemory p_vkMapMemory;
     PFN_vkMapMemory2KHR p_vkMapMemory2KHR;
+    PFN_vkQueueBindSparse p_vkQueueBindSparse;
     PFN_vkQueuePresentKHR p_vkQueuePresentKHR;
     PFN_vkQueueSubmit p_vkQueueSubmit;
     PFN_vkQueueSubmit2 p_vkQueueSubmit2;
     PFN_vkQueueSubmit2KHR p_vkQueueSubmit2KHR;
+    PFN_vkSignalSemaphore p_vkSignalSemaphore;
+    PFN_vkSignalSemaphoreKHR p_vkSignalSemaphoreKHR;
     PFN_vkUnmapMemory p_vkUnmapMemory;
     PFN_vkUnmapMemory2KHR p_vkUnmapMemory2KHR;
+    PFN_vkWaitSemaphores p_vkWaitSemaphores;
+    PFN_vkWaitSemaphoresKHR p_vkWaitSemaphoresKHR;
 };
 
 /* interface between win32u and the user drivers */
-- 
GitLab


https://gitlab.winehq.org/wine/wine/-/merge_requests/9510

[PATCH v2 2/4] win32u: Spawn a device thread for timeline semaphores.

[Rémi Bernon]

From: Rémi Bernon <rbernon(a)codeweavers.com>

---
 dlls/win32u/main.c           |  21 +++++-
 dlls/win32u/ntuser_private.h |   3 +
 dlls/win32u/syscall.c        |  13 +++-
 dlls/win32u/sysparams.c      |   3 +
 dlls/win32u/unixlib.h        |  38 +++++++++++
 dlls/win32u/vulkan.c         | 125 +++++++++++++++++++++++++++++++++--
 include/ntuser.h             |   1 +
 include/wine/vulkan_driver.h |   4 ++
 8 files changed, 199 insertions(+), 9 deletions(-)
 create mode 100644 dlls/win32u/unixlib.h

diff --git a/dlls/win32u/main.c b/dlls/win32u/main.c
index 193c2c048bf..c5349266028 100644
--- a/dlls/win32u/main.c
+++ b/dlls/win32u/main.c
@@ -31,6 +31,8 @@
 #include "wine/asm.h"
 #include "win32syscalls.h"
 
+#include "unixlib.h"
+
 void *__wine_syscall_dispatcher = NULL;
 
 /*******************************************************************
@@ -2554,8 +2556,25 @@ ALL_SYSCALLS
 
 void *dummy = NtQueryVirtualMemory;  /* forced import to avoid link error with winecrt0 */
 
+static void set_thread_name( const WCHAR *name )
+{
+    THREAD_NAME_INFORMATION info;
+    RtlInitUnicodeString( &info.ThreadName, name );
+    NtSetInformationThread( GetCurrentThread(), ThreadNameInformation, &info, sizeof(info) );
+}
+
+static NTSTATUS WINAPI timeline_thread( void *arg )
+{
+    set_thread_name( L"wine_timeline_thread" );
+    RtlExitUserThread( NtUserCallNoParam( NtUserCallNoParam_TimelineThread ) );
+}
+
 BOOL WINAPI DllMain( HINSTANCE inst, DWORD reason, void *reserved )
 {
+    struct process_attach_params params =
+    {
+        .timeline_thread_proc = (UINT_PTR)timeline_thread,
+    };
     HMODULE ntdll;
     void **dispatcher_ptr;
     const UNICODE_STRING ntdll_name = RTL_CONSTANT_STRING( L"ntdll.dll" );
@@ -2568,7 +2587,7 @@ BOOL WINAPI DllMain( HINSTANCE inst, DWORD reason, void *reserved )
         LdrGetDllHandle( NULL, 0, &ntdll_name, &ntdll );
         dispatcher_ptr = RtlFindExportedRoutineByName( ntdll, "__wine_syscall_dispatcher" );
         __wine_syscall_dispatcher = *dispatcher_ptr;
-        if (!__wine_init_unix_call()) WINE_UNIX_CALL( 0, NULL );
+        if (!__wine_init_unix_call()) WINE_UNIX_CALL( unix_process_attach, ¶ms );
         break;
     }
     return TRUE;
diff --git a/dlls/win32u/ntuser_private.h b/dlls/win32u/ntuser_private.h
index c38b6d0e029..2265f6f873e 100644
--- a/dlls/win32u/ntuser_private.h
+++ b/dlls/win32u/ntuser_private.h
@@ -222,6 +222,9 @@ extern LRESULT system_tray_call( HWND hwnd, UINT msg, WPARAM wparam, LPARAM lpar
 extern BOOL set_dc_pixel_format_internal( HDC hdc, int format, struct opengl_drawable **drawable );
 
 /* vulkan.c */
+extern PRTL_THREAD_START_ROUTINE timeline_thread_proc;
+extern NTSTATUS timeline_thread(void);
+
 extern struct vulkan_instance *vulkan_instance_create( const struct vulkan_instance_extensions *extensions );
 
 /* window.c */
diff --git a/dlls/win32u/syscall.c b/dlls/win32u/syscall.c
index 59913497915..352c496660e 100644
--- a/dlls/win32u/syscall.c
+++ b/dlls/win32u/syscall.c
@@ -34,6 +34,8 @@
 #include "wine/unixlib.h"
 #include "win32syscalls.h"
 
+#include "unixlib.h"
+
 ULONG_PTR zero_bits = 0;
 
 static void stub_syscall( const char *name )
@@ -89,8 +91,14 @@ static const char *usercall_names[NtUserCallCount] =
 #undef USER32_CALLBACK_ENTRY
 };
 
-static NTSTATUS init( void *args )
+static PRTL_THREAD_START_ROUTINE get_thread_start_routine( UINT64 value )
+{
+    return (PRTL_THREAD_START_ROUTINE)(UINT_PTR)value;
+}
+
+static NTSTATUS process_attach( void *args )
 {
+    struct process_attach_params *params = args;
 #ifdef _WIN64
     if (NtCurrentTeb()->WowTebOffset)
     {
@@ -102,10 +110,11 @@ static NTSTATUS init( void *args )
 #endif
     KeAddSystemServiceTable( syscalls, NULL, ARRAY_SIZE(syscalls), arguments, 1 );
     ntdll_add_syscall_debug_info( 1, syscall_names, usercall_names );
+    timeline_thread_proc = get_thread_start_routine( params->timeline_thread_proc );
     return STATUS_SUCCESS;
 }
 
 const unixlib_entry_t __wine_unix_call_funcs[] =
 {
-    init,
+    [unix_process_attach] = process_attach,
 };
diff --git a/dlls/win32u/sysparams.c b/dlls/win32u/sysparams.c
index 536a417bd87..8f7f71d6391 100644
--- a/dlls/win32u/sysparams.c
+++ b/dlls/win32u/sysparams.c
@@ -7444,6 +7444,9 @@ ULONG_PTR WINAPI NtUserCallNoParam( ULONG code )
         display_mode_changed( FALSE );
         return TRUE;
 
+    case NtUserCallNoParam_TimelineThread:
+        return timeline_thread();
+
     /* temporary exports */
     case NtUserExitingThread:
         exiting_thread_id = GetCurrentThreadId();
diff --git a/dlls/win32u/unixlib.h b/dlls/win32u/unixlib.h
new file mode 100644
index 00000000000..e1f063abea8
--- /dev/null
+++ b/dlls/win32u/unixlib.h
@@ -0,0 +1,38 @@
+/*
+ * Copyright 2021 Rémi Bernon for CodeWeavers
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA
+ */
+
+#ifndef __WIN32U_UNIXLIB_H
+#define __WIN32U_UNIXLIB_H
+
+#include <stdarg.h>
+
+#include <windef.h>
+#include <winbase.h>
+#include <winternl.h>
+
+struct process_attach_params
+{
+    UINT64 timeline_thread_proc; /* PE-side function pointer */
+};
+
+enum unix_funcs
+{
+    unix_process_attach,
+};
+
+#endif /* __WIN32U_UNIXLIB_H */
diff --git a/dlls/win32u/vulkan.c b/dlls/win32u/vulkan.c
index fd34ce8b968..3bcf5d62c35 100644
--- a/dlls/win32u/vulkan.c
+++ b/dlls/win32u/vulkan.c
@@ -332,6 +332,103 @@ static HANDLE open_shared_resource_from_name( const WCHAR *name )
     return open_name.hNtHandle;
 }
 
+static VkResult timeline_semaphore_create( struct vulkan_device *device, VkSemaphore *semaphore )
+{
+    VkSemaphoreTypeCreateInfo type_info = {.sType = VK_STRUCTURE_TYPE_SEMAPHORE_TYPE_CREATE_INFO};
+    VkSemaphoreCreateInfo create_info = {.sType = VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO, .pNext = &type_info};
+    type_info.semaphoreType = VK_SEMAPHORE_TYPE_TIMELINE;
+    return device->p_vkCreateSemaphore( device->host.device, &create_info, NULL, semaphore );
+}
+
+static void timeline_semaphore_signal( struct vulkan_device *device, VkSemaphore semaphore, uint64_t value )
+{
+    PFN_vkSignalSemaphore p_vkSignalSemaphore = device->p_vkSignalSemaphore ? device->p_vkSignalSemaphore : device->p_vkSignalSemaphoreKHR;
+    VkSemaphoreSignalInfo info = {.sType = VK_STRUCTURE_TYPE_SEMAPHORE_SIGNAL_INFO, .semaphore = semaphore, .value = value};
+    VkResult res = p_vkSignalSemaphore( device->host.device, &info );
+    if (res) ERR( "Failed to signal timeline semaphore %#jx value %#jx, res %d\n", (uintmax_t)semaphore, (uintmax_t)value, res );
+}
+
+static VkResult timeline_semaphore_wait_any( struct vulkan_device *device, uint32_t count, VkSemaphore *semaphores, uint64_t *values, uint64_t timeout )
+{
+    PFN_vkWaitSemaphores p_vkWaitSemaphores = device->p_vkWaitSemaphores ? device->p_vkWaitSemaphores : device->p_vkWaitSemaphoresKHR;
+    VkSemaphoreWaitInfo wait_info = {.sType = VK_STRUCTURE_TYPE_SEMAPHORE_WAIT_INFO, .flags = VK_SEMAPHORE_WAIT_ANY_BIT, .semaphoreCount = count,
+                                     .pSemaphores = semaphores, .pValues = values};
+    VkResult res = p_vkWaitSemaphores( device->host.device, &wait_info, timeout );
+    if (res && res != VK_TIMEOUT) ERR( "Failed to wait on %u timeline semaphores, res %d\n", count, res );
+    return res;
+}
+
+static pthread_mutex_t timeline_lock = PTHREAD_MUTEX_INITIALIZER;
+static struct vulkan_device *timeline_thread_param;
+PRTL_THREAD_START_ROUTINE timeline_thread_proc;
+
+NTSTATUS timeline_thread(void)
+{
+    struct vulkan_device *device = timeline_thread_param;
+    struct mempool pool = {0};
+    VkResult res;
+
+    TRACE( "Timeline thread running for device %p\n", device );
+
+    TRACE( "Signaling timeline %p/%#jx\n", device, (uintmax_t)device->timeline_value );
+    timeline_semaphore_signal( device, device->timeline_semaphore, device->timeline_value++ );
+
+    do
+    {
+        VkSemaphore *wait_semaphores, *wait_semaphore;
+        uint64_t *wait_values, *wait_value;
+        uint32_t count = 0;
+
+        pthread_mutex_lock( &timeline_lock );
+
+        TRACE( "Waiting on semaphores:\n" );
+        wait_semaphores = wait_semaphore = mem_alloc( &pool, (count * 2 + 1) * sizeof(*wait_semaphores) );
+        wait_values = wait_value = mem_alloc( &pool, (count * 2 + 1) * sizeof(*wait_values) );
+
+        TRACE( "  - timeline %p/%#jx\n", device, (uintmax_t)device->timeline_value );
+        *wait_semaphore++ = device->timeline_semaphore;
+        *wait_value++ = device->timeline_value;
+        count = wait_semaphore - wait_semaphores;
+
+        pthread_mutex_unlock( &timeline_lock );
+
+        res = timeline_semaphore_wait_any( device, count, wait_semaphores, wait_values, -1 );
+        mem_free( &pool );
+    } while (!res && InterlockedCompareExchangePointer( &device->timeline_thread, NULL, NULL ));
+
+    if (res) ERR( "Timeline thread for device %p exiting with res %d\n", device, res );
+    return 0;
+}
+
+static void timeline_thread_notify( struct vulkan_device *device )
+{
+    NTSTATUS status;
+
+    pthread_mutex_lock( &timeline_lock );
+
+    if (!device->timeline_semaphore) timeline_semaphore_create( device, &device->timeline_semaphore );
+    if (!device->timeline_semaphore) ERR( "Failed to create device timeline semaphore\n" );
+
+    TRACE( "Signaling timeline %p/%#jx\n", device, (uintmax_t)device->timeline_value );
+    timeline_semaphore_signal( device, device->timeline_semaphore, device->timeline_value++ );
+
+    if (!device->timeline_thread)
+    {
+        timeline_thread_param = device;
+
+        status = NtCreateThreadEx( &device->timeline_thread, THREAD_ALL_ACCESS, NULL, NtCurrentProcess(),
+                                   timeline_thread_proc, NULL, 0, 0, 0, 0, NULL );
+        if (status) ERR( "Failed to start timeline thread, status %#x\n", status );
+
+        TRACE( "Waiting on timeline %p/%#jx\n", device, (uintmax_t)device->timeline_value );
+        timeline_semaphore_wait_any( device, 1, &device->timeline_semaphore, &device->timeline_value, -1 );
+
+        timeline_thread_param = NULL;
+    }
+
+    pthread_mutex_unlock( &timeline_lock );
+}
+
 static const void *find_next_struct( const VkBaseInStructure *header, VkStructureType type )
 {
     for (; header; header = header->pNext) if (header->sType == type) return header;
@@ -781,6 +878,7 @@ static VkResult win32u_vkCreateDevice( VkPhysicalDevice client_physical_device,
 
     if (!(device = calloc( 1, offsetof(struct vulkan_device, queues[queue_count]) ))) return VK_ERROR_OUT_OF_HOST_MEMORY;
     device->extensions = client_device->extensions;
+    device->timeline_value = 1;
 
     if ((res = convert_device_create_info( physical_device, create_info, &pool, device ))) goto failed;
     if ((res = instance->p_vkCreateDevice( physical_device->host.physical_device, create_info, NULL /* allocator */, &host_device ))) goto failed;
@@ -816,9 +914,16 @@ static void win32u_vkDestroyDevice( VkDevice client_device, const VkAllocationCa
     struct vulkan_device *device = vulkan_device_from_handle( client_device );
     struct vulkan_instance *instance = device->physical_device->instance;
     unsigned int i;
+    HANDLE thread;
 
     if (!device) return;
 
+    if ((thread = InterlockedExchangePointer( &device->timeline_thread, NULL )))
+    {
+        timeline_semaphore_signal( device, device->timeline_semaphore, device->timeline_value++ );
+        NtWaitForSingleObject( thread, FALSE, NULL );
+    }
+
     device->p_vkDestroyDevice( device->host.device, NULL /* pAllocator */ );
     for (i = 0; i < device->queue_count; i++)
         instance->p_remove_object( instance, &device->queues[i].obj );
@@ -946,12 +1051,9 @@ static VkResult win32u_vkAllocateMemory( VkDevice client_device, const VkMemoryA
 
         if (device->client.device->extensions.has_VK_KHR_win32_keyed_mutex && memory->sync)
         {
-            VkSemaphoreTypeCreateInfo semaphore_type = {.sType = VK_STRUCTURE_TYPE_SEMAPHORE_TYPE_CREATE_INFO};
-            VkSemaphoreCreateInfo semaphore_create = {.sType = VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO, .pNext = &semaphore_type};
             VkImportSemaphoreFdInfoKHR fd_info = {.sType = VK_STRUCTURE_TYPE_IMPORT_SEMAPHORE_FD_INFO_KHR};
 
-            semaphore_type.semaphoreType = VK_SEMAPHORE_TYPE_TIMELINE;
-            if ((res = device->p_vkCreateSemaphore( device->host.device, &semaphore_create, NULL, &memory->semaphore ))) goto failed;
+            if ((res = timeline_semaphore_create( device, &memory->semaphore ))) goto failed;
 
             fd_info.handleType = get_host_external_semaphore_type();
             fd_info.semaphore = memory->semaphore;
@@ -2096,6 +2198,7 @@ static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count,
     struct vulkan_fence *fence = client_fence ? vulkan_fence_from_handle( client_fence ) : NULL;
     struct vulkan_queue *queue = vulkan_queue_from_handle( client_queue );
     struct vulkan_device *device = queue->device;
+    VkResult res;
 
     TRACE( "queue %p, count %u, binds %p, fence %p\n", queue, count, binds, fence );
 
@@ -2161,7 +2264,10 @@ static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count,
         }
     }
 
-    return device->p_vkQueueBindSparse( queue->host.queue, count, binds, fence ? fence->host.fence : 0 );
+    res = device->p_vkQueueBindSparse( queue->host.queue, count, binds, fence ? fence->host.fence : 0 );
+    if (!res) timeline_thread_notify( queue->device );
+
+    return res;
 }
 
 static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, const VkSubmitInfo *submits, VkFence client_fence )
@@ -2312,6 +2418,7 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
     }
 
     res = device->p_vkQueueSubmit( queue->host.queue, count, submits, fence ? fence->host.fence : 0 );
+    if (!res) timeline_thread_notify( queue->device );
 
 failed:
     mem_free( &pool );
@@ -2375,6 +2482,7 @@ static VkResult queue_submit( struct vulkan_queue *queue, uint32_t count, const
     }
 
     res = p_vkQueueSubmit2( queue->host.queue, count, submits, fence ? fence->host.fence : 0 );
+    if (!res) timeline_thread_notify( queue->device );
 
 failed:
     mem_free( &pool );
@@ -2564,12 +2672,16 @@ static VkResult signal_semaphore( struct vulkan_device *device, const VkSemaphor
 {
     struct semaphore *semaphore = semaphore_from_handle( signal_info->semaphore );
     VkSemaphoreSignalInfo info = *signal_info;
+    VkResult res;
 
     if (info.pNext) FIXME( "pNext not implemented\n" );
     info.pNext = NULL;
 
     info.semaphore = semaphore->obj.host.semaphore;
-    return p_vkSignalSemaphore( device->host.device, &info );
+    res = p_vkSignalSemaphore( device->host.device, &info );
+    if (!res) timeline_thread_notify( device );
+
+    return res;
 }
 
 static VkResult win32u_vkSignalSemaphore( VkDevice client_device, const VkSemaphoreSignalInfo *signal_info )
@@ -2609,6 +2721,7 @@ static VkResult wait_semaphores( struct vulkan_device *device, const VkSemaphore
         struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[i] );
         semaphores[i] = semaphore->host.semaphore;
     }
+    timeline_thread_notify( device );
 
     res = p_vkWaitSemaphores( device->host.device, &info, timeout );
 
diff --git a/include/ntuser.h b/include/ntuser.h
index 664bcfe459d..1c740d98f70 100644
--- a/include/ntuser.h
+++ b/include/ntuser.h
@@ -1072,6 +1072,7 @@ enum
     NtUserCallNoParam_GetShellWindow,
     NtUserCallNoParam_GetTaskmanWindow,
     NtUserCallNoParam_DisplayModeChanged,
+    NtUserCallNoParam_TimelineThread,
     /* temporary exports */
     NtUserExitingThread,
     NtUserThreadDetach,
diff --git a/include/wine/vulkan_driver.h b/include/wine/vulkan_driver.h
index 6e37a462ef2..1dbaed13f1f 100644
--- a/include/wine/vulkan_driver.h
+++ b/include/wine/vulkan_driver.h
@@ -212,6 +212,10 @@ struct vulkan_device
     ALL_VK_DEVICE_FUNCS
 #undef USE_VK_FUNC
 
+    HANDLE      timeline_thread;
+    VkSemaphore timeline_semaphore;
+    uint64_t    timeline_value;
+
     uint64_t queue_count;
     struct vulkan_queue queues[];
 };
-- 
GitLab


https://gitlab.winehq.org/wine/wine/-/merge_requests/9510

[PATCH v2 3/4] win32u: Make a copy of the timeline semaphore value arrays.

[Rémi Bernon]

From: Rémi Bernon <rbernon(a)codeweavers.com>

---
 dlls/win32u/vulkan.c | 106 ++++++++++++++++++++++++++++++-------------
 1 file changed, 74 insertions(+), 32 deletions(-)

diff --git a/dlls/win32u/vulkan.c b/dlls/win32u/vulkan.c
index 3bcf5d62c35..ff9112db0b0 100644
--- a/dlls/win32u/vulkan.c
+++ b/dlls/win32u/vulkan.c
@@ -2198,14 +2198,21 @@ static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count,
     struct vulkan_fence *fence = client_fence ? vulkan_fence_from_handle( client_fence ) : NULL;
     struct vulkan_queue *queue = vulkan_queue_from_handle( client_queue );
     struct vulkan_device *device = queue->device;
-    VkResult res;
+    VkResult res = VK_ERROR_OUT_OF_HOST_MEMORY;
+    VkTimelineSemaphoreSubmitInfo *timelines;
+    struct mempool pool = {0};
 
     TRACE( "queue %p, count %u, binds %p, fence %p\n", queue, count, binds, fence );
 
+    if (!(timelines = mem_alloc( &pool, count * sizeof(*timelines) ))) return VK_ERROR_OUT_OF_HOST_MEMORY;
+    memset( timelines, 0, count * sizeof(*timelines) );
+
     for (uint32_t i = 0; i < count; i++)
     {
         VkBindSparseInfo *bind = (VkBindSparseInfo *)binds + i; /* cast away const, chain has been copied in the thunks */
         VkBaseOutStructure **next, *prev = (VkBaseOutStructure *)bind;
+        VkTimelineSemaphoreSubmitInfo *timeline = timelines + i;
+        uint64_t *wait_values = NULL, *signal_values = NULL;
 
         for (const VkSparseBufferMemoryBindInfo *buffer = bind->pBufferBinds, *end = buffer + bind->bufferBindCount; buffer < end; buffer++)
         {
@@ -2244,11 +2251,22 @@ static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count,
             case VK_STRUCTURE_TYPE_DEVICE_GROUP_BIND_SPARSE_INFO: break;
             case VK_STRUCTURE_TYPE_FRAME_BOUNDARY_EXT: break;
             case VK_STRUCTURE_TYPE_FRAME_BOUNDARY_TENSORS_ARM: break;
-            case VK_STRUCTURE_TYPE_TIMELINE_SEMAPHORE_SUBMIT_INFO: break;
+            case VK_STRUCTURE_TYPE_TIMELINE_SEMAPHORE_SUBMIT_INFO:
+                if (timeline->sType) ERR( "Duplicated timeline semaphore submit info!\n" );
+                *timeline = *(VkTimelineSemaphoreSubmitInfo *)*next;
+                *next = (*next)->pNext; next = &prev; /* remove it from the chain, we'll add it back below */
+                break;
             default: FIXME( "Unhandled sType %u.\n", (*next)->sType ); break;
             }
         }
 
+        if (timeline->sType && timeline->waitSemaphoreValueCount)
+        {
+            if (!(wait_values = mem_alloc( &pool, timeline->waitSemaphoreValueCount * sizeof(*wait_values) ))) goto failed;
+            memcpy( wait_values, timeline->pWaitSemaphoreValues, timeline->waitSemaphoreValueCount * sizeof(*wait_values) );
+            timeline->pWaitSemaphoreValues = wait_values;
+        }
+
         for (uint32_t j = 0; j < bind->waitSemaphoreCount; j++)
         {
             VkSemaphore *semaphores = (VkSemaphore *)bind->pWaitSemaphores; /* cast away const, it has been copied in the thunks */
@@ -2256,17 +2274,33 @@ static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count,
             semaphores[j] = semaphore->host.semaphore;
         }
 
+        if (timeline->sType && timeline->signalSemaphoreValueCount)
+        {
+            if (!(signal_values = mem_alloc( &pool, timeline->signalSemaphoreValueCount * sizeof(*signal_values) ))) goto failed;
+            memcpy( signal_values, timeline->pSignalSemaphoreValues, timeline->signalSemaphoreValueCount * sizeof(*signal_values) );
+            timeline->pSignalSemaphoreValues = signal_values;
+        }
+
         for (uint32_t j = 0; j < bind->signalSemaphoreCount; j++)
         {
             VkSemaphore *semaphores = (VkSemaphore *)bind->pSignalSemaphores; /* cast away const, it has been copied in the thunks */
             struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
             semaphores[j] = semaphore->host.semaphore;
         }
+
+        /* insert the timeline semaphore values in the chain if it was there or has been created */
+        if (timeline->sType)
+        {
+            timeline->pNext = bind->pNext;
+            bind->pNext = timeline;
+        }
     }
 
     res = device->p_vkQueueBindSparse( queue->host.queue, count, binds, fence ? fence->host.fence : 0 );
     if (!res) timeline_thread_notify( queue->device );
 
+failed:
+    mem_free( &pool );
     return res;
 }
 
@@ -2290,33 +2324,12 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
         const VkSemaphoreSubmitInfo *wait_infos = NULL, *signal_infos = NULL;
         VkBaseOutStructure **next, *prev = (VkBaseOutStructure *)submit;
         VkTimelineSemaphoreSubmitInfo *timeline = timelines + i;
+        uint64_t *wait_values = NULL, *signal_values = NULL;
         VkSemaphore *wait_semaphores, *signal_semaphores;
         VkDeviceGroupSubmitInfo *device_group = NULL;
         UINT wait_count = 0, signal_count = 0;
         VkPipelineStageFlags *wait_stages;
         uint32_t *indexes;
-        uint64_t *values;
-
-        for (uint32_t j = 0; j < submit->commandBufferCount; j++)
-        {
-            VkCommandBuffer *command_buffers = (VkCommandBuffer *)submit->pCommandBuffers; /* cast away const, chain has been copied in the thunks */
-            struct vulkan_command_buffer *command_buffer = vulkan_command_buffer_from_handle( command_buffers[j] );
-            command_buffers[j] = command_buffer->host.command_buffer;
-        }
-
-        for (uint32_t j = 0; j < submit->waitSemaphoreCount; j++)
-        {
-            VkSemaphore *semaphores = (VkSemaphore *)submit->pWaitSemaphores; /* cast away const, it has been copied in the thunks */
-            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
-            semaphores[j] = semaphore->host.semaphore;
-        }
-
-        for (uint32_t j = 0; j < submit->signalSemaphoreCount; j++)
-        {
-            VkSemaphore *semaphores = (VkSemaphore *)submit->pSignalSemaphores; /* cast away const, it has been copied in the thunks */
-            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
-            semaphores[j] = semaphore->host.semaphore;
-        }
 
         for (next = &prev->pNext; *next; prev = *next, next = &(*next)->pNext)
         {
@@ -2351,6 +2364,20 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
             }
         }
 
+        if ((timeline->sType && timeline->waitSemaphoreValueCount) || wait_count)
+        {
+            if (!(wait_values = mem_alloc( &pool, (timeline->waitSemaphoreValueCount + wait_count) * sizeof(*wait_values) ))) goto failed;
+            memcpy( wait_values, timeline->pWaitSemaphoreValues, timeline->waitSemaphoreValueCount * sizeof(*wait_values) );
+            timeline->pWaitSemaphoreValues = wait_values;
+        }
+
+        for (uint32_t j = 0; j < submit->waitSemaphoreCount; j++)
+        {
+            VkSemaphore *semaphores = (VkSemaphore *)submit->pWaitSemaphores; /* cast away const, it has been copied in the thunks */
+            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
+            semaphores[j] = semaphore->host.semaphore;
+        }
+
         if (wait_count) /* extra wait semaphores, need to update arrays and counts */
         {
             if (!(wait_semaphores = mem_alloc( &pool, (submit->waitSemaphoreCount + wait_count) * sizeof(*wait_semaphores) ))) goto failed;
@@ -2368,11 +2395,8 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
             }
             submit->waitSemaphoreCount += wait_count;
 
-            if (!(values = mem_alloc( &pool, (timeline->waitSemaphoreValueCount + wait_count) * sizeof(*values) ))) goto failed;
-            memcpy( values, timeline->pWaitSemaphoreValues, timeline->waitSemaphoreValueCount * sizeof(*values) );
-            for (uint32_t j = 0; j < wait_count; j++) values[submit->waitSemaphoreCount + j] = wait_infos[j].value;
+            for (uint32_t j = 0; j < wait_count; j++) wait_values[submit->waitSemaphoreCount + j] = wait_infos[j].value;
             timeline->waitSemaphoreValueCount = submit->waitSemaphoreCount;
-            timeline->pWaitSemaphoreValues = values;
 
             if (device_group)
             {
@@ -2384,6 +2408,27 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
             }
         }
 
+        for (uint32_t j = 0; j < submit->commandBufferCount; j++)
+        {
+            VkCommandBuffer *command_buffers = (VkCommandBuffer *)submit->pCommandBuffers; /* cast away const, chain has been copied in the thunks */
+            struct vulkan_command_buffer *command_buffer = vulkan_command_buffer_from_handle( command_buffers[j] );
+            command_buffers[j] = command_buffer->host.command_buffer;
+        }
+
+        if ((timeline->sType && timeline->signalSemaphoreValueCount) || signal_count)
+        {
+            if (!(signal_values = mem_alloc( &pool, (timeline->signalSemaphoreValueCount + signal_count) * sizeof(*signal_values) ))) goto failed;
+            memcpy( signal_values, timeline->pSignalSemaphoreValues, timeline->signalSemaphoreValueCount * sizeof(*signal_values) );
+            timeline->pSignalSemaphoreValues = signal_values;
+        }
+
+        for (uint32_t j = 0; j < submit->signalSemaphoreCount; j++)
+        {
+            VkSemaphore *semaphores = (VkSemaphore *)submit->pSignalSemaphores; /* cast away const, it has been copied in the thunks */
+            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
+            semaphores[j] = semaphore->host.semaphore;
+        }
+
         if (signal_count) /* extra signal semaphores, need to update arrays and counts */
         {
             if (!(signal_semaphores = mem_alloc( &pool, (submit->signalSemaphoreCount + signal_count) * sizeof(*signal_semaphores) ))) goto failed;
@@ -2392,11 +2437,8 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
             submit->signalSemaphoreCount += signal_count;
             submit->pSignalSemaphores = signal_semaphores;
 
-            if (!(values = mem_alloc( &pool, submit->signalSemaphoreCount * sizeof(*values) ))) goto failed;
-            memcpy( values, timeline->pSignalSemaphoreValues, timeline->signalSemaphoreValueCount * sizeof(*values) );
-            for (uint32_t j = 0; j < signal_count; j++) values[submit->signalSemaphoreCount + j] = signal_infos[j].value;
+            for (uint32_t j = 0; j < signal_count; j++) signal_values[submit->signalSemaphoreCount + j] = signal_infos[j].value;
             timeline->signalSemaphoreValueCount = submit->signalSemaphoreCount;
-            timeline->pSignalSemaphoreValues = values;
 
             if (device_group)
             {
-- 
GitLab


https://gitlab.winehq.org/wine/wine/-/merge_requests/9510

[PATCH v2 4/4] win32u: Implement timeline semaphore rewinding support.

[Rémi Bernon]

From: Rémi Bernon <rbernon(a)codeweavers.com>

---
 dlls/vulkan-1/tests/vulkan.c |   5 -
 dlls/win32u/tests/d3dkmt.c   |   6 +-
 dlls/win32u/vulkan.c         | 322 ++++++++++++++++++++++++++++++++---
 include/wine/vulkan_driver.h |   3 +
 4 files changed, 300 insertions(+), 36 deletions(-)

diff --git a/dlls/vulkan-1/tests/vulkan.c b/dlls/vulkan-1/tests/vulkan.c
index d4d5cc86ab0..98a9f2c6dad 100644
--- a/dlls/vulkan-1/tests/vulkan.c
+++ b/dlls/vulkan-1/tests/vulkan.c
@@ -1573,10 +1573,6 @@ static void test_timeline_semaphore(VkInstance vk_instance, VkPhysicalDevice vk_
     ok(value == 1, "got %#I64x\n", value);
 
 
-    /* following tests are confusing the vulkan driver on Linux */
-    if (winetest_platform_is_wine) goto done;
-
-
     /* large value jumps are allowed */
 
     signal_info.semaphore = vk_semaphore;
@@ -1686,7 +1682,6 @@ static void test_timeline_semaphore(VkInstance vk_instance, VkPhysicalDevice vk_
     ok(value == 3, "got %#I64x\n", value);
 
 
-done:
     vkDestroySemaphore(vk_device, vk_semaphore, NULL);
     vkDestroySemaphore(vk_device, vk_semaphore2, NULL);
     vkDestroyDevice(vk_device, NULL);
diff --git a/dlls/win32u/tests/d3dkmt.c b/dlls/win32u/tests/d3dkmt.c
index b3797f9ac99..c851ee59fc7 100644
--- a/dlls/win32u/tests/d3dkmt.c
+++ b/dlls/win32u/tests/d3dkmt.c
@@ -6084,9 +6084,9 @@ static void test_d3d12_fence_from_vulkan_timeline( ID3D12Device *device, struct
     HRESULT hr;
     DWORD ret;
 
-    p_vkGetSemaphoreCounterValue = (void *)p_vkGetDeviceProcAddr( dev->device, "vkGetSemaphoreCounterValue" );
-    p_vkSignalSemaphore = (void *)p_vkGetDeviceProcAddr( dev->device, "vkSignalSemaphore" );
-    p_vkWaitSemaphores = (void *)p_vkGetDeviceProcAddr( dev->device, "vkWaitSemaphores" );
+    p_vkGetSemaphoreCounterValue = (void *)p_vkGetDeviceProcAddr( dev->device, "vkGetSemaphoreCounterValueKHR" );
+    p_vkSignalSemaphore = (void *)p_vkGetDeviceProcAddr( dev->device, "vkSignalSemaphoreKHR" );
+    p_vkWaitSemaphores = (void *)p_vkGetDeviceProcAddr( dev->device, "vkWaitSemaphoresKHR" );
 
     hr = ID3D12Device_OpenSharedHandle( device, handle, &IID_ID3D12Fence, (void **)&fence );
     if (FAILED(hr))
diff --git a/dlls/win32u/vulkan.c b/dlls/win32u/vulkan.c
index ff9112db0b0..d4a64dd8d95 100644
--- a/dlls/win32u/vulkan.c
+++ b/dlls/win32u/vulkan.c
@@ -169,9 +169,14 @@ static struct swapchain *swapchain_from_handle( VkSwapchainKHR handle )
 struct semaphore
 {
     struct vulkan_semaphore obj;
+    struct vulkan_device *device;
+    VkSemaphoreType type;
+
     D3DKMT_HANDLE local;
     D3DKMT_HANDLE global;
     HANDLE shared;
+
+    uint64_t semaphore_value;
 };
 
 static struct semaphore *semaphore_from_handle( VkSemaphore handle )
@@ -348,6 +353,16 @@ static void timeline_semaphore_signal( struct vulkan_device *device, VkSemaphore
     if (res) ERR( "Failed to signal timeline semaphore %#jx value %#jx, res %d\n", (uintmax_t)semaphore, (uintmax_t)value, res );
 }
 
+static uint64_t timeline_semaphore_value( struct vulkan_device *device, VkSemaphore semaphore )
+{
+    PFN_vkGetSemaphoreCounterValue p_vkGetSemaphoreCounterValue = device->p_vkGetSemaphoreCounterValue ? device->p_vkGetSemaphoreCounterValue
+                                                                                                       : device->p_vkGetSemaphoreCounterValueKHR;
+    uint64_t value = 0;
+    VkResult res = p_vkGetSemaphoreCounterValue( device->host.device, semaphore, &value );
+    if (res) ERR( "Failed to read timeline semaphore %#jx, res %d\n", (uintmax_t)semaphore, res );
+    return value;
+}
+
 static VkResult timeline_semaphore_wait_any( struct vulkan_device *device, uint32_t count, VkSemaphore *semaphores, uint64_t *values, uint64_t timeout )
 {
     PFN_vkWaitSemaphores p_vkWaitSemaphores = device->p_vkWaitSemaphores ? device->p_vkWaitSemaphores : device->p_vkWaitSemaphoresKHR;
@@ -358,13 +373,127 @@ static VkResult timeline_semaphore_wait_any( struct vulkan_device *device, uint3
     return res;
 }
 
+struct timeline_event
+{
+    struct vulkan_semaphore obj;
+    D3DKMT_HANDLE global;
+    struct list entry;
+    uint64_t timeout;
+    uint64_t event_value;
+    uint64_t semaphore_value;
+};
+
 static pthread_mutex_t timeline_lock = PTHREAD_MUTEX_INITIALIZER;
 static struct vulkan_device *timeline_thread_param;
 PRTL_THREAD_START_ROUTINE timeline_thread_proc;
 
+static struct timeline_event *timeline_event_create( struct vulkan_device *device )
+{
+    struct timeline_event *event;
+    VkSemaphore host_semaphore;
+
+    if (!(event = calloc( 1, sizeof(*event) ))) return NULL;
+    if (timeline_semaphore_create( device, &host_semaphore ))
+    {
+        free( event );
+        return NULL;
+    }
+    event->event_value = 1;
+
+    vulkan_object_init( &event->obj.obj, host_semaphore );
+    TRACE( "device %p, event %p\n", device, event );
+    return event;
+}
+
+static void timeline_event_destroy( struct vulkan_device *device, struct timeline_event *event )
+{
+    struct vulkan_instance *instance = device->physical_device->instance;
+
+    TRACE( "device %p, event %p\n", device, event );
+
+    if (event->obj.obj.client_handle) instance->p_remove_object( instance, &event->obj.obj );
+    device->p_vkDestroySemaphore( device->host.device, event->obj.host.semaphore, NULL /* allocator */ );
+    free( event );
+}
+
+static struct timeline_event *timeline_event_acquire( struct vulkan_device *device, VkSemaphore client_semaphore, uint64_t semaphore_value )
+{
+    struct vulkan_instance *instance = device->physical_device->instance;
+    struct timeline_event *event = NULL;
+    struct list *ptr;
+
+    pthread_mutex_lock( &timeline_lock );
+    if (device->timeline_thread && (ptr = list_head( &device->timeline_events )))
+    {
+        event = LIST_ENTRY( ptr, struct timeline_event, entry );
+        if (!event->event_value) event = NULL;
+        else list_remove( &event->entry );
+    }
+    pthread_mutex_unlock( &timeline_lock );
+
+    if (!event && !(event = timeline_event_create( device ))) return NULL;
+    event->semaphore_value = semaphore_value;
+    event->obj.client.semaphore = client_semaphore;
+    instance->p_insert_object( instance, &event->obj.obj );
+
+    return event;
+}
+
+static void timeline_event_release( struct vulkan_device *device, struct timeline_event *event )
+{
+    struct vulkan_instance *instance = device->physical_device->instance;
+
+    instance->p_remove_object( instance, &event->obj.obj );
+    event->obj.client.semaphore = 0;
+    event->semaphore_value = 0;
+
+    if (event->event_value) list_add_head( &device->timeline_events, &event->entry );
+    else list_add_tail( &device->timeline_events, &event->entry );
+}
+
+static void timeline_release_wait( struct vulkan_device *device, struct timeline_event *wait )
+{
+    wait->event_value++;
+    list_remove( &wait->entry );
+    timeline_event_release( device, wait );
+}
+
+static void timeline_satisfy_wait( struct vulkan_device *device, struct timeline_event *wait )
+{
+    struct semaphore *semaphore = semaphore_from_handle( wait->obj.client.semaphore );
+
+    TRACE( "Signaling event %p/%#jx for semaphore %p wait %#jx\n", wait, (uintmax_t)wait->event_value,
+           semaphore, (uintmax_t)wait->semaphore_value );
+    timeline_semaphore_signal( device, wait->obj.host.semaphore, wait->event_value );
+    timeline_release_wait( device, wait );
+}
+
+static void timeline_satisfied( struct vulkan_device *device, struct timeline_event *signal )
+{
+    struct semaphore *semaphore = semaphore_from_handle( signal->obj.client.semaphore );
+    struct timeline_event *wait, *next;
+
+    TRACE( "Satisfied event %p/%#jx for semaphore %p signal %#jx\n", signal, (uintmax_t)signal->event_value,
+           semaphore, (uintmax_t)signal->semaphore_value );
+
+    LIST_FOR_EACH_ENTRY_SAFE( wait, next, &device->timeline_waits, struct timeline_event, entry )
+    {
+        if (wait->obj.client.semaphore != signal->obj.client.semaphore) continue;
+        if (wait->semaphore_value > signal->semaphore_value) continue;
+        timeline_satisfy_wait( device, wait );
+    }
+
+    semaphore->semaphore_value = signal->semaphore_value;
+    signal->event_value++;
+    list_remove( &signal->entry );
+    timeline_event_release( device, signal );
+}
+
 NTSTATUS timeline_thread(void)
 {
+    struct list waits = LIST_INIT( waits ), signals = LIST_INIT( signals );
     struct vulkan_device *device = timeline_thread_param;
+    struct timeline_event *signal, *wait, *next;
     struct mempool pool = {0};
     VkResult res;
 
@@ -376,15 +505,44 @@ NTSTATUS timeline_thread(void)
     do
     {
         VkSemaphore *wait_semaphores, *wait_semaphore;
-        uint64_t *wait_values, *wait_value;
+        uint64_t *wait_values, *wait_value, time;
+        LARGE_INTEGER counter, freq;
         uint32_t count = 0;
 
+        NtQueryPerformanceCounter( &counter, &freq );
+        time = counter.QuadPart * 100;
+
         pthread_mutex_lock( &timeline_lock );
 
+        list_move_tail( &waits, &device->timeline_waits );
+        list_move_tail( &signals, &device->timeline_signals );
+
+        LIST_FOR_EACH_ENTRY_SAFE( signal, next, &signals, struct timeline_event, entry )
+        {
+            uint64_t value = timeline_semaphore_value( device, signal->obj.host.semaphore );
+            if (value < signal->event_value) count++;
+            else timeline_satisfied( device, signal );
+        }
+
+        LIST_FOR_EACH_ENTRY_SAFE( wait, next, &waits, struct timeline_event, entry )
+        {
+            struct semaphore *semaphore = semaphore_from_handle( wait->obj.client.semaphore );
+            uint64_t semaphore_value = semaphore->semaphore_value;
+            if (semaphore_value >= wait->semaphore_value) timeline_satisfy_wait( device, wait );
+            else if (wait->timeout < time) timeline_release_wait( device, wait );
+        }
+
         TRACE( "Waiting on semaphores:\n" );
         wait_semaphores = wait_semaphore = mem_alloc( &pool, (count * 2 + 1) * sizeof(*wait_semaphores) );
         wait_values = wait_value = mem_alloc( &pool, (count * 2 + 1) * sizeof(*wait_values) );
-
+        LIST_FOR_EACH_ENTRY_SAFE( signal, next, &signals, struct timeline_event, entry )
+        {
+            struct semaphore *semaphore = semaphore_from_handle( signal->obj.client.semaphore );
+            TRACE( "  - event %p/%#jx for semaphore %p signal %#jx\n", signal, (uintmax_t)signal->event_value,
+                   semaphore, (uintmax_t)signal->semaphore_value );
+            *wait_semaphore++ = signal->obj.host.semaphore;
+            *wait_value++ = signal->event_value;
+        }
         TRACE( "  - timeline %p/%#jx\n", device, (uintmax_t)device->timeline_value );
         *wait_semaphore++ = device->timeline_semaphore;
         *wait_value++ = device->timeline_value;
@@ -396,16 +554,33 @@ NTSTATUS timeline_thread(void)
         mem_free( &pool );
     } while (!res && InterlockedCompareExchangePointer( &device->timeline_thread, NULL, NULL ));
 
+    pthread_mutex_lock( &timeline_lock );
+
+    list_move_tail( &waits, &device->timeline_waits );
+    list_move_tail( &signals, &device->timeline_signals );
+
+    LIST_FOR_EACH_ENTRY_SAFE( signal, next, &signals, struct timeline_event, entry )
+        timeline_satisfied( device, signal );
+    LIST_FOR_EACH_ENTRY_SAFE( wait, next, &waits, struct timeline_event, entry )
+        timeline_satisfy_wait( device, wait );
+
+    pthread_mutex_unlock( &timeline_lock );
+
     if (res) ERR( "Timeline thread for device %p exiting with res %d\n", device, res );
     return 0;
 }
 
-static void timeline_thread_notify( struct vulkan_device *device )
+static void timeline_thread_notify( struct vulkan_device *device, struct list *waits, struct list *signals, BOOL force )
 {
     NTSTATUS status;
 
+    if (list_empty( waits ) && list_empty( signals ) && !force) return;
+
     pthread_mutex_lock( &timeline_lock );
 
+    list_move_tail( &device->timeline_waits, waits );
+    list_move_tail( &device->timeline_signals, signals );
+
     if (!device->timeline_semaphore) timeline_semaphore_create( device, &device->timeline_semaphore );
     if (!device->timeline_semaphore) ERR( "Failed to create device timeline semaphore\n" );
 
@@ -429,6 +604,18 @@ static void timeline_thread_notify( struct vulkan_device *device )
     pthread_mutex_unlock( &timeline_lock );
 }
 
+static void free_timeline_events( struct vulkan_device *device, struct list *events )
+{
+    struct list *ptr;
+
+    while ((ptr = list_head( events )))
+    {
+        struct timeline_event *event = LIST_ENTRY( ptr, struct timeline_event, entry );
+        list_remove( &event->entry );
+        timeline_event_destroy( device, event );
+    }
+}
+
 static const void *find_next_struct( const VkBaseInStructure *header, VkStructureType type )
 {
     for (; header; header = header->pNext) if (header->sType == type) return header;
@@ -878,6 +1065,9 @@ static VkResult win32u_vkCreateDevice( VkPhysicalDevice client_physical_device,
 
     if (!(device = calloc( 1, offsetof(struct vulkan_device, queues[queue_count]) ))) return VK_ERROR_OUT_OF_HOST_MEMORY;
     device->extensions = client_device->extensions;
+    list_init( &device->timeline_waits );
+    list_init( &device->timeline_signals );
+    list_init( &device->timeline_events );
     device->timeline_value = 1;
 
     if ((res = convert_device_create_info( physical_device, create_info, &pool, device ))) goto failed;
@@ -922,6 +1112,7 @@ static void win32u_vkDestroyDevice( VkDevice client_device, const VkAllocationCa
     {
         timeline_semaphore_signal( device, device->timeline_semaphore, device->timeline_value++ );
         NtWaitForSingleObject( thread, FALSE, NULL );
+        free_timeline_events( device, &device->timeline_events );
     }
 
     device->p_vkDestroyDevice( device->host.device, NULL /* pAllocator */ );
@@ -2193,9 +2384,37 @@ failed:
     return VK_ERROR_UNKNOWN;
 }
 
+static VkSemaphore semaphore_map_signal( struct semaphore *semaphore, uint64_t *value, struct list *signals )
+{
+    struct vulkan_device *device = semaphore->device;
+    struct timeline_event *event;
+
+    if (semaphore->type != VK_SEMAPHORE_TYPE_TIMELINE || semaphore->local) return semaphore->obj.host.semaphore;
+    if (!(event = timeline_event_acquire( device, semaphore->obj.client.semaphore, *value ))) return VK_NULL_HANDLE;
+    TRACE( "Mapped semaphore %p signal %#jx to event %p/%#jx\n", semaphore, (uintmax_t)*value, event, (uintmax_t)event->event_value );
+    list_add_tail( signals, &event->entry );
+    *value = event->event_value;
+    return event->obj.host.semaphore;
+}
+
+static VkSemaphore semaphore_map_wait( struct semaphore *semaphore, uint64_t *value, uint64_t timeout, struct list *waits )
+{
+    struct vulkan_device *device = semaphore->device;
+    struct timeline_event *event;
+
+    if (semaphore->type != VK_SEMAPHORE_TYPE_TIMELINE || semaphore->local) return semaphore->obj.host.semaphore;
+    if (!(event = timeline_event_acquire( device, semaphore->obj.client.semaphore, *value ))) return VK_NULL_HANDLE;
+    TRACE( "Mapped semaphore %p wait %#jx to event %p/%#jx\n", semaphore, (uintmax_t)*value, event, (uintmax_t)event->event_value );
+    list_add_tail( waits, &event->entry );
+    event->timeout = timeout;
+    *value = event->event_value;
+    return event->obj.host.semaphore;
+}
+
 static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count, const VkBindSparseInfo *binds, VkFence client_fence )
 {
     struct vulkan_fence *fence = client_fence ? vulkan_fence_from_handle( client_fence ) : NULL;
+    struct list waits = LIST_INIT( waits ), signals = LIST_INIT( signals );
     struct vulkan_queue *queue = vulkan_queue_from_handle( client_queue );
     struct vulkan_device *device = queue->device;
     VkResult res = VK_ERROR_OUT_OF_HOST_MEMORY;
@@ -2270,8 +2489,8 @@ static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count,
         for (uint32_t j = 0; j < bind->waitSemaphoreCount; j++)
         {
             VkSemaphore *semaphores = (VkSemaphore *)bind->pWaitSemaphores; /* cast away const, it has been copied in the thunks */
-            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
-            semaphores[j] = semaphore->host.semaphore;
+            struct semaphore *semaphore = semaphore_from_handle( semaphores[j] );
+            semaphores[j] = semaphore_map_wait( semaphore, wait_values + j, -1, &waits );
         }
 
         if (timeline->sType && timeline->signalSemaphoreValueCount)
@@ -2284,8 +2503,8 @@ static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count,
         for (uint32_t j = 0; j < bind->signalSemaphoreCount; j++)
         {
             VkSemaphore *semaphores = (VkSemaphore *)bind->pSignalSemaphores; /* cast away const, it has been copied in the thunks */
-            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
-            semaphores[j] = semaphore->host.semaphore;
+            struct semaphore *semaphore = semaphore_from_handle( semaphores[j] );
+            semaphores[j] = semaphore_map_signal( semaphore, signal_values + j, &signals );
         }
 
         /* insert the timeline semaphore values in the chain if it was there or has been created */
@@ -2297,7 +2516,9 @@ static VkResult win32u_vkQueueBindSparse( VkQueue client_queue, uint32_t count,
     }
 
     res = device->p_vkQueueBindSparse( queue->host.queue, count, binds, fence ? fence->host.fence : 0 );
-    if (!res) timeline_thread_notify( queue->device );
+    if (!res) timeline_thread_notify( queue->device, &waits, &signals, FALSE );
+    free_timeline_events( queue->device, &waits );
+    free_timeline_events( queue->device, &signals );
 
 failed:
     mem_free( &pool );
@@ -2307,6 +2528,7 @@ failed:
 static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, const VkSubmitInfo *submits, VkFence client_fence )
 {
     struct vulkan_fence *fence = client_fence ? vulkan_fence_from_handle( client_fence ) : NULL;
+    struct list waits = LIST_INIT( waits ), signals = LIST_INIT( signals );
     struct vulkan_queue *queue = vulkan_queue_from_handle( client_queue );
     struct vulkan_device *device = queue->device;
     VkResult res = VK_ERROR_OUT_OF_HOST_MEMORY;
@@ -2336,9 +2558,17 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
             switch ((*next)->sType)
             {
             case VK_STRUCTURE_TYPE_D3D12_FENCE_SUBMIT_INFO_KHR:
-                FIXME( "VK_STRUCTURE_TYPE_D3D12_FENCE_SUBMIT_INFO_KHR not implemented!\n" );
+            {
+                VkD3D12FenceSubmitInfoKHR *fence_info = (VkD3D12FenceSubmitInfoKHR *)*next;
+                if (timeline->sType) ERR( "Duplicated timeline semaphore/d3d12 fence submit info!\n" );
+                timeline->sType = VK_STRUCTURE_TYPE_TIMELINE_SEMAPHORE_SUBMIT_INFO;
+                timeline->waitSemaphoreValueCount = fence_info->waitSemaphoreValuesCount;
+                timeline->pWaitSemaphoreValues = fence_info->pWaitSemaphoreValues;
+                timeline->signalSemaphoreValueCount = fence_info->signalSemaphoreValuesCount;
+                timeline->pSignalSemaphoreValues = fence_info->pSignalSemaphoreValues;
                 *next = (*next)->pNext; next = &prev;
                 break;
+            }
             case VK_STRUCTURE_TYPE_DEVICE_GROUP_SUBMIT_INFO:
                 device_group = (VkDeviceGroupSubmitInfo *)*next;
                 break;
@@ -2374,8 +2604,8 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
         for (uint32_t j = 0; j < submit->waitSemaphoreCount; j++)
         {
             VkSemaphore *semaphores = (VkSemaphore *)submit->pWaitSemaphores; /* cast away const, it has been copied in the thunks */
-            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
-            semaphores[j] = semaphore->host.semaphore;
+            struct semaphore *semaphore = semaphore_from_handle( semaphores[j] );
+            semaphores[j] = semaphore_map_wait( semaphore, wait_values + j, -1, &waits );
         }
 
         if (wait_count) /* extra wait semaphores, need to update arrays and counts */
@@ -2425,8 +2655,8 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
         for (uint32_t j = 0; j < submit->signalSemaphoreCount; j++)
         {
             VkSemaphore *semaphores = (VkSemaphore *)submit->pSignalSemaphores; /* cast away const, it has been copied in the thunks */
-            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[j] );
-            semaphores[j] = semaphore->host.semaphore;
+            struct semaphore *semaphore = semaphore_from_handle( semaphores[j] );
+            semaphores[j] = semaphore_map_signal( semaphore, signal_values + j, &signals );
         }
 
         if (signal_count) /* extra signal semaphores, need to update arrays and counts */
@@ -2460,7 +2690,9 @@ static VkResult win32u_vkQueueSubmit( VkQueue client_queue, uint32_t count, cons
     }
 
     res = device->p_vkQueueSubmit( queue->host.queue, count, submits, fence ? fence->host.fence : 0 );
-    if (!res) timeline_thread_notify( queue->device );
+    if (!res) timeline_thread_notify( queue->device, &waits, &signals, FALSE );
+    free_timeline_events( queue->device, &waits );
+    free_timeline_events( queue->device, &signals );
 
 failed:
     mem_free( &pool );
@@ -2470,6 +2702,7 @@ failed:
 static VkResult queue_submit( struct vulkan_queue *queue, uint32_t count, const VkSubmitInfo2 *submits, VkFence client_fence, PFN_vkQueueSubmit2 p_vkQueueSubmit2 )
 {
     struct vulkan_fence *fence = client_fence ? vulkan_fence_from_handle( client_fence ) : NULL;
+    struct list waits = LIST_INIT( waits ), signals = LIST_INIT( signals );
     struct mempool pool = {0};
     VkResult res;
 
@@ -2489,16 +2722,16 @@ static VkResult queue_submit( struct vulkan_queue *queue, uint32_t count, const
         for (uint32_t j = 0; j < submit->waitSemaphoreInfoCount; j++)
         {
             VkSemaphoreSubmitInfo *semaphore_infos = (VkSemaphoreSubmitInfo *)submit->pWaitSemaphoreInfos; /* cast away const, it has been copied in the thunks */
-            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphore_infos[j].semaphore );
-            semaphore_infos[j].semaphore = semaphore->host.semaphore;
+            struct semaphore *semaphore = semaphore_from_handle( semaphore_infos[j].semaphore );
+            semaphore_infos[j].semaphore = semaphore_map_wait( semaphore, &semaphore_infos[j].value, -1, &waits );
             if (semaphore_infos->pNext) FIXME( "Unhandled struct chain\n" );
         }
 
         for (uint32_t j = 0; j < submit->signalSemaphoreInfoCount; j++)
         {
             VkSemaphoreSubmitInfo *semaphore_infos = (VkSemaphoreSubmitInfo *)submit->pSignalSemaphoreInfos; /* cast away const, it has been copied in the thunks */
-            struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphore_infos[j].semaphore );
-            semaphore_infos[j].semaphore = semaphore->host.semaphore;
+            struct semaphore *semaphore = semaphore_from_handle( semaphore_infos[j].semaphore );
+            semaphore_infos[j].semaphore = semaphore_map_signal( semaphore, &semaphore_infos[j].value, &signals );
             if (semaphore_infos->pNext) FIXME( "Unhandled struct chain\n" );
         }
 
@@ -2524,7 +2757,9 @@ static VkResult queue_submit( struct vulkan_queue *queue, uint32_t count, const
     }
 
     res = p_vkQueueSubmit2( queue->host.queue, count, submits, fence ? fence->host.fence : 0 );
-    if (!res) timeline_thread_notify( queue->device );
+    if (!res) timeline_thread_notify( queue->device, &waits, &signals, FALSE );
+    free_timeline_events( queue->device, &waits );
+    free_timeline_events( queue->device, &signals );
 
 failed:
     mem_free( &pool );
@@ -2597,6 +2832,7 @@ static VkResult win32u_vkCreateSemaphore( VkDevice client_device, const VkSemaph
     VkBaseOutStructure **next, *prev = (VkBaseOutStructure *)create_info;
     struct vulkan_instance *instance = device->physical_device->instance;
     VkExportSemaphoreCreateInfoKHR *export_info = NULL;
+    VkSemaphoreTypeCreateInfo type_info = {0};
     struct semaphore *semaphore;
     VkSemaphore host_semaphore;
     BOOL nt_shared = FALSE;
@@ -2623,12 +2859,18 @@ static VkResult win32u_vkCreateSemaphore( VkDevice client_device, const VkSemaph
             *next = (*next)->pNext; next = &prev;
             break;
         case VK_STRUCTURE_TYPE_QUERY_LOW_LATENCY_SUPPORT_NV: break;
-        case VK_STRUCTURE_TYPE_SEMAPHORE_TYPE_CREATE_INFO: break;
+        case VK_STRUCTURE_TYPE_SEMAPHORE_TYPE_CREATE_INFO:
+            type_info = *(VkSemaphoreTypeCreateInfo *)*next;
+            break;
         default: FIXME( "Unhandled sType %u.\n", (*next)->sType ); break;
         }
     }
 
     if (!(semaphore = calloc( 1, sizeof(*semaphore) ))) return VK_ERROR_OUT_OF_HOST_MEMORY;
+    semaphore->type = type_info.semaphoreType;
+    semaphore->device = device;
+    semaphore->semaphore_value = type_info.initialValue;
+    type_info.initialValue = 0;
 
     if ((res = device->p_vkCreateSemaphore( device->host.device, create_info, NULL /* allocator */, &host_semaphore )))
     {
@@ -2657,6 +2899,7 @@ static VkResult win32u_vkCreateSemaphore( VkDevice client_device, const VkSemaph
 
     vulkan_object_init( &semaphore->obj.obj, host_semaphore );
     instance->p_insert_object( instance, &semaphore->obj.obj );
+    TRACE( "Created semaphore %p with value %#jx\n", semaphore, (uintmax_t)semaphore->semaphore_value );
 
     *ret = semaphore->obj.client.semaphore;
     return res;
@@ -2690,6 +2933,14 @@ static void win32u_vkDestroySemaphore( VkDevice client_device, VkSemaphore clien
 static VkResult get_semaphore_counter_value( struct vulkan_device *device, struct semaphore *semaphore, uint64_t *value,
                                              PFN_vkGetSemaphoreCounterValue p_vkGetSemaphoreCounterValue )
 {
+    if (semaphore->type == VK_SEMAPHORE_TYPE_TIMELINE && !semaphore->local)
+    {
+        pthread_mutex_lock( &timeline_lock );
+        *value = semaphore->semaphore_value;
+        pthread_mutex_unlock( &timeline_lock );
+        return VK_SUCCESS;
+    }
+
     return p_vkGetSemaphoreCounterValue( device->host.device, semaphore->obj.host.semaphore, value );
 }
 
@@ -2713,17 +2964,25 @@ static VkResult signal_semaphore( struct vulkan_device *device, const VkSemaphor
                                   PFN_vkSignalSemaphore p_vkSignalSemaphore )
 {
     struct semaphore *semaphore = semaphore_from_handle( signal_info->semaphore );
+    struct list waits = LIST_INIT( waits ), signals = LIST_INIT( signals );
     VkSemaphoreSignalInfo info = *signal_info;
-    VkResult res;
 
     if (info.pNext) FIXME( "pNext not implemented\n" );
     info.pNext = NULL;
 
-    info.semaphore = semaphore->obj.host.semaphore;
-    res = p_vkSignalSemaphore( device->host.device, &info );
-    if (!res) timeline_thread_notify( device );
+    if (!semaphore->local)
+    {
+        pthread_mutex_lock( &timeline_lock );
+        TRACE( "Setting semaphore %p value %#jx\n", semaphore, (uintmax_t)signal_info->value );
+        semaphore->semaphore_value = signal_info->value;
+        pthread_mutex_unlock( &timeline_lock );
+        timeline_thread_notify( device, &waits, &signals, TRUE );
 
-    return res;
+        return VK_SUCCESS;
+    }
+
+    info.semaphore = semaphore->obj.host.semaphore;
+    return p_vkSignalSemaphore( device->host.device, &info );
 }
 
 static VkResult win32u_vkSignalSemaphore( VkDevice client_device, const VkSemaphoreSignalInfo *signal_info )
@@ -2743,12 +3002,18 @@ static VkResult win32u_vkSignalSemaphoreKHR( VkDevice client_device, const VkSem
 static VkResult wait_semaphores( struct vulkan_device *device, const VkSemaphoreWaitInfo *wait_info, uint64_t timeout,
                                  PFN_vkWaitSemaphores p_vkWaitSemaphores )
 {
+    struct list waits = LIST_INIT( waits ), signals = LIST_INIT( signals );
     VkResult res = VK_ERROR_OUT_OF_HOST_MEMORY;
     VkSemaphoreWaitInfo info = *wait_info;
+    LARGE_INTEGER counter, freq;
     struct mempool pool = {0};
     VkSemaphore *semaphores;
     uint64_t *values;
 
+    NtQueryPerformanceCounter( &counter, &freq );
+    if (timeout == -1) counter.QuadPart = -1;
+    else counter.QuadPart = (counter.QuadPart + 1) * (1000000000 / freq.QuadPart) + timeout;
+
     if (!(semaphores = mem_alloc( &pool, info.semaphoreCount * sizeof(semaphores) ))) return VK_ERROR_OUT_OF_HOST_MEMORY;
     info.pSemaphores = memcpy( semaphores, info.pSemaphores, info.semaphoreCount * sizeof(semaphores) );
 
@@ -2760,10 +3025,11 @@ static VkResult wait_semaphores( struct vulkan_device *device, const VkSemaphore
 
     for (uint32_t i = 0; i < info.semaphoreCount; i++)
     {
-        struct vulkan_semaphore *semaphore = vulkan_semaphore_from_handle( semaphores[i] );
-        semaphores[i] = semaphore->host.semaphore;
+        struct semaphore *semaphore = semaphore_from_handle( semaphores[i] );
+        semaphores[i] = semaphore_map_wait( semaphore, values + i, counter.QuadPart, &waits );
     }
-    timeline_thread_notify( device );
+    timeline_thread_notify( device, &waits, &signals, FALSE );
+    free_timeline_events( device, &waits );
 
     res = p_vkWaitSemaphores( device->host.device, &info, timeout );
 
diff --git a/include/wine/vulkan_driver.h b/include/wine/vulkan_driver.h
index 1dbaed13f1f..6f7cd6d79e9 100644
--- a/include/wine/vulkan_driver.h
+++ b/include/wine/vulkan_driver.h
@@ -215,6 +215,9 @@ struct vulkan_device
     HANDLE      timeline_thread;
     VkSemaphore timeline_semaphore;
     uint64_t    timeline_value;
+    struct list timeline_waits;
+    struct list timeline_signals;
+    struct list timeline_events;
 
     uint64_t queue_count;
     struct vulkan_queue queues[];
-- 
GitLab

https://gitlab.winehq.org/wine/wine/-/merge_requests/9510

[PATCH v2 0/4] MR9510: win32u: Implement timeline semaphore rewinding.

[Rémi Bernon]

v2: Make sure to copy the timeline value arrays, they aren't actually copied in the thunks and changing the values there caused vkd3d to get confused and tests to timeout.

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/9510#note_122800

[PATCH v2 0/4] MR9510: win32u: Implement timeline semaphore rewinding.

[Paul Gofman (＠gofman)]

> Pretty much the same kind of thing Proton does, this is only for non-shared semaphores for now.

Well, Proton doesn't do that for timeline semaphores, it is only done for d3d12 shared fences. I think the main issue here is that doing this for every timeline semaphore is a no go performance wise. I know timeline semaphores may be showing similar semantics on Windows at least on NVidia (but is that the same on AMD, Intel, not sure?) but nothing is known to depend on such behaviour.

The (severe) problem with performance comes from the fact such wrapping routes GPU side waits through CPU side scheduling. That really unfortunate and stalls the GPU pipelines. That is especially unfortunate on NVidia where for some reason CPU waits on timeline semaphores have significant delay waking up. I think we cannot afford that in general Vulkan translation (even if there will be some native Vulkan app depending on this out of spec behaviour). Proton only gets away with that with d3d12 shared fences because those are not used extensively, unlike Vulkan timeline semaphores (both in Vulkan games and d3d translation layers). I think we should only be doing this (between the lines of this patchset) for backing d3d12 fences.

Then, a server call to signal shared fence value seems like entirely no go on its own. Maybe we can do that as a special case for cross process fence (when a fence is known to be shared between different processes; I honestly doubt we encountered such a case in practice so far but not sure). But most of the time at least those are used within the same process between different 3d devices, it is important to avoid any server calls for those (besides creation / destruction).

Then, if we are talking about d3d12 fences semantics, the implementation here doesn't seem to mind "pulse timeline" case. When d3d12 fence gets higher value all the queued waiters for lower values are supposed to be woken, similar to SetEvent() WRT waiters in queue. While the value can be reset to lower value (either explicitly or through queued fence signal which happened to end later than some other set to some higher value; the cases is similar to PulseEvent or ResetEvent followed by SetEvent). This is inherently hard to do without having real wait queue in place (while, actually, we do have one?). Proton currently does it by maintaining fence value history with some fixed depth. Maybe it can be done without the history by ordering signals + waits appropriately within the scheduling pass in timeline_thread() and throw the multi-process shared fence "pulse" case under the bus.

Then, this is probably more subtle (while issues from the similar problem in vkd3d-proton non-shared d3d12 fences implementation were encountered in practice) it seems like both Proton implementation and this one may allow for out of order signaling of events. If there are dependent submits signaling (different) fences within a signle timeline_thread() scheduling pass, I think now the order of signal is arbitrary while it should be making sure that dependent signals are signaled not before those they are depend on. That is not something which so far came up in practice with shared fences (while bugs in such a place depending on subtle timing are very hard to deal with and it is better to have that right). But if we imagine we do timeline semaphores this way that will be likely breaking things at once.

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/9510#note_122818

[Rémi Bernon]

Right, we could probably limit this to cases where the semaphores are shared (and/or as D3D12 fences), as this is a practical case we need it for proper interop. However, and with the other points you mention I'm worried that it's not going to be possible to implement this properly.

The ordering problem for instance seems tricky to handle correctly. Some signals ordering might indeed be well defined on the GPU, but it becomes undefined as we wait and process signals in the order they were queued. I had this hunch before, and this seems to confirm it, that we won't be able to respect the ordering without tracking all the event dependencies, and that doesn't seem realistic.

In a similar fashion, I think it also introduces race conditions between the submission fences and the signaled semaphores: the fences are signaled by the GPU directly as soon as it has executed the submission and signaled all the events, but the thread might not have processed the signals yet and not have updated the CPU-visible semaphore values to the values that they should have.

This also seems difficult to solve, without swapping submit fences too, which would require tracking their semaphore dependencies, or adding ping-pongs between the timeline thread and the GPU signals to make sure the thread has processed all the signals before the GPU signals the fence.

Overall I'm starting to wonder whether it is a good idea to add all this logic now, and end up with a half assed implementation which will never be able to implement every thing we need properly. It seems to me that the only option here is to have host Vulkan driver support for rewinding.

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/9510#note_122825

[Paul Gofman (＠gofman)]

> In a similar fashion, I think it also introduces race conditions between the submission fences and the signaled semaphores: the fences are signaled by the GPU directly as soon as it has executed the submission and signaled all the events, but the thread might not have processed the signals yet and not have updated the CPU-visible semaphore values to the values that they should have.

yes, maybe it can be made fully correct only if we handle all the d3d12 fences on the winevulkan . win32u side, both shared and not, also knowing d3d12 queue configuration and submits... which maybe pulls a noticable share of d3d12 translation layer implementation there. This is unlikely much appealing and shared fences implementation will have to have some trade offs, which however worked in practice so far, given limited usage of those.
> Overall I'm starting to wonder whether it is a good idea to add all this logic now, and end up with a half assed implementation which will never be able to implement every thing we need properly. It seems to me that the only option here is to have host Vulkan driver support for rewinding.

Not only for rewindind per se but a bit of other timeline semaphores constraints. That would be great not only for shared fences but for d3d12 local fences implementation but AFAIU that meets issues on the Vulkan speccing side (as far as I heard that meets some opposition because Vulkan is supported on a wide range of hardware and on some that is hard to implement; but don't quote me on that, I am not up to date with those details). But I guess we will unlikely have that done anytime soon, and even if that is specced chances that will be an optional extension and that will be limited to a subset of GPU hardware / drivers. While that is not so extensively used within rendering workflow, it is used relatively often in modern games. So maybe a solution with known limitations but generally working is still a good call.

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/9510#note_122847

[PATCH v2 0/4] MR9510: Draft: win32u: Implement timeline semaphore rewinding. - closed

[Rémi Bernon (＠rbernon)]

This merge request was closed by Rémi Bernon.
-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/9510