I got into this because I'm writing firmware for a custom USB device to use in an embedded Linux system and the vendor tools are only available for Windows. Sigh. And of course Wine handles everything fine *except* the USB snooping. I've spent my entire professional life working the in Unix workstation & HPC world staring with PDP-11 and later VAX "boot me" tapes from Bell Labs / Western Electric. I think the last time I had to write a Windows GUI app was when the original Windows 95 was released :-( So I'm woefully out of date on Windows.
I probably shouldn't tell you this because I *really* want USB support in Wine, but you can currently do this currently using a VM. Run your application in a Windows in VirtualBox, then you can use Linux's usbmon to sniff the traffic.
To do the capture I usually start off with wireshark, and then use this command line tool that I maintain:
https://github.com/jhol/usbdump
With a command line packet trace, you can then do greps, diffs, vidiff etc. to reverse engineer the protocol.
Kind Regards Joel