The issue I was a bit concerned about was that threads which cancel async would wait for every async in the "cancelled_asyncs" list, not just the ones it added there. But like you said it's probably not a big issue if it's only a transient state.
Ah got it, yeah that doesn't seem too bad to me in practice.