When using a video sample allocator, the session will allocate samples from it, then try to get some output from a transform, and if there's no output to be pulled, release the sample immediately. This will trigger the sample allocator notify callback, which will then trigger another try at pulling samples, ending up in a callback loop that can starve other working threads.
Another option here would be to keep the last returned status, and only re-try to pull samples from the allocator notify callback if we previously failed to process output because of a failed allocation.