Fixing asynchronous I/O, again

Real applications that are using async operations don't wait on a specific operation; they wait for notification that any I/O has completed. A high level overview of the application I work on is that it has multiple threads that perform various operations within a pipeline. There are parts of the system that face the network and parse data coming in over TCP. The parser then formats the request into a message (which is easier to work on and agnostic of the actual on-the-wire protocol being used) and then sends those messages to various threads that then run their FSMs and potentially send messages to other threads. Some of the FSMs cause disk reads/writes to be issued. All threads are structured to spin in their main event loops which poll the internal queues between threads, run various FSMs that have been scheduled, and some of which also check the AIO ring buffer for notification of completion events for I/O. Under heavy load, nothing blocks; there is always more work to do. It is highly undesirable for any thread to block in the kernel and starve other processing from occurring, as that will add latency to the response time the end user sees. Waiting on a specific AIO to complete is simply not an idiom that is used.