[Demo] Nested epoll fd avoid to call add_interest, because there is a global kernel lock when doing epoll_ctl

[beef9999]

Cascading engine has a performance issue when used in multi-thread program.

Its wait_for_events will firstly call wait_for_fd, and then add_interest with one-shot.

For epoll engine, the kernel epoll_ctl will compete in multi-threads to acquire one global mutex, if the fd it watches on is epoll fd. See https://elixir.bootlin.com/linux/v5.15.125/source/fs/eventpoll.c#L2130

io_uring engine doesn't have this problem.

According to my observation, in a 24 threads program, the lock acquisition can consume as much as 80% CPU workload, which is totally unacceptable.

The solution is to use multi-shot poll to replace those epoll_ctl, so we need to keep tracks on the epoll fd.

After applying this demo changes, the osq_lock has disappered. You can run this demo in your environment.

The demo has used a limited size of arrays (8 elements) to store fd, instead of map. Comparisons are also made.

Search Time (nanoseconds)	array	std::map	std::unordered_map
Result at index 1	<1	2.8	9.6
Result at index 4	1.8	4.6	9.7
Result at index 8	3.8	5.6	9.7

The idea is that we should only allow a small number of nested epoll fd to be registered, and use array to reduce overhead in the I/O path.

Finally, this is just a demo. We need formal patch.