I’m trying to solve the problem of request blocking caused by CPU-intensive tasks through multi-threading. But I found that when there are CPU-intensive computing tasks executing, new partial requests may still be blocked. Of course, some requests remain unaffected.
I guess, should be a new request Fiber was assigned to a busy thread (a Fiber that is performing CPU-intensive tasks already exists), cause new fibers to wait.
If a thread is occupied by a Fiber that performs CPU-intensive tasks, it will not allocate a new Kemal request Fiber. Can it be done?
I have a lot of questions in this regard as well.
For one, I believe that spawn(:other_thread) defaults to true. So you should already have the behavior you’re looking for. But is it working right?
Also if there are threads that are assigned both socket fibers and “cpu intensive” fibers is there a chance for starvation?
MT is still a work in progress so some of these things aren’t solidified I think…
Currently there know there are two possible paths.
(1) Add some Fiber.yield in cpu intensive loops. Essentially making everything interruptible. But will penalize a bit some parts of your system.
(2) Change how HTTP::Server#listen delegates to a fiber how a request is processed. With @waj we checked on different alternatives to contemplate this scenario but for non cpu-intensive web servers the performance was a bit worse. The idea is to ensure the are one worker fiber per crystal worker that will accept socket connections. Currently there is no way to ensure a set of fibers work on different crystal workers, but due to the current scheduler round robin logic, a loop is very likely to distribute fibers among workers. So monkey-patching HTTP::Server#listen might help you test how this strategy can work on your scenario.
If you want to have more granular control you might need to play with Channels. Communicating some fibers that accepts connections with others that run the handlers might work. But for simple scenarios it will have, again, some penalties.
@bcardiff Thanks in advance for your reply and help.
CPU-intensive tasks are brought by the Native library, so I cannot insert Fiber.yield into it.The second way I will try.
Multithreading does not seem to be enough to cope with more situations, and there is a lot of room for improvement. If there is a certain preemptive ability, or more fine-grained control, it will be perfect!
You could also consider to split background processing (which I assume your long-running fiber is) and handling HTTP request into separate processes. That might prove beneficial anyway for decoupling and separate deployment/scaling. I don’t know about your exact use case but might likley be a good idea.
I forgot you might not be running “multi thread” yet in which case it’s single thread. If you’re calling out to C code then yeah it’s likely you’ll get pretty blocked unless you can maybe go to MT and the :other_thread => true stuff is working as one would expect…I haven’t checked if it does or not.
Ideally crystal would be able to check periodically (even if CPU intensive tasks are occurring) if new socket activity has made some new fibers runnable. I think it might not at current. I’ll try and look into it sometime and maybe file an issue (non-block select might be a good option, for instance…) :)