ForkJoin - a work-first execution context for Crystal

Just pushed fork_join shard, a multi-threaded execution context for recursively generated and nested fiber workloads.

The core idea comes from the work-first model used by Cilk and Java’s ForkJoinPool. Work stays with the worker that created it, while idle workers can take work from busier ones. That means every spawn doesn’t have to pass through a single shared queue, and recursive workloads can spread across the available workers as they grow.

The implementation is not a direct port. Its queueing and scheduling policy combines ideas proven in several runtimes:

  • Tokio influenced the bounded worker queues, batch stealing, overflow handling, and limits on concurrent searching.
  • Go influenced fair-share global injection, periodic attention to external work, and the wakeup discipline between searching and parked workers.
  • Kotlin’s coroutine scheduler influenced the separation of local and external publication, immediate-task locality, and blocking-worker handling.
  • OpenJDK’s ForkJoinPool influenced randomized stealing, contention control, and compensation around blocking work.

The scheduler works with regular Crystal fibers, so there is no new task type to adopt. Channels, timers, I/O, blocking calls, nested workloads, and live pool resizing all continue to work through Crystal’s existing concurrency model.

Feedback and real-world workload results are welcome.

It’s nice to see custom implementation of execution context strategies.
We’ve had that in mind from the beginning, but the API might not be exactly polished for that. Please share any issues you might’ve found along the way!

About the context itself, I’m wondering what specific use cases benefit from such a design. It’s clear that this context is meant for algorithms that produce additional work items recursively. The critical point is that these work items are represented as individual fibers.
A different approach would represent work items as plain job descriptions, and a couple of fixed fibers for executing them (which may include a similar, depths-first scheduling logic). That would be my first choice, because it avoids the overhead of potentially very many fiber stack allocations.

Do you have any insights on the pros and cons of using fibers as work items?

Thanks! On the execution-context SPI, the main issue was that a custom multi-threaded context needs several pieces that are currently :nodoc:: context registration, thread-pool checkout/checkin, scheduler enumeration for syscall monitoring, and the event-loop lifecycle. The event-loop API was the only compatibility issue I hit directly; fork_join has a small compile-time branch for the change from the Fiber::List form in 1.21 to the block-based form in 1.22-dev.

A reusable base for multi-threaded execution contexts might help here. It could take care of registration, thread attachment, syscall-monitor participation, and event-loop setup, while leaving queueing, stealing, parking, and wakeup policy to the implementation. Short of that, documenting and stabilizing the small set of hooks required by a custom scheduler would already make experimentation easier.

About the context itself, I’m wondering what specific use cases benefit from such a design. It’s clear that this context is meant for algorithms that produce additional work items recursively. The critical point is that these work items are represented as individual fibers.
A different approach would represent work items as plain job descriptions, and a couple of fixed fibers for executing them (which may include a similar, depths-first scheduling logic). That would be my first choice, because it avoids the overhead of potentially very many fiber stack allocations.

Do you have any insights on the pros and cons of using fibers as work items?

The use case I have in mind is nested Crystal workloads where fibers discover and spawn more work as they run. A recursive filesystem processor is one example: walking directories produces more directory and file work, while individual branches may read, parse, hash, or send results through channels. The tree is irregular, so keeping new work local and allowing idle workers to steal from busy ones helps it spread naturally.

Java’s fork/join framework and Rayon use the same scheduling idea for recursive sorting, tree traversal, search, and parallel collection processing. Those systems generally represent work as lightweight tasks, so I agree that plain job descriptions executed by fixed fibers are a better choice for tiny, CPU-only operations.

The reason fork_join uses fibers is that it is an execution context for existing Crystal code. Spawned work remains an ordinary fiber and can use channels, timers, I/O, or other suspension points without introducing a separate task API. The cost is one fiber per work item, so the work needs to be coarse enough to justify that cost, with recursive algorithms stopping at a sensible cutoff.

The current benchmark compares Parallel and fork_join using the same recursive fiber workload. It measures the scheduling-policy difference, not fibers versus lightweight jobs. That would be a useful separate comparison.

Interesting. I guess it is very high time to document what I’m cooking in the area as well, even if there are still to-do items :)

Ok, whipped up a quick readme. GitHub - yxhuvud/nest: Streamlined grouping of fibers focused on structured concurrency · GitHub

What is still missing is a: Stitching backtraces so that they are followable across nestings, and b: cancelations/deadlines in all its forms. Of those a is solvable and b require solutions in crystal itself. And maybe a smarter solution for recursive work is necessary. The current design for backpressure is for when all work is known up front.

EDIT: Created a separate topic in Nest - a shard for simple structured concurrency . Any comments go there, please.

Work stays with the worker that created it, while idle workers can take work from busier ones. That means every spawn doesn’t have to pass through a single shared queue, and recursive workloads can spread across the available workers as they grow.

Keeping new work local and allowing idle workers to steal from busy ones helps it spread naturally

This is exactly what the Parallel context does, so, I’m puzzled on the need for a custom context. Fibers spawns and enqueues are always local first (with an overflow mechanism) and starving schedulers can steal work (and dequeue from the overflow), plus a regular division of work (by starting more schedulers) to auto-scale the load across CPU cores.

Thanks. I’d push back on “exactly what Parallel does.” Same family — both are bounded-deque + overflow + steal-half machines that share the stdlib’s ThreadPool and monitor. But the actual scheduling architecture differs in three code-visible ways:

  1. Steal surface and run order. fork_join’s per-worker queue is three-tier: a LIFO slot for the just-spawned child, a handoff slot for the pending continuation, and a FIFO ring. The owner runs newest-first (depth-first on recursive work); thieves can only pull from the ring. Parallel’s local queue is a single flat FIFO ring, and its steal path takes half of everything, including work the owner would have run next.
  2. Search contention. fork_join gates victim scans: an atomic counter cap at (workers+1)/2 concurrent searchers, so an idle worker can’t stampede all others on every empty. Parallel’s steal path starts at a random index and walks the whole scheduler list with no scan limit — each empty worker probes everything, every pass.
  3. Worker lifecycle. fork_join mounts a fixed pool eagerly and only changes it via explicit resize: retiring workers cooperatively evacuate their queues into the context, are reaped, and return to the thread pool. Parallel treats capacity as a ceiling and dynamically grows/auto-scales threads under load — which, to be fair, is a real advantage for workloads without a fixed shape.

The benchmark isolates #1 and #2 on recursive fiber load — it’s not about whether stealing exists, both steal; it’s who can steal and run what.

3 sounds like something that could be fixed with an extra parameter on creation of the context, and should be easy to add if it is wanted. Which it very well may be, it is definitely something I have wondered about as well.

Do you have any benchmarks showing 1 and/or 2? If it is problematic perhaps it should be fixed in execution contexts too.

Yeah agree.

fork_join/guides/BENCHMARKS.md at main · naqvis/fork_join · GitHub and bench folder contains those benchmarks.