This blog post goes into the depths comparing mmap and pread for reading positional data from a file.
It offers quite a few insights. One of the main observations is that the runtime is aware that pread is a syscall that may block. But reading mapped-memory is just a memory read. But it will also block if the accessed page is not resident. The mechanism for that is not obvious like a syscall and can affect performance: When multiple fibers block waiting on data, they can block the entire thread. On contrast, when a fiber blocks on a syscall, it gets unscheduled and the thread is free for other fibers.
The post discusses this problem for the Go runtime, but it’s very similar in Crystal. It might even be more prominent because in Go it affects the global limit GOMAXPROCS whereas Crystal’s execution contexts might have much smaller parallelism and thus fewer blocking fibers can obstruct the entire context.
Looks like a complicated approach. At least on linux there is also the normal uring read operation that takes a position to read from (the case where the OS keeps track of the position is the special case with a magic position), which would solve at least some of the issues raised in the article, as it then wouldn’t actually block the thread (unless it is the only thing running as it would end up waiting on io_uring_enter then). It would also be very quick for reads that is already in page cache.
And on that event machine we could also perhaps consider RWF_UNCACHED for less overhead from the page cache. The latter should be pretty pointless for buffered file operations anyhow (and counterproductive if the use case is a lot of random reads)..
Yeah, uring would skip a lot of the overhead.
mmap might still have some performance benefits though. When performing small reads on a file from different fibers, perhaps in similar regions, there’s a good chance that the respective pages might already be in memory. Then a mmap read would cause no delay at all.
Of course, it always depends on the specific characteristics.
It surely wouldn’t be a good idea to wrap every memory read in a Fiber.syscall.
But perhaps, if you know you’re performing a read from an mmapped file, that could be an option?
Another interesting point is in the section SIGBUS and mmap:
When reading at the end of an mmaped slice it might happen that some runtime primitive reads beyond that for performance reasons. That’s usually not a problem with contiguous memory. But in case of mmapped memory, that’ll trigger a SIGBUS signal.
The recommended way to prevent that is to always scale the size of the mmapped memory to multiple of the page size. The extra room not filled with file contents doesn’t matter and prevents SIGBUS issues.
That’s an interesting approach, and directly relatable to our model.
@yxhuvud pread through io_uring avoids the direct syscall and the “runtime doesn’t know about hidden syscall” but it’s still significantly slower than direct memory access: write sqe, maybe enter(getevents), fiber cs, wait for cqe, fiber cs.
Aside: while reading the article, I noticed fasttime.UnixTimestamp() and went to look at it. It starts a thread that ticks every 200ms to update a global. Checking the time is a mere read global rather than calling clock_gettime. Precision is coarser, but fine enough for many cases.
With execution contexts, we already have a thread that ticks every 10ms and checks the current time (the monitor thread). We could store it, and we’d have a fast timestamp
@ysbaddaden The thing is that with the kind of complexity budget such a hybrid solution to interact with what the page cache is doing, then you can instead do away with the page cache interactions (either through direct io or through RWF_UNCACHED) and thus totally avoid the black box magic of trying to keep track of what is in memory - and instead manage it yourself. The stuff that is in memory will be regular memory reads in that case as well, and you won’t get uncontrollable overhead in cases where it tries to readahead, or funky interactions when something else is also heavily using the page cache or whatever. My understanding is that the most common databases tend to not use mmap.
(and for writes you get tools like fsync as well, instead of magic deferred syncing work that you don’t have any control over)
