Each fiber stack is allocated with a call to mmap, so this is probably reaching the limit of mmaped segments for a single process. In Linux the current limit can be obtained with sysctl vm.max_map_count. Note that not only fibers use mmap, so the actual fibers limit could be much lower than the limit.
The limit can also be increased with sysctl -w vm.max_map_count=NEW_LIMIT. That should change the number of fibers that can be created before that error is raised.