The Crystal Programming Language Forum

Multi-thread program: Unhandled exception: Cannot allocate new fiber stack: Cannot allocate memory (Errno)

my parallel code demo.cr is this:

def test(num : Int32)
	num.times do |line|
  		proc = ->(x : Int32) do
		    spawn do
				# to do something
		    end
		  end
		proc.call(line)
	end
end

inf = ARGV[0].to_i
test(inf)

Fiber.yield
$crystal --version
Crystal 0.31.1 [0e2e1d067] (2019-09-30)

LLVM: 8.0.0
Default target: x86_64-unknown-linux-gnu

after build with --release -Dpreview_mt, then run get error:

$ export CRYSTAL_WORKERS=2;./demo 100 # no error,when 100
$ export CRYSTAL_WORKERS=2;./demo 1000000
Unhandled exception: Cannot allocate new fiber stack: Cannot allocate memory (Errno)
GC Warning: Failed to expand heap by 2228224 bytes
GC Warning: Failed to expand heap by 262144 bytes
GC Warning: Out of Memory! Heap size: 6 MiB. Returning NULL!
Invalid memory access (signal 11) at address 0x6018
[0x4380b6] ???
[0x437a47] __crystal_sigfault_handler +263
[0x3143a0f130] ???
[0x423415] ???
[0x417fd2] ???
[0x415151] ???
[0x42fb93] ???
[0x42dfce] ???
[0x4108c7] main +55
[0x3142e21b45] __libc_start_main +245
[0x40c719] ???
[0x0] ???

$export CRYSTAL_WORKERS=3;./demo 1000000
Unhandled exception: Cannot allocate new fiber stack: Cannot allocate memory (Errno)
  from ???

thanks ~

Regards

What happens with CRYSTAL_WORKERS=1 or if you compile without -Dpreview_mt?

Would also be great to know how much memory the application allocates in total before crashing.

get the same error:

$export CRYSTAL_WORKERS=1;./demo 100000
Unhandled exception: Cannot allocate new fiber stack: Cannot allocate memory (Errno)
GC Warning: Failed to expand heap by 1703936 bytes
GC Warning: Failed to expand heap by 262144 bytes
GC Warning: Out of Memory! Heap size: 4 MiB. Returning NULL!
Invalid memory access (signal 11) at address 0xc018
[0x4380b6] ???
[0x437a47] __crystal_sigfault_handler +263
[0x3143a0f130] ???
[0x423415] ???
[0x417fd2] ???
[0x415151] ???
[0x42fb93] ???
[0x42dfce] ???
[0x4108c7] main +55
[0x3142e21b45] __libc_start_main +245
[0x40c719] ???
[0x0] ???

What OS are you in? 32 bits or 64 bits?

I’m running this on a mac, no problem.

CentOS release 6.9 (Final) and 64bits

https://crystal-lang.org/reference/guides/concurrency.html said this:
On a 64-bit machine it lets us spawn millions and millions of fibers,
my situation is that I have a array which have more than millions elements, and I wish to do something for each elements independently by parallel.

But in your example above you’re only spawning 100K fibers and it already runs out of memory. So it would be great if you could tell us how much memory the program actually uses. The number of fibers that can be spawned is strictly limited by the amount of memory available.

Regarding the general use case, when dealing with large numbers of jobs, it’s probably better not to spawn a fiber for each item. It doesn’t make sense when most of them are going to be locked anyway for a long time until it’s their time to process. Instead, use a number of worker fibers which loop through the list.

Thanks for the guide, I understand more about why fiber number is limited~ In most situation, the required memory for all task is often changed, so I am not sure that.
You said Instead, use a number of worker fibers which loop through the list how can I use worker fibers? Sorry, I just don’t understand the how to parallellism instead of concurrency.

@orangeSi something like this maybe?

def test(num : Int32, ch, threads)
	num.times do |line|
        ch.send(line)
    end
end

def signal_finished(ch, count)
    count.times do
        ch.send(nil)
    end
end

def process(num : Int32)
    # Do something slow, sleep to simulate slow operation
    sleep(0.1)
end

def launch_workers(count)
    count.times do
        spawn do
            # When gets nil from the channel ends the working thread loop
            # ch.receive blocks until the channel has a message available to be read
            while line = ch.receive
                process(line)
            end
        end
    end
end

ch = Channel(Int32 | Nil).new
threads = 4
launch_workers(threads)

inf = ARGV[0].to_i

test(inf, ch)
signal_finished(ch,threads)
1 Like

How much RAM does the box have that you’re running it on?