How to output progress message in parallel processing with Crystal?

Help & Support
1

I am using Crystal’s spawn for parallel processing. How can I output the progress neatly?

channel = Channel(Result1).new(16)
results = [] of Result1

records.each_with_index do |record, index|
  spawn do
    # Process each file and send the result to the channel
    result = process_file(record)
    channel.send(result)
  end
end

records.size.times do
  result = channel.receive
  results << result
  print_message(result) # Output the result for each record
end

The issue is that the terminal remains blank for a while, and then all the progress messages are printed at once near the end. Using STDOUT.flush hasn’t resolved this.

How can I ensure smooth and real-time output of progress messages during parallel processing?

I am not fully aware of the difference between “Concurrency” and “Parallelism”, but I would like it to work with or without the preview_mt flag.

I would appreciate any advice. Thank you!

2

How are you printing the results? Via like #puts or the Log module?

3

Yes

1 Like
4

So which one? #puts or the Log module?

1 Like
5

Thanks for the reply. However, my guess was that this was a context switching issue and not really related to the output method.

I use puts because I want to display the results in real time in the terminal, but is there any difference between puts and the Log module?

6

The Log module can output to any IO, including STDOUT. But in this context, its catch is that it emits the messages async by default. So if you have a tight loop in a fiber that doesn’t ever yield, then it doesn’t get a chance to actually emit the messages until it finishes, or blocks in some way.

If I had to guess what’s happening, it’s that all the fibers are spawned and the delay in the output is all the fibers executing in the background. After some period of time the first one finishes and has its results printed, but because they likely all take a similar amount of time, it seems like they all print at the end at once.

EDIT: In regards to the buffered channel, afaik that shouldn’t make a difference here since the #send calls are in their own fiber.

7

I guess the answer is -Dpreview_mt, please try run following example with/without preview_mt compile flag to see the difference.

  def fib(n)
    return n if n < 2

    fib(n - 1) + fib(n - 2)
  end

  def spinner(delay : Float64)
    loop do
      ['-', '\\', '|', '/'].each do |e|
	print "\r#{e}"
	sleep delay.seconds
      end
    end
  end

  spawn spinner(0.1)

  n = 45
  fiber_result = fib(n)
  puts "Fibonacci(#{n}) = #{fiber_result}"