Hi guys, I was wondering how is it going for the parallelism feature ? Because to be honest I really would like to use it.
If I enable this preview feature, is it quite stable or not ? Basically I would like to use it to boost the dependency calculation in my package manager ( and something else too)
It’s pretty stable these days yes, however it’s not something you turn on and it just magically increases performance. You’ll need to ensure you’re making proper use of fibers and have your code designed in such a way to make use of it. Also if your dependency calculation is mostly IO bound it’s unlikely MT will even help all that much, but could be worth looking into if it’s mostly CPU bound.
Okay got it. And where can I find the documentation for using parallelism ? Because the website just show concurrency examples
Enabling MT mode essentially just allows more than 1 fiber to execute at once. So if you’re currently properly handling concurrency (using channels to communicate data, using mutexes as needed, etc) then you should be mostly good.
Ah okay got it. Thank you very much !
Other than this blog post, I am not aware of any official documentation on parallelsism.
This may not be a good example, but I wrote a code to calculate pi.
N = ARGV[0].to_i
M = 32
channel = Channel(Float64).new(N)
M.times do |value|
spawn do
c = 0
N.times do
x = Random.rand
y = Random.rand
r = x ** 2 + y ** 2
c += 1 if r < 1
end
channel.send(c / N.to_f)
end
end
r = Array.new(M){ channel.receive }.sum / M
puts r * 4
Build with single thread
crystal build --release -o pi_st pi.cr
Build with multi-threading
crystal build --release -D preview_mt -o pi_mt pi.cr
It can be easily benchmarked using a command line tool called hyperfine.
hyperfine --warmup 1 --export-csv crystal-parallel-bench.csv \
'CRYSTAL_WORKERS=1 ./pi_st 5000000' \
'CRYSTAL_WORKERS=1 ./pi_mt 5000000' \
'CRYSTAL_WORKERS=2 ./pi_mt 5000000' \
'CRYSTAL_WORKERS=4 ./pi_mt 5000000' \
'CRYSTAL_WORKERS=6 ./pi_mt 5000000' \
'CRYSTAL_WORKERS=8 ./pi_mt 5000000'
Visualize the obtained data using Python’s Seaborn. (I used the Crystal language here for fun.)
require "../src/crython"
Crython.session do
pd = Crython.import("pandas")
plt = Crython.import("matplotlib.pyplot")
file_path = "/path/to/crystal-parallel-bench.csv"
df = pd.read_csv(file_path)
error_lower = df["mean"] - df["min"]
error_upper = df["max"] - df["mean"]
error = [error_lower, error_upper]
plt.figure(figsize: {10, 6})
plt.barh(df["command"], df["mean"], xerr: error, capsize: 5)
plt.ylabel("Command")
plt.xlabel("Mean Execution Time (s)")
plt.title("Benchmark Results with Corrected Min-Max Error Bars (Horizontal)")
plt.tight_layout
plt.savefig("/path/to/crystal-parallel-bench.png")
end
The results were about as expected.
Recently, WaitGroup was added by ysbaddaden and syntax was added by jgaskins.
If you want to understand the whole thing, you need to follow the core members and contributors, or take the time to read the code and explore it deeply by yourself (with the help of AI).
Understanding the cutting edge is like that in any field. There is a certain mental cost. (Whether you can pay that cost depends on how you allocate your time in life, rather than on your ability. Talk to yourself. (Unfortunately, I can’t pay that cost, so I’m waiting and trusting the core team))
Appreciate the link to your “fun” link above. Specifically highlighting it in case others pass over it: Crython: Crystal meets Python. Crython is a tool that lets you use [Python] libraries in [Crystal], a programming language. GitHub - kojix2/crython: Crystal meets Python
This is not the place to introduce crython, but to explain a little, in addition to the concept created by the original author RomainFranceschini a few years ago, I have added Python function calls via method missing macros, implicit type conversion of common classes, control of Python reference counters from the finalize methods of Crystal classes, and more C functions support. As is often the case with open source project, sometimes even simple code does not work well. Of course, Im happy to let you try it out.
