Just thought I’d throw this in here … I know that there’re utils such as sha256sum, 512sum, stuff like that for Linux, certutil.exe for Windows that’ll display file hashes … I made my own with Crystal as I wanted a couple of specialized command line options, just using the standard library, and found that the Crystal version is significantly faster hashing files than either OS’s boxed utils. Just a fun fact that I had to share!
7 Likes
Thank you, BloodFeastMan, for sharing your report.
From my understanding, the hash calculation speed in Linux should be the same as that of standard OS tools. This is because the hash calculation in Crystal’s standard library doesn’t use its own implementation. instead, it calls external functions from OpenSSL’s libcrypto bindings.
I also created a hash calculation command-line tool using Crystal’s standard library for my own use, and in my environment, the calculation speed with md5sum is about the same as the OS tools. (There might be a slight overhead, but it isn’t essential compared to the file I/O and checksum calculation for large files.)
I’m sure Crystal’s standard library is faster in your environment. However, I’m curious about what is really happening here. If there is a significant difference, it might not just be that Crystal is faster; there could be other reasons as well.
At home, Debian is running on an i5 at 3.2g, 4 processors and 16g ram. I made a Ruby script do ten million aes encryption iterations with a different string and password each time just to start from a clean slate, and hashed a file of size 2,717,007,872 bytes, and sha512sum did it in 8.334 seconds, and the Crystal wrapper did it in 5.802 seconds. Doing it in reverse, (running the Crystal util first), Crystal: 5.806 seconds, sha512sum: 8.304 seconds. I did that since I don’t know how this stuff is cached up.
Here at work, Windows 10 runs on an i5 at 2.6g, 12 processors and 32g ram, the same exact tests including the Ruby scripts, and using certutil.exe, certutil: 8.158 seconds, Crystal: 3.939 seconds, and in reverse, Crystal: 3.881 seconds and certutil: 5.652 seconds.
Fwiw, I called these tests from a TCL script to be totally fair:
#!/usr/bin/tclsh
set time_start [clock milliseconds]
#puts [exec sha512sum database.db]
puts [exec certutil -hashfile database.db sha512]
#puts [exec ft_digest sha512 database.db]
set time_stop [clock milliseconds]
set total_time [expr ($time_stop - $time_start) / 1000.000]
puts "elapsed time: $total_time seconds"
1 Like