I have this really large file I want to read in to my app. I noticed that my app wouldn’t build after an hour, so my guess was it was something related to this. I broke it out to a separate file to test.
Running crystal build --release test.cr
builds this file in about 11 seconds for me. The domains.txt
file contains 119,261 lines.
# test.cr
DOMAINS = {{ read_file("./domains.txt").split('\n') }}
Now, when I try to iterate over that array, that compile time shoots up from 11 seconds to infinity (as far as I can tell because it never finishes building).
DOMAINS = {{ read_file("./domains.txt").split('\n') }}
value = "test@test.com"
found = DOMAINS.find do |domain|
value.ends_with?(domain)
end
pp found
❯ crystal build --stats --release test.cr -o zzz
Parse: 00:00:00.000129278 ( 0.75MB)
Semantic (top level): 00:00:00.320183866 ( 43.35MB)
Semantic (new): 00:00:00.000973178 ( 43.35MB)
Semantic (type declarations): 00:00:00.015073108 ( 43.35MB)
Semantic (abstract def check): 00:00:00.003208983 ( 43.35MB)
Semantic (ivars initializers): 00:00:00.013159808 ( 57.80MB)
Semantic (cvars initializers): 00:00:00.068934745 ( 73.80MB)
Semantic (main): 00:00:00.666560999 ( 234.23MB)
Semantic (cleanup): 00:00:00.007756949 ( 234.23MB)
Semantic (recursive struct check): 00:00:00.001534194 ( 234.23MB)
Codegen (crystal): 00:00:01.745678227 ( 314.23MB)
Codegen (bc+obj):
It basically sits at this spot…
Has anyone seen anything similar? Anyone have a suggestion for getting around it?