# Issue reading really large file with read\_file **URL:** \ **Category:** Help & Support\ **Created:** [February 22, 2021, 9:38pm UTC](https://forum.crystal-lang.org/t/issue-reading-really-large-file-with-read-file/2983 "2021-02-22T21:38:50Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jwoertink](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jwoertink/32/50_2.png) [@jwoertink](https://forum.crystal-lang.org/u/jwoertink)\ **Post date:** [February 22, 2021, 9:38pm UTC](https://forum.crystal-lang.org/t/issue-reading-really-large-file-with-read-file/2983/1 "2021-02-22T21:38:50Z")
I have this [really large file](https://github.com/wesbos/burner-email-providers/blob/master/emails.txt) I want to read in to my app. I noticed that my app wouldn’t build after an hour, so my guess was it was something related to this. I broke it out to a separate file to test. Running `crystal build --release test.cr` builds this file in about 11 seconds for me. The `domains.txt` file contains 119,261 lines. ```crystal # test.cr DOMAINS = {{ read_file("./domains.txt").split('\n') }} ``` Now, when I try to iterate over that array, that compile time shoots up from 11 seconds to infinity (as far as I can tell because it never finishes building). ```crystal DOMAINS = {{ read_file("./domains.txt").split('\n') }} value = "test@test.com" found = DOMAINS.find do |domain| value.ends_with?(domain) end pp found ``` ```auto ❯ crystal build --stats --release test.cr -o zzz Parse: 00:00:00.000129278 ( 0.75MB) Semantic (top level): 00:00:00.320183866 ( 43.35MB) Semantic (new): 00:00:00.000973178 ( 43.35MB) Semantic (type declarations): 00:00:00.015073108 ( 43.35MB) Semantic (abstract def check): 00:00:00.003208983 ( 43.35MB) Semantic (ivars initializers): 00:00:00.013159808 ( 57.80MB) Semantic (cvars initializers): 00:00:00.068934745 ( 73.80MB) Semantic (main): 00:00:00.666560999 ( 234.23MB) Semantic (cleanup): 00:00:00.007756949 ( 234.23MB) Semantic (recursive struct check): 00:00:00.001534194 ( 234.23MB) Codegen (crystal): 00:00:01.745678227 ( 314.23MB) Codegen (bc+obj): ``` It basically sits at this spot… Has anyone seen anything similar? Anyone have a suggestion for getting around it? ---
**Author:** ![jwoertink](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jwoertink/32/50_2.png) [@jwoertink](https://forum.crystal-lang.org/u/jwoertink)\ **Post date:** [February 22, 2021, 9:50pm UTC](https://forum.crystal-lang.org/t/issue-reading-really-large-file-with-read-file/2983/2 "2021-02-22T21:50:00Z")
I wonder if it’s not the `read_file` but just the massive Array? I removed the macro and just dropped in a regular array ```crystal DOMAINS = %w[ ... super long ... ] ``` and after 5 minutes, it’s still trying to build. Normally my app builds in ~1minute ---
**Author:** ![asterite](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/asterite/32/60_2.png) [@asterite](https://forum.crystal-lang.org/u/asterite)\ **Post date:** [February 22, 2021, 10:21pm UTC](https://forum.crystal-lang.org/t/issue-reading-really-large-file-with-read-file/2983/3 "2021-02-22T22:21:31Z")
I tried this: ```crystal A = {% begin %} %w[ {% for i in 0..100_000 %} {{i.stringify}} {% end %} ] {% end %} puts A.size ``` It compiles in about 4 seconds and prints 100000 as an output. Do you have code that we can copy/paste? ---
**Author:** ![jwoertink](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jwoertink/32/50_2.png) [@jwoertink](https://forum.crystal-lang.org/u/jwoertink)\ **Post date:** [February 22, 2021, 10:37pm UTC](https://forum.crystal-lang.org/t/issue-reading-really-large-file-with-read-file/2983/4 "2021-02-22T22:37:55Z")
> You’ll have to just download these two files, then build it with the release flag ---
**Author:** ![straight-shoota](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/straight-shoota/32/36_2.png) [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)\ **Post date:** [February 22, 2021, 10:52pm UTC](https://forum.crystal-lang.org/t/issue-reading-really-large-file-with-read-file/2983/5 "2021-02-22T22:52:39Z")
Are you sure you compiled with `--release`? For me your code hangs at codegen: ```auto $ crystal run --release --stats test-array.cr Parse: 00:00:00.000045500 ( 0.75MB) Semantic (top level): 00:00:00.184131000 ( 44.82MB) Semantic (new): 00:00:00.001422800 ( 44.82MB) Semantic (type declarations): 00:00:00.028435500 ( 44.82MB) Semantic (abstract def check): 00:00:00.005617600 ( 44.82MB) Semantic (ivars initializers): 00:00:00.015214200 ( 59.77MB) Semantic (cvars initializers): 00:00:00.066107500 ( 75.77MB) Semantic (main): 00:00:00.802865400 ( 204.20MB) Semantic (cleanup): 00:00:00.006389700 ( 204.20MB) Semantic (recursive struct check): 00:00:00.000689500 ( 204.20MB) Codegen (crystal): 00:00:01.448765600 ( 236.20MB) Codegen (bc+obj): ``` There is already an issue on this: > > > I have a file that holds a single multi-dimensional array with over 40000 entries in it here. When I attempt to... > > > \# Crystal version 15, MacBook Pro (Retina, 13-inch, Early 2015, 16GB RAM) > \#0.5 seconds : crystal build staticarrays.cr ; ./staticarrays > \#55 seconds :... ---
**Author:** ![straight-shoota](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/straight-shoota/32/36_2.png) [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)\ **Post date:** [February 22, 2021, 10:54pm UTC](https://forum.crystal-lang.org/t/issue-reading-really-large-file-with-read-file/2983/6 "2021-02-22T22:54:28Z")
Workaround is just to move `.split('\n')` out of the macro expression which lets the array be created at runtime. ---
**Author:** ![jwoertink](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jwoertink/32/50_2.png) [@jwoertink](https://forum.crystal-lang.org/u/jwoertink)\ **Post date:** [February 22, 2021, 11:08pm UTC](https://forum.crystal-lang.org/t/issue-reading-really-large-file-with-read-file/2983/7 "2021-02-22T23:08:42Z")
Ok cool. I’ll give that a try, thanks! At least it’s a known issue.