Another Asset/Resource Embedding question

Help & Support
1

Hey,

I’m working on a small library that requires to load a file and convert that file into a hash. As this is doing the exact same thing whenever this loading happens I’ve been asking myself if I can skip this loading step on runtime and maybe create that hash on compile time, so access is as fast as possible when the binary is executed.

I have read about embedding assets using

{{ `cat #{__DIR__}/README.md`.stringify }}

but this is only half of the work, as parsing the string into a hash still would require work on runtime.

What would be the best approach to this? Macros? Any other feature that I am not too aware of?

Thanks for any pointers or ideas to dig into :slight_smile:

Wishing everyone nice and calm (aka a little bit of coding) holidays

–Alex

2

I had a similar idea to have a lot of data in a hash. I’ve tried several approaches:

  • Loading from external file at runtime. Even with original CSV source this was pretty quick, like 4 seconds to import the whole data on start. Ruby version was close to 1 minute importing this data into memory.
  • Code generation. The compile speed can be really slow if you have a lot of data and define hashes normally using literals. But there is a workaround for this.
  • Using Messagepack. It’s fast, like really really fast. Loading time won’t be an issue unless you are doing something crazy.

I’m currently using code generation since I’ve optimized to have less huge hashes and don’t worry about compile times yet, or applying workarounds.

I plan to switch to Messagepack approach though, since I believe it’s implementation for Crystal is now better and have support for custom types, which it hadn’t before. Also there was no MessagePack::Serializable when I’ve tried it, so now it’ll be much easier and cleaner.

3

If you go with macro there is a better way to do it https://crystal-lang.org/api/0.32.1/Crystal/Macros.html#read_file(filename):StringLiteral-instance-method

4

You can run a crystal code at compile-time and insert it’s output as a text.
https://crystal-lang.org/api/0.32.1/Crystal/Macros.html#run(filename,*args):MacroId-instance-method

You can see GitHub - schovi/baked_file_system: Virtual File System for Crystal language. Embedding your assets into final binary. for an example - this is actually pretty simple.