Crystal shard RocksDB memory leak

Help & Support
1

Hey! I’m a software developer working for Diploid, and only recently started working with Crystal. I’m using RocksDB with the RocksDB shard (https://github.com/maiha/rocksdb.cr) to store and retrieve data. I noticed that the memory usage of the program I wrote using this gem increases over time, leading me to believe I have a memory leak.

Firstly, I tried calling GC.collect in the loop of the program, which didn’t result in any noticeable changes. Then, I created a memory dump using gcore, which seemed to indicate that the used memory contains the values that are returned by the rocksdb_get function. So I ran the program without using rocksdb_get (note that this is not a solution, since it changes the program’s functionality). Doing this indeed avoids the memory usage pattern I saw before, strengthening my believe that I found my culprit. I also tried releasing the string that was returned manually, as described on StackOverflow : Crystal lang, is it possible to explicitly dispose (free) of instance (object) without waiting for GC? - Stack Overflow. This resulted in a segfault. This last thing might be normal behaviour though, as I’m not familiar with Crystal’s GC.

Lastly, I tried using a minimal example. Initialise the database, and input some values. Running this program then results in the same behaviour.

require "rocksdb"
ROCKSDB = RocksDB::DB.new("path/to/rocksdb")
keys = ROCKSDB.keys
loop do
  keys.each do |k|
    puts ROCKSDB.get(k)
  end
  GC.collect
end

First question: Is this indeed a memory leak?
Second question: What can I do about it? If it is a memory leak, what can I do to fix it? If it isn’t, how can I decrease the memory usage of my program? I saw the StackOverflow issue (sorry, can’t post more than 2 links) questions/35304158/binding-glib-into-crystal-lang-gc-issue, but I don’t think the solution is applicable here.

2

Hi! Please report an issue in that repository. I took a quick look, memory return from rocksdb_get needs to be freed but isn’t. That’s the problem. I’m sure many other functions don’t release the memory but should.

It seems one can also specify the memory allocator to use. It might be possible then to hook with Crystal’s GC and not have to worry about manually freeing memory.

1 Like
3

Thanks, I’ll do that! I wanted to make sure I was not missing something before creating an issue in the repo.