Memory leak

Help & Support
1

I’ve been looking into a memory leak that occurs in our drivers that make use of HTTP::Client

Running stats = GC.stats (and adjusting to give MB values I am seeing this)

"memory": {
        "stats_free": "63.5MiB", # free_bytes
        "stats_heap": "170.3MiB", # heap_size
        "stats_total": "6628.9MiB", # total_bytes
        "stats_unmapped": "0.0MiB"
    },

what does total_bytes represent in this case?
It would be nice if there were descriptions added: GC::Stats - Crystal 1.16.0

2

/* Return the total number of bytes allocated in this process. /
/ Never decreases, except due to wrapping. This is an unsynchronized /
/ getter (see GC_get_heap_size comment regarding thread-safety). */
GC_API size_t GC_CALL GC_get_total_bytes(void);

3

Bdw has a GitHub - ivmai/bdwgc: The Boehm-Demers-Weiser conservative C/C++ Garbage Collector (bdwgc, also known as bdw-gc, boehm-gc, libgc) that I never tried before. Maybe it can be helpful if you are out of tricks to track the leak

4

actually looks to be solved moving from 1.14 to 1.16 looking at our metrics which simplifies things
the leak looked to be related to SSL sockets

thanks for the pointers!

2 Likes
5

We’ve recently encountered a memory leak caused by the system lib of openssl 3.3 in AlpineLinux.
Any chance you’re using the same one?
This shouldn’t be fixed by 1.16.0, though. Only by using a different library (such as a custom build).
/cc @ysbaddaden

6

Interesting… It does sound like it could have been the cause as we do use Alpine to build our images - we did bump the Alpine container version at the same time

7

Indeed, we must document these:

  • heap_size: bytes of OS memory the GC allocated for its HEAP (170MB);
  • free_bytes: how many free bytes in the GC HEAP (63MB);
  • total_bytes: how many bytes the GC has allocated since the program started (keeps growing indefinitely until it the integer overflows back to 0MB).

Note: the openssl leak is happening outside of Crystal and the GC reach. The libssl and/or libcrypto from the Alpine image are showcasing an unidentified leak: the VIRT and RES memory kept growing, but the GC always came back to the same amount of heap size and free bytes.

To search the leak: compile your program, send some HTTP requests, when the program is idling run GC.collect a few times in a loop (for finalizers to run and free more memory).

Monitor the GC stats and the OS VIRT/RES memory: they should come back to the same levels (roughly). If something keeps growing: there’s a leak. If the GC HEAP is stable, there are some libc malloc or mmap that aren’t freed.

8

@bcardiff Sadly the leak detector of BDWGC is for identifying leaks in programs that don’t use a GC: it replaces the malloc, realloc and free methods and reports allocations that became unreachable.