Weird memory behaviour

Help & Support
1

I have a weird memory usage problem that I’m looking to solve
Have an application that is using up a lot of memory in real terms, however internally it’s not reporting much memory usage at all.

I added an signal handler to my app so I could have it output GC memory stats

Memory Usage:
 - Free: 0.7MiB
 - Heap: 46.7MiB
 - Total: 144.7MiB
 - Unmapped 0.0MiB
 - Non GC Bytes 0.0MiB

However when I use top:

VSZ    VSZRW   RSS   (SHR)  DIRTY  (SHR)  STACK
509m   500m  64040       4  55544      0    520

Memory usage seems to be increasing by ~100MB per instance of a class being instantiated. However GC class doesn’t seem to report on it and instead reports what looks like a much more reasonable increase of a few KB…

Anyone have an idea of why this might be happening?
My current guess is memory fragmentation, i.e. the OS can’t technically free the memory so we’re just increasing the amount of virtual memory used but not increasing the physical memory used.

I used gcore to grab a core dump of the running process and can see basically all the memory is unused - but not really sure how best to debug this.

2

I would suggest

  1. compiling binary with debug symols
  2. going with gc_none (to take GC out of equation)
  3. performing a RAM profiling using heaptrack or similar tools

to understand what is causing either huge block allocation or segmentation. Having such information at hand will enable you make more informed decisions instead of shooting in the dark.

HIH

3 Likes
3

Does your program use any C bindings? Memory allocated by C functions won’t be reported by the GC because it’s memory unknown to the GC. If there’s a memory leak, one thing might be that C resources are acquired but never freed.

2 Likes
4

OpenSSL and LibSSH2 but in this case nothing is being initialized.
So fairly sure neither should allocating anything.

5

If you are allocating that much at once it may also show up during regular profiling.

6

What’s the difference between all the GC.stats values?
There is

  • free_bytes (allocated by the OS but currently unused?)
  • heap_size (size of all objects being tracked by the GC?)
  • total_bytes (what is this?)

I sometimes see total being smaller than the free and heap sizes

image

7

A quick, not very helpful answer is in the bdwgc include file. This said, information can sometimes be confusing.

3 Likes
8

Thanks for the link. It does mention that total never decreases, so I assume it’s total allocations over the lifetime of the process - not total current usage. I agree that the comments are not exactly clear

Good news is I think I’ve solved my memory leak (refactored as I couldn’t see anything obvious)

4 Likes