Alright, after some serious debugging, we have a somewhat repeatable way to tank our app. It doesn’t seem to take too much effort, but for example if someone is scraping our site, then we get this:
GC Warning: Failed to expand heap by 607793152 bytes
GC Warning: Failed to expand heap by 574238720 bytes
GC Warning: Out of Memory! Heap size: 1174 MiB. Returning NULL!
Invalid memory access (signal 11) at address 0x0
[0x6a2fd6] ???
[0x451c1f] __crystal_sigfault_handler +479
[0x7f8f8abf4390] ???
[0x7f8f8a123a9e] ???
[0x4ec16a] ???
[0x4fd5bb] ???
[0x4fc4be] ???
[0x512750] ???
[0x588e0d] ???
[0x5cb50c] ???
[0x62b3ac] ???
[0x66c1fa] ???
[0x546b1f] ???
[0x53182c] ???
[0x5312c1] ???
[0x52691f] ???
[0x526431] ???
[0x52577e] ???
[0x52591b] ???
[0x68e480] ???
[0x4d2a1a] ???
[0x44d29b] ???
[0x0] ???
Then after enough times of doing this, we get this fun one: (tailing docker logs)
*** Error in `/app/app': corrupted double-linked list: 0x0000000001cf1030 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f7af95b27e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7d814)[0x7f7af95b8814]
/lib/x86_64-linux-gnu/libc.so.6(+0x8288e)[0x7f7af95bd88e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f7af95bf184]
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0(CRYPTO_malloc+0x58)[0x7f7afa965e78]
/lib/x86_64-linux-gnu/libssl.so.1.0.0(+0x28a05)[0x7f7afad70a05]
/lib/x86_64-linux-gnu/libssl.so.1.0.0(+0x28b59)[0x7f7afad70b59]
/lib/x86_64-linux-gnu/libssl.so.1.0.0(+0x2ab6c)[0x7f7afad72b6c]
/app/app[0x4e3b62]
/app/app[0x4e1b8f]
....tons more here...
/app/app[0x44c61b]
======= Memory map: ========
00400000-00753000 r-xp 00000000 fd:04 2888380 /app/app
00952000-00953000 r--p 00352000 fd:04 2888380 /app/app
00953000-00955000 rw-p 00353000 fd:04 2888380 /app/app
00955000-00bab000 rw-p 00000000 00:00 0
01b01000-02937000 rw-p 00000000 00:00 0 [heap]
7f7a88000000-7f7a88021000 rw-p 00000000 00:00 0
7f7a88021000-7f7a8c000000 ---p 00000000 00:00 0
.....and this goes on forever....
This also seems to be similar to what is here: https://github.com/omarroth/invidious/issues/773
If anyone has any clue as to direction I should look in to, that would be greatly helpful