Memory usage of HTTP one-shot vs maintaining a persistent connection

News
1

I deployed a Mastodon bot a couple weeks ago that polls a US National Weather Service RSS feed for an airport in my county and posts to Mastodon when it updates. I was using the HTTP “one-shot” methods that spin up an HTTP connection, send the request, then terminate the connection.

I expected this bot to use very little RAM (less than 10MB) because it’s doing very little:

  1. Makes an HTTP request to get the RSS feed
  2. Parses the RSS feed
  3. Iterates over each feed item (there’s only one)
  4. If it hasn’t been posted yet (tracked by a key in Redis or a record in SQLite), make an HTTP request to the Mastodon API
  5. Sleep for 1 second
  6. GOTO 1

That’s all it does. The whole thing is only 61 lines of code including whitespace.

What surprised me was that RAM consumption grew over time.

Graph of memory usage during various experiments
Graph of memory usage during various experiments1856×1034 118 KB

The first two lines show linear memory growth. I restarted the app and it happened again. Then I tried a few more experiments:

  • Replace SQLite with Redis
  • Replace the RSS feed parser with using stdlib XML directly
  • Remove all XML parsing and just use regexes (yes, I know, don’t worry about it)
  • Compile the app using --mcpu for the specific aarch64 CPUs I’m using on GCP
  • Cap memory usage to 20MB so it’ll just restart when it exceeds it
  • Use a persistent HTTP connection

The first 4 didn’t do anything useful. Capping memory usage actually caused an additional problem — it caused the app to crash after persisting that it was posting to Mastodon but before actually posting.

The punchline is in the image above, but after almost 24 hours using a persistent HTTP connection (with a connection pool) for both the RSS feed and the Mastodon API, memory usage has tapered at about 5MB. This is the only thing that kept RAM stable at single-digit MB.

That makes me wonder if there’s something in OpenSSL (either in the version of OpenSSL itself that’s included in the 84codes Alpine-based container images or in the Crystal bindings for it) that’s causing it to leak memory. I’ve only ever seen this kind of memory growth in Ktistec, which is why I thought it might be something in SQLite but it happening when running one-shot HTTP requests would also track.

Regardless of the reason, if you want to minimize memory consumption in an app that talks to other HTTP services, reusing HTTP connections will do that.

3 Likes
2

Using the latest Crystal version? Does remind me of like Fix memory leak in `OpenSSL::SSL::Socket#peer_certificate` by compumike · Pull Request #13785 · crystal-lang/crystal · GitHub, but that was quite a while ago at this point.

1 Like
3

@jgaskins We recently identified a memory leak with the OpenSSL library in the Alpine docker images. Removing the openssl packages and compiling it from source “fixes” the issue for us:

apk del openssl-dev openssl-libs-static
apk add curl perl linux-headers
curl -Ls "https://github.com/openssl/openssl/releases/download/openssl-3.3.3/openssl-3.3.3.tar.gz" | tar xz
cd openssl-3.3.3
./Configure --openssldir=/etc/ssl && make -j

Then:

PKG_CONFIG_PATH=$PWD/openssl-3.3.3 crystal build ...

We haven’t been able to define what’s at fault, even after lots and lots of debugging sessions. Some SSL free methods don’t are called but don’t do some internal calls or something.

Maybe a configure feature in the openssl package that doesn’t work well with something we do in the Crystal bindings?

2 Likes
4

Yep. From my Dockerfile:

FROM 84codes/crystal:1.16.3-alpine AS builder

Fascinating. Since I’m using the 84codes container images (because I need aarch64 support), I’ll open an issue on their repo. Thanks!

5

Did you folks manage to create a reproduction repository/app to see the issue?

Looking at how Alpine compiles the package here it definitely changes some parameters versus your local configure

I would love to see if that is reproducible between 3.3.4 and 3.5.x builds from Alpine.

Cheers.

6

It’s easy to replicate: start a HTTPS server in Crystal in an Alpine Docker image, configure a GC loop (see below), start it and make a bunch of HTTPS requests, let the GC loop collect the memory, and repeat again. The GC memory should always settle to the same amounts (roughly) but the RES/VIRT memory of the process should keep growing slightly.

spawn do
  loop do
    sleep 2.seconds
    10.times { GC.collect }
    p GC.stats
  end
end
7

NOTE: keeping the same crystal executable, but swapping the libssl.so and libcrypto.so from the alpine package to the custom build (same version, of course) with LD_LIBRARY_PATH fixes the leak: the RES/VIRT memory always returns to the same rough amount.