# Memory usage of HTTP one-shot vs maintaining a persistent connection **URL:** \ **Category:** News\ **Created:** [July 15, 2025, 7:58pm UTC](https://forum.crystal-lang.org/t/memory-usage-of-http-one-shot-vs-maintaining-a-persistent-connection/8237 "2025-07-15T19:58:05Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jgaskins](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jgaskins/32/2449_2.png) [@jgaskins](https://forum.crystal-lang.org/u/jgaskins)\ **Post date:** [July 15, 2025, 7:58pm UTC](https://forum.crystal-lang.org/t/memory-usage-of-http-one-shot-vs-maintaining-a-persistent-connection/8237/1 "2025-07-15T19:58:05Z")
I deployed [a Mastodon bot](https://zomglol.wtf/@joco_ks_weather) a couple weeks ago that polls a US National Weather Service RSS feed for an airport in my county and posts to Mastodon when it updates. I was using the `HTTP` “one-shot” methods that spin up an HTTP connection, send the request, then terminate the connection. I expected this bot to use very little RAM (less than 10MB) because it’s _doing_ very little: 1. Makes an HTTP request to get the RSS feed 2. Parses the RSS feed 3. Iterates over each feed item (there’s only one) 4. If it hasn’t been posted yet (tracked by a key in Redis or a record in SQLite), make an HTTP request to the Mastodon API 5. Sleep for 1 second 6. `GOTO 1` That’s all it does. The whole thing is [only 61 lines of code](https://gist.github.com/jgaskins/cc6b66c4624d65577f91800b14573dc5#file-weather_bot-cr) including whitespace. What surprised me was that RAM consumption grew over time. ![Graph of memory usage during various experiments](https://canada1.discourse-cdn.com/flex036/uploads/crystal_lang/original/2X/7/73cc4a054207eb6ef56533db8a3893a30ee3882d.png) The first two lines show linear memory growth. I restarted the app and it happened again. Then I tried a few more experiments: - Replace SQLite with Redis - Replace the RSS feed parser with using stdlib `XML` directly - Remove all XML parsing and just use regexes (yes, [I know](https://stackoverflow.com/a/1732454), don’t worry about it) - Compile the app using `--mcpu` for the specific aarch64 CPUs I’m using on GCP - Cap memory usage to 20MB so it’ll just restart when it exceeds it - Use a persistent HTTP connection The first 4 didn’t do anything useful. Capping memory usage actually caused an additional problem — it caused the app to crash _after_ persisting that it was posting to Mastodon but _before_ actually posting. The punchline is in the image above, but after almost 24 hours using a persistent HTTP connection ([with a connection pool](https://github.com/jgaskins/http_client)) for both the RSS feed and the Mastodon API, memory usage has tapered at about 5MB. This is the only thing that kept RAM stable at single-digit MB. That makes me wonder if there’s something in OpenSSL (either in the version of OpenSSL itself that’s included in [the 84codes Alpine-based container images](https://hub.docker.com/r/84codes/crystal/tags) or in the Crystal bindings for it) that’s causing it to leak memory. I’ve only ever seen this kind of memory growth in [Ktistec](https://github.com/toddsundsted/ktistec), which is why I thought it might be something in SQLite but it happening when running one-shot HTTP requests would also track. Regardless of the reason, if you want to minimize memory consumption in an app that talks to other HTTP services, reusing HTTP connections will do that. ---
**Author:** ![Blacksmoke16](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/blacksmoke16/32/1241_2.png) [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)\ **Post date:** [July 15, 2025, 8:27pm UTC](https://forum.crystal-lang.org/t/memory-usage-of-http-one-shot-vs-maintaining-a-persistent-connection/8237/2 "2025-07-15T20:27:06Z")
Using the latest Crystal version? Does remind me of like [Fix memory leak in `OpenSSL::SSL::Socket#peer_certificate` by compumike · Pull Request #13785 · crystal-lang/crystal · GitHub](https://github.com/crystal-lang/crystal/pull/13785), but that was quite a while ago at this point. ---
**Author:** ![ysbaddaden](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/ysbaddaden/32/2252_2.png) [@ysbaddaden](https://forum.crystal-lang.org/u/ysbaddaden)\ **Post date:** [July 15, 2025, 9:05pm UTC](https://forum.crystal-lang.org/t/memory-usage-of-http-one-shot-vs-maintaining-a-persistent-connection/8237/3 "2025-07-15T21:05:42Z")
@jgaskins We recently identified a memory leak with the OpenSSL library in the Alpine docker images. Removing the openssl packages and compiling it from source “fixes” the issue for us: ``` apk del openssl-dev openssl-libs-static apk add curl perl linux-headers curl -Ls "https://github.com/openssl/openssl/releases/download/openssl-3.3.3/openssl-3.3.3.tar.gz" | tar xz cd openssl-3.3.3 ./Configure --openssldir=/etc/ssl && make -j ``` Then: ``` PKG_CONFIG_PATH=$PWD/openssl-3.3.3 crystal build ... ``` We haven’t been able to define what’s at fault, even after lots and lots of debugging sessions. Some SSL free methods don’t are called but don’t do some internal calls or something. Maybe a configure feature in the openssl package that doesn’t work well with something we do in the Crystal bindings? ---
**Author:** ![jgaskins](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jgaskins/32/2449_2.png) [@jgaskins](https://forum.crystal-lang.org/u/jgaskins)\ **Post date:** [July 15, 2025, 9:49pm UTC](https://forum.crystal-lang.org/t/memory-usage-of-http-one-shot-vs-maintaining-a-persistent-connection/8237/4 "2025-07-15T21:49:37Z")
> [@Blacksmoke16](#): > > Using the latest Crystal version? Yep. From my `Dockerfile`: ```Dockerfile FROM 84codes/crystal:1.16.3-alpine AS builder ``` > [@ysbaddaden](#): > > We recently identified a memory leak with the OpenSSL library in the Alpine docker images. Removing the openssl packages and compiling it from source “fixes” the issue for us Fascinating. Since I’m using the 84codes container images (because I need aarch64 support), I’ll open an issue on [their repo](https://github.com/84codes/crystal-packages). Thanks! ---
**Author:** ![luislavena](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/luislavena/32/572_2.png) [@luislavena](https://forum.crystal-lang.org/u/luislavena)\ **Post date:** [July 16, 2025, 10:34am UTC](https://forum.crystal-lang.org/t/memory-usage-of-http-one-shot-vs-maintaining-a-persistent-connection/8237/5 "2025-07-16T10:34:46Z")
> [@ysbaddaden](#): > > Maybe a configure feature in the openssl package that doesn’t work well with something we do in the Crystal bindings? Did you folks manage to create a reproduction repository/app to see the issue? Looking at how Alpine compiles the package [here](https://gitlab.alpinelinux.org/alpine/aports/-/blob/3.22-stable/main/openssl/APKBUILD#L167-187) it definitely changes some parameters versus your local configure I would love to see if that is reproducible between 3.3.4 and 3.5.x builds from Alpine. Cheers. ---
**Author:** ![ysbaddaden](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/ysbaddaden/32/2252_2.png) [@ysbaddaden](https://forum.crystal-lang.org/u/ysbaddaden)\ **Post date:** [July 16, 2025, 9:27pm UTC](https://forum.crystal-lang.org/t/memory-usage-of-http-one-shot-vs-maintaining-a-persistent-connection/8237/6 "2025-07-16T21:27:29Z")
It’s easy to replicate: start a HTTPS server in Crystal in an Alpine Docker image, configure a GC loop (see below), start it and make a bunch of HTTPS requests, let the GC loop collect the memory, and repeat again. The GC memory should always settle to the same amounts (roughly) but the RES/VIRT memory of the process should keep growing slightly. ```crystal spawn do loop do sleep 2.seconds 10.times { GC.collect } p GC.stats end end ``` ---
**Author:** ![ysbaddaden](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/ysbaddaden/32/2252_2.png) [@ysbaddaden](https://forum.crystal-lang.org/u/ysbaddaden)\ **Post date:** [July 16, 2025, 9:29pm UTC](https://forum.crystal-lang.org/t/memory-usage-of-http-one-shot-vs-maintaining-a-persistent-connection/8237/7 "2025-07-16T21:29:19Z")
NOTE: keeping the _same_ crystal executable, but swapping the `libssl.so` and `libcrypto.so` from the alpine package to the custom build (same version, of course) with `LD_LIBRARY_PATH` fixes the leak: the RES/VIRT memory always returns to the same rough amount.