Alys, a memory allocation tracer for Crystal

[refi64]

https://sr.ht/~refi64/alys/

Alys is a project I’ve been working on-and-off for a few months now, originating from some discussion on the Lucky Discord on finding the source of a memory “leak”. It’s in EARLY ALPHA STATE, but it’s quite easy to plug into an application and generate tracing events, which can then be converted into a custom JSON format or Go’s pprof format (which can be viewed in an interactive web UI via the pprof tool).

Note that it has only been tested on Linux thus far!

As a quick preview, this is what it looks like loading the pprof web UI for tracing data generated from Lucky’s website application:

I would appreciate any feedback and thoughts on real-world usage!

EDIT: Also worth noting this is NOT multithreading-safe! I plan on moving some logic to a separate thread, which will probably improve perf but also make room for preview_mt support.

[rogerdpack]

How does it work?

[refi64]

Right now it just overrides the GC allocation/deallocation functions to also take a backtrace and write info to a file, not too exciting.

[bararchy]

I just found out about this amazing shard and will be testing it soon, thank you for the amazing work!

[refi64]

It’s been quite a while, but Alys 0.2.0 is now released! There are quite a few highlights:

MessagePack traces

The .alys format is now based on MessagePack. This can result in
massive decreases in file size, largely thanks to variable-length
integer encoding.

Sampled events via ALYS_SAMPLE_INTERVAL

On large applications, writing an event on every single allocation is
slow and can result in massive, hard-to-inspect trace files. Now, you
can set ALYS_SAMPLE_INTERVAL=bytes:N to only write events for every
N bytes allocated, trading granularity for significantly better
performance and a smaller file size.

PIE executable support for address-only backtraces (ELF only)

On systems that use ELF binaries, address-only backtraces will now embed
information in the trace file that allows alys_converter to symbolize
them regardless of the executable’s base address. (Support for
macOS/Mach-O will come in the future.)

Significant alys_converter performance improvements

alys_converter now uses a single llvm-symbolizer process for all
its symbolization, which can result in performance improvements of over
8x.

Flamegraphs (and folded stacks)

alys_converter can now output folded stacks (primarily used for
feeding into various flamegraph tools) and flamegraphs
(via inferno-flamegraph).

Here’s a screenshot of a flamegraph taken from running the lucky website with a sample interval of 4KiB:

ALYS_BACKTRACE_TYPE=addr is the default (ELF only)

Thanks to all of the above changes, making alys_converter
significantly faster and usable with PIE binaries,
ALYS_BACKTRACE_TYPE=addr is now the default trace type on ELF systems.
The main advantages are the smaller perf hit and trace file size, as
well as alys_converter’s symbolization giving significantly better
results.