Sharing VirtualFs: A virtual filesystem for Crystal.
I built it because I needed it for another, bigger project I’ve been working on for years. It lets you mount folders, archives, and in-memory stores into one merged tree, then read through stdlib-style File/Dir/IO APIs. Layer patches by priority, read byte ranges without loading whole files, and serve files over HTTP.
It ships with a custom .vpak archive format built for fast random access: content-defined chunking, per-chunk compression, deduplication, and optional AES-256-GCM or ChaCha20-Poly1305 encryption.
Feedback, bug reports, and contributions are welcome.
I’d love to have some kind of file-system abstraction in the standard library in order to easily use custom file systems across shards.
What’s the motivation for inventing a new archive format?
An existing standard such as squashfs for example is battle tested, has tooling readily available and can be easily used in other environments.
Fair question — squashfs is the right tool for “compressed read-only filesystem mounted at the OS level.” vpak targets a different niche: in-process asset packaging for Crystal apps, where the following matter more than OS mountability.
Why a new format:
Cross-file dedup via content-defined chunking (FastCDC). Cut points are derived from the content itself via a rolling hash, so identical regions in two different files — even at different offsets — produce identical chunks and dedup. Squashfs dedups fixed-size blocks, so identical content at misaligned offsets doesn’t match. Practical case: a texture atlas and a sprite sheet sharing an alpha mask, or repeated boilerplate across many small files.
Range reads with partial decompression + decryption. Each chunk is compressed and encrypted independently, so a byte-range read only touches the overlapping chunk(s). Squashfs has block-level random access for decompression but no encryption layer — “decrypt only the slice I need” doesn’t exist.
Per-entry AEAD encryption (AES-256-GCM / ChaCha20-Poly1305) built into the format. Squashfs has none; you’d bolt on LUKS, which is block-device-level, not per-asset.
Priority mounting / shadowing. base.vpak + patch.vpak + dlc.vpak mount into one merged tree, higher priority shadows per-file. A VFS concern the format is designed around, not a separate layer on top.
Crystal-native, zero FFI. Plugs into stdlib File/Dir/IO, supports mmap, async/prefetch, and custom codecs registered in-process. squashfs from Crystal means FFI to libsquashfs or shelling out — and losing those hooks.
App-level per-entry metadata (mime, tags, key/value attrs, version) beyond POSIX stat. The package doubles as the asset manifest.
Honest tradeoff: on Linux, if you just want a compressed ROFS you can mount, use squashfs — kernel-backed, decades of tooling. vpak trades OS-level mountability and that ecosystem for the asset-packaging feature set above plus no native dependencies. Different niche, not a replacement.
One note: the VFS layer already mounts .zip and .tar as container drivers. A squashfs reader would fit the same seam if anyone wants OS-portable squashfs read access through the merged-tree API — contributions welcome.
