Serve static files over HTTP (kemal)

Hello! I’m new to crystal (coming from go) and I’m trying to statically compile files into my webserver (kemal) using https://github.com/schovi/baked_file_system

I’m using

class StaticFiles
  extend BakedFileSystem

  bake_folder "../public"
end

Now i’m wondering if there is an easy way to plug this into kemal, in go i would use https://github.com/gobuffalo/packr their box struct implements the http.FileSystem interface thus allowing me to plug it directly into my webserver, is there something similar i can do in crystal?

Any help is appreciated :smiley:

Hey, welcome to Crystal!

There is no ready-to-use solution for this, but it should be really easy to adapt HTTP::StaticFileHandler to load files from the baked file system. It’s just a couple of calls to File that need to be mapped to StaticFiles.

At one point it would be great to have this kind of file system access plugable, so you can either use a local file system, or some kind of virtual file system, or a merge between several ones (like overlayFS). But that’s ideas for the future…

1 Like

I created a handler for it a while back:

# This is a handler to serve files from a file system that
# is coupled with the executable, essentially serving files from
# it, using https://github.com/schovi/baked_file_system
class BakedFileSystemHandler
  include HTTP::Handler

  def initialize(@files : BakedFileSystem)
  end

  def call(context)
    return call_next context if context.request.method != "GET"
    path = context.request.path.to_s.sub(/^\//, "")
    serve context, @files.get(path)
  rescue ::BakedFileSystem::NoSuchFileError
    serve context, @files.get("index.html")
  end

  def serve(context, file)
    mime_type = file.name.ends_with?(".js") ? "application/javascript" : file.mime_type
    context.response.content_type = mime_type + "; charset=utf-8"
    context.response.content_length = file.size
    context.response.write file.to_slice
  end
end

This was for an earlier version of Crystal so it might not work, if not it could be probably updated and it has a fallback to index.html which you might not need.

Thanks so much! I’ll use that @gdotdesign, might be worth PR’ing BakedFileSystem as i think this should be something included.

Though long term, i think it would be nice if HTTP::StaticFileHandler used an abstraction instead of a string to serve files from, in go we have http.Dir that is a wrapper around a string but it adds the Open(name string) (File, error) method, i think something similar would be great to have in crystal, here’s my attempt at translating the same abstraction we have in go.

# https://godoc.org/net/http#FileSystem
abstract class HTTP::FileSystem
  abstract def open(path : String) : IO
  end
end

# https://godoc.org/net/http#Dir
class HTTP::Dir < HTTP::FileSystem
  def open(path : String) : IO
    # ...
  end
end

Then HTTP::StaticFileHandler could work like this

class HTTP::StaticFileHandler
    def initialize(@fs : HTTP::FileSystem, fallthrough = true, directory_listing = true)
    @fallthrough = !!fallthrough
    @directory_listing = !!directory_listing
  end
end

I’m sure my implementation is wrong, but i hope you get the idea.

I’m going to move the HTTP::Dir and HTTP::FileSystem suggestion over to #crystal-contrib

Hi, can i know how to use that ( BakedFileSystemHandler) in kemal?

1 Like

If we have plan to make HTTP::StaticFileHandler can optional accept normal File or backed file system as a drive or plugin?

Maybe i can try contribute if we have this plan, anyway, i consider hack the Crystal standard library directly is not recommended, instead, if we have a plugins/driver standard,
it better.

Hi, @straight-shoota, I consider there is a more elegant way to achieve this.

I wan working on a new kemal project, it use a layout like this:

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <%= yield_content "title" %>
    <link rel="stylesheet" href="/materialize/css/materialize.min.css" />
  </head>
  <body>
      <%= yield_content "footer" %>
  </body>
</html>

I checked yesterday, i can confirm this solution should works, though, i consider this is not a good solution, because we need hack not only one library files.

I want to compile the css file src/assets//materialize/css/materialize.min.css into binary use backed_file_system, when kemal server is starting, those file can be serve directly.

your’s solution should works, though, i consider this is not a good solution, because we need hack several source code file, e.g.

src/kemal/static_file_handler.cr
src/kemal/helpers/helpers.cr

replace all File IO to with backed_file_system, then it should work. HTTP::StaticFileHandler in Crystal std-lib probably don’t need change.

Anyway, i found a more elegant solution:

  1. save file into binary use backed_file_system,

  2. instead hack source code read from backed_file_system directly, we can recreate those file in the default public/ asset folder when starting kemal sever.

We can achieve this use macro, like this: (may need improve, but it works)

require "baked_file_system"

private macro mount(from, to)
  {% begin %}
    {% root = system("pwd").strip.id %}

    class FileStorage
      extend BakedFileSystem
      bake_folder "{{root}}/{{from.id}}"
    end

    def create_public_assets
      Dir.glob("{{from.id}}/**/*").each do |filename|
        next unless File.file? filename

        target_file_name = filename.sub("{{from.id}}", "{{to.id}}")

        FileUtils.mkdir_p File.dirname(target_file_name) unless File.exists?(target_file_name)

        File.write(target_file_name, FileStorage.get(filename.sub("{{from.id}}/", "")).gets_to_end)
      end
    end
  {% end %}
end

mount "src/assets", "public"
create_public_assets

 ╰─ $ tree src/assets/ public/
src/assets/
└── materialize
    ├── css
    │   └── materialize.min.css
    └── js
        └── materialize.min.js
public/
└── materialize
    ├── css
    │   └── materialize.min.css
    └── js
        └── materialize.min.js

3 directories, 2 files

For my cases, it will save files in src/assets into binary use backed_file_system when build, then, it will read those files from backed_file_system, then them in real file system public folder again when start kemal.

Maybe it be worth to create a shards to make kemal support it.

EDIT:

Sorry, above code not work, i fix it and release it as a shard. check here

I wouldn’t call it elegant if a static webserver who really just needs to serve some data now needs write access to the filesystem just in order to serve files that are already baked into the binary.

1 Like

Yes, i think you are right too, but, AFAIK, for now no other solution can achieve this easily, hack source code directly is really not a good idea too, maybe we should make this plugin, but it hard because need change many things.

BTW: we can configure to mount on /tmp or whatever similiar writeable folder.

If the file contents are in memory, I think it’s a shame that they are written to the file system, then read.

I would suggest looking at middlewares: Kemal - Guide

The idea would be to add a Middleware before the static file handler that checks for the baked filesystem. If found, it reads the contents from memory.

No hacks, no modifying code. And I think that shard you created would be just a Middleware.

1 Like

I have to copy those files into same file position manually where i put the execute binary before i add this shard, it just make this process automation anyway.

Cool, i didn’t realize it, thank you, will check.