File-uploader-crystal: A simple file uploader made on crystal using Kemal

Community
1

Hi. I did an uguu replacement on Crystal. Is pretty straight forward. A simple server where you can upload any media temporarily or permanently (depending on server configuration).

It supports:

  • Random filenames with custom length
  • Files can be retrieved with or without the extension
  • Unix sockets (Although I need to fix the socket permissions in some way or NGINX or any other reverse proxy is not able to access to the server)
  • Deletion link

Special thanks to:

Most of the development was made in less than a week so if you find a bug, just tell me. Feel free to break it too. Suggestions and recommendations are appreciated.

The front-end could be better but I don’t like front-end development at all so I just asked ChatGPT to make it lol.

4 Likes
2

Nice work!

Side note, It’s funny how many base58 shards there are :joy:

And these are just the ones that I know of lol

1 Like
3

Great project, thanks a lot for using Kemal :heart:

1 Like
4

So, anyone remembers this project? It has been heavily rewritten and now it’s called “Patchy”

Is no longer a simple file uploader as I plan to add Abuse reports and a proper Admin panel and other features to make moderation of the content way easier instead of having to delete the files manually when something undesirable is uploaded.

I still need to make some improvements like limiting how many concurrent accesses can be made to a single file to prevent DoS attacks due to I/O exhaustion, so if a specific file ID is being retrieved at more than 100 req/s, we just limit the access to that specific file, from all IP addresses; Maybe not the best approach since this kind of rate limit can be done per IP address at the reverse proxy level like NGINX, but generally, people that do DDoS attacks to services use more than 1 IP address, so I think it’s better to restrict the access to the whole file temporarily.

The I/O exhaustation doesn’t happen with files that are cached since they are saved on ram of course.

And talking of cache, I also implemented a LRU cache which caches certain files depending of the size of them on memory.

It doesn’t make things any more faster if file cached is being retrieved just a few times, but I did it in case a file is accessed hundreds/thousand of times.

Here are some benchmarks with the cache using a ~498KiB file ;)

No cache, file being read directly from a SSD spamming openat syscalls

$ wrk -c 100 http://localhost:10006/-/file/vZ
Running 10s test @ http://localhost:10006/-/file/vZ
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.20ms   16.82ms 100.25ms   73.27%
    Req/Sec     1.42k   173.58     2.17k    76.00%
  28296 requests in 10.02s, 13.46GB read
Requests/sec:   2823.61
Transfer/sec:      1.34GB

$ wrk -c 100 http://localhost:10006/-/file/vZ
Running 10s test @ http://localhost:10006/-/file/vZ
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    18.87ms    3.08ms  39.52ms   83.04%
    Req/Sec     2.60k   252.05     3.02k    67.50%
  51913 requests in 10.04s, 24.61GB read
Requests/sec:   5168.75
Transfer/sec:      2.45GB

Around 58% more req/s while using Cache. This may be useful for people that have a lot of RAM on their servers and serve a lot of files so that is why I did that in the first place ;3

The old code that existed before I renamed it to “patchy” was awful, I didn’t use struct nor classes at all, just, Tuples and some NamedTuples, it was horrible, lol.

3 Likes