So, anyone remembers this project? It has been heavily rewritten and now it’s called “Patchy”
Is no longer a simple file uploader as I plan to add Abuse reports and a proper Admin panel and other features to make moderation of the content way easier instead of having to delete the files manually when something undesirable is uploaded.
I still need to make some improvements like limiting how many concurrent accesses can be made to a single file to prevent DoS attacks due to I/O exhaustion, so if a specific file ID is being retrieved at more than 100 req/s, we just limit the access to that specific file, from all IP addresses; Maybe not the best approach since this kind of rate limit can be done per IP address at the reverse proxy level like NGINX, but generally, people that do DDoS attacks to services use more than 1 IP address, so I think it’s better to restrict the access to the whole file temporarily.
The I/O exhaustation doesn’t happen with files that are cached since they are saved on ram of course.
And talking of cache, I also implemented a LRU cache which caches certain files depending of the size of them on memory.
It doesn’t make things any more faster if file cached is being retrieved just a few times, but I did it in case a file is accessed hundreds/thousand of times.
Here are some benchmarks with the cache using a ~498KiB file ;)
No cache, file being read directly from a SSD spamming openat syscalls
$ wrk -c 100 http://localhost:10006/-/file/vZ
Running 10s test @ http://localhost:10006/-/file/vZ
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 35.20ms 16.82ms 100.25ms 73.27%
Req/Sec 1.42k 173.58 2.17k 76.00%
28296 requests in 10.02s, 13.46GB read
Requests/sec: 2823.61
Transfer/sec: 1.34GB
$ wrk -c 100 http://localhost:10006/-/file/vZ
Running 10s test @ http://localhost:10006/-/file/vZ
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 18.87ms 3.08ms 39.52ms 83.04%
Req/Sec 2.60k 252.05 3.02k 67.50%
51913 requests in 10.04s, 24.61GB read
Requests/sec: 5168.75
Transfer/sec: 2.45GB
Around 58% more req/s while using Cache. This may be useful for people that have a lot of RAM on their servers and serve a lot of files so that is why I did that in the first place ;3
The old code that existed before I renamed it to “patchy” was awful, I didn’t use struct nor classes at all, just, Tuples and some NamedTuples, it was horrible, lol.