A curl like tool for file streaming

Community
1

It is my second app in Crystal and it is still just a rough draft. I would like to co-lab with other experts (to give hints and ideas) and beginners to learn and code!

The need for this tool came from trying to upload a larger amount of files to a Function App endpoint in Azure. The idea was that I should use find to feed a list of files (many thousands of files) to a script that would consume those filenames and upload them to the endpoint. Somewhat reliable or at least do reasonable error control.

It started out in Iteration 1 like shown below. Note that the script should be receiving a list of files from say find or similar:

#!/bin/sh
. ./.env
while read -r NAME
do
	curl -XPOST -H'content-type:application/json' "$URL" -d "@$NAME"
done

and that eventually turned into this monster:

#!/bin/sh
. ./.env
set --
while read -r NAME
do
	set -- "$@" "$NAME"
	if test $# -ge 100
	then
		FL=$(printf "%s\n" "$@" | ~/bin/join -d,)
		curl --http2 -Z -XPOST -H 'content-type:application/json' -T "{$FL}" "$URL" &
		set --
	fi
	if test $(jobs | wc -l) -ge 8
	then
		wait -n
	fi
done

if test $# -gt 0
then
	FL=$(printf "%s\n" "$@" | ~/bin/join -d,)
	curl --http2 -Z -XPOST -H'content-type:application/json' -T"{$FL}" "$URL"
	set --
fi

That’s round about when I considered “I should maybe make a tool for this. This should not be this hard.”. Note that I am not doing proper error control in this script yet either and it requires a custom CLI tool called join (that can be found here: https://github.com/cli-tools/join)