Hello,
Question
It appears I am using spawn incorrectly (please see the code below).
Why am I getting the “Error opening file…Too many open files error”
Can someone please help? Thank you. The folder that is giving me trouble has 83 files in it. That does not seem enough to run into file descriptors limit problem, or am I wrong. ulimit -n shows 256 on my Mac.
Relevant detail
I am trying to run a crystal script that uploads some local files to
digital ocean spaces (which is like Amazon S3). The command is:
./msbcli uploadfiles --dest images --src /Users/username/images/testfolder
msbcli is my crystal lang script that calls aws command line tool to do the job.
The command worked great for some folders. However for one folder that contain 83 files, I get the following errors:
Unhandled exception in spawn: Could not create pipe: Too many open files (Errno)
Unhandled exception in spawn: Error opening file ‘/dev/null’ with mode ‘r’: Too many open files (Errno)
Unhandled exception in spawn: Error opening file ‘/dev/null’ with mode ‘r’: Too many open files (Errno)
Unhandled exception in spawn: Error opening file ‘/dev/null’ with mode ‘r’: Too many open files (Errno)
Unhandled exception in spawn: Error opening file ‘/dev/null’ with mode ‘r’: Too many open files (Errno)
.
.
.
many lines like the above followed by some successful upload messages.
.
.
hangs up here (probably because channel.send or channel.receive is blocking after the above errors.)
Relevant Code
The code that the above command calls is:
def put_multiple_local_files(path_to_dir : String,
key_prefix : String,
ext_incl : String = "",
ext_excl : String = "",
permission : String = "public-read")
channel = Channel(String).new
num_files = 0
Dir.each_child(path_to_dir) do |file|
num_files += 1
spawn do
res = put_local_file(File.join(path_to_dir, file), key_prefix)
channel.send(res.to_json)
end
end
num_files.times do |_|
val = channel.receive
puts val
end
end
The put_local_file function is:
def put_local_file(local_file : String, key_prefix : String, filename : String = "", permission : String = "public-read")
if filename == ""
content_type = MIME.from_filename?(local_file) || "application/octet-stream"
fname = File.basename(local_file)
else
content_type = MIME.from_filename?(filename) || "application/octet-stream"
fname = filename
end
cmd = %(aws s3api put-object \
--bucket #{BUCKET_NAME} \
--profile #{PROFILE} \
--endpoint-url #{ENDPOINT_URL} \
--key #{key_prefix}/#{fname} \
--body #{local_file} \
--acl #{permission} \
--content-type #{content_type})
stdout = IO::Memory.new
stderr = IO::Memory.new
status = Process.run(cmd, shell: true, output: stdout, error: stderr)
if status.success?
## on success, the command returns a json object with ETag (the MD5-hash)
## of the file. The ETag has extra double quotes in it, which need to
## be stripped to get the MD5-hash.
output = JSON.parse(stdout.to_s)["ETag"].as_s
result = output.gsub(/\"/, "")
return {type: "success", msg: "#{fname} uploaded to #{key_prefix}.", result: result}
else
return {type: "error", msg: stderr.to_s, result: ""}
end
end
. But thanks, I try 
.