Process CSV from pipe

Help & Support
1

Not sure if I’m doing something wrong here but given the following program:

require "option_parser"
require "csv"

module Tail
  VERSION = "0.1.0"

  src = STDIN
  process_csv = false

  OptionParser.parse do |parser|
    parser.on("-f path", "file path to process") { |path| src = File.new(path) }
    parser.on("-c", "process with CSV") { process_csv = true }
  end

  if process_csv
    CSV.new(src) do |csv|
      pp! csv.row.to_a
    end
  else
    src.each_line do |l|
      puts l
    end
  end
end

If I have the following in t.csv:

a,b,c
d,e,f
e,f,g
f,g,h
g,h,i
h,i,j

Running the program outputting each line (no csv parsing) gives this:

[~/git/tail]$ tail -f t.csv | bin/tail
a,b,c
d,e,f
e,f,g
f,g,h
g,h,i
h,i,j
^C

But in ‘csv’ mode I don’t get the last line:

[~/git/tail]$ tail -f t.csv | bin/tail -c
csv.row.to_a # => ["a", "b", "c"]
csv.row.to_a # => ["d", "e", "f"]
csv.row.to_a # => ["e", "f", "g"]
csv.row.to_a # => ["f", "g", "h"]
csv.row.to_a # => ["g", "h", "i"]
^C

Of course just reading the file in csv mode is fine:

[~/git/tail]$ bin/tail -f t.csv -c
csv.row.to_a # => ["a", "b", "c"]
csv.row.to_a # => ["d", "e", "f"]
csv.row.to_a # => ["e", "f", "g"]
csv.row.to_a # => ["f", "g", "h"]
csv.row.to_a # => ["g", "h", "i"]
csv.row.to_a # => ["h", "i", "j"]

Is this expected behaviour? In case it matters, this is crystal 1.1.1 on MacOS 11.5.2.

Thanks,

Steve

2

Hi!

This is because when the CSV lexer finds a newline, it checks if there are more chars afterwards. If not, it produces an EOF token instead of a newline. However, tail won’t give that next char after newline so the lexer blocks and doesn’t even return the newline token.

Please feel free to open an issue for this. I’m not sure how it can be fixed without breaking backwards compatibility.

3

Actually, even if it’s a minor breaking change, I think it’s fine. Instead of EOF produced in this case, NEWLINE followed by EOF will be produced, which is almost the same.

1 Like