Crystal JSON Lexer is too strict on single unpaired UTF-16 surrogate for skipped fields

Help & Support
1

A little code sample:

require "json"
 
# raw JSON frame with a lone-surrogate ESCAPE (not raw invalid bytes)
raw = %q({"post_data":"before\uda41after","x":1})

puts "raw valid_encoding? #{raw.valid_encoding?}" 

struct X
  include JSON::Serializable
  
  @[JSON::Field(ignore: true)]
  @post_data : String? 

  @x : Int32
end
 
X.from_json(raw)

Output:

raw valid_encoding? true
Unhandled exception: Unterminated UTF-16 sequence at line 1, column 27 (JSON::ParseException)
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/lexer.cr:334:5 in 'raise'
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/lexer.cr:199:11 in 'consume_string_escape_sequence'
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/lexer/string_based.cr:45:18 in 'consume_string_slow_path'
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/lexer/string_based.cr:22:16 in 'consume_string'
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/lexer.cr:64:37 in 'next_token'
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/pull_parser.cr:678:5 in 'next_token'
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/pull_parser.cr:554:14 in 'read_next_internal'
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/pull_parser.cr:410:5 in 'read_next'
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/pull_parser.cr:311:25 in 'read_string'
  from /home/damir/.asdf/installs/crystal/1.19.2/share/crystal/src/json/pull_parser.cr:179:5 in 'read_object_key'
  from play:8:3 in 'initialize:__pull_for_json_serializable'

Actually no way to parse such raw data to get x value without monkey-patch stdlib or scrub data by the hand before parsing. Change the field type or use a converter also does not work, raise is at tokenization, before type handling.

I think when field ignore: true or not inscluded (same lexer.skip used) Lexer can use relaxed token rules, instead of full checking

Browsers’ JSON.parse, Go’s encoding/json, Python’s json, and serde_json all accept lone surrogate escapes in input and decode them to U+FFFD (or keep them, in Go’s and Browser case) rather than aborting.

Browser:

JSON.parse('{"post_data":"before\uda41after","x":1}')
{post_data: 'before\uDA41after', x: 1}

The JSON spec (RFC 8259 sec. 8.2) explicitly notes that unpaired surrogates are allowed to appear and that behavior is implementation-defined; it does not require rejecting the whole document. Crystal is stricter than the ecosystem here.

2

Would you mind posting this as a bug report on Sign in to GitHub · GitHub ?

3

Sure #17390