# Complex JSON deserialization
**URL:** https://forum.crystal-lang.org/t/complex-json-deserialization/6095
**Category:** Help & Support
**Created:** [October 17, 2023, 4:14am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095 "2023-10-17T04:14:48Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@plambert](https://forum.crystal-lang.org/u/plambert)
#### Post date: [October 17, 2023, 4:14am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/1 "2023-10-17T04:14:48Z")
I’m reading a JSON document (and then putting it in a sqlite database). The document is much more complicated than I need to actually track, so I’m trying to figure out how to do two things:
> First question:
There is a list of associated objects (an array of strings) that is buried deep in nested JSON objects that I otherwise don’t need to extract. For example, imagine the document looks like…
```auto
{
"id": 12345,
"name": "Paul",
"complicated": {
"object": {
"nested": {
"deeply": [
"a",
"b",
"c"
]
}
}
}
}
```
And my class looks like:
```auto
class Thing
include JSON::Serializable
property id : Int64
property name : String
property nested_list : Array(String)
# ...
end
```
Since the array is nested more than one object deep, is there any way to use JSON::Serializable? Or am I going to have to write a custom `from_json` function?
Or… is there a way I can have JSON::Serializable trigger on the “complicated” key, and call a custom method to decode it?
> Second question:
I would really like my class to have a property, `raw_json`, which is a String containing the full raw JSON from which it was decoded.
Is this possible with JSON::Serializable in any way?
Using the example above:
```auto
class Thing
include JSON::Serializable
property id : Int64
property name : String
property nested_list : Array(String)
property raw_json : String
end
json_string="{"id":12345,"name":"Paul","complicated":{"object":{"nested":{"deeply":["a","b","c"]}}}}"
thing=Thing.from_json(json_string)
thing.raw_json == json_string # => true
```
I know that’s likely to be difficult; I know instead that I can make it nilable, and assign it after creating the object. For example:
```auto
class Thing
def self.new(json_string : String)
thing = Thing.from_json(json_string)
thing.raw_json=json_string
thing
end
end
```
But it would be nice if I could do it inside the `Thing.from_json` class method, since then I would be able to parse it when it’s embedded in other documents.
I hope these questions are clear enough that someone can make some suggestions. The first question is definitely more important than the second…
Thanks!
Paul
---
### Author:  [@sablokgaurav](https://forum.crystal-lang.org/u/sablokgaurav)
#### Post date: [October 17, 2023, 4:53am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/2 "2023-10-17T04:53:00Z")
i started crystal a little time ago but coded in ruby. In ruby you can directly call the pandas bindings as require “Pandas” and then you can call the function. pd.from\_json(“json\_file”) and it will give you a direct normalized frame on which you can put any iteration.
---
### Author:  [@plambert](https://forum.crystal-lang.org/u/plambert)
#### Post date: [October 17, 2023, 5:04am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/3 "2023-10-17T05:04:37Z")
Thanks for the response.
I think Pandas is a python library, is it not? And the ruby “pandas” module calls the python code? I can’t embed python and the pandas library in my program, and I’m not a python programmer.
I apologize for not seeing what you’re suggesting that I do. Can you explain it a different way, so that I might be able to understand?
Thanks again!
Paul
---
### Author:  [@sablokgaurav](https://forum.crystal-lang.org/u/sablokgaurav)
#### Post date: [October 17, 2023, 5:08am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/4 "2023-10-17T05:08:20Z")
Yes pandas is a python library but with the bindings you can call in ruby also. This is what we calling in the first line require “Pandas” and this will attach the pandas bindings and then you can call those functions. If you want a crystal based compiler then attach the polars as for that also there is a ruby binding. see here, you can also install the gem directly and call the same. [GitHub - mrkn/pandas.rb: Pandas wrapper for Ruby](https://github.com/mrkn/pandas.rb)
---
### Author:  [@sablokgaurav](https://forum.crystal-lang.org/u/sablokgaurav)
#### Post date: [October 17, 2023, 5:10am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/5 "2023-10-17T05:10:51Z")
if you just want to put the data into a sqlite database then call the jq . like this with the backticks with in the crystal function and it will parse the json and will put much easier. A single line of jq will solve and will remove all those iterations and back tracking the for loops.
---
### Author:  [@npn](https://forum.crystal-lang.org/u/npn)
#### Post date: [October 17, 2023, 8:28am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/6 "2023-10-17T08:28:15Z")
Pretty sure you can use root for that. Check out the api document.
---
### Author:  [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)
#### Post date: [October 17, 2023, 8:45am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/7 "2023-10-17T08:45:48Z")
> [@plambert](#):
>
> Since the array is nested more than one object deep, is there any way to use JSON::Serializable? Or am I going to have to write a custom `from_json` function?
>
> Or… is there a way I can have JSON::Serializable trigger on the “complicated” key, and call a custom method to decode it?
Yes you can do the latter with custom converter for this particular, nested field.
```crystal
@[JSON::Field(key: "complicated", converter: Thing::NestedArrayConverter)]
property nested_array : Array(String)
module NestedArrayConverter
def self.from_json(pull)
pull.on_key!("object") do
pull.on_key!("nested") do
pull.on_key!("deeply") do
return Array(String).new(pull)
end
end
end
end
end
```
I suppose it would be nice if the `root` property of `JSON::Field` could receive a list of keys instead of a single key. (see [`root` property of `JSON::Field` should support nested keys · Issue #13894 · crystal-lang/crystal · GitHub](https://github.com/crystal-lang/crystal/issues/13894))
---
### Author:  [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)
#### Post date: [October 17, 2023, 8:50am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/8 "2023-10-17T08:50:26Z")
For the `raw_json` property I don’t think there is any way to make `Serializable` do that for you. The JSON parser operates on a stream and only consumes that stream once. So you’ll have to duplicate the contents outside of the deserialization logic.
Instead of `.new` you can override `.from_json` for this:
```auto
class Thing
def self.from_json(string : String)
super.tap do |thing|
thing.raw_json = string
end
end
end
```
Note that this won’t cover the signature `Thing.from_json(IO)`.
---
### Author:  [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)
#### Post date: [October 17, 2023, 1:10pm UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/9 "2023-10-17T13:10:22Z")
For one at least, Athena Serializer supports this. Could be an option until the stdlib has support:
```crystal
class Thing
include ASR::Serializable
property id : Int64
property name : String
@[ASRA::Accessor(path: {"complicated", "object", "nested", "deeply"})]
property nested_list : Array(String)
end
ASR.serializer.deserialize Thing, DATA, :json
# => #
```
---
### Author:  [@hugopl](https://forum.crystal-lang.org/u/hugopl)
#### Post date: [October 29, 2023, 3:39am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/10 "2023-10-29T03:39:26Z")
> [@straight-shoota](#):
>
> Yes you can do the latter with custom converter for this particular, nested field.
>
> ```auto
> @[JSON::Field(key: "complicated", converter: Thing::NestedArrayConverter)]
> property nested_array : Array(String)
>
> module NestedArrayConverter
> def self.from_json(pull)
> pull.on_key!("object") do
> pull.on_key!("nested") do
> pull.on_key!("deeply") do
> return Array(String).new(pull)
> end
> end
> end
> end
> end
>
> ```
This make me think that I wont be the only one that could find [my imcomplete proposal to stdlib](https://forum.crystal-lang.org/t/let-json-pullparser-on-key-work-with-nested-keys/6077) useful.
The above code would be
```auto
@[JSON::Field(key: "complicated", converter: Thing::NestedArrayConverter)]
property nested_array : Array(String)
module NestedArrayConverter
def self.from_json(pull)
pull.on_key!("object", "nested", "deeply") do
return Array(String).new(pull)
end
end
end
```
---
### Author:  [@girng](https://forum.crystal-lang.org/u/girng)
#### Post date: [November 1, 2023, 5:25am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/11 "2023-11-01T05:25:57Z")
a trick i do is just parse the data with `JSON.parse` so it handles all the deserialization/nested objects automatically. then use `raw.to_xx` on them and put that data in a class or wherever i need. this way you don’t need to fiddle around and keep annotations or other hard-coded code synced
i remember suggesting a way to handle this years ago where you could specify the keys and their types, and let crystal convert the JSON.parse result (JSON::Any) to their static types
in fact, JSON.parse is extremely underrated for what it does, no reason we can’t harness what it returns and put static types on those results dynamically. just not sure how, but it would def help clear up all the dynamic JSON problems we’ve been running into for years
---
### Author:  [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)
#### Post date: [November 1, 2023, 12:43pm UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/12 "2023-11-01T12:43:46Z")
> [@girng](#):
>
> i remember suggesting a way to handle this years ago where you could specify the keys and their types, and let crystal convert the JSON.parse result (JSON::Any) to their static types
Isn’t this just what `JSON::Serializable` is?
---
### Author:  [@Devonte](https://forum.crystal-lang.org/u/Devonte)
#### Post date: [November 1, 2023, 4:44pm UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/13 "2023-11-01T16:44:32Z")
> [@girng](#):
>
> in fact, JSON.parse is extremely underrated for what it does, no reason we can’t harness what it returns and put static types on those results dynamically. just not sure how, but it would def help clear up all the dynamic JSON problems we’ve been running into for years
Agreed, I implemented a similar concept [here](https://github.com/devnote-dev/strict-yaml/blob/ac720cf2aab6198f267f4330e76e2f9253023b67/src/strict-yaml/objects.cr) in an experimental shard (with a `JSON::Any`-style struct defined [here](https://github.com/devnote-dev/strict-yaml/blob/ac720cf2aab6198f267f4330e76e2f9253023b67/src/strict-yaml/any.cr)) and it worked well. There is a minor performance difference between the two though, so I wonder if that would affect decisions for changing the implementation.
```crystal
require "benchmark"
require "json"
class Vector
include JSON::Serializable
getter x : Float64
getter y : Float64
getter z : Float64
def initialize(@x, @y, @z)
end
end
SOURCE = %({"x": 51.4534, "y": 65.89, "z": 32.112})
c = false
x : Vector
y : Vector
Benchmark.ips do |ips|
ips.report("control") { c = false }
ips.report("JSON::Serializable") do
x = Vector.from_json SOURCE
end
ips.report("JSON.parse") do
json = JSON.parse SOURCE
y = Vector.new(json["x"].as_f, json["y"].as_f, json["z"].as_f)
end
end
```
I ran `crystal build --no-debug --release bench.cr` then ran the binaries twice (first just for system warmup, second for results).
### Windows
```auto
control 653.56M ( 1.53ns) (± 6.31%) 0.0B/op fastest
JSON::Serializable 417.11k ( 2.40µs) (± 6.43%) 880B/op 1566.88× slower
JSON.parse 390.84k ( 2.56µs) (± 4.68%) 992B/op 1672.20× slower
```
### WSL (Ubuntu)
```auto
control 638.57M ( 1.57ns) (±11.05%) 0.0B/op fastest
JSON::Serializable 767.70k ( 1.30µs) (± 5.57%) 880B/op 831.80× slower
JSON.parse 742.27k ( 1.35µs) (± 4.77%) 992B/op 860.30× slower
```
That looks like a lot of big numbers, but that’s because of the control variable. For Windows it’s ±26.27k iterations, being ~105.32x slower. For WSL it’s ±25.44 iterations, being ~28.5x slower. I don’t know if there are optimisations that could be made to make `JSON::Any` faster than `JSON::Serializable` within normal constraints.
---
### Author:  [@HertzDevil](https://forum.crystal-lang.org/u/HertzDevil)
#### Post date: [November 1, 2023, 5:10pm UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/14 "2023-11-01T17:10:43Z")
> [@Devonte](#):
>
> For Windows it’s ±26.27k iterations, being ~105.32x slower. For WSL it’s ±25.44 iterations, being ~28.5x slower.
That’s not how you interpret the ratios when a control is present, since the denominator is not in terms of the control. On Windows it is `(1672.20 - 1) ÷ (1566.88 - 1) = 1.067×` slower, on WSL it’s `(860.30 - 1) ÷ (831.80 - 1) = 1.034×` slower. The control time differs by 3 orders of magnitude so it is not very useful here.
---
### Author:  [@Devonte](https://forum.crystal-lang.org/u/Devonte)
#### Post date: [November 1, 2023, 5:35pm UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/15 "2023-11-01T17:35:21Z")
Oh right 🤦 Thanks for that, given that’s such a small difference I guess the change could be made then? For nested keys `JSON::Any#dig?` could be taken advantage of.
Also, is there a guide for calculations and using control variables in specs?
---
### Author:  [@Blacksmoke16](https://forum.crystal-lang.org/u/Blacksmoke16)
#### Post date: [November 1, 2023, 6:56pm UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/16 "2023-11-01T18:56:32Z")
> [@Devonte](#):
>
> Also, is there a guide for calculations and using control variables in specs?
Related: [Iterations-per-second benchmarks with control jobs / `#before_each` · Issue #13284 · crystal-lang/crystal · GitHub](https://github.com/crystal-lang/crystal/issues/13284)
---
### Author:  [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)
#### Post date: [November 1, 2023, 7:27pm UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/17 "2023-11-01T19:27:30Z")
> [@Devonte](#):
>
> I guess the change could be made then?
What change do you refer to?
---
### Author:  [@Devonte](https://forum.crystal-lang.org/u/Devonte)
#### Post date: [November 1, 2023, 7:36pm UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/18 "2023-11-01T19:36:04Z")
Changing `JSON::Serializable` to use `JSON::Any`. It would provide a solution for the original post and solve a lot of dynamic JSON issues with the module, as pointed out by @girng. Then again, it may be easier to just add on top of the pull parser code to handle these cases.
---
### Author:  [@hugopl](https://forum.crystal-lang.org/u/hugopl)
#### Post date: [November 1, 2023, 8:14pm UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/19 "2023-11-01T20:14:24Z")
In a project of mine I remember I did a macro to build Objects from `JSON::Any`, like `JSON::Serializable` does with a `PullParser`. But in the end I was in doubt if this approach was just a result of a bad design of my code used to read the complex JSON or if it was really necessary and a good approach.
>
---
### Author:  [@straight-shoota](https://forum.crystal-lang.org/u/straight-shoota)
#### Post date: [November 2, 2023, 11:57am UTC](https://forum.crystal-lang.org/t/complex-json-deserialization/6095/20 "2023-11-02T11:57:47Z")
> [@Devonte](#):
>
> Changing `JSON::Serializable` to use `JSON::Any`.
I’m not sure that would be a good idea. `JSON::Any` needs to be read entirely into memory before being usable. The pull parser processes data as a stream, enables custom converters and skipping over portions of the document that are irrelevant.
That’s a very different approach resulting in many efficiency benefits. This is certainly noticable when handling non-trivial JSON documents.
[Next page](https://forum.crystal-lang.org/t/complex-json-deserialization/6095.md?page=2)