Have an alternative to JSON.parse that works on dynamic structures, but statically types the values

1

I think this would be very beneficial for a shard.

To statically type JSON data, a developer can use JSON.mapping or include JSON::Serializable (of which you still need to type out the structure of the JSON data).

Since JSON.parse already exists for dynamic data, I don’t see a reason why we can’t parse dynamic JSON, and have their values be casted to their correct type instead of JSON::Any. This way, the developer doesn’t need to use .as_s, to_i etc all over their code.

Is something like this possible with macros?

2

No. Macros run at compile time. Dynamic JSON data is only available at runtime.

That’s also the reason why this can’t work. The compiler needs to know the structure of the JSON data at compile time in order to apply type-safe mappings. That’s what JSON.mapping and JSON::Serializable do. Without that, parsed JSON can only return a union of possible JSON values, which is implement in JSON::Any. The individual values are actually casted to the matching type, that’s not the issue. But the type of the expression can by any of them.

3

@straight-shoota @oprypin helped with a CSV to dictionary method a few months ago that statically casted values. I think it’s definitely possible to do the same with JSON data. I remember the method used macros

It’s also possible to read the JSON data at compile time. Thus, I’m positive a shard could be created to do this, I’m just not sure how

1 Like
4

Yes, if the JSON structure is available to the compiler this should be possible, as I’ve mentioned. But then there isn’t much point to it because it can’t handle arbitrary JSON data and when the JSON structure is known, your probably better off declaring explicit mappings right away.

5

Take a look at ProtoBuf. There is a shard already for this.

6

Why should you take a look at ProtoBuf, when this is about JSON?

7

Well, because currently it is the only way to achieve strict typing when working with external clients. And as @girng is working on a game server, I assumed that the only client he would have is his own game. Therefore, using ProtoBuf over JSON is a valuable alternative in this case.

1 Like
8

I rather suggest msgpack, which describe itself as similar to JSON.

A shard exists.

There is this example https://github.com/crystal-community/msgpack-crystal/blob/master/examples/pack_unpack.cr, perhaps that’s what you’re looking for.

9

There are literally tons of data formats and you can pick whatever you like for whatever use case. But the OP doesn’t mention any specific use case (so don’t assume any) and specifically asks for JSON. Suggesting alternative formats without any substantial justification is not really helpful and the discussion drifts off with everyone jumping in with their favourite data format.

And of course you can have type safe JSON mapping when you specify the serialization format using JSON.mapping or include JSON::Serializable.

10

What do you mean by “arbitrary JSON data”?
For example, I was looking at the run example here:
https://crystal-lang.org/api/0.27.2/Crystal/Macros.html#run(filename%2C*args)%3AMacroId-instance-method

The file, however, is read at compile time

Surely, this means we could statically type JSON values (dynamically!) with macros?

I wonder, if it’s possible to loop through the JSON data after it’s been parsed, and then statically cast the values. @scott has stated this is not possible however, which brings me back to the “reading the JSON data at compile time is possible, surely there must be a way”.

If we already have a JSON.parse() method, I think a shard that does some magic under the hood, and converts all the JSON::Any types to their statically typed value would work too. Thus, removing the need for the developer to maintain a JSON.mapping, use to_i, as_i, as_i64, as_f, etc.

It would be a lot easier if a developer could just use JSON.parse, and access the values without having to type check them all over their code. If this is never going to be possible, another solution is a developer can specify what keys they want, and their corresponding type. And when JSON.parse runs (or a macro that parses JSON data at compile time), and when a key matches, it is statically casted to that type. Thus, removing the need for type checking. But instead, we have to maintain a huge JSON.mapping for dynamic data and it feels cumbersome.

If there is already JSON.parse (which goes against crystal’s “statically typed language”), then there should be a way for it to be even easier. Right now, IMO, it seems like JSON.parse is in the middle of statically typed and dynamically typed. Which is confusing I think (all the type checking makes my head explode, and makes code look convoluted / verbose)

I’m not against JSON.parse working with a dynamic structure and the developer having type checking methods littered in their code (hell, that’s what I do!). However, a developer should also not be required to use type checking for their JSON data. Which makes life a lot easier, and I’d argue… cleaner code! And most importantly… the JSON data is statically typed!

1 Like
11

Also, I posted this on the forum instead of github because I want to get everyone’s thoughts first. Please don’t hesitate to chime in!

12

But how would you know what to cast each value to if its not predefined?

1 Like
13

If we can read data at compile time, I’m sure we could have a file like json_types_xxx (for that specific json data), and then the macro can read which key is for what type, then while looping through the JSON data at compile time (in a macro), statically cast their values? Am not sure how to really do it myself, but it has to be possible

14

So how is that different than just reading a JSON file into a struct/class using JSON.mapping or JSON::Serializable if you still have to tell it what key is of what type.

1 Like
15

Because you don’t have to write out the structure / maintain it with JSON.mapping / JSON::Serializable. You can just specify the key names and their types to use, and it will work on the JSON data (even if the data is dynamic, because it’s just checking for matching key names recursively, and then the values of those will be statically casted to their corresponding type)

edit: When I say recursively, I mean when traversing through the dynamic JSON data

16

I think it would really help if you could provide the code that you’d want to have (even if it doesn’t work right now). Otherwise, at least for me, I can’t understand exactly what you want.

1 Like
17

@asterite For sure. Let me create an example:

Okay got something: https://play.crystal-lang.org/#/r/6k6c

Please read my comments, starting at line 28, then comment line 17, and use line 18. Then run. You will see more errors and not only that, but it breaks code.

A much better solution would be something like this:
https://play.crystal-lang.org/#/r/6k6g

require "json"

struct Level
  JSON.map_keys({
      amount: Int32,
      mob_id: Int32,
      name: String,
      position: String,
      size: String,
      type: String
  })
end

Levels   = Hash(String, Level).new

Levels["testarea"] = Level.from_json(%({"entities":[{"amount":5,"mob_id":"Null","name":"crate_pack","position":"1053.91,861.24","size":"144.18,45.52","type":null},{"amount":5,"mob_id":"Null","name":"crate_pack2","position":"681.33,916.61","size":"144.18,45.52","type":null},{"amount":5,"mob_id":"9","name":"mob_pack","position":"1352.06,971.27","size":"190.91,90.42","type":"normal"}]}))

Levels["testarea"].entities.each do |entity|
  amount = entity["amount"] # we don't need to do .to_i on this, it's statically typed as an Int32
  # And, if ANY key is shown up in this dynamic JSON data, ANYWHERE (outside of the entities array), it will be
  # statically typed. This way, no matter the dynamic JSON structure, static typing is ensured.
  amount.times do |x|
    puts x
  end
end

Obviously we need to include entities somewhere in the map_keys macro, so the compiler knows it’s an array. But you get the idea

In the example above, the map_keys macro could work with JSON.parse, doesn’t have to be bound within a Struct

The developer shouldn’t have to worry about the JSON structure, they should just explicitly write the keys and their types, and that’s it. That’s all I’m trying to convey

18

I see what you’re trying to do, but you said it yourself

So you still need to specify the structure of the JSON data, and your keys are basically just a JSON.mapping still.

JSON is a structured format, I just don’t see the benefit/difference between this and JSON.mapping for example. You wouldn’t be able to have truly dynamic JSON support since if one structure has a same key as another you would need another mapping to differentiate the keys/types from the one set to another. At that point you’re doing JSON.mapping, mapping a set of structured data to a struct/class.

19

I don’t like type checking and type conversion. Crystal is a STATICALLY typed language for pete’s sake, why am I even messing with them? Just to ultimately have type checking and conversion methods littered all over my code? Makes absolutely zero sense IMO.

Something is definitely not right, I don’t know what it is at this point as I feel like I’m a broken record.

1 Like
20

I don’t think you’re using whats available correctly then.

https://play.crystal-lang.org/#/r/6k6v

Isn’t that what you want? Now you dont have to do any type checking/conversion as each ivar is guaranteed to be the type you give it. Granted you’ll still have to check for nils, but that is a given.