Open, read and write binary into a file

Hello, today I tried to understand how to open a file to read or write binary data into it. I would like to save a class (an object) into a file, and read it after.

It’s a part of my code:


        def checkSoftwareDatabase
            if !Dir.exists?(ISM::Default::Path::SoftwaresDirectory)
                Dir.mkdir(ISM::Default::Path::SoftwaresDirectory)
            end

            if File.exists?(ISM::Default::Path::DatabaseFile)
                file = File.open(ISM::Default::Path::DatabaseFile,"rb")
                @softwares = file.read
            else
                list = Dir.entries(ISM::Default::Path::SoftwaresDirectory)

                list.each do |argument|
                    puts argument
                end
            end
        end

In this part, I would like to open a saved class into a file as binary data. (But I don’t know as well how to write object data as binary into a file)

Crystal has no implicit serialization of data types. Their data layout is not specificied and may change between different compilations of a program. While you could technically write the memory representation of a data type to a file, there is no guarantee that you can load it again later.

You can use an explicit serialization format, though. The standard library ships with JSON and YAML serialization libraries. You can find shards (including ones that provide binary serialization) at Data Formats on Shardbox and Network Protocols on Shardbox

It is also possible and often fairly easy not depend on something predefined but instead implement read and write by yourself. For an example, I implement a serialization format defined by a separate project in the Reader and Writer classes of HDR histogram . In that case the header is readable, but there is a blob of binary data that is handled by the Codec class.

I’m sure I saw a post asking about this.

Basically, for saving objects to files", you need:

  • require "json"
  • include ::JSON::Serializable (or yaml instead; likewise below)
  • def initialize(json_params)...
  • save/load via self.from_json(some_json_string) and self.to_json and save contents of to_json results to file, etc
  • Some things don’t serialize well (e.g.: Symbols), so find alternatives.
  • Performance Note: Writing to file will probably be slower than writing to memory.

See also: Carcin

require "json"

class Foo
  include ::JSON::Serializable # This along with `require "json"` enables `.from_json` and `.to_json` for many classes/structures.
  
    # Symbol don't serialize, but you can use Enum's like:
    enum Widget
      Alpha = 19
      Beta  = 28
      Gamma = 37
    end

    getter bar : Widget

    def initialize(@bar = Widget::Beta)
    # You'll need an initialize that handles all serializable params to be saved/loaded via the json version.
    end
end

# example usage:

foo = Foo.new(bar: Foo::Widget::Gamma)

puts "foo: #{foo}"
puts "foo as json: #{foo.to_json}" # or print to file, etc

foo2 = Foo.from_json(foo.to_json)

puts "foo2: #{foo2}"
puts "foo2.bar: #{foo2.bar}"
puts "foo2.bar as int: #{foo2.bar.to_i}"
puts "foo2.bar's class: #{foo2.bar.class}"
puts "foo2 as json: #{foo2.to_json}"

This one?

1 Like

@Blacksmoke16/@straight-shoota , thanks for helping me find the original post and moving my reply. I was beginning to think maybe the original post was deleted. :slight_smile:

Okay, this look interesting, but can you just provide me an example of a serialization of data in a file with json ? I’m not sure to understand how you can read and write data in json file.

I think I will prefer something like xml

Hey @Zohran, here is a concrete example:

  1. Define your serializable events
  2. Define a method to persist them to IO
  3. Open the file where you persisted your objects and read them up

To summarise, it all boils down to:

  1. Defining an Event super class (or super struct) that includes JSON::Serializable - or the equivalent XML module
  2. Persisting instances of events - e.g. event.to_json(file_io)
  3. Reading events from file with - e.g. when reading a file line by line: Event.from_json(line)

I wrote this article about JSON serialization specifically a while ago, but you might be able to apply the same concepts to other serialization formats - this doesn’t cover reading from and writing to file, though.

3 Likes

Unfortunately, we don’t yet have an XML::Serializable module. As I understand it, the reason comes down to 1) XML is more complicated than JSON, and 2) there hasn’t been enough of a demonstrated need in the Crystal community for this functionality. I’d be happy to help contribute to such a module, but I’ve messed with the idea a bit in the past and it would be a big task.

2 Likes

Finally I came back to JSON xD

Just one thing, I wrote this json file to test, but something is wrong with the syntax, but what?

{
  "name": "Binutils",
  "architectures": ["x86_64"],
  "description": "The GNU collection of binary tools",
  "website": "https://www.gnu.org/software/binutils/",
  "downloadLinks": ["https://ftp.gnu.org/gnu/binutils/"],
  "signatureLinks": ["https://ftp.gnu.org/gnu/binutils/"],
  "options": [
    {
      "name": "Pass1",
      "description": "Build the first pass to make a system from scratch",
    },
    {
      "name": "Pass2",
      "description": "Build the second pass to make a system from scratch",
      "secretIdentity": "Unknown"
    }
  ]
}

I have this error:

 zohran  ~  Documents  Programmation  Test  1  crystal Test.cr 
Unhandled exception: unexpected token '}' at line 12, column 5 (JSON::ParseException)
  from /usr/lib64/crystal/json/parser.cr:125:5 in 'parse_exception'
  from /usr/lib64/crystal/json/parser.cr:121:5 in 'unexpected_token'
  from /usr/lib64/crystal/json/parser.cr:92:13 in 'parse_object'
  from /usr/lib64/crystal/json/parser.cr:35:7 in 'parse_value'
  from /usr/lib64/crystal/json/parser.cr:49:18 in 'parse_array'
  from /usr/lib64/crystal/json/parser.cr:33:7 in 'parse_value'
  from /usr/lib64/crystal/json/parser.cr:85:19 in 'parse_object'
  from /usr/lib64/crystal/json/parser.cr:35:7 in 'parse_value'
  from /usr/lib64/crystal/json/parser.cr:13:12 in 'parse'
  from /usr/lib64/crystal/json.cr:135:5 in 'parse'
  from Test.cr:4:8 in '__crystal_main'
  from /usr/lib64/crystal/crystal/main.cr:110:5 in 'main_user_code'
  from /usr/lib64/crystal/crystal/main.cr:96:7 in 'main'
  from /usr/lib64/crystal/crystal/main.cr:119:3 in 'main'
  from ???
  from __libc_start_main
  from _start
  from ???

My Test.cr crystal file:

require "json"

file = File.read("Information.json")

text = JSON.parse(file)

puts text

You have an extra comma at the end of line 12 (the first “description” line). That happens to me all the time.

OOOOOOh thanks you ! I’m happy to discover json file, thanks you so much !

1 Like

A workaround for this is using GitHub - Blacksmoke16/oq: A performant, and portable jq wrapper to facilitate the consumption and output of formats other than JSON; using jq filters to transform the data. as a library to leverage the conversion. It does require jq however.

Is it normal I have an error when I’m doing something like that ?

@information.architectures = Array(String).from_json(information["architectures"])

In this part of my code:

def initialize
        super
        informationFile = File.read("Information.json")
        information = JSON.parse(informationFile)

        @information.name = information["name"].as_s
        @information.architectures = Array(String).from_json(information["architectures"])
        @information.description = information["description"].as_s
        @information.website = information["website"].as_s
        #@information.downloadLinks = information["downloadLinks"].as_s
        #@information.signatureLinks = information["signatureLinks"].as_s
        #puts @information.signatureLinks

        option1 = ISM::SoftwareOption.new
        option1.name = "Pass1"
        option1.description = "Build the first pass to make a system from scratch"

        option2 = ISM::SoftwareOption.new
        option2.name = "Pass2"
        option2.description = "Build the second pass to make a system from scratch"

        @information.options = [option1, option2] of ISM::SoftwareOption
    end

From the docs

The general type-safe interface for parsing JSON is to invoke T.from_json on a target type T and pass either a String or IO as an argument.

You’re passing a JSON::Any object to Array(String).from_json instead of a String or an IO.

You should aim to make the following work

information = Information.from_json(informationFile)

where Information is a type you’ve defined. If this sounds confusing, then I suggest you read some of the resources shared above :muscle:

I’m a little bit lost.

The thing I don’t understand, is, if I can access to one element of my json array, why I can’t access to a "sub"element

You’re mixing two approaches:

  • Approach 1: parse json string into JSON::Any with JSON.parse. This is very useful to explore data when you are not sure about the shape of the payload.
  • Approach 2: parse json string into Crystal primitive types like String or Array(String) or into types defined by you, like classes or structs. You usually take this approach when you know the json schema and would like to rely on the Crystal type system and possibly define functions on such types for convenience.

You should pick one: either you read json text into JSON::Any and never have to call from_json again, but will often have to call as_s, as_a, etc. OR you deserialise the json text into a type that you defined, such as Information or InformationOption.

If all you want is for your program to treat information["architectures"] as an array, then information["architectures"].as_a will do the trick.

Okay, but I tried already that, but I had this error:

Error:

Showing last frame. Use --error-trace for full trace.

There was a problem expanding macro 'property'

Code in /home/zohran/Documents/Programmation/ISM/ISM/SoftwareInformation.cr:6:5

 6 | property architectures = ISM::Default::SoftwareInformation::Architectures
     ^
Called macro defined in macro 'macro_140431119652272'

 630 | macro property(*names, &block)

Which expanded to:

 >  8 |             end
 >  9 | 
 > 10 |             def architectures=(@architectures)
                                       ^------------
Error: instance variable '@architectures' of ISM::SoftwareInformation must be Array(String), not Array(JSON::Any)

Code:

def initialize
        super
        informationFile = File.read("Information.json")
        information = JSON.parse(informationFile)

        @information.name = information["name"].as_s
        @information.architectures = information["architectures"].as_a
        @information.description = information["description"].as_s
        @information.website = information["website"].as_s
        #@information.downloadLinks = information["downloadLinks"].as_a
        #@information.signatureLinks = information["signatureLinks"].as_a

        option1 = ISM::SoftwareOption.new
        option1.name = "Pass1"
        option1.description = "Build the first pass to make a system from scratch"

        option2 = ISM::SoftwareOption.new
        option2.name = "Pass2"
        option2.description = "Build the second pass to make a system from scratch"

        @information.options = [option1, option2] of ISM::SoftwareOption
    end

My json file:

{
  "name": "Binutils",
  "architectures": ["x86_64"],
  "description": "The GNU collection of binary tools",
  "website": "https://www.gnu.org/software/binutils/",
  "downloadLinks": ["https://ftp.gnu.org/gnu/binutils/"],
  "signatureLinks": ["https://ftp.gnu.org/gnu/binutils/"],
  "options": [
    {
      "name": "Pass1",
      "description": "Build the first pass to make a system from scratch"
    },
    {
      "name": "Pass2",
      "description": "Build the second pass to make a system from scratch"
    }
  ]
}

It’s the reason I’m lost, because I just would like to do with the simple way

Yes, that’s one of the inconveniences of using JSON.parse: you are sort of stuck with the JSON::Any type. If you read the compiler error carefully, you’ll notice:

Error: instance variable ‘@architectures’ of ISM::SoftwareInformation must be Array(String), not Array(JSON::Any)

This is because you’re trying to assign an Array(JSON::Any) to an Array(String) field. Now, you could get around this by fixing the type of the object on the right-hand side:

@information.architectures = information["architectures"].as_a.map(&.to_s)

but I’d actually recommend you look into parsing the json text directly into your Information type using JSON::Serializable. Here is how I’d do it

2 Likes

Thanks you very much for your video !!! It’s so nice of you. I will listen again to be sure I understood.

Just sorry for the SecretIdentity, it’s just a pure error xD. It’s because I copied a json code to make the body of my json file.

I will come back to you if I have a question

1 Like