Open, read and write binary into a file

Fulgurance · December 7, 2021, 6:03pm

Hello, today I tried to understand how to open a file to read or write binary data into it. I would like to save a class (an object) into a file, and read it after.

It’s a part of my code:


        def checkSoftwareDatabase
            if !Dir.exists?(ISM::Default::Path::SoftwaresDirectory)
                Dir.mkdir(ISM::Default::Path::SoftwaresDirectory)
            end

            if File.exists?(ISM::Default::Path::DatabaseFile)
                file = File.open(ISM::Default::Path::DatabaseFile,"rb")
                @softwares = file.read
            else
                list = Dir.entries(ISM::Default::Path::SoftwaresDirectory)

                list.each do |argument|
                    puts argument
                end
            end
        end

In this part, I would like to open a saved class into a file as binary data. (But I don’t know as well how to write object data as binary into a file)

straight-shoota · December 7, 2021, 6:43pm

Crystal has no implicit serialization of data types. Their data layout is not specificied and may change between different compilations of a program. While you could technically write the memory representation of a data type to a file, there is no guarantee that you can load it again later.

You can use an explicit serialization format, though. The standard library ships with JSON and YAML serialization libraries. You can find shards (including ones that provide binary serialization) at Data Formats on Shardbox and Network Protocols on Shardbox

yxhuvud · December 8, 2021, 12:52pm

It is also possible and often fairly easy not depend on something predefined but instead implement read and write by yourself. For an example, I implement a serialization format defined by a separate project in the Reader and Writer classes of HDR histogram . In that case the header is readable, but there is a blob of binary data that is handled by the Codec class.

drhuffman12 · December 8, 2021, 9:10am

I’m sure I saw a post asking about this.

Basically, for saving objects to files", you need:

require "json"
include ::JSON::Serializable (or yaml instead; likewise below)
def initialize(json_params)...
save/load via self.from_json(some_json_string) and self.to_json and save contents of to_json results to file, etc
Some things don’t serialize well (e.g.: Symbols), so find alternatives.
Performance Note: Writing to file will probably be slower than writing to memory.

See also: Carcin

require "json"

class Foo
  include ::JSON::Serializable # This along with `require "json"` enables `.from_json` and `.to_json` for many classes/structures.
  
    # Symbol don't serialize, but you can use Enum's like:
    enum Widget
      Alpha = 19
      Beta  = 28
      Gamma = 37
    end

    getter bar : Widget

    def initialize(@bar = Widget::Beta)
    # You'll need an initialize that handles all serializable params to be saved/loaded via the json version.
    end
end

# example usage:

foo = Foo.new(bar: Foo::Widget::Gamma)

puts "foo: #{foo}"
puts "foo as json: #{foo.to_json}" # or print to file, etc

foo2 = Foo.from_json(foo.to_json)

puts "foo2: #{foo2}"
puts "foo2.bar: #{foo2.bar}"
puts "foo2.bar as int: #{foo2.bar.to_i}"
puts "foo2.bar's class: #{foo2.bar.class}"
puts "foo2 as json: #{foo2.to_json}"

straight-shoota · December 8, 2021, 10:14am

This one?

drhuffman12 · December 9, 2021, 2:28am

@Blacksmoke16/@straight-shoota , thanks for helping me find the original post and moving my reply. I was beginning to think maybe the original post was deleted.

Fulgurance · December 13, 2021, 8:03pm

Okay, this look interesting, but can you just provide me an example of a serialization of data in a file with json ? I’m not sure to understand how you can read and write data in json file.

I think I will prefer something like xml

lbarasti · December 14, 2021, 4:44pm

Hey @Fulgurance, here is a concrete example:

Define your serializable events
Define a method to persist them to IO
Open the file where you persisted your objects and read them up

To summarise, it all boils down to:

Defining an Event super class (or super struct) that includes JSON::Serializable - or the equivalent XML module
Persisting instances of events - e.g. event.to_json(file_io)
Reading events from file with - e.g. when reading a file line by line: Event.from_json(line)

I wrote this article about JSON serialization specifically a while ago, but you might be able to apply the same concepts to other serialization formats - this doesn’t cover reading from and writing to file, though.

RespiteSage · December 14, 2021, 4:52pm

Unfortunately, we don’t yet have an XML::Serializable module. As I understand it, the reason comes down to 1) XML is more complicated than JSON, and 2) there hasn’t been enough of a demonstrated need in the Crystal community for this functionality. I’d be happy to help contribute to such a module, but I’ve messed with the idea a bit in the past and it would be a big task.

Fulgurance · December 14, 2021, 4:53pm

Finally I came back to JSON xD

Just one thing, I wrote this json file to test, but something is wrong with the syntax, but what?

{
  "name": "Binutils",
  "architectures": ["x86_64"],
  "description": "The GNU collection of binary tools",
  "website": "https://www.gnu.org/software/binutils/",
  "downloadLinks": ["https://ftp.gnu.org/gnu/binutils/"],
  "signatureLinks": ["https://ftp.gnu.org/gnu/binutils/"],
  "options": [
    {
      "name": "Pass1",
      "description": "Build the first pass to make a system from scratch",
    },
    {
      "name": "Pass2",
      "description": "Build the second pass to make a system from scratch",
      "secretIdentity": "Unknown"
    }
  ]
}

I have this error:

 zohran  ~  Documents  Programmation  Test  1  crystal Test.cr 
Unhandled exception: unexpected token '}' at line 12, column 5 (JSON::ParseException)
  from /usr/lib64/crystal/json/parser.cr:125:5 in 'parse_exception'
  from /usr/lib64/crystal/json/parser.cr:121:5 in 'unexpected_token'
  from /usr/lib64/crystal/json/parser.cr:92:13 in 'parse_object'
  from /usr/lib64/crystal/json/parser.cr:35:7 in 'parse_value'
  from /usr/lib64/crystal/json/parser.cr:49:18 in 'parse_array'
  from /usr/lib64/crystal/json/parser.cr:33:7 in 'parse_value'
  from /usr/lib64/crystal/json/parser.cr:85:19 in 'parse_object'
  from /usr/lib64/crystal/json/parser.cr:35:7 in 'parse_value'
  from /usr/lib64/crystal/json/parser.cr:13:12 in 'parse'
  from /usr/lib64/crystal/json.cr:135:5 in 'parse'
  from Test.cr:4:8 in '__crystal_main'
  from /usr/lib64/crystal/crystal/main.cr:110:5 in 'main_user_code'
  from /usr/lib64/crystal/crystal/main.cr:96:7 in 'main'
  from /usr/lib64/crystal/crystal/main.cr:119:3 in 'main'
  from ???
  from __libc_start_main
  from _start
  from ???

My Test.cr crystal file:

require "json"

file = File.read("Information.json")

text = JSON.parse(file)

puts text

RespiteSage · December 14, 2021, 4:55pm

You have an extra comma at the end of line 12 (the first “description” line). That happens to me all the time.

Fulgurance · December 14, 2021, 4:57pm

OOOOOOh thanks you ! I’m happy to discover json file, thanks you so much !

Blacksmoke16 · December 14, 2021, 5:05pm

A workaround for this is using GitHub - Blacksmoke16/oq: A performant, and portable jq wrapper to facilitate the consumption and output of formats other than JSON; using jq filters to transform the data. as a library to leverage the conversion. It does require jq however.

Fulgurance · December 14, 2021, 5:39pm

Is it normal I have an error when I’m doing something like that ?

@information.architectures = Array(String).from_json(information["architectures"])

In this part of my code:

def initialize
        super
        informationFile = File.read("Information.json")
        information = JSON.parse(informationFile)

        @information.name = information["name"].as_s
        @information.architectures = Array(String).from_json(information["architectures"])
        @information.description = information["description"].as_s
        @information.website = information["website"].as_s
        #@information.downloadLinks = information["downloadLinks"].as_s
        #@information.signatureLinks = information["signatureLinks"].as_s
        #puts @information.signatureLinks

        option1 = ISM::SoftwareOption.new
        option1.name = "Pass1"
        option1.description = "Build the first pass to make a system from scratch"

        option2 = ISM::SoftwareOption.new
        option2.name = "Pass2"
        option2.description = "Build the second pass to make a system from scratch"

        @information.options = [option1, option2] of ISM::SoftwareOption
    end

lbarasti · December 14, 2021, 8:12pm

From the docs

The general type-safe interface for parsing JSON is to invoke T.from_json on a target type T and pass either a String or IO as an argument.

You’re passing a JSON::Any object to Array(String).from_json instead of a String or an IO.

You should aim to make the following work

information = Information.from_json(informationFile)

where Information is a type you’ve defined. If this sounds confusing, then I suggest you read some of the resources shared above

Fulgurance · December 14, 2021, 8:27pm

I’m a little bit lost.

The thing I don’t understand, is, if I can access to one element of my json array, why I can’t access to a "sub"element

lbarasti · December 14, 2021, 8:57pm

You’re mixing two approaches:

Approach 1: parse json string into JSON::Any with JSON.parse. This is very useful to explore data when you are not sure about the shape of the payload.
Approach 2: parse json string into Crystal primitive types like String or Array(String) or into types defined by you, like classes or structs. You usually take this approach when you know the json schema and would like to rely on the Crystal type system and possibly define functions on such types for convenience.

You should pick one: either you read json text into JSON::Any and never have to call from_json again, but will often have to call as_s, as_a, etc. OR you deserialise the json text into a type that you defined, such as Information or InformationOption.

If all you want is for your program to treat information["architectures"] as an array, then information["architectures"].as_a will do the trick.

Fulgurance · December 14, 2021, 9:08pm

Okay, but I tried already that, but I had this error:

Error:

Showing last frame. Use --error-trace for full trace.

There was a problem expanding macro 'property'

Code in /home/zohran/Documents/Programmation/ISM/ISM/SoftwareInformation.cr:6:5

 6 | property architectures = ISM::Default::SoftwareInformation::Architectures
     ^
Called macro defined in macro 'macro_140431119652272'

 630 | macro property(*names, &block)

Which expanded to:

 >  8 |             end
 >  9 | 
 > 10 |             def architectures=(@architectures)
                                       ^------------
Error: instance variable '@architectures' of ISM::SoftwareInformation must be Array(String), not Array(JSON::Any)

Code:

def initialize
        super
        informationFile = File.read("Information.json")
        information = JSON.parse(informationFile)

        @information.name = information["name"].as_s
        @information.architectures = information["architectures"].as_a
        @information.description = information["description"].as_s
        @information.website = information["website"].as_s
        #@information.downloadLinks = information["downloadLinks"].as_a
        #@information.signatureLinks = information["signatureLinks"].as_a

        option1 = ISM::SoftwareOption.new
        option1.name = "Pass1"
        option1.description = "Build the first pass to make a system from scratch"

        option2 = ISM::SoftwareOption.new
        option2.name = "Pass2"
        option2.description = "Build the second pass to make a system from scratch"

        @information.options = [option1, option2] of ISM::SoftwareOption
    end

My json file:

{
  "name": "Binutils",
  "architectures": ["x86_64"],
  "description": "The GNU collection of binary tools",
  "website": "https://www.gnu.org/software/binutils/",
  "downloadLinks": ["https://ftp.gnu.org/gnu/binutils/"],
  "signatureLinks": ["https://ftp.gnu.org/gnu/binutils/"],
  "options": [
    {
      "name": "Pass1",
      "description": "Build the first pass to make a system from scratch"
    },
    {
      "name": "Pass2",
      "description": "Build the second pass to make a system from scratch"
    }
  ]
}

It’s the reason I’m lost, because I just would like to do with the simple way

lbarasti · December 14, 2021, 9:14pm

Yes, that’s one of the inconveniences of using JSON.parse: you are sort of stuck with the JSON::Any type. If you read the compiler error carefully, you’ll notice:

Error: instance variable ‘@architectures’ of ISM::SoftwareInformation must be Array(String), not Array(JSON::Any)

This is because you’re trying to assign an Array(JSON::Any) to an Array(String) field. Now, you could get around this by fixing the type of the object on the right-hand side:

@information.architectures = information["architectures"].as_a.map(&.to_s)

but I’d actually recommend you look into parsing the json text directly into your Information type using JSON::Serializable. Here is how I’d do it

Fulgurance · December 14, 2021, 9:29pm

Thanks you very much for your video !!! It’s so nice of you. I will listen again to be sure I understood.

Just sorry for the SecretIdentity, it’s just a pure error xD. It’s because I copied a json code to make the body of my json file.

I will come back to you if I have a question

Topic		Replies	Views
Is there a better way to use YAML? Help & Support	2	461	September 16, 2019
Possible to serializer/deserialize and object like Ruby marshaling? Help & Support	26	3371	May 25, 2020
Logging to a file Learning Resources	8	934	November 30, 2020
Idea for a simple compressed key value store in Crystal Learning Resources	3	352	March 2, 2023
Using module IO::ByteFormat to write properties of an object Help & Support	1	155	May 24, 2023

Open, read and write binary into a file

Related Topics