Is increasing array index size in the future?

Crystal Contrib
1

Currently the index of arrays is maxed at Int32::MAX, i.e. (2**31 - 1) => 2147483647

Are there plans to increase that to at least Int64::MAX?
I seriously need to create|address larger arrays.

2

Related: Arrays have a 'silent' length limit due to Int32 type on size. · Issue #4011 · crystal-lang/crystal · GitHub.

3

I think any addition of larger array would be it’s own array type, optimized to not create the kind of massive contiguous allocations required by an array larger than u32_max. What’s the usecase you’re running into here @jzakiya?

4 Likes
4

I want to genrate as many prime numbers as possilbe.

The program below will generate primes until you run out of memory.

def sozp240(val, residues)
  res  = residues   
  md, rscnt = res.last-1, res.size  

  kmax = (val - 2) // md + 1  
  prms = Array(UInt64).new(kmax, 0) 
  modk, r, k = 0, -1, 0  

  loop do 
    if (r += 1) == rscnt; r = 0; modk += md; k += 1 end
    next if prms[k] & (1u64 << r) != 0   
    prm_r = res[r]         
    prime = modk + prm_r  
    break if prime > Math.isqrt(val)  
    res.each do |ri|   
      kn,rn = (prm_r * ri - 2).divmod md  
      bit_r = 1u64 << res.index(rn+2).not_nil! 
      kpm = k * (prime + ri) + kn  
      while kpm < kmax; prms[kpm] |= bit_r; kpm += prime end
  end end
  prms
end

The program encodes the primes within the primes candidates array prms.
I extract the primes individually from prms to use in another routine that uses them.

Currently I’m array size limited to only doing 19 Billion primes (~45 minutes with single core).

Max array index is: (2**31 - 1) = 2,147,483,647 elements.

The 19th Bil prime is 491_701_819_267.

491_701_819_267 // 240 = 2,048,757,581 residue groups of 64-bits.

Thats 2_048_757_581 * 8 = 16,390,060,648 bytes (16.39 GB) of memory.

I have 40GB on my laptop, so I can|want to do a lot more.

40th Billion prime is 1,066,173,339,601

That would take an array of: 1,066,173,339,601 // 240 = 4,442,388,915 resgroups (index values)
That’s 35,539,111,320 bytes (35.5GB) of memory.
I can easily do this, and find max possible with my memory.

That’s why I need to be able to create|address larger indexed arrays.