# AI library for Crystal **URL:** \ **Category:** Help & Support\ **Created:** [July 15, 2024, 7:07pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019 "2024-07-15T19:07:59Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 18, 2024, 1:59pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/42 "2024-07-18T13:59:40Z")
I used the one you put as screenshot on your previous post. For the number of threads, on Linux you can get the number by just running that command: ```auto nproc ``` ---
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 18, 2024, 2:07pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/43 "2024-07-18T14:07:47Z")
So crimson, just to let you know. Basically what I would like to do (if I can get an example), it’s I send a question to the AI by setting a prompt (I would like to load it from a file), then get the answer, and save it in a file, or in a var inside my crystal code. And then close. It’s what I want to do basically. I want that var or that file just contain the answer, nothing else. I don’t know if you get my point ? ---
**Author:** ![crimson-knight](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/crimson-knight/32/1643_2.png) [@crimson-knight](https://forum.crystal-lang.org/u/crimson-knight)\ **Post date:** [July 18, 2024, 2:30pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/44 "2024-07-18T14:30:08Z")
Okay I had assumed you were using Apple hardware. The way llama.cpp works is very dependent on your hardware, and I haven’t had a chance to thoroughly test on non-Apple hardware. `nproc` lets you know the number of CPU cores available for threads, but you will want to set the threads to the number of threads available for your GPU. Also, depending on the amount of VRAM you have, the model will get loaded only partially into memory and processing gets shared between CPU and GPU. Performance takes a big degradation between CPU and GPU, so if you’re using CPU only I would recommend using a smaller model like Llama3 8b. You can definitely do what you’re trying to do, I think my last code example illustrates that, you’ll just need to read from a file and create the prompt you want. Then read from the response to save your file. I do this know with creating an OpenAPI spec. I can probably update my example later today or tomorrow morning for you. Do you have an example of a command you want it to run and I can aim to demo that? I’ll try to see if I can figure out why the Mixtral model is working on my work computer but not my personal computer (both Apple M-series) and see if I can offer some settings to help. ---
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 18, 2024, 2:35pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/45 "2024-07-18T14:35:07Z")
I find how to do ah ah. I just passed a context file, and redirect the answer to another file, in completely quiet mode to get the answer without any other text: ```auto ./llama-cli -m ../dolphin-2.7-mixtral-8x7b.Q4_K_M.gguf --chat-template vicuna-orca --n-predict 512 --threads 16 --ctx-size 2048 --temp 0.9 --top-k 80 --repeat-penalty 1.1 --no-display-prompt -f context.txt > result.txt ``` **Context file:** ```plaintext The user interact with the assistant named epsilon. If the user ask for the time, you must just answer COMMAND:time --show, nothing else. When you answer, you must never continue any conversation. ### User: what's time is it ? ``` **File with result:** ```plaintext COMMAND:time --show ``` So basically I just need to use Process.run and that it … ---
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 18, 2024, 2:41pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/46 "2024-07-18T14:41:41Z")
It’s crazy but I am thinking actually, probably soon most of the linux distribution component we know actually will become obsolete with AI. Like the init system, the shell… etc ---
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 18, 2024, 9:12pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/47 "2024-07-18T21:12:03Z")
I have one question. Why on my laptop the model take quite a bit of time to answer ? When I test llama online it’s very quick. And my laptop is quite powerful … ---
**Author:** ![kojix2](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/kojix2/32/1583_2.png) [@kojix2](https://forum.crystal-lang.org/u/kojix2)\ **Post date:** [July 19, 2024, 3:03am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/48 "2024-07-19T03:03:53Z")
Oh, I usually try to avoid making negative comments, but I can’t hold back on this topic. In the AI programming, I think the most important libraries are those for matrix computation. In Python, this is [NumPy](https://github.com/numpy/numpy). The Ruby community has also struggled a lot in this area. Ruby has [Numo::NArray](https://github.com/ruby-numo/numo-narray) and [Cumo](https://github.com/sonots/cumo), which work on GPUs. There used to be a library called [NMatrix](https://github.com/SciRuby/nmatrix), but in the end, Numo became the de-facto standard library. In Ruby, [ankane](https://github.com/ankane) and [yoshoku](https://github.com/yoshoku) have created many AI-related libraries [1](https://ankane.org/new-ml-gems) [2](https://ankane.org/more-ml-gems), which are built using C library bindings and Numo. In Crystal, developing matrix computation libraries is even rarer than in Ruby. Still, there is [num.cr](https://github.com/crystal-data/num.cr). Maintaining foundational libraries like NumPy or Numo is very difficult. Languages other than Python and C++ have not been very successful in doing this. Even popular languages like Rust seem to face challenges. (It may be hard for statically typed languages to handle matrix computations when the dimensions and sizes of the matrices are not known. [rust-ndarray](https://github.com/rust-ndarray/ndarray)) Therefore, making bindings for libraries like [ONNXRuntime](https://onnxruntime.ai/) or connecting to API servers provided by tools like [Ollama](https://ollama.com/) seems to be the practical approach. However, this is not interesting at all! Of course, wrapping llama.cpp or re-implementing it in Crystal would be much more amazing. But, oh, it would feel as challenging as climbing Everest. With my skills, not only is climbing Everest impossible, but even climbing the hills around me is difficult. Still, I understand how tough it is… (This text was written with the help of ChatGPT) ---
**Author:** ![zw963](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/zw963/32/1623_2.png) [@zw963](https://forum.crystal-lang.org/u/zw963)\ **Post date:** [July 19, 2024, 9:01am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/49 "2024-07-19T09:01:29Z")
Hi, @crimson-knight , is there possible to use AMD [rocm][GitHub - rocm-arch/rocm-arch: A collection of Arch Linux PKGBUILDS for the ROCm platform](https://github.com/rocm-arch/rocm-arch) to speed up the process of talking with AI? it so slow when i running following code on my Arch Linux, with AMD 7480hs (780M integrated GPU), 64G memory, and use all my 16 core of CPU, temp to 86, but GPU is free, waiting a long time, still could get the answer. ```crystal require "llamero" model = Llamero::BaseModel.new( model_root_path: Path.new("/home/zw963/models/"), model_name: "Meta-Llama-3-8B-Instruct-Q5_K_M.gguf" ) puts model.quick_chat([{role: "user", content: "Hi!"}]) ``` ```auto ╰─ $ ./ai_example 2024-07-19T08:54:48.274588Z INFO - Interacting with the model... 2024-07-19T08:54:48.285018Z INFO - The AI is now processing... please wait ``` ![image](https://canada1.discourse-cdn.com/flex036/uploads/crystal_lang/original/2X/c/c686bdb70ca48618a075b3c398bb6176af37f03b.png) ---
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 19, 2024, 7:29pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/51 "2024-07-19T19:29:21Z")
I think I will try to find another AI optimized for Linux. I thought Llama was more for Linux… Because we loose a lot of performance ---
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 19, 2024, 8:15pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/52 "2024-07-19T20:15:56Z")
That one is really impressive: [https://pi.ai/desktop](https://pi.ai/desktop) Do you know how can I just simply send a request in Crystal to that AI with http client, and get the response ? I am just extremely bad with network ---
**Author:** ![kojix2](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/kojix2/32/1583_2.png) [@kojix2](https://forum.crystal-lang.org/u/kojix2)\ **Post date:** [July 20, 2024, 12:56am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/53 "2024-07-20T00:56:52Z")
Actually, calling HTTP APIs is not very difficult. You can use Crystal’s [standard library](https://crystal-lang.org/api/HTTP/Client.html), but the standard library is often not user-friendly, so I prefer [Crest](https://github.com/mamantoha/crest). I have made some command-line tools to call APIs. Basically, you open the API reference page and write the query in Crest as described there. (I couldn’t find the API page for pi.ai right away. Do I need to register for the wishlist?) I have created several command-line tools that call HTTP APIs. These may not be very useful, but here are the links: - [GitHub - kojix2/deepl-cli: Simple command line tool for DeepL](https://github.com/kojix2/deepl-cli) - [GitHub - kojix2/deepl.cr: Crystal library for the DeepL language translation API.](https://github.com/kojix2/deepl.cr) - [GitHub - kojix2/chatgpt-cli: Yet another ChatGPT command line tool](https://github.com/kojix2/chatgpt-cli) - [GitHub - kojix2/claude-cli](https://github.com/kojix2/claude-cli) Based on the following Ruby code, it should be easy to write Crystal code to call Ollama’s API. - [GitHub - gbaptista/ollama-ai: A Ruby gem for interacting with Ollama's API that allows you to run open source AI LLMs (Large Language Models) locally.](https://github.com/gbaptista/ollama-ai) This kind of work is not very difficult. What is really important for the Crystal community is developing basic libraries for matrix calculations. Another important task is creating binding libraries for C and C++ libraries. I would greatly appreciate such efforts. (This text was written with the help of ChatGPT) ---
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 20, 2024, 8:45am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/54 "2024-07-20T08:45:50Z")
This is the documentation . If I can just have on example > **[GitHub - pi-apps/pi-platform-docs](https://github.com/pi-apps/pi-platform-docs/tree/master)** > > master ---
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 20, 2024, 12:49pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/55 "2024-07-20T12:49:03Z")
I think I find the documentation: [https://docs.aveva.com/bundle/pi-web-api-reference/page/help.html](https://docs.aveva.com/bundle/pi-web-api-reference/page/help.html) ---
**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\ **Post date:** [July 20, 2024, 3:57pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/56 "2024-07-20T15:57:39Z")
So I finally find why it was lagging. You must set the number of the cpu cores for ONE of the processor. For example in my case, if I get the information about /proc/info, I should use the number of cpu cores for a **single core.** In my case it’s 8, not 16. Now the answer is really fast. So I will back to llama now ! Look at that post crimson-knight for the explanation: [https://www.reddit.com/r/LocalLLaMA/comments/190v426/llamacpp\_cpu\_optimization/](https://www.reddit.com/r/LocalLLaMA/comments/190v426/llamacpp_cpu_optimization/) ---
**Author:** ![kojix2](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/kojix2/32/1583_2.png) [@kojix2](https://forum.crystal-lang.org/u/kojix2)\ **Post date:** [March 13, 2025, 8:34am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/57 "2025-03-13T08:34:24Z")
Over the past few weeks, there’s been a sensation about programming by AI agents. I, too, am amazed by the power of this technology. Today, I asked Cline (claude-sonnet) to implement either NumPy or Numo::NArray (for those who aren’t familiar, this is the standard library in Ruby equivalent to NumPy) using the Crystal language, and the result was quite impressive. > **[GitHub - kojix2/narray: AI quickly developed a NumPy-like library for...](https://github.com/kojix2/narray)** > > AI quickly developed a NumPy-like library for Crystal In just another two years, regardless of how obscure the language may be, we may soon be able to simply request an implementation of a matrix computation library like NumPy. Before long, I believe we will discover new technological gaps or areas where human expertise is indispensable, but for now, I am simply in awe of this tremendous technological revolution. (Translated from Japanese by ChatGPT) ---
**Author:** ![kojix2](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/kojix2/32/1583_2.png) [@kojix2](https://forum.crystal-lang.org/u/kojix2)\ **Post date:** [March 14, 2025, 2:52pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/58 "2025-03-14T14:52:45Z")
I realized something extremely obvious: using a matrix library in Crystal doesn’t provide any performance boost. In Python or Ruby, using NumPy or Numo::NArray naturally results in a speedup, so this feels strange (though it’s not strange at all). It seems that in a language like Crystal, matrix computation libraries exist primarily for readability and ease of writing rather than for performance improvements. ---
**Author:** ![HertzDevil](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/hertzdevil/32/1023_2.png) [@HertzDevil](https://forum.crystal-lang.org/u/HertzDevil)\ **Post date:** [March 14, 2025, 3:26pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/59 "2025-03-14T15:26:37Z")
This is because the parts of NumPy like `linalg` are written in [C translated from Fortran code](https://github.com/numpy/numpy/blob/main/numpy/linalg/lapack_lite/f2c_c_lapack.c) that has been continually improved for over 45 years (BLAS was released in 1979), rather than textbook implementations that LLMs have no issues regurgitating. It might sound counterintuitive that Python and Ruby are faster than Crystal, but this is because the fast parts are written in neither Python nor Ruby (nor in this case, C) to begin with. You are better off using something like [GitHub - konovod/linalg: Linear algebra library based on LAPACK](https://github.com/konovod/linalg) instead. ---
**Author:** ![kojix2](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/kojix2/32/1583_2.png) [@kojix2](https://forum.crystal-lang.org/u/kojix2)\ **Post date:** [March 14, 2025, 4:14pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/60 "2025-03-14T16:14:01Z")
I’m not aiming to solve a real-world problem with this, so I prefer a simple, textbook-style library implemented in Crystal rather than a LAPACK binding. But apart from that, I feel that things that seemed impossible a few months ago—like AI creating a faster BLAS in 20 minutes or LLVM making a huge performance leap—are now quite possible in the future. No one knows what the future holds, though. ---
**Author:** ![renich](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/renich/32/2137_2.png) [@renich](https://forum.crystal-lang.org/u/renich)\ **Post date:** [July 31, 2025, 6:41am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/61 "2025-07-31T06:41:22Z")
I just want to state my interest in machine learning/AI related libraries in Crystal. Python is like the worst language they could’ve picked for AI, yet, the most successful. ---
**Author:** ![watzon](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/watzon/32/481_2.png) [@watzon](https://forum.crystal-lang.org/u/watzon)\ **Post date:** [August 1, 2025, 5:01pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/62 "2025-08-01T17:01:31Z")
I feel the same. The reason Python was chosen is because: 1. It’s a language that a lot of academics and researchers are familiar with, and those people have been the ones working on machine learning stuff for the longest. 2. It does have a fairly good interop with C, which made it easy to write the majority of the libraries in C and then just provide a Python entry point into it. What I like about the idea of doing it through Crystal is that for the most part you could get very similar performance to what you get out of the existing libraries, but with something implemented entirely in Crystal, which would make it much easier to get in and modify things without having to go through a cross-language barrier. Even in the case though that you’re binding to existing libraries, though, Crystal does have a much more user-friendly C FFI. I do believe @bararchy has been working hard on a pure Crystal LLM. [Previous page](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019.md?page=2) [Next page](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019.md?page=4)