# AI library for Crystal

**URL:** <https://forum.crystal-lang.org/t/ai-library-for-crystal/7019>\
**Category:** Help & Support\
**Created:** [July 15, 2024, 7:07pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019 "2024-07-15T19:07:59Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 16, 2024, 10:52pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/21 "2024-07-16T22:52:45Z")

</div>

Okay got it. When you said large language models only generate text, does it mean some model can do it ?

---

<div class="post-metadata">

**Author:** ![jgaskins](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jgaskins/32/2449_2.png) [@jgaskins](https://forum.crystal-lang.org/u/jgaskins)\
**Post date:** [July 16, 2024, 10:54pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/22 "2024-07-16T22:54:17Z")

</div>

By “can do it”, do you mean can other models run commands themselves?

---

<div class="post-metadata">

**Author:** ![jgaskins](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/jgaskins/32/2449_2.png) [@jgaskins](https://forum.crystal-lang.org/u/jgaskins)\
**Post date:** [July 16, 2024, 11:33pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/23 "2024-07-16T23:33:38Z")

</div>

If that’s what you meant: no, AI models really only operate on numbers, vectors, and matrices. The “model” that we’re talking about is a purely mathematical model. With language models, there are words (or parts of words called “tokens”) associated with those numbers so it can generate text for you. But that’s really where it stops.

What you’re trying to do requires a higher-level abstraction on top of the model itself. An analogy might be, let’s you want a cappuccino. The AI model is just espresso beans. The beans are a crucial component but you have to make the rest of the cappuccino.

---

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 17, 2024, 7:10am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/24 "2024-07-17T07:10:32Z")

</div>

Okay so I got the point basically with your example . Thank you, I like how you explain 😊

Very helpful everyone , thanks a lot !

So basically , I just need to establish a context to the AI and just implement the process to run code.

---

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 17, 2024, 8:55am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/25 "2024-07-17T08:55:01Z")

</div>

Is it normal when I try to load my model, the app try always to find it in a very strange path ? Look like a Windows like path

```auto
zohran@alienware-m17-r3 ~/Downloads/Test $ crystal Test.cr 
Unhandled exception: Error opening file with mode 'r': '/Users/zohran/models/dolphin-2.7-mixtral-8x7b.Q4_K_M.gguf': No such file or directory (File::NotFoundError)
  from /usr/lib64/crystal/crystal/system/unix/file.cr:12:7 in 'open'
  from /usr/lib64/crystal/file.cr:175:10 in 'new'
  from /usr/lib64/crystal/file.cr:741:12 in 'read_meta_data'
  from lib/llamero/src/models/meta_data/meta_data_reader.cr:34:5 in 'initialize'
  from lib/llamero/src/models/meta_data/meta_data_reader.cr:32:3 in 'new'
  from lib/llamero/src/models/base_model.cr:118:25 in 'initialize:model_name'
  from lib/llamero/src/models/base_model.cr:84:3 in 'new:model_name'
  from Test.cr:3:9 in '__crystal_main'
  from /usr/lib64/crystal/crystal/main.cr:118:5 in 'main_user_code'
  from /usr/lib64/crystal/crystal/main.cr:104:7 in 'main'
  from /usr/lib64/crystal/crystal/main.cr:130:3 in 'main'
  from /usr/lib64/libc.so.6 in '??'
  from /usr/lib64/libc.so.6 in '__libc_start_main'
  from /home/zohran/.cache/crystal/crystal-run-Test.tmp in '_start'
  from ???

```

---

<div class="post-metadata">

**Author:** ![crimson-knight](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/crimson-knight/32/1643_2.png) [@crimson-knight](https://forum.crystal-lang.org/u/crimson-knight)\
**Post date:** [July 17, 2024, 9:51am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/26 "2024-07-17T09:51:44Z")

</div>

Yes that’s correct, it looks for models relative to your system user instead of your projects source. There’s an ivar you can overload to change that if you need to, but I don’t recommend putting models in your project source like that. Since they are so large it makes more sense to have 1 folder that any app can access to run the model

---

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 17, 2024, 10:52am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/27 "2024-07-17T10:52:25Z")

</div>

So which var should I set to do that ? If I can just see how to do that please

---

<div class="post-metadata">

**Author:** ![crimson-knight](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/crimson-knight/32/1643_2.png) [@crimson-knight](https://forum.crystal-lang.org/u/crimson-knight)\
**Post date:** [July 17, 2024, 12:11pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/28 "2024-07-17T12:11:38Z")

</div>

What does that mean?

---

<div class="post-metadata">

**Author:** ![zw963](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/zw963/32/1623_2.png) [@zw963](https://forum.crystal-lang.org/u/zw963)\
**Post date:** [July 17, 2024, 1:20pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/29 "2024-07-17T13:20:12Z")

</div>

He said he changed the title from `IA` to `AI`.

But, he forgot to modify the same part in the post content.

---

<div class="post-metadata">

**Author:** ![crimson-knight](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/crimson-knight/32/1643_2.png) [@crimson-knight](https://forum.crystal-lang.org/u/crimson-knight)\
**Post date:** [July 17, 2024, 1:28pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/30 "2024-07-17T13:28:51Z")

</div>

Before I explain how to do the configuration, I’m going to write some introductory docs here for you (and everyone). It’s important to understand these basic concepts because Llamero introduces a slightly new variation on how to write applications that let your AI models interact with the user and system.

# Getting Started With Llamero

Llamero is built to act as an interface between a Large Language Model (LLM) and a locally running application. This interface makes a pleasant, simple and way to build apps that utilize locally running models to take advantage of your own hardware for offline and unlimited use.

In particular, this is to take advantage of the Apple M-series hardware, but it is not limited to Apple hardware.

By utilizing Llamero, a developer can write applications that interact with an LLM and allow the LLM to control the flow of the application. This includes developing agents and agentic workflows.

Llamero is currently developed to be single threaded and _ **does not** _ manage memory for the models being used. This means you must use 1 model at a time, and choosing a model that is larger than your systems memory has can cause hard crashes.

## Key Concepts

- Models
- Prompts
- Grammars
- Tokens (not documented here, I’ll write more about this later)

### About Models

Llamero currently is only designed to work with LLMs, specifically _chat_ LLMs. Local LLMs vary in size and therefore capabilities. When we work with LLMs, the number of parameters is a good indicator of how capable a model is in it’s ability to use logic and reasoning.

Models are typically sized with a parameter like `7b` or `22b`. This means the model has `7 billion` or `22 billion` parameters respectively. The general rule of thumb for an open source model is that the smaller models use less memory and process faster but have very limited logic and reasoning. The larger the amount of parameters, the more capable a model becomes.

Llamero provides a `Llamero::BaseModel` class that comes with basic default configuration that should allow for quick agent development, and will let you use multiple models throughout your application.

All models have a limited **context window**. This is very important as this represents the length of total text a model can keep track of a topic in the conversation. All prompts can exceed the context window, but the models responses will start veering off topic and become unreliable once it is exceeded.

The default path where models are looked for is in the `~/models` folder, but if you need to change this you can set

```crystal
property model_root_path : Path

```

On your model class of the full path to the folder with your model file.

You can check the `src/models/base_model.cr` file for all of the default properties, their descriptions include how/what they do. Some are specific to AI models, not just run-time configuration. Things like `top_k_sampling` and `repeat_penalty` are examples of AI fine tuning properties that you can customize. The `model_root_path` is an example of a run-time configuration.

### About Prompts

Prompts are a simple collection of interactions that are provided to the model. Every LLM uses some form of a `system` prompt, which is meant to provide explicit instructions and rules that set the models perspective before getting further instructions.

The `user` role is the text provided by your application for the model to infer from.

The `assistant` role is the models response.

You can create a prompt chain, which is the series of back and forth in the chat, typically starting with the `system` prompt.

### About Grammars

This is the part that is most unique to Llamero. Grammars are a convenient way to get the LLMs to respond using a consistent and parsable format. Llamero handles parsing that response for you. Grammars both influence the output from the model and hold the response from the model. By using a grammar, you don’t have to provide instructions within your prompt for your expected response and then parse it.

Let’s say we have some text that we want to extract a customers information from. This is unstructured text and we want the model to perform the extraction for us.

When using LLMs from any other service, your prompt will look something like this:

```auto
You are a client services professional and a new customer has emailed you inquiring about our company's services. Here is the customers email, I want you to extract the customers first name, last name, and their email address. Your response must be JSON using this template: { "first_name": "", "last_name": "", "email_address": "" }

#{Insert some unstructured user input here}

```

You would then need to parse this into a class, hopefully it parses correctly and you would need to write the process for if it fails to re-query the model to try again.

When using Llamero models with grammars, this is all done for you.  
This entire portion of the prompt is eliminated:

```auto
Your response must be JSON using this template: { "first_name": "", "last_name": "", "email_address": "" }

```

This includes the need to retry if parsing fails, because Llamero handles this.

Grammars are a super power, and they influence the output from the model.

```crystal
# A simple example from the above scenario
class NewCustomerStructuredResponse < Llamero::BaseGrammar
  property first_name : String = ""
  property last_name : String = ""
  property email : String = ""
end

```

```crystal
# Property names influence the models output, play around with an example like this
class ExampleOfInfluenceStructuredResponse < Llamero::BaseGrammar
  property a_random_number_between_zero_and_ten : Int32 = 0
  property a_random_number_greater_than_one_hundred : Int32 = 0
end

```

Now if you want to have your model write and execute commands on your local machine:

```crystal
class ExecutableCommandStructuredResponse < Llamero::BaseGrammar
  property system_command_to_execute : String
end

base_prompt = Llamero::BasePrompt.new(
      system_prompt: "You are a Senior Ruby on Rails developer who is working on an application. Your job is to write the system command that is necessary to perform the next step",
      messages: [
        Llamero::PromptMessage.new(role: "user", content: "Create a new scaffold for a Customer that has the following properties: first_name, last_name, email_address"),
      ]
    )

base_model = Llamero::BaseModel.new(model_name: "meta-llama-3-8b-instruct-Q6_K.gguf")

# the `ai_response` is now our structured response
ai_response = base_model.chat(base_prompt, ExecutableCommandStructuredResponse.from_json(%({}))

# Let's execute the command we have from the model
`#{ai_response.system_command_to_execture}`

```

There’s enough to be dangerous and write something cool! I’ll continue adding docs to the repo, especially for Cursor. I’ve found it to be incredibly helpful.

---

<div class="post-metadata">

**Author:** ![crimson-knight](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/crimson-knight/32/1643_2.png) [@crimson-knight](https://forum.crystal-lang.org/u/crimson-knight)\
**Post date:** [July 17, 2024, 1:36pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/31 "2024-07-17T13:36:46Z")

</div>

> Actually it’s weird, if you run it locally, even when you just ask simple question like what’s time is it, it answer wrong

Think of the AI model like it’s a service that runs on your computer, but is isolated. It does not perform any actions, it just talks. It’ll make stuff up because it’s job is to answer. The goal for Llamero was to create a way to talk to the model, have the responses be in a format your app can then perform the tasks like getting the time, checking folders, going to the web for things, etc.

The model is just a brain. It’s our job to write the “body” that makes the brain capable of performing tasks.

---

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 17, 2024, 3:39pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/33 "2024-07-17T15:39:31Z")

</div>

Thanks a lot for your guide.

So before I start to teach to the model how to run any command. I just try actually to understand how to start a chat with the AI like a chatgpt assistant.

So I did that:

```crystal
require "llamero"

model = Llamero::BaseModel.new(model_root_path: Path.new("/home/zohran/Downloads/Test/"),model_name: "dolphin-2.7-mixtral-8x7b.Q4_K_M.gguf")

answer = model.quick_chat([{ role: "user", content: "Hi !" }])
puts answer

```

Why I don’t see any answer, and I can’t write anything ?

---

<div class="post-metadata">

**Author:** ![Sunrise](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/sunrise/32/2339_2.png) [@Sunrise](https://forum.crystal-lang.org/u/Sunrise)\
**Post date:** [July 18, 2024, 11:09am UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/34 "2024-07-18T11:09:19Z")

</div>

I’m not sure how to teach AI to run commands. The method I can think of is:

1. Give the AI an instruction: I need to do something, please provide the code between `<code> </code>`.
2. Use regular expressions to match the code enclosed within `<code></code>`.
3. Check the correctness of the code and execute it.  
If you only want to integrate AI into your application without making adjustments to the AI internally, there are many ready-made AI service providers available, or you can use the open-source ollama.

I once wrote a program that simply converts ollama’s interface to OpenAI’s interface and provides the services. It implements a basic command-line program, reading and outputting complete large trunk of content, and reading and outputting streaming multi-section content.

[Here](https://github.com/yanecc/MockGPT) is my code repository, hope this helps you.

---

<div class="post-metadata">

**Author:** ![crimson-knight](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/crimson-knight/32/1643_2.png) [@crimson-knight](https://forum.crystal-lang.org/u/crimson-knight)\
**Post date:** [July 18, 2024, 12:08pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/35 "2024-07-18T12:08:56Z")

</div>

I did some testing with the Mixtral and I was getting the same issue. Which is _weird_ because I use that same model on my work laptop and it doesn’t freeze up like that.

Anyway, I redid the test with Llama3 8b and it worked fine. Llama is pretty… strange. It tends to follow directions 60-75% of the time, but definitely not well.

Relatively speaking, ChatGPT and Claude are models (and Mixture Of Experts) that have several hundred to over 1 trillion params. So you will not get the same kind of experience with an untrained local model as you do with GPT4o or Claude.

However, I do plan to create a way to train lora filters as part of Llamero. Based on research I’ve been finding (watch [https://www.youtube.com/watch?v=IeLxmeHdHWg](https://www.youtube.com/watch?v=IeLxmeHdHWg) for a great presentation on this) this is how you’ll get a local model to be as accurate and effective as the hosted larger models. Right now there isn’t an easy way to create the fine-tuning filters, so once I get that done everything will really come together!

Also, @Sunrise you may find this interesting, I wrote a quick example to show how you can get a local AI model to execute commands on your local computer.

 ![Screenshot 2024-07-18 at 7.59.11 AM](https://canada1.discourse-cdn.com/flex036/uploads/crystal_lang/original/2X/5/5cff03912e9629703162dcde048e3412937ff168.png)  
 ![Screenshot 2024-07-18 at 7.59.38 AM](https://canada1.discourse-cdn.com/flex036/uploads/crystal_lang/original/2X/3/3266f8ecbee8e0f936056fab936c354f0b5027cf.png)

I want to emphasize something important here: Llamero is to AI models as a db adapter is to a database.

I have another shard I plan to release that will be an agent development framework, which will make working with various models a lot easier by creating common abstractions for standard workflows. Basically like a langchain for Crystal and local models.

Definitely still experiment with Llamero and let me know any issues you run into. I appreciate your attempts so far. This is the wild wild west of tech right now, and fortune favors the bold!

---

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 18, 2024, 1:24pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/36 "2024-07-18T13:24:12Z")

</div>

So I am happy it’s not related just to me that issue x)

So I should stick on the version you ask on your repository or I can migrate to the last one ?

Basically, if I want to get the answers of the AI one by one without any shard, how we are suppose to process normally ?

So I will try again with your example now. I will come back to you

---

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 18, 2024, 1:39pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/37 "2024-07-18T13:39:09Z")

</div>

So I still have the same problem, just hanging after that:

```bash
zohran@alienware-m17-r3 ~/Downloads/Llama Test $ crystal test.cr 
2024-07-18T13:37:39.867706Z ERROR - An error was encountered: Capacity too big
2024-07-18T13:37:39.867717Z ERROR - An error was encountered: Capacity too big
2024-07-18T13:37:39.867925Z INFO - Interacting with the model...
2024-07-18T13:37:39.869880Z INFO - The AI is now processing... please wait

```

---

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 18, 2024, 1:41pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/38 "2024-07-18T13:41:34Z")

</div>

I think I find a part of the problem. When I started to inspect the log just by curiosity, I just noticed your shard pass the wrong number of threads on my computer:

```bash
[1721309859] Log start
[1721309859] Cmd: /usr/local/bin/llamacpp -m "/home/zohran/Downloads/Llama Test/dolphin-2.7-mixtral-8x7b.Q4_K_M.gguf" --grammar-file tmp/grammar_file.gbnf --n-predict 512 --threads 18 --ctx-size 2048 --temp 0.9 --top-k 80 --repeat-penalty 1.1 --prompt "
You are a Ruby on Rails expert, your are building an application using the rails CLI commands. Your response 

```

My computer can maximum use 16 …

---

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 18, 2024, 1:48pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/39 "2024-07-18T13:48:29Z")

</div>

So I did a test with the last version. It look like it work better …

```auto
zohran@alienware-m17-r3 ~/Downloads/Llama Test $ crystal test.cr 
2024-07-18T13:46:48.226821Z ERROR - An error was encountered: Capacity too big
2024-07-18T13:46:48.226832Z ERROR - An error was encountered: Capacity too big
2024-07-18T13:46:48.227003Z INFO - Interacting with the model...
2024-07-18T13:46:48.231427Z INFO - The AI is now processing... please wait
2024-07-18T13:47:10.648968Z INFO - The process completed with a status of: Process::Status[0]
2024-07-18T13:47:10.648988Z INFO - Recieved a successful response from the model
#<StructuredResponse:0x7f68d5fe4380 @command_to_run="">
2024-07-18T13:47:10.649155Z INFO - We have recieved the the output from the AI, and parsed into the response
sh: -c: line 2: syntax error: unexpected end of file

```

---

<div class="post-metadata">

**Author:** ![Fulgurance](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/fulgurance/32/1463_2.png) [@Fulgurance](https://forum.crystal-lang.org/u/Fulgurance)\
**Post date:** [July 18, 2024, 1:51pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/40 "2024-07-18T13:51:54Z")

</div>

But how should I proceed to get the answer from the AI, parse the output of the process ? Is there any way in your shard to launch the process in quiet mode ?

---

<div class="post-metadata">

**Author:** ![crimson-knight](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/crimson-knight/32/1643_2.png) [@crimson-knight](https://forum.crystal-lang.org/u/crimson-knight)\
**Post date:** [July 18, 2024, 1:55pm UTC](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019/41 "2024-07-18T13:55:56Z")

</div>

Can you share the code snippet you used?

The `quick_chat` method is just for getting back a raw string output from the process. In my example I used the `chat` method and it works quietly and handles parsing the response.

And nice catch with the number of threads. I haven’t found a way to detect how many GPU threads a processor has available yet to make this dynamic. I think that’s going to have to come once I bind the Metal library directly in Crystal instead of wrapping llama.cpp

[Previous page](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019.md?page=1)

[Next page](https://forum.crystal-lang.org/t/ai-library-for-crystal/7019.md?page=3)