I think we’re a ways off from being able to run inference inside Crystal code unless someone’s published something I haven’t found yet. We had a discussion about the topic of AI in this thread recently, too, which may be helpful for you.
There are a few shards you can use to send AI prompts to hosted LLMs.
You can also use Ollama on hardware you control or OpenRouter to have access to a wider selection of models. Ollama and OpenRouter both use the OpenAI API (for chat completions, anyway), so you can use the openai shard to talk to their APIs by setting the OPENAI_API_BASE env var. Just keep in mind that that shard doesn’t use an HTTP connection pool (the Anthropic and Google ones do) so if you’re using it in multiple fibers, such as when calling out from your own backend service, you need to maintain your own connection pool.
If you want to use OpenAI, Anthropic, Google, or OpenRouter, you’ll also need an API key from them. Ollama can be used with any API key.