Vector databases in Crystal?

Community
1

Are there Crystal API for vector databases?

2

for ChatGPT Retrieval Plugin?

3

No, I was asking the question more generally.

A couple of years ago, I wrote an experimental search engine, using vector space, in Go and in order to make the database persistent I used SQLite to store the vector data (and some Go code to implement queries based on proximity).

Since then, owing to the role of vectors to represent data in AI applications, I was wondering if I could not rewrite my earlier code using a ready made solution based on current existing vector databases.

On the other hand, I would like try to rewrite said code in Crystal for a couple of reasons:

  • Out of curiosity.

  • Out of personal preferences over Go (easier OO, more conciseness, better readability, less boiler plate, String are classes (or Structs), not a simple package; Chaining, etc, Shards for NLP, Shards for Wordnet…

So I was wondering if anybody knows of a possibility to use vector databases from Crystal or alternatively if anybody is working on the topic or on something similar.

4

Here is one possibility (via PostgresSQL) : GitHub - pgvector/pgvector: Open-source vector similarity search for Postgres

5

I’d really like to see an ORM for using Pinecone https://pinecone.io/

Maybe that’s something I can include into Amber 2.0, I’m trying to focus the framework on leaning into the AI trend.

1 Like
6

On the other hand, the Postgres option is probably easier to implement as an API or perhaps as a shard.

Also, it is free (financially) to use, I think.