Hey, I have an idea for a sync tool that’s not strictly related to Crystal other than the potential for a client library, but I’m testing the waters to see if there’s any interest in this tool or if I should just keep it for myself.
*Update*
I’m initially just trying to do some research. If you could help out by answering this short survey, you’ll be the first to know if I end up creating anything.
TLDR;
I’ve built a library that makes it surprisingly easy to sync data between clients and a server with a lot of really interesting benefits. All the client has to do is define some events (just a few lines of code per event) and a handler for each event. Now, whenever the client needs to change some state it just needs to record a new event. Syncing happens magically.
A little more detail
A few years ago I started working on a project that needed to sync data to a server in environments with unreliable internet, maintain a granular history of changes, and automatically resolve conflicts while preserving the details about the conflict. In the end, I came up with a really nice solution. Now, I’m working on another project that has a few more requirements.
Here are a list of my combined requirements including a few dreams for the future:
- no-knowledge server (dream). e.g. the server knows nothing about what kind of data it is syncing.
- support e2e encryption (dream).
- online/offline peer-to-peer syncing.
- eventual consistency across all devices and servers.
- support syncing to multiple clients and multiple servers (dream).
- permanent and granular history of changes.
- scope-based access to segments of the synced data.
- easy client configuration.
In my experience the more data you need to sync the more complicated it gets. However, with this library I’m able to define new data to sync, in constant developer time. (I’m syncing 28 different types of data mutations in one application, and it’s a piece of cake to add more).
The design approach I took borrows a few Domain Driven Design principles, which might be a bit overwhelming to new developers, but makes life a whole lot easier on large or long-lived projects.
I keep getting projects that require some sort of syncing, and it would be nice not to have to keep duplicating server infrastructure and client code. I’d love to have some sort of SaaS that would make this actually feel like plug and play in most cases.
I’m looking for input
Have you had to deal with syncing before? Does this sound like something cool or just meh…?
I’d love to pick your brain a little to see if I can make this library usable for a general audience.
A few things this is not
Someone recently pointed out some other syncing alternatives like https://zero.rocicorp.dev and https://electric-sql.com.
However, those don’t offer quite the same thing. What I’ve built is like a stream of events that relies heavily on Domain Driven Design. I basically implemented this Event-Sourcing as a DDD pattern. About Event-sourcing and how to combine… | by Eliran Natan | The Startup | Medium