Moku — federated social network

News
1

I’ve been working on a project for a few weeks called Moku, a federated social network similar to Mastodon, written in Crystal. I have an example of it running at https://jgaskins.wtf/ — check out the public timeline (currently implemented as just showing every post in the instance’s public timeline all within a single page).

It’s still super early development (and the code is still a mess as a result), but it works pretty well. You can follow people from other instances, post, like, and boost. Replying isn’t built yet, unfortunately. Notifications could work if I just build the UI for it. :slight_smile:

Mostly this was an experiment for 3 things:

  • Build a social graph with an actual graph DB
    • Mastodon uses Postgres, which is amazing and flexible and all that, but isn’t great with graph queries
    • Most requests to Mastodon involve around 20 SQL queries, most of which could be coalesced into a single graph query
  • Running the Neo4j Crystal driver against Neo4j’s new DBaaS (called Aura)
    • Needs a little work because Aura uses clusters exclusively but the driver is designed around single connections, and your DB subdomain doesn’t guarantee that you connect to a writable node in the cluster
    • Basically, it means I need to build up a unified interface for both clusters and connections the way the official drivers use
  • Building a social app using pure server-rendered HTML rather than a SPA like Mastodon and Twitter use
    • I don’t know if this is something that will stick around but I really like it
    • Coming back to the app on a mobile browser will put you back where you were, not at the top of the feed
    • Liking/boosting a post does take you to the top of the page but the timeline is paginated so it’s not too far from where you were

I compared its performance to Mastodon for giggles by repeatedly hammering the home timeline. Both apps running within the same Kubernetes cluster). The difference is hilarious.

Mastodon vs Moku performance
Mastodon vs Moku performance2200×1704 480 KB

Mastodon uses 61.7% CPU just for the Rails server process and has local maxima of around 80%. Moku reached a whopping 1.1% local maximum and averaged around 0.7% CPU for the same workload. I think a big part of Mastodon’s performance woes are in ActiveRecord. With the Crystal Neo4j driver, I don’t even use an ORM. I map nodes in the DB to objects in memory, certainly, but I use query objects to fetch data and make use of streaming results.

Memorywise, Mastodon’s web process alone took up 470MB, but it also uses a Node.js websocket server and a Sidekiq process, which bumps the Mastodon installation up to about 800MB of RAM consumed. Moku uses 25MB. Because it’s still early development, I can just use Crystal fibers as a stopgap for background jobs so I don’t need to mess with message queues yet, but I do expect that memory consumption to grow (omg, it could be FIFTY MB) once I build that out for durability’s sake.

It’s also not using a web framework. I’ve built a mixin that lets me handle routing similarly to Roda and sessions are stored in Redis. Rendering is done as partials, I’m not even using a CSS preprocessor (or even CSS files) yet. That’s all just hardcoded into the HTML currently. I’m enjoying the minimalist development approach here. Maybe as the project grows I’ll be less interested in that and want to move to more conventional practices, but I’m finding this works really well for now.

Anyway, I’d love to hear y’all’s thoughts on the project. This is my first time using Crystal to do anything HTML-based. I’ve mainly been using it for back-end services, passing around JSON and Protobufs over HTTP and RabbitMQ.

11 Likes
2

Ugh I remember those rails active_record slowdowns…so confusing…go crystal! :)

2 Likes
3

Very cool. Can’t wait to see where this goes.

2 Likes
4

Very cool I cant wait to see where this project goes.

1 Like
5

I think this is great!

If there’s a way to bring Moku’s functionality to the same level as Mastodon and show that:

  • CPU and memory usage is way lower than Ruby/Rails
  • The development cycles are similar (maybe just the trade-off between compile-times and type safety)

then I think this could be a really good use case for Crystal. If people from Mastodon start working on this too it might mean more people will contribute to Crystal and its ecosystem, which ultimately benefits all of us.

I personally don’t care whether Crystal is popular or not: as long as it’s here and people can use it and it makes them happy then it’s fine for me. But of course if it gets more popular it means it can get even better (more collaborators!).

Out of curiosity: how long does it take to compile it on your machine?

I’m also curious: does Mastodon support any form of runtime plugins, probably written in Ruby or something else? Because if that’s the case I don’t know how something equivalent could be done in Crystal.

4 Likes
6

Ooo, this looks very promising. For the last 25-30 years a group of my friends have been running a chat server written in LambdaMOO. It’d be great to replace that with something written in crystal. (there are very few people who know how to program a LambdaMOO database!). I’ll be interested to see how your project works out.

7

It might be tricky to have full-featured plug-ins (whatever that means…), but it shouldn’t be too hard to support simple filter-like plugins. A program that the Moku server could start up on each connection to it, Moku would write to stdin of that program and then that program would write what the user receives on the other end of the connection.

1 Like
8

I’m away from my dev machines and just have my old fanless MacBook with me on vacation, but:

➜  moku git:(master) ✗ time shards build
Dependencies are satisfied
Building: moku
shards build  4.79s user 0.89s system 135% cpu 4.193 total

Not fast, but this is a cold compile on an underclocked laptop. On my iMac it compiles in a pretty reasonable amount of time with Sentry. Every now and then I have to refresh the page a couple times before the changes show up, but it’s otherwise fine. The same thing happens when I work on Ruby apps that don’t use Rails — I use the rerun gem to reload the app when I make changes to files and it takes a couple seconds for the new instance of the app to load.

Not that I know of and a cursory web search doesn’t return anything useful. But I bet we could probably support plugins as part of the build process if it came down to it.

1 Like