Croupier, a tasks / dataflow library

ralsina · August 25, 2023, 2:45pm

I posted this on reddit but a comment said the forum was where people are nowadays, so here it is

I have published the 1st sort-of-feature-complete version of Croupier at GitHub - ralsina/croupier: A library to create and execute tasks with dependencies

You may wonder what a tasks / dataflow library is?

Well, it lets you define tasks which are procs and connect them in a dependency graph. Some of these tasks may consume/produce files or values from a k/v store.

So, when you tell Croupier to “run” it will examine the state of the whole system and execute exactly the things that are needed, in an order that guarantees their dependencies are ready for them.

Why? Well, for example for another project, GitHub - ralsina/nicolino: A not-quite-minimalisting SSG written in Crystal. In a SSG you want to take markdown files, images, CSS, JS, etc, smush them through templates and generate HTML. Also you want to classify them into categories, and create indexes, and so on.

Doing that in a large site takes time, so you want to build things incrementally when needed.

Using a dataflow library like Croupier you don’t need to care about how to do that. Just create the tasks, link them via dependencies, declare what file it generates, and ask it to run things.

Here’s a toy example:

require "croupier"

b1 = Croupier::TaskProc.new{
  puts "task1 running"
  File.read("input.txt").downcase
}

Croupier::Task.new(
  output: "fileA",
  inputs: ["input.txt"],
  proc: b1
)

b2 = Croupier::TaskProc.new{
  puts "task2 running"
  File.read("fileA").upcase
}

Croupier::Task.new(
  output: "fileB",
  inputs: ["fileA"],
  proc: b2
)

Croupier::Task.run_tasks

Moving that complexity into a generic library means Nicolino can stay small and each feature is just a small fragment of uncoupled code with defined inputs and outputs.

I know it’s a pretty niche thing, but maybe someone will find it useful.

Blacksmoke16 · August 25, 2023, 3:23pm

One thing you may consider doing to improve the DX is to allow Croupier::Task.new to accept a block, vs needing to manually pass it a proc. E.g.

Croupier::Task.new output: "fileA", inputs: ["input.txt"] do 
  puts "task1 running"
  File.read("input.txt").downcase
end

This could be a pretty easy integration since captured blocks are essentially procs, so would be able to pass the captured block to the existing constructor’s proc parameter.

ref: Capturing blocks - Crystal