Live coding a URL status checker in Crystal

Learning Resources
1

Hi all :wave: I’d like to share a bunch of live coding sessions I’ve streamed on Twitch.

The idea is to explore Crystal’s concurrency, with emphasis on Channels, Fibers and the CSP model, by means of building a simple terminal-based app.

Here is the table of content so far.

Session 1

  1. Initialising a Crystal app
  2. Making HTTP calls
  3. Reading from config files
  4. Concurrently checking URLs with Channels and Fibers

Session 2

  1. Classes and aliases
  2. Extracting tasks into modules
  3. Scheduling periodic tasks

Session 3

  1. Type-safe config handling
  2. Sensible monkey patching
  3. Logging across Fibers
  4. Signals, timers and select statement

Session 4

  1. Fibers owning Channels
  2. Terminating groups of Fibers
  3. Propagating Channel closure throughout a pipeline
  4. Waiting for a pipeline to be done

I recommend watching the videos at 1.5x the regular speed for better enjoyment :robot:
I’ll add new ones as they come, in the comment section.

EDIT (Feb 2020): I collected all the videos from the series, plus some comments and errata into a handy blog post.

16 Likes
2

Thank you so much for doing this! Especially Fibers related stuff.

1 Like
3

Session 5 is out!

We talk about two-way communication between Fibers, taking inspiration from Elixir’s GenServer and Akka’s Actor. Hope you enjoy this!

3 Likes
4

Session 6 is out :tada:

We talk about

  1. Partitioning and merging channels
  2. Processing data on a sliding window. In particular, we define a processor to compute the moving average on a set of response times.

Make sure you don’t watch all the videos in parallel :grin:

4 Likes
5

Session 7 was all about testing concurrent code.

  1. Writing robust tests for our channel partitioning method :weight_lifting_woman:
  2. Refactoring to decouple concurrency and business logic :merman:
  3. Testing non-deterministic output :woman_shrugging:
  4. Writing time-dependent tests :timer_clock:

I feel like much more can be said on the topic, It would be great if you could share your testing strategies for concurrent code.

4 Likes
6

I have worked through the first two sessions today and I must say: I love these videos.

Having a knowledgable and measured host is lovely, having that host be exceedingly likable and considerate is divine.

I’ve been trudging through the Programming Crystal book these past few days, though stumbling upon these videos has accelerated the progress of my learning in a most appealing manner.

Thank you for them :)

3 Likes
7

Thanks for your kind words. It means a lot to know that someone is finding the series useful :blue_heart:

I’ll be doing some more live coding on Twitch today at 8pm UTC.
You can preview the slides here :clipboard:

We’ll spend some more time on graceful termination of concurrent apps, implement a simple alerting system and polish the terminal UI :sparkles:

4 Likes
8

cool.

9

The season finale is out! We talk about

  1. Bringing rogue fibers to order with a more robust termination strategy
  2. Adding an alerting stage to our pipeline :rotating_light:
  3. Polishing the terminal UI with ncurses
1 Like
10

Nice tutorial, but I wonder how you configered your editor for completion as that is a nice feature.

11

In which of the videos is there autocomplete?

I think the autocomplete you see is just the standard one that comes with the editor, or maybe a plugin: suggest words fine in any of the open files.

12

Hey @Alain, I’m using Visual Studio Code with Faustino’s Crystal plugin.

Contextual autocompletion is provided by VS directly - nothing too fancy.

If you are looking for more advanced autocompletion / go-to-definition type of features, then you’ll have to install scry and have the VS Crystal plugin point to it’s executable.

I’m currently experimenting with the VS plugin + scry integration. It works fairly well with the standard library, but can get a bit confused with your application’s code, in my experience.

13

Hi @lbarasti I’m happy you are enjoying VSCode :smile: , this extension still requires a lot of work, although, scry can be a way to get more features and fix some issues. We still need to figure out a better way to detach crystal tools from compiler itself and use them inside scry via Language Server Protocol

2 Likes
14

@faustinoaq I was planning to extract parser info as a JSON. I was planning to modify ToSVisitor to make it as toJsonVisitor, so we can dump parser info as JSON. You can perfectly access Crystal parser classes from your cade as in this example:

require "compiler/crystal/syntax"

module Crast
  VERSION = "0.1.0"

  source = File.read(ARGV[0]).strip

  ast = Crystal::Parser.parse(source);
  pp ast
end

so if I can properly implement JSON AST dumper and cache it somewhere (and invalidate cache if that file was modified) it will help tremendously.

1 Like
15

Nice :smile:

That’s a better way to get things done, I was reviewing scry code and currently we using regex to get completion done:

def parse_context
  lines = @text_document.source.lines[0..@position.line]
  lines[-1] = lines.last[0..@position.character - 1]
  lines = lines.join(" ")
  case lines
  when METHOD_CALL_REGEX
    Completion::MethodCallContext.new(lines, $~["target"], $~["method"], @method_db)
  when CLASS_NAME_REGEX
    Completion::ClassModuleContext.new(lines, $~["target"], @method_db)
  when INSTANCE_VARIABLE_REGEX
    Completion::InstanceVariableContext.new($~["var"], lines, @text_document)
  when REQUIRE_MODULE_REGEX
    Completion::RequireModuleContext.new($~["import"], @text_document)
  else
    UnrecognizedContext.new
  end
end

So, using crystal parser to get detailed code information would be really good and I’m guessing it can improve tool performance as well.

See: https://github.com/crystal-lang-tools/scry/blob/master/src/scry/completion_provider.cr#L30

16

@lbarasti Can you please share more about linking the plugin to scry?

I’m using VSC with the plugin as well.
I put the scry executable in “/usr/local/bin” on macOS Catalina, so the path I’ve entered in the plugin’s settings is “/usr/local/bin/scry”.

Scry successfully starts up, but then generates a folder structure like “Users/zolo/.cache/…” in project the working directory instead of creating its files in the actual Users/zolo/.cache/… folder on my machine. After that, it promptly throws a “File too long” error.

working_dir