I’m regularly annoyed by the usability issues of shards’ postinstall feature.
Some context on that: Shards' `postinstall` considered harmful
There are different reasons for shards to use it. I would like to identify the main use case profiles for postinstall to spark a discussion about possible improvements.
In this thread I would ask to report specific use cases in which you have used in your own shards or seen in the wild. I possible, a link to code is appreciated.
One use case is where you distribute some C file you need to compile for the shard to run. This can be everything from an entire separate project (eww) or something simpler, like including a shim-file to make it possible to use stuff that is not available using the regular C lib interaction that crystal supports. That can happen either because the .h file include macros or defines that needs to be available, or because functions are declared with the static inline modifier.
The use of static inline varies a lot between different libraries, but GitHub - yxhuvud/ior: IO Uring bindings for Crystal. is an example that make use of it. I also have a different project that is not published yet that have had to make MANY, MANY shim functions as basically everything is static inline. Perhaps the compiler could generate it so that I don’t have to? That’d be one way of solving the issue with static inline.
Though that would not solve the problem that sometimes C files need to be compiled. There are C libraries out there that are not available through package-management (in at least one gruesome case they are supposed to be generated based on packaged xml files. I really wonder what the wayland people are smoking).
Aside: I so hate the C library tarpit. I hope someone at some point say enough is enough and invent a format that is actually language-independent.
Aside 2: And before someone invoke the portability argument against makefiles: I don’t care, the libraries I work with only work on Linuxes anyhow. Having to build things manually is a nonstarter. You don’t want to do that with a moderately complex shards structure anyhow. Especially not for transitive dependencies.
A common use case is building tools that aid users of a shard in development.
The shard typically builds and installs itself in the main project’s bin/ directory via executables in order to be easily accessible.
An example that is encountered very frequently is ameba. It builds and installs itself for use in the main project.
Another aspect of this use case seen in ameba is that it also installs a Crystal source file in the main bin/ (that’s the make run_file part). It’s intended for customizing the ameba code by including additional plugins. Building that is the user’s responsibility, though.
This use case applies to many development tools, for purposes such as database migration (e.g. micrate), framework tools (e.g. lucky), or code coverage (e.g. crystal-coverage).
Main issues associated with the use case:
EDIT: I made a propsal to enhance shards’ behaviour for this use case by automatically building executables from target defintions: Connect `executables` with `targets` · Issue #591 · crystal-lang/shards · GitHub
What about automatically building and installing the proper version of a library on shards install? It’s something I am considering for raylib-cr to make it easier for newbies to “one-click install” the proper version of raylib with no fiddling. This is easier for Windows than it is Linux though because users may want to install shared objects in a different place than the default and postinstall doesn’t give me any way to let a user specify arguments. However, I can just as easily just let a user run an install script and let the power users just do their own thing.
Our main use with it in Lucky is just to run shards build, but conditionally so it doesn’t run when your app is in CI or being deployed to production.
scripts:
postinstall: BUILD_WITHOUT_DEVELOPMENT=true script/precompile_tasks
Could you elaborate a bit more on the use case? What gets build and why?
When you’re working on a Lucky app locally in development, you’ll get a lucky watch CLI task. This task would never be used in production or in CI. So when you run shards install on your app, the Lucky postinstall script checks that you’re not in CI or production, then it builds the lucky watch command in to bin/lucky.watch. When you boot your app you’ll have the lucky watch command.
It would technically be possible for us to remove the use of postinstall and tell people they have to build that command by hand before booting their app, but that becomes a pain for new users.That’s only 1 example, but there’s several targets that get built using this same method. At least 10 or 11 targets that get built.
Wow, that’s… quite a lot. I figure it takes some time for all of them to build? And that would happen every time you update the lucky framework dpendency, right?
Yeah… it’s not the greatest, but we haven’t come up with a better solution yet. The idea was actually to wait for the interpreter to become native out of the box, then just run all tasks in interpreted mode and skip all post installs 
It would be a balancing act with speed and compilation times.
I haven’t had a use for postinstall, and the few times I’ve seen it in shards in the wild where it was used to build a C library (zstandard, specifically), it was a bit of an eyebrow raiser. Not a little binding shim to interface with a C++ library, but the ZStandard itself as a library shared library. If it’s just a small shim, I can understand that maybe being part of the shards build process, but not postinstall.
If the program is used in development, I would think a better place for that would be within something like a Makefile or some other actual build system since Shards is about managing dependencies. Shards may install the Ameba code, for example, but the Makefile or whatever would build Ameba if needed.
To me, Shards kinda seems like it wants to be a dependency manager sometimes, and a build system other times.
I use the postinstall option for Anyolite (see https://github.com/Anyolite/anyolite/blob/main/shard.yml) to configure and build the required mruby/Ruby libraries.
Essentially, the postinstall option here is to run another Crystal program, which then runs the actual Rake build script.
If this seems overly complex, you’re completely right and I’m not too happy about it - but I don’t really see a better way to do this (the postinstall option on Windows allows only for running executables, so commands like make or rake do not work - unless I missed a possible recent change to this).
I decided to use the postinstall option to simplify the process of using Anyolite. If you want to start developing with Anyolite and have to install mruby or Ruby manually (which is both not trivial on Windows), you might get easily frustrated. Especially, since the point of Anyolite is to use Ruby with Crystal and NOT care for the whole C infrastructure behind it.
I don’t really have a good proposal to change postinstall. But at the current state it feels like a powerful solution to help solving the problem of platform dependencies - if it weren’t so restricted and, sadly, platform dependent.
The only use case for now is, when integrating ameba to my shards, post install can help build a new ameba binary in my shards ./bin folder, then i can always run ./bin/ameba in the CI or development env conveniently after shards install successful.
Wow. Building Ruby seems awefully huge and complex for a postinstall script. 
And it’s also completely unnecessary if you have the required libraries already available.
I don’t think this is quite right. If it was, it would definitely be a bug. But I’m pretty sure you can run make or rake on Windows as well (assuming the executable is available of course). So the intermediary step with compiling a Crystal binary seems unnecessary.
I would argue this isn’t simple or convenient at all. Building mruby/ruby has some prerequisites which cannot be expected to be available on a typical dev environment (particularly ruby and rake). So chances are that running shards install with anyolite as a dependency will fail. That failure prevents the entire install command from being successful and no dependencies will actually be installed. That means the documentation for anyolites build prerequisites is not even available.
In my opinion this is quite a poor developer experience.
My use case is pretty much the same as @jwoertink’s. A few shards I maintain use it to build executables:
interro builds the migration tool. Interro migrations are raw SQL files on disk rather than Crystal code so this is the only time it’s compiled.protobuf uses it to build the codegen so you can compile protobuf definitions into Crystal code. Protobuf code generators are not invoked by the user directly but instead by the protoc process as protoc-gen-#{language}.In the case of Anyolite, this isn’t really a postinstall issue but rather crystal-lang/crystal#9030 (which should be resolved by crystal-lang/crystal#13567) but this is definitely a justified reason to use postinstall, users would just need to ensure that other dependencies like Ruby and Rake are installed before installing (which one can assume so if you are using a library like Anyolite).
I don’t see how #9030 applies here.
That issue is about the argument array which you can optionally pass to Process.run, but that’s not used when running postinstall scripts.
They are a single string command executed in a shell. There is no issue with that on Windows.
It applies because simply setting postinstall: rake build_shard works on Linux environments (tested on WSL) but fails on Windows with the following error:
The postinstall hook itself (ideally) works fine in all other cases though.
Actually, the requirement of having Ruby and Rake installed is something I’d like to circumvent in the future, but as of now, there isn’t really a similar build tool in Crystal I’m completely happy with (and thus willing to invest time for porting and testing).
Also the build process for mruby is actually dependent on Rake, so the best thing (besides reimplementing the whole mruby build process in Crystal) seems to be to just stick with it.
For GTK, the bindings generator, gi-crystal, gets built on postinstall https://github.com/hugopl/gi-crystal/blob/main/shard.yml#L15
which is a bit painful to do in environments such as flatpak:
This is indeed true because LibC.CreateProcessW will append .exe to the process name automatically only if shell mode is used and a file extension wasn’t given; in other cases it won’t automatically locate an executable using %PATHEXT%, which PowerShell and the command prompt do. If Ruby is installed using RubyInstaller for Windows then only rake (Ruby script without extension) and rake.bat would be available in %PATH%.
So either that PR goes forward or we emulate %PATHEXT% processing ourselves.
