Maybe I’m wrong about this but doesn’t the macro {% run some_crystal.cr %} have the same problem? I was thinking about trying to use it instead of postinstall since I could do some sort of checking via crystal (instead of sh) to see if they actually have the proper library installed. I think this would give more control but I’ve never tried it.
I’m going to sit down tonight and work on raylib-cr and see if I can get it’s install process working with this method instead of making users run install scripts, or via postinstall.
Depends on what problem you refer to… It doesn’t have the same problems as postinstall, but some are similar, others different.
I really don’t get why there’s such a conviction that a shard should somehow automagically install libraries that it links against.
Sure, it could be a nice thing if it worked. But building dependencies is quite a challenge, especially if you want it to be reliable (don’t even start with broad portability).
Whether it’s postinstall, macro run or any other mechanism that somehow hooks into the Shards/Crystal build process: The only thing you can reliably expect to be available in a build environment is the Crystal compiler and maybe shards (even that might not always be true).
There’s not necessarily a shell (at least not any specific one). There’s not necessarily a toolchain for C or any other programming language. Or any generic build tools.
A host for cross-compiling might not even have a linker and the libraries that the Crystal program links. Those would only be needed on the target host for linking. (for this use case you wouldn’t even want to build libraries automatically because you won’t need them on the build host)
If you can build your dependencies with basically no other dependencies than a Crystal compiler, congratulations.
If not, your setup is likely going to miss some use cases. That means you should take good care that your shard is usable even if any automatic build steps fail. Ideally it should be able to skip this entirely, in order to provide pre-installed libraries for example.
Anyway, before doing any automatic dependency building, I think the first step should be to have proper documentation about the dependencies. Which libraries your shard needs, maybe which versions are compatible. A reference where to get the library would be nice and maybe some build instructions for it, or a link to a resource about this.
This should give all users of your shard enough information to make the library available in a way that works for them.
Maybe they already have the library. Maybe their package manager provides it easily. No need to self-build, then. Maybe there’s a build system where dependencies are supposed to integrate.
There’s many unknowns when it comes to building and installing software (which version to build? where to install? …). So it requires a lot of flexibility. There are complex build systems which deal with these kind of problems. I don’t think any Crystal shard (or whatever source package) can realistically deal with all these challenges.
Looking at the readme of this shard, I feel I’m missing a lot of relevant information about dependencies.
It’s a shard with bindings to raylib, so it’s safe to assume that this library is required.
But besides from that there’s just a reference to run some install script which is available in a couple of flavours for different OSes.
Looking at these scripts, I see that the Ubuntu one installs a bunch more libraries from apt. Not sure if those are strictly build dependencies for raylib. And aren’t they necessary on other platforms?
I further learned that the install script (at least on Ubuntu) also builds raygui and a bundled library miniaudiohelpers.
This is all information that I would love to see documented more explicitly. That would be more helpful than an automatic build script. The build script may work for me, or may not. If I know what exactly is needed, I can provide that myself.
Sorry for picking on this project specifically. You brought it up though I think it fits well as an example that we can learn and improve on.
I was more thinking about using macro run to do a last resort style dependency checking. Like, just go through the path and check if raylib exists on the system where crystal was going to look for it anyways, and swiftly add it into the path/env/compile raylib or something only if it would fail… but I’m not sure if its possible, was going to play around with it. Is that a more acceptable solution?
miniaudiohelpers is actually something I’d like to get advice on, because I know I’ve asked a bit around but no one has had a great fix for it. Basically, a very important audio function in raylib relies on an AudioCallback which takes a struct which has a member whose actual size is different per system/implementation/compilation?. Basically I can’t detect what mode someone compiled miniaudio in (like using 32bit floats or 64bit floats or something like that) and so I need to detect the actual struct size to insert into crystal using sizeof in C. Technically I only need to do this once per system but there is no other nice way to do that unless I use postinstall to generate crystal code or this method with macro run generating code. That’s what miniaudiohelpers does. It may be totally unnecessary, and maybe I just got lost in the sauce when developing this thing but I couldn’t find any nice way around it and I did want to add android and raspi support at some point and this is needed.
IDK if that may be a good reason for postinstall if someone has to install an extension of some kind.
If that was reliably possible, yeah. The thing is, even the Crystal compiler doesn’t know which libraries are available. Realistically, only the linker will know when it tries to link them
The compiler gains some information via pkg-config (which is useful for exactly this purpose). But that’s is not always available, and packages might not be registered with pkg-config. So in general, the information from pkg-config is inconclusive. Even if a package cannot be found that way, it may still exist and be available for the linker.
So, you would have to ask the linker whether it can find the package.
To not further derail this one, maybe you could create a new thread for miniaudiohelpers?
I don’t really see how this is different from dependencies on any other library. It still require action before using the shard. Of course, in an optimal world the different tool chains would work together and make the experience seamless across language barriers, but I have seen no packager/build system try to implement that yet.
The only real difference is if you are a packager that wants to repack into distro packages, but then you are also in the right place to improve the situation (for you. The difference doesn’t matter for most).
I’m with @straight-shoota here, not too fond of shards just running things.
PHPs composer has the ability. Then someone noticed that this might present a security issue, so now composer asks if it should allow the package to run its hooks. Which 99% of developers then blindly allow. Hard to spot the improvement.
Thing is, when you go down the “installing dependencies should take care of everything” route (composer, npm), you end up doing a bad make imitation, because the only thing you can really be sure is available is the language itself.
So, how about going in the other direction and remove postinstall and replace it with a facility for a shard to get a message printed to the user on install/update? The message could then point the user at documentation/scripts and let it be up to the user whether to install by hand or add it to their Makefile/Taskfile/Whateverfile.
Shards could also implement this, I don’t see this as any reason to not use postinstall though. It could also just print out the running command/script. If the user sees something in it that they don’t like then they can stop the program.
Removing postinstall feels like a step backwards. The fact of the matter is that postinstall is needed for quite a few shards as it is, and that number is only going to increase as the community gets bigger, so I believe the conversation should instead be about making it safer to use — and even that is just speculation from this thread as no one has reported any actual issues with postinstall security thus far.
@straight-shoota came up with a concept for platform-specific postinstall scripts in this post, I implemented this in an upcoming project and it works for the most part (minus a few Crystal-specific target triple issues). I don’t see why this can’t be used to support cross-platform postinstall scripts.
I’d disagree on that as an absolute statement. It may be needed for a certain way that shard authors may intended their shard to be used. But it’s not needed per se. There’s always alternatives. Those may require an extra step, and thus is a bit less convenient. But ultimately much more flexible and powerful.
I think postinstall is inherently flawed from a security perspective and impossible to fix. The only “fix” could be not run it automatically and ask for explicit approval. But then there’s not much benefit over an independent solution.
This would overcome one hurdle for having distinct entry points. But there’s still a bazillion other prerequisites that most install scripts would have. Fulfilling those is a much bigger hurdle for actual portability.
The shards depend on postinstall as part of the installation process, for example, with Ameba. If the team suddenly decided to stop using postinstall and have users start manually running the build command, it would lengthen the install process and would be a less developer-friendly experience. So I’d say it is definitely needed.
Agreed, but prompting for explicit approval to run scripts could be configurable with environment variables just like other options in Shards.
I’d say that’s where things go beyond the scope of Shards. I’ve seen in this thread (and others) people are confusing the idea of Shards being a package manager and Shards being the next Make (or something of that nature). If a shard has complex external dependencies then it shouldn’t be included in the postinstall script; clear instructions should be provided for installing those. Shards should only be concerned with managing things that it has certain control over, of which most shards using postinstall fall into that category.
Most of the postinstall scripts in this dataset have requirements beyond shards. They’ll fail if the environment doesn’t provide some specific prerequisites.
I would suggest that instead of affirming that postinstall is definitely needed, let’s imagine it wouldn’t exist yet. Let’s look at the problems that shard authors and users have and consider solutions for those problems without being preoccupied by postinstall.
The same is true of every single library linking against a .so. That is perfectly fine, and not a reason to break automation. There is nothing special about build environments in this regard.
But it’s failing at different stages. The main purpose of shards is to make the sources of dependencies available. There’s a lot of things you can do with that without a dynamic library for linking.
But if you make it configurable, you’re kinda defeating the purpose, as you’ll end up with a lot of people just setting the var to get rid of those annoying prompts.
re-reads the thread We seem to be going a bit in circles.
That leaves the difficult build cases that @straight-shoota argues isn’t really possible to automate in a way that satisfies everybody and doesn’t fail in annoying ways in edge cases.
It’s been suggested agreeing on a canonical non-crystal build tool. I’d like to suggest https://taskfile.dev/ . YAML should be familiar to any Crystal developer, and it does have the advantage of being a static binary you can just download if your local package system doesn’t have it.
True, but much like this thread, this all stems from theoretical security issues that have yet to come up. I’m not saying it’s not good to plan ahead, but we do need to be realistic given the current context.
Except that’s not the case… People originally asked for postinstall to handle these cases, and apart from cross-platform support it has worked fine.
This is just my opinion but Taskfile, much like YAML, takes what should be a fairly straightforward concept and runs with it. I personally don’t think it would work with Crystal’s ecosystem, and many people agree it shares resemblance with nightmare YAML configurations like the OpenAI Spec and Kubernetes.
Anyways, opinions aside, Shards is already doing a fraction of this (minus cross-platform support), and the majority of libraries won’t need the full capability of Taskfile, so I don’t see how this is better. There also isn’t any difference in terms of security between them, so the original issue isn’t being solved.
It might be theoretical at this point, but security stuff is one of the more painful places to go “whoops, I didn’t think of that”. And it’s not just the security thing, but the much more slippery notion of “should we really?”.
Whenever someone goes “wouldn’t it be nice if” and goes implement something, it has consequences. Someone thought it would be nice to run something postinstall and now we have shards that build tools whether you need them or not. Sure, you can do stop it with a switch, but you’re putting the cognitive load on those who do not, in favor of those do or don’t care.
And when you provide a feature, it’s the hammer that people is going to use to hammer ind their screws. We’ll see postcommands that’s Crystal programs that’s a lousy re-implementation of Make as someone is sure to think “well, Crystal is the only thing I can be guaranteed is available, so…”.
Good design isn’t just about what to add, it’s just as much about what to leave out.
Well, it’s a bit of a lousy solution if you ask me. “I would like to do X”, “Here’s a hook that’ll allow you to do whatever the hell you want, knock yourself out”.
But, yes, it works fine if you chose to ignore cross-platform support and postinstall hooks failing on missing external dependencies. But “cross-platform” apparently includes Alpine, which is a tad too narrow a definition for my taste.
Besides, you’re sneaking in external dependencies without declaring them, and replacing them with a recipe on how to create them, which might or might not work in the user’s case. Shards never going to be the perfect tool for working with dependencies in everything else under the sun, so if we end up calling make from postinstall, why not skip the middleman?
Tell me about it, some of the Taskfiles I’ve seen at work… I survived the XML based build tools (my god…), so maybe I’m a tad pragmatic, but a dependency you can just wget gets a plus in my book. I’d love to see a Crystal build tool that uses a file with a syntax resembling a simple makefile, but without the eccentricities, but considering the amount of projects ending in “make” or “ake”, it would seem that it’s a very tough problem to crack to everybody’s (or even the majority’s) satisfaction. And cake has already been taken (thought crake isn’t and is apparently a bird, so that would go with the Crystal mascot).
Oh, and while I agree that Kubernetes is painfully verbose, it does have one redeeming characteristic: the YAML files describe what is needed, not how to do it. Which is what I want from a shard.yml.
I didn’t read all of the recent replies, so apologies of this was already suggested - what if running postinstall was an explicit property in the shards file? Something like:
Any direct dependencies that have that property (copy pasted from GitHub readmes, as is tradition) trigger the postinstall, while indirect dependencies that have it trigger a warning like:
Indirect dependency blah has a postinstall, but it’s not being run. To run postinstall, please add this dependency as a direct dependency by adding {shard text here} to your shard.yml and rerun install
Or however it’s best handled.
Thoughts?
(On phone, no idea how if this formatted correctly)
compiling ameba everytime is annoying. I have ameba and it is applied globally already, why I have to recompile it everytime I install/update a shard?
skipping postinstall is not ideal either as some C-native library needs it.
=> better just printing some messages as postinstall hook instead.
I think it fits well as an example that we can learn and improve on.