I think this is a no issue:
run macro.shards --skip-postinstall
Just need to decide what’s more annoying, have skip-post-install ON by default and run post install manually later with something like shards post-install or have it OFF by default like now and remember to use --skip-post-install when you think isn’t needed, with a detail, the later is already ready and used by a lot of people.
Exactly!
You also download a Ruby gem and run the code. Do you check all dependencies? Do you check all the code to see if it’s doing something malicious? I don’t think so…
I don’t think it’s a good security concept to do nothing because other things are similarly bad. Sure, it will not make everything secure, but it removes at least some vectors. And we can try to improve things like run macro as well, but that’s a different topic.
It also makes a difference that postinstall runs directly after downloading dependencies, thus before you even get a chance to look at the code and see what it does. This implicitness establishes a mentality that it’s good/acceptable to just run random code downloaded from the internet without checking 
Perhaps the most practical problem with the way postinstall is used is the catastrophic portability UX. The vast majority of scripts has quite limited platform support. Many only work on some flavours of GNU/Linux, some on MacOS. Linux with musl or BSDs are even rarer. And Windows support is pretty much nonexistent.
The problem is not that those systems are unsupported. It’s hard to take many different platforms into account, especially when the shard owner doesn’t even use them.
The problem is that the postinstall tries to runs indiscriminately on any platform and then fails. IMO that’s an unacceptable UX failure when you just want to check out some code and the whole thing explodes.
I obviously can’t speak for all crystal devs, but if I’m going to check out a shards code, I’ll likely git clone it or look at it through a web view before using shards to install it. By the shards install time, I’ve already made the decision to use it, and accept all consequences thereof. Usage of postinstall does not add nor take away trust from my perspective.
Again, if the you just want to check out some code, I think git is a much better tool for this than shards. I’ll posit that having a postinstall explode on unsupported platforms is the best possible UX, since you’re not in a state of:
postinstall failing silently and being under the impression the shard will workpostinstall not running and being under the impression the shard will work until a shim explodes (much later)postinstall succeeding (and ignoring any errors) and being under the impression the shard will work.I fully agree. What I don’t agree with is pretending that the problems that people use postinstall to solve goes away by pretending the problems don’t exist and simply remove postinstall. For example, it would be much nicer if it was possible to declare how things are to be built on different platforms in shards.yml, and then having a shards command to either run or output the commands necessary, for either a single shard or all shards for a project.
This would not necessarily have to be shards itself, but it does need to be aware of how shards work, and it would also be very nice to have the possibility to output the relevant shards to run the update scripts for on shards update.
Also different use cases may have different needs - a function to install a command in the shell may be totally different from a function to build dependencies. The latter may fail if platform doesn’t support a linked library, for example. It is also likely to have additional preconditions (eg installed system packages to link against). So it may be that several different functionalities may be necessary to avoid having to use postinstall.
So the shard doesn’t support windows, a shard without post-install that doesn’t support windows could explode too when trying to compile or even worst, when running.
I would recommend avoid pairing the fate of postinstall to whatever happens with macro run, as you can still do git clone and get bitten by a macro shelling out.
Compare and justify what Crystal decides against what Rust/Cargo, Ruby/RubyGems, NodeJS/NPM/Yarn or other languages and their package managers is also a limiting view.
A package in Cargo can be using unsafe code, a RubyGems can be performing malicious actions during extension compilation phase (extconf.rb) and someone can do package squatting and you end installing a crypto wallet steal program
Shards have no definition or enforcing of supported platforms, and adding that can also hinder what you do or how you use the shard when attempting to cross-compile projects… it just gets more complicated.
An alternative could be disabling post install by default and show indication that postinstall actions were defined, allowing the user to shards postinstall ameba to run Ameba’s postinstall actions, or allow --dry-run or similar to only show what ameba is going to execute.
As for Crystal macro run itself, a safer --warn-macro-run or --no-macro-run that shows a warning or error, respectively could be an option for those that want more security 
Cheers,
Luis
I also prefer the download, view code (if desired, but at least an option before run), then run
From the angle of safety, i consider a better solution is score on shards, good maintained shards very likely not exists big problem for postinstall, though, how to score is a issue.
Macro means you’ve decided to compile the code. but with post install you can get stuff injected and ran just by updating dependencies.
If cross-platform portability is the sole issue here, that sounds like a use case for CMake to me, and there are already many C libraries that provide a CMakeLists.txt. As a last resort one can simply run another Crystal script from postinstall since at least Crystal’s compile-time flags can detect some OS configurations. So I don’t think “shards that depend on a POSIX shell will break on MSVC” is a compelling reason to drop postinstall.
Operating system and shell are not the only challenges for portability. If the only problem would be that you need different sets of commands for POSIX and Windows, that would be a nobrainer. Build environments can be much more complex with the need for specific libraries and toolchains.
Lack of configurability is another concern of automatic postinstall scripts. Maybe you want your tools to be built in release mode. Or with debug symbols. Or whatever. If the build implicitly happens as part of shards install, there’s little chance for the user to customize the instructions.
It’s much easier to first have shards install pull in the dependency code and then you have a chance to apply necessary configuration before building.
IMO building dependencies fits much better as part of the build step (e.g. shards build), not the dependency install step (e.g. shards install).
I’m responding cross-thread because I think this fits better here and I would like Survey: Use cases for shards' `postinstall` to be focused on describing use cases, not judging them.
I strongly object to this. Requiring a bunch of dependencies even just for installing the sources of a dependency is quite demanding. It forces every user to have Ruby and rake available. Even if they don’t need them or their product (the mruby/ruby libraries). For example if the C libraries are already available, there’s no need to built them. Or if there no intention to actually build the code that links these libraries, there’s also no need to build them.
I understand the motivation to package everything conveniently, but I think this is not really developer friendly. You are forced into a specific workflow and the only way to opt out is to skip postinstall entirely.
I agree with this argument but at the same time, there are very few platform-independent build tools that are easily accessible and/or provide the level of functionality required for libraries like Anyolite. If not through the postinstall hook, users are required to install the necessary tool(s) to build/use a particular library, so you fall into the same issue anyway.
instead of relied on postinstall, shards should provide some way so user can invoke the commands instead, aka opt-in instead of opt-out, and have a choice of which command to run, instead of just a single one.
for example supposedly a library named foo has as shards.yml file like this:
scripts:
- postinstall: ....
- postinstal-win: ....
- random_setup_script: ...
then after install/update you can futher invoke the command by typing shards exec foo.postinstall.
granted with this approach you need to run the command for every shard you install (if needed).
but you need to pay extra attention when install/update a shard anyway.
or we can figure out which implicit conversions to follow (the current postinstall one) so you can invoke them all by a single command. also need some postinstall hook to print the installation guide (if needed) for the shard (this is just a string to print to console, not the script to run).
I think this is going in the right direction. But at this point the question arises why use shards at all. There are better dedicated tools for running tasks / build recipes (e.g Make, Rake Task, etc.).
And it doesn’t really make much difference if you run shards exec foo.postinstall or make -C lib/foo postinstall for example.
I suppose the main merit of using shards is that it’s already available and doesn’t require an additional dependency. But then it’s also quite limiting and you get much more value out of a dedicated tool - and I don’t think it makes sense to extend shards into yet another buid tool.
Yeah, I think that could be a reasonable use for postinstall. However maybe this could be handled by shards with declarative configuration, i.e. a install_message property in shard.yml? This would be more versatile and implicitly portable.
Maybe a prebuild hook could be useful for building libraries…?
Yeah, I think that could be a reasonable use for postinstall. However maybe this could be handled by shards with declarative configuration, i.e. a
install_messageproperty inshard.yml? This would be more versatile and implicitly portable.
Yeah this is what I meant, another property to set the install_message that will be printed to the console every time the library is installed/updated.
Maybe a
prebuildhook could be useful for building libraries…?
I was thinking of another command, like shards exec-all postinstall that will run every postinstall script of the installed scripts. Or shards install --exec-all=postinstall might be better.
But at this point the question arises why use shards at all. There are better dedicated tools for running tasks / build recipes (e.g Make, Rake Task, etc.).
And it doesn’t really make much difference if you runshards exec foo.postinstallormake -C lib/foo postinstallfor example.
I suppose the main merit of using shards is that it’s already available and doesn’t require an additional dependency. But then it’s also quite limiting and you get much more value out of a dedicated tool - and I don’t think it makes sense to extend shards into yet another buid tool.
Yeah this is my question for sometime: what shards is aiming to be, just a simple package manager to install some crystal libraries, or a full featured project manager?
shards currently is not very useful, I only run it when I need to install/update a library, which is a rare occasion. shards build supposedly should be a frequently useful one, but it’s faster to me to type yarn br than shards build --release -s. In my opinion, shards does not need to compete with build systems like make, rake, it just need to provide some convenience way to invoke those frequently used commands.
For developing an application, there are some actions that need to run all the time: building program, clean then build the program, build the program then upload somewhere, fetch some external resources as dependencies, run some generator scripts, run migration tools etc. It is pretty inconvenient that we need to install other tool just to do it.
Also, passing arguments to make feels dirty, so I’m not very fond of using make as task invoker.
p/s: if shards is aiming to be a full-featured project manager, then stuff like project scaffolding (currently crystal init) should move to shards, too. And then might be we can provide an another common feature of newer generation package managers: generate projects from online templates. The ecosystem can be more uniformed this way.
There are better dedicated tools for running tasks / build recipes
For tasks, perhaps. For (starting) build recipies? No, not really - shards is after all the thing that knows if something has been updated/installed or not. Shard updates can pull transitive dependencies and keeping track of what have to be rebuilt (or built at all) is a task that needs to be automated with a tie-in to shards. It doesn’t need to be the postinstall command in particular, but it needs to be automated and recursive.
Maybe a
prebuildhook could be useful for building libraries…?
That would make sense. Perhaps a separate shards command would suffice. There just needs to be something.
Any resolution to this? I’m creating a shard that’ll benefit from a post install. For example this shard needs to build a rust library and then link against it.
Thanks
So does it install a Rust toolchain as well?