Shards support for Dependabot

Community
1

Hey all, so I’m pleased to announce Shards support for dependabot is finally is in a reviewable state!

Needless to say the vast majority of the files are just boilerplate, but i’d love to get at least some initial community feedback/testing before i go and open a PR against upstream. I opened a PR in my fork to make reviewing easier, which ultimately i’ll squash (based on author) once it’s in a good spot.

This is a good starting point to understand the architecture, but I also wanted to outline my approach at a high level to make it easier to review than first having to fully understand how dependabot internals work. Each section includes some option questions that would be :pray: to focus on. Tho some of these may be better suited to the dependabot maintainers themselves.

Implementation

FileFetcher

Parses shard.yml from host, and shard.lock if it exists. Don’t really think we need to worry about path dependencies for now, if ever.

Open Questions

  • All the other ecosystems have a concept of “supported versions”, for now I just made this >= 0.19 as other ecosystems have versions like >= 0.13 so seems reasonable.
  • Is the lib/.shards.info file important at all? Seems to be on version 1.0 unlike shard.lock

FileParser

Parses shard.yml and shard.lock into either a runtime or development dependency group depending on if they’re dependencies or development_dependencies. This is one of the areas where i’m not 100%. shard.yml dependencies are treated as having a * requirement if they only provide github.

Open Questions

  • If there is not a shard.lock, how to determine the version of a dependency? My thinking is if there is no shard.lock then it’s not installed so nil makes sense. Otherwise we could figure out what would be installed via git but don’t think that’s appropriate.
  • For branch/commit/tag, the current logic just does latest commit on a branch for branch, latest tag for tag, and no-ops for commit since there is no upgradable path for that.
  • For runtime vs development we don’t really have a way to know if a given dep is one or the other outside of them being directly listed in dependencies or development_dependencies. So as of now it’ll just assume they’re a runtime if not explicitly included within the top level keys of development_dependencies.
  • The format (0.13.0+git.commit.7fff589e026412646b33cef80f78cd1c7fd072aa) of branch/commit/tag is kinda strange because it includes the commit and the last tag is there a reason for that vs making it specific to tag? Actually just seems to use the version of the shard in shard.yml as even if you don’t have any tags, it’ll use 0.1.0+git.commit... when pinned to master branch for example

UpdateChecker

#fetch_latest_version is basically the implementation of the second open in the FileParser section. We also can’t really rely on dependabot’s built-in git features (tho its been a while as to exactly why). The RequirementsUpdater helper type is what actually handles changing the version of a dep one to another. The logic in there and #fetch_latest_version makes sense to me, but another pair of eyes wouldn’t hurt. It just tries to keep the version requirement similar to what it was before.

Vast majority of the determining the latest “resolvable” version is basically just running like shards lock --update ameba in a temp dir and reading the updated lock file. Might be a more robust way to handle it, but this was easiest and seems to work fine.

Open Questions

  • #fetch_latest_version/RequirementsUpdater logic makes sense?
  • Not totally sure what “full unlock” means
  • Probably fine to keep using shards lock command?

FileUpdater

Feel pretty good about this one, just handles writing the updates to the underlying files

Requirement, Version, MetadataFinder, PackageManager, Language

Feel pretty good about these, mostly just boilerplate/meta information about the ecosystem and such. Dependabot makes use of some Ruby Gem related types like Requirement for representing a dependency’s version constraint. E.g. ~> 1.2.3. It supports mostly the same values, other than * at least.

Final Thoughts

  • Currently only supports git, probably fine
  • Some repos to test on would be :fire:
  • Want to take another pass on the specs, they dont feel all that robust to me even tho coverage is good. Tho might just be rspec feeling weird to me :sweat_smile:
  • Don’t think we need to handle any security/credential stuff since it’s all just git
  • Ultimately will need to implement the cooldown feature, at first glance seems doable as it just uses metadata related to a tag.
10 Likes
2

As far as I understand, dependabot is only concerned about the contents of the dependency files (shard.yml and shard.lock). The shards tool itself doesn’t need to be installed. So its version is of no relevance, right?
All we should care about is the version of the config files.

lib/.shards.info describes which versions are actually installed in lib/. It shouldn’t be relevant for dependency resolution.

This seems relevant for other dependency managers, so it might be interesting to see how they’re handling that.

Tags can be deleted and then it’s possible to retag a different commit. This is not recommended, but can happen (including deliberatly by a malicious actor).

1 Like
3

We probably don’t need to worry about them ever. Path dependencies are either pointing outside the repository, then it’s just broken because there’s no way to resolve that. Or they’re internal to the repo (pointing to a subdirectory), which means the dependency is implicitly up to date.

2 Likes
4

I’m actually using shards to determine what’s resolvable. So we do actually need it. That seems to be fairly common pattern between ecosystems.

EDIT: We could probably get away with only installing shards tho. Atm I’m just using the crystal install script because that was easier, but longer term don’t really want to use Crystal version to determine what version of shards we use.

But yea I’m not 100%. Seems a lot of the other ecosystems tie support to the version of the package manager. E.g. composer v1 vs v2. There also seems to be some notion of supported vs deprecated versions within package_manager.rb. But given shards has drifted between its version and the shard.lock version, maybe the latter is the way to go…

To be clear I was suggesting if you use branch or commit just make the format like git.commit.abc123 and leave the tag prefix only for tags just to make it more clear.

1 Like
5

Ah, gotcha. Yeah, that might need some improvement.

6

Oh right, it’s not sufficient to lookup if there are newer versions. We need to actually run dependency resolution to figure out if there are installable newer versions. And for that we need shards :+1:

1 Like
7

Technically we could copy the resolution logic (molnillo resolver IIRC) although I don’t think it’s worth it.

8

Dependabot makes use of some Ruby Gem related types like Requirement for representing a dependency’s version constraint. E.g. ~> 1.2.3. It supports mostly the same values, other than * at least.

If just * same as not specify version, maybe we can drop support it instead.