Crater is a tool for compiling and running tests for every crate on crates.io (and a few on GitHub). It is mainly used for checking the extent of breakage when implementing potentially breaking changes and ensuring lack of breakage by running beta vs stable compiler versions.
Essentially it runs some
cargo command on every crate twice; once against the “start” toolchain and again against the “end” toolchain.
For example, “start” could be the stable release, and “end” could be beta.
If it passes in “start” but fails with “end”, then that is reported as a regression.
There is a bot called craterbot which is used to run crater on hardware managed by the rust-lang organization.
Crater is run by the release team during the beta cycle. If there are any regressions that look like they are caused by Cargo, they should contact the Cargo team to decide how to handle it.
If you have a change that you want to test before the beta release, or you want to test behavior that is not normally exercised by crater, you can do a manual run of crater. Roughly the steps are:
Create a branch with your changes.
In your clone of cargo, make the changes to incorporate whatever new thing you want to test and push it to a branch on your fork on GitHub.
Get a clone of https://github.com/rust-lang/rust
Create a branch in your rust-lang/rust clone to add your changes.
src/tools/cargosubmodule to point to your new branch.
.gitmodulesto point to your clone and branch of cargo with the changes you want to test. For example:
git submodule set-url src/tools/cargo https://github.com/ehuss/cargo.git git submodule set-branch --branch my-awesome-feature src/tools/cargo git submodule update --remote src/tools/cargo git add .gitmodules src/tools/cargo git commit
Create an PR on rust-lang/rust.
Push your submodule changes to GitHub and make a PR. Start the PR title with
[EXPERIMENT]to make it clear what the PR is for and assign yourself or @ghost.
Make a “try” build.
A “try” build creates a full release of x86_64-unknown-linux-gnu and stores it on rust-lang servers. This can be done with a comment
@bors tryon the PR (all Cargo team members should have permission to do this).
Look at the craterbot docs to determine the command that you want to run. There are different modes like
You can also choose how many crates to run against. If you are uncertain if your cargo changes will work correctly, it might be a good idea to run against
top-100first to check its behavior. This will run much faster. You can do a full run afterwards.
After the try build finishes (which should take a couple hours), ask someone to make a crater run. The Cargo team does not have that permission, so just ask someone on Zulip. They will need to write a comment to
@craterbotwith the command that you have specified.
Crater can take anywhere from a few hours to a few weeks to run depending on how long the craterbot queue is and which mode you picked and the priority of your job. When the crater run finishes, craterbot will post a comment to the PR with a link to a report of the results.
Investigate the report.
Look through the report which contains links to build logs for any regressions or errors.
Close the PR.
Whenever you are done doing crater runs, close your PR.
Crater only has a few built-in modes, such as running
cargo check or
You can pass extra flags with
More complex tests can be accomplished by customizing Cargo to perform whatever actions you want.
Since crater essentially runs
cargo check, you can modify the
check command to perform whichever actions you want.
For example, to test
cargo fix --edition, this commit intercepted
cargo check and modified it to instead:
- Only run on crates with the 2018 edition.
cargo fix --edition.
- Modify the manifest to switch to the 2021 edition.
cargo checkto verify.
If you need to compare the before and after of a command that is not part of crater’s built-in modes, that can be more difficult. Two possible options:
- Work with the infra team to add a new mode.
- Build two custom try builds.
Each one should modify the
cargo checkcommand as described above. The “start” build should perform whichever action you want with an otherwise unmodified cargo. The “end” build should perform whichever action you want with your modified cargo. Then, in the
@craterbotcommand, specify the start and end hashes of the two try builds.
There are some limitations of crater to consider when running Cargo:
- A crater run without regressions is not a green light to move forward.
- A large portion of Rust code is not tested, such as closed-source projects or things otherwise not collected by crater.
- Many crates can’t build in crater’s environment or are otherwise broken.
- Some crates have flaky tests.
- Crater runs in an isolated environment.
- It only runs on Linux x86-64.
- It does not have network access.
- The crate source is in a read-only mount.
- Crater does several steps before running the test (using its own copy of the stable toolchain):
- It generates a lockfile using
-Zno-index-updateto prevent index updates (which makes it run much faster).
- All dependencies are downloaded ahead-of-time with
- It generates a lockfile using
- The built-in modes pass several flags to cargo such as
--message-format=json. It will sometimes use
--all-targetsand sometimes not. Check the crater source for more details on how it works.