Skip to content

Fix flaky CI test

Sergej Kostjucenko requested to merge cmichi-debug-flaky-test into master

Created by: cmichi

It's hunting season for https://github.com/paritytech/cargo-contract/issues/234 🏹!

I found it!

The issue was that we use

CARGO_TARGET_DIR: "/ci-cache/${CI_PROJECT_NAME}/targets/${CI_COMMIT_REF_NAME}/${CI_JOB_NAME}"

in our CI configuration. This made all tests build to the same target directory. Since we have a number of tests which build a contract called new_project this made our tmp_dir's obsolete and resulted in overlapping contracts build to e.g. /ci-cache/cargo-contract/targets/263/test/ink/new_project.contract.

The way I fixed it now is by appending a hash of a contract's manifest_path to the target_directory (for tests this path contains the tmp_dir path). This results in the paths now being e.g. /ci-cache/cargo-contract/targets/263/test/0xd82f746d702f636172676f2d636f6e74726163742e746573742e5479444c634b2f6e65775f70726f6a6563742f436172676f2e746f6d6c/ink/new_project.contract.

I'm not super-happy with that solution, since it makes caching obsolete for the contract build in those particular tests (since the hash is unique each time a test is run, due to tmp_dir being included in it).

I have a better idea, but it's a bit more involved:

  • We could use the full test path (e.g. cmd::metadata::tests::generate_metadata) for the hash instead.
  • This would be achieved by writing a proc. macro (could also be called #[test]), which wraps around the Rust #[test] macro.
  • The macro would then extract the module_path! and function_name! and put those into a thread-local variable.
  • This thread-local variable would be used for hashing.

The disadvantage is that we would require something like use crate::test; for the macro to become available for tests. But we would then again have proper caching.

But: The CI hasn't really slowed down with the current fix in this PR ‒ a run of the test stage still takes 4-5 minutes. So maybe the current fix is already good enough. Though it fills up the cache and I'm not sure when it is cleared.

Wdyt?

Merge request reports