Summary of the issues that might be lingering there:

• PR build checks generally seem to get confused when multiple crates are involved:
  • This seems to result in the behaviour that for the final result only the last build is taken into account.
  • Hence in a scenario where a PR changes crate A and B, building of crate A fails, but B is being built successfully after, the final check result may be considered successful.
• PR build checks seem to build the latest version of a crate, possibly regardless of the version of the crate that changed.

If you compare these runs:

https://github.com/alire-project/alire-index/runs/1270111342?check_suite_focus=true (failed)
https://github.com/alire-project/alire-index/runs/1270157219?check_suite_focus=true (success)

The first run ends with:

stderr: ADA.IO_EXCEPTIONS.USE_ERROR
stderr: new name "alire/alire.toml.upstream" designates a file that already exists
stderr: raised ADA.IO_EXCEPTIONS.USE_ERROR : new name "alire/alire.toml.upstream" designates a file that already exists
Call stack traceback locations:
0x5644729b977a 0x5644726e6d43 0x564472872098 0x5644728778c0 0x564472830e79 0x564472830121 0x56447224db6f 0x56447225054b 0x7f1081a89099 0x56447224d748 0xfffffffffffffffe

ERROR at line 113 (code: 1)

From the logs you see there are two changed crate files (both new, just different versions):

Changed files: index/ai/aicwl/aicwl-3.24.0.toml index/ai/aicwl/aicwl-3.24.1.toml

but it seems then that alr will always try to build the latest version of the crate. That is probably intended for the general use case, I still think it should build the exact crate that has been specified, one could imagine a scenario where an existing crate will be modified. Or maybe alr just got utterly confused, because it was the same crate with two different versions.

If I understand the log correctly, the result of the first failing run (search for USE_ERROR) does not immediately lead to a failing build, alire continues after the first failure, and only the second run leads to the build failing.

Which could indicate that even in a scenario where multiple different crates are tested at once (which seems a more likely use case than my attempt here), only the result of the last build counts.