Conversation
|
Why do you need to bump deps of stable benchmarks? We try to touch those as minimally as we can. For Loongaarch it should be enough to run the normal benchmarks, right? |
We’re also running the stable benchmarks on LoongArch, and those results are shown on the dashboard here: https://rust-perf.loongarch.win/dashboard.html The changes I made are intentionally minimal and only aimed at getting the benchmarks to build and run on LoongArch, without touching the benchmark logic itself. Thanks! |
|
The problem is that changing these benchmarks might perturb our own x64 dashboard and its data. Since we don't modify the stable benchmarks very often, can't you just keep this as a downstream patch? Same as with some of the other loongaarch changes. |
|
I understand the concern here; the key point is to make sure we don’t alter the performance baselines. I'm happy to verify locally whether these changes have any real impact on x64. It’d be great if CI could catch this as well. |
|
The following data was collected on an AMD Ryzen 9 7950X. “Before” corresponds to commit 8ee2ffb, and “After” includes the additional changes from this PR. Both the Before and After runs were sampled three times. Based on the results, there’s no observable impact on the stable benchmarks. The raw data is attached for reference, and you’re very welcome to review or double-check it. Thanks!
|
|
I mean, that's not the whole point.. updating the benchmarks also moves the lockfiles to newer versions, and it won't be possible to rebenchmark them so easily - we plan to add an ARM architecture this year, and for that we'll probably rebenchmark the stable releases going a few years back. We will likely have to use older git versions to make that work anyway, but any change made to the stable benchmarks complicates that. As a more general point, we already struggle with maintenance of our bots, including rustc-perf, and I don't know if I want to make changes that do not benefit our use-case (and could in fact cause problems, we don't really have many useful tests, and benchmarking is notoriously finicky) for downstream usage. This project is not really intended to be deployed anywhere else than on perf.rlo. I understand that reimplementing all of rustc-perf would take a lot of work, and you might not want to hardfork it to keep receiving new features and changes. However, I'd like to hear what stops you from keeping a few loongaarch-specific patches on top of the rustc-perf codebase and just keep rebasing them on top of |
|
Ah, I see your point. You're worried that older rust toolchains won't be able to build them after the bump. That could definitely be an issue. Fair enough, I'll drop the changes to the stable benchmarks in this PR. Thanks for the patient explanation. |
This PR bumps the collector dependency to the minimal version required for LoongArch support.