It's often necessary to create some special state for a benchmark before it's run, and performing the setup as part of the benchmark can decrease the confidence in the actual result, introduce bias, or other unrelated variables which are best avoided.
For instance, a benchmark might want to test something over a vector of size 1'000'000, but the allocation of such a vector would impact the benchmark results (e.g. see #11361).