Right, the idea of using the sandbox for tests is that each test has to checkout a connection at the beginning of the test, the connection is wrapped in a transaction which is then rolled back at the end of the test. So each test can work with its own data without interfering with other tests (with some caveats). The connection is checked out in the setup callback, which is called once for each test in the case. The setup_all callback is called once for the entire case, and it’s called before any setup call and therefore before any connection is checked out.
The sandbox connection is checked out once for every test, so “delaying” the execution of your setup_all logic until then would mean running it once for each test 
it would be just like adding that logic to the setup call.
I personally wouldn’t worry about it. Your test case is async so will run in parallel with other tests. The benefits of having isolated tests, each with their own test data are IMO greater then the performance gains of inserting the test data once for all tests. At least in the general case.
However, if you think that you’d benefit from having tests share some data, you could seed the test database with the appropriate data before running the test suite (of course, tests shouldn’t modify that data, otherwise they may fail in funny ways). See this thread: Fixture on tests load once and keep for all the tests
