Improves test suite performance by 8.5-25x [experiment!] - #22917
Improves test suite performance by 8.5-25x [experiment!]#22917NickSdot wants to merge 26 commits into
Conversation
github-actions
Bot
added
Extension: openssl
Extension: curl
Extension: opcache
Extension: pdo (core)
Extension: pdo_dblib
Extension: phar
Extension: shmop
Extension: snmp
Extension: sockets
Extension: sodium
Extension: standard
Extension: sysvmsg
Extension: sysvshm
SAPI: cli
Category: CI
labels
|
Didn't expect this to be smooth in the very first CI run; will look into these. |
|
Interesting work! Related: #20980 Please submit each commit as a separate PR to make reviewing easier (maybe not all at once to avoid spamming the repos. Start with the most impacting ones). I will review at least some of them. |
Thanks, Arnaud! Would you, for now, be fine with reviewing the individual commit diffs here? Like, so that we can pre-decide a few things here first, and based on that I am happily splitting up into individual PRs. The commits table in the PR desc brings you straight to the same diff you would have in a dedicated PR. #20980 is indeed (admittedly naively) solved by this change. By decreasing batch size, much longer running batches get automatically more rare. Though, this is something that surely could be further improved in the future. |
2 participants
While working on #22799 I was bothered by the test suite being so slow. CI needs ~45 minutes from push to finish; running tests locally also takes quite a while. I was wondering why that is and whether it must be like that. Hence, I decided to run an experiment to find out if we could make the developer experience notably better.
Note
This experiment was fully executed by Codex. I was throwing some money on OpenAI to let GPT 5.6 Sol pursue the
/goalto make the test suite "massively" faster. This PR is the result of Codex trying exactly that -- for 14 hours straight. Consequently, I will not pretend that I understand the implications of every proposal -- and I also did not review everything in detail yet. Though, I did spend plenty of time to confirm the claims and reworking things.The result are interesting enough to open this clearly flagged as LLM-executed experiment, hoping that it can serve as a discussion starter to find out whether anything can be picked up. Would love to get feedback from people with more experience than me, to pin down which of these are legit ideas and which are dead-ends right away.
Results
Values in seconds; measured on macOS. The full suite improvements are already awesome on their own. But targeted runs also show up to 25x faster results (
-j10vs.-j10), many at least 2x faster, and very few slightly slower.Full Suite
Partial Runs
The -j1 row compares the master default with the explicit sequential opt-out.
Commits
-j1remains the serial opt-out. Includes a regression proving the cap.-j2. Requires Windows CI measurement.new_oomsubprocesses-j10, median of three runs, disabled versus enabled. Includes signal-safe child tracking, environment-aware caching, timeout/failure fallback coverage, and ZTS-safe compile guards.ext/pdo_dblib,-j10, final branch versus master. A one-second timeout applies only to the default availability probe. Configured connection attributes are preserved.3/23/3MAX_CONCURRENCYandSCOPED_CONFLICTS. OrdinaryCONFLICTSremains strict. Includes regressions for both semantics.ext/snmp,-j10, final branch versus master. Only failed probes are cached.DIR_W32.offsetregression.proc_opentests without fixed delaysext/openssl,-j10, final branch versus master. Uses isolated fixtures, dynamic ports, scopedserverconflicts and a cap of 10.proc_open()microbenchmark. Preserves environment isolation and falls back where shell semantics are required.ext/curl,-j10, final branch versus master. Isolates duplicate fixtures and caps the directory at four workers.open_basedirfixturesopen_basedirconflicts, a cap of 10 and symlink-safe cleanup.Legend:
(D)development-step measurement;(C)cumulative module measurement;(F)focused local measurement;(E)environment-specific measurement;(V)correctness verification;(W)Windows CI pending.Misc
existing, internal
--repeatflag does exactly that.-j2, and parallel not being the default. I, however, could not find out what these reasons are; hence, I kept changes Codex proposed.MAX_CONCURRENCY, but it shows gains.My repeated benchmark runs exposed existing bugs, fixed here in f2ba2d7 and d932481 (also PR'ed separately in #22911), and c577411 and 22022f1 (also PR'ed separately in #22912).