Recreate venv for each build + install from pylock.toml when available

[hugovk]

Re: python/cpython#153227 (comment)

python/cpython#153228 switched to use a dev version of pygments from GitHub, but it's not being picked up on the doc server.

This is because:

• We re-use a venv per Python version on the docs server
• pip install --upgrade -rrequirements.txt still sees pygments with the same version number in the venv, so doesn't need to upgrade it

This PR does two things:

• Clear the venv for each build. We still have a pip cache on the server, so it's not downloading everything every time. This means we'll always a fresh venv matching what we expect.
• We now have pylock.toml on newer branches (3.15+), which locks the actual required version, so this explicitly pin the dev version from GitHub. Let's use pylock.toml when when available.

[hugovk]

This ensures a fresh venv.

[hugovk]

Upgrade pip, because otherwise we use the bundled pip, and we need at least 25.1 for pylock.toml support.

Could do this just for the pylock.toml part, but might as well update for both.

[StanFromIreland]

Just a little clarification for anyone reading this in the future, it needs 26.1 for pylock.toml support.

[ZeroIntensity]

I'm definitely not the most qualified reviewer here, but nothing jumps out at me as wrong (and I really want the highlighting 😃)

[hugovk]

You'll be happy to hear that Discourse (which uses highlight.js) will also get lazy highlighting at some point :)

highlightjs/highlight.js#4450

[ZeroIntensity]

Hugo, do you realize I was the one who implemented that 😅: highlightjs/highlight.js#4398

But yeah, I think GitHub uses hljs too. Hopefully we see adoption before the 3.15 release.

[StanFromIreland]

We'll now be build[ing]_venv[s] for every version, but also every language. That's quite a lot considering the amount of translations we have. I'd rather avoid the rebuilds, I think running it once per version will keep the venv fresh enough?

[hugovk]

Yes, good idea, updated to create each venv at most once per run.

It also now deletes the venvs at the end of the run, we don't need to keep those hanging around, especially as they're recreated at the start of a run.

How does it look now?

[StanFromIreland]

It looks great!

[StanFromIreland]

Our Integration Test currently runs on 3.14, any objections to also running it for main (maybe this was tried previously, but was found brittle)?

[hugovk]

Our Integration Test currently runs on 3.14, any objections to also running it for main (maybe this was tried previously, but was found brittle)?

I don't think we tried it before, it was added last year: #275.

No objection, let's see how it goes.

[StanFromIreland]

I'll write a patch.

[hugovk]

I've confirmed this is creating new venv dirs for 3.13-3.16, like venv-3.15-only-html-en below, when it's rebuilding:

docsbuild@docs:/srv/docsbuild$ du -sh venv*
32M	venv
299M	venv-3.10
297M	venv-3.11
294M	venv-3.12
299M	venv-3.13
299M	venv-3.14
299M	venv-3.15
300M	venv-3.15-only-html-en
299M	venv-3.16

And then tidying up afterwards:

docsbuild@docs:/srv/docsbuild$ du -sh venv*
32M	venv
299M	venv-3.10
297M	venv-3.11
294M	venv-3.12
299M	venv-3.13
299M	venv-3.14
299M	venv-3.15
299M	venv-3.16

I've moved venv-3.13 - venv-3.16 to /tmp for now, and we can delete them later, and also the older ones.

[hugovk]

Translations also working:

https://docs.python.org/pl/3.16/whatsnew/3.15.html#whatsnew315-lazy-imports
https://docs.python.org/fr/3.16/whatsnew/3.15.html#whatsnew315-lazy-imports

I've now deleted all the old venv*s.