gh-155003: Don't keep file descriptor in multiprocessing SharedMemory object

[takluyver]

Once we've created the mmap, we can close the file descriptor immediately, instead of keeping it around in the SharedMemory object.

Without a file descriptor, we can also get rid of the __del__ method, and let the mmap & memoryview finalizers deal with cleanup. This allows shmem.buf to remain usable without having to keep a reference to the SharedMemory object.

• Issue: multiprocessing SharedMemory.__del__ can fail, leak file descriptor #155003

[gpshead]

Going back and forth on this PR with Claude (Fable 5), things we should consider:

grow-in-place resize hacks. SharedMemory has no resize API, so recipes in the wild do os.ftruncate(shm._fd, newsize) — sometimes paired with os.fstat(shm._fd) in the other process to detect the new size before re-mmapping. A few also pass shm._fd over Unix sockets with SCM_RIGHTS. All of that dies with this PR.

Those are "abusing" a private-api via ._fd access, but primarily because they had no other choice. We should attempt to provide them something... which turns out to be for multiple reasons (letting Claude do the digging):

Real _fd uses in the wild:

1. sglang — os.posix_fallocate(shm._fd, 0, nbytes) at line 1924 to avoid SIGBUS on /dev/shm exhaustion:
   https://github.com/sgl-project/sglang/blob/9dcaf6bfdff89b4b29611725ef44161be4e429dd/python/sglang/srt/managers/mm_utils.py#L1919-L1924
2. autonomi-ai/nos — os.chown/os.chmod on shm._fd (lines 45–51) for cross-user permission fixup:
   https://github.com/autonomi-ai/nos/blob/2761f7b50fa3173c74ec63a3321527fbb980b9ac/nos/common/shm.py
3. KolinGuo/RealRobot — reader/writer fcntl locks on shm._fd (lines 486–487) and the ftruncate-resize plan (line 755):
   https://github.com/KolinGuo/RealRobot/blob/42cb3a12c5e0b98023465d54535b3e8faa3692ef/real_robot/utils/multiprocessing/shared_object.py

Related CPython issue:

4. gh-114390 — "SharedMemory crashes on linux with SIGBUS on insufficient shm" (the fallocate-on-create fix):

... me:

The first one for sglang I think could be addressed in this PR as a bugfix: Fix gh-114390 by calling os.posix_fallocate(fd, 0, size) on create (where the platform supports it). This eliminates sglang's need entirely, and it composes with this PR since it happens in __init__ before the fd is closed. Arguably a bugfix and potentially backportable (at least to 3.15). sglang is very widely used so avoiding breaking it by coordinating on this would be good.

But adding support for a mode= parameter like nos wants or public APIs for what the RealRobot code wants to do would be squarely features (3.16+).

... digging in on the things passing the ._fd over sockets. Claude:

1. ChrisJR035/Talos-O-Architecture — the textbook case. It allocates a SharedMemory, then sends shm._fd to a C++ daemon over an AF_UNIX socket for zero-copy tensor handoff:

def send_fd(sock, fd):
    ancillary_data = [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", [fd]))]
    sock.sendmsg([b"1"], ancillary_data)
...
send_fd(client, shm._fd)

1. https://github.com/ChrisJR035/Talos-O-Architecture/blob/1d90b5e6ae322279efc5ee65cd2c1b1cfdbd58bf/cognitive_plane/cortex/ignite_native_ipc.py

2. sujaldev/pywaygui — a Wayland client using SharedMemory as its pixel buffer pool. It hands shm._fd to the compositor via conn.flush([shm._fd]); Wayland's wire protocol transmits fds over the unix socket with SCM_RIGHTS (that's the only way wl_shm works). It also creates a second mmap directly from shm._fd with explicit PROT_READ|PROT_WRITE, MAP_SHARED:

3. https://github.com/sujaldev/pywaygui/blob/11c355971886bddd938e47c7c9bfd9fc4230bf24/waygui/client.py

Notable: the fd-passing cases are ones where the held fd matters most, because SCM_RIGHTS needs a live fd at send time. Under PR 155007 these users would have to reopen by name — which works, but ...

... me: is another point in favor of a (beyond this PR) public shared memory opening API to be used for that.

In terms of what to change for this, could we at least do the posix_fallocate and sglang coordination? a sentinel for sglang to use is the existence of the _fd attribute. i've found other code at least acknowledging it being private does not guarantee it exists doing that. sglang today just checks "linux" and assumes. (ugh)

[gpshead]

should we add trackfd=False here? BUT see my other overall comment.

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.