fix(system): do not fail memory snapshots on inaccessible processes by anxkhn · Pull Request #2078 · apify/crawlee-python

anxkhn · 2026-07-16T06:28:09Z

get_memory_info() no longer raises when a process cannot be inspected. In restricted environments (hidepid, hardened containers) it failed on every sample, which meant a traceback in the log every second and an autoscaler running on a stale memory history. Processes that deny inspection, exit mid-measurement, or lose their /proc entry are now skipped instead.
Failure to list the child processes no longer aborts the snapshot either - the parent's own usage is still reported.
A kernel that exposes no smaps at all makes psutil alias memory_full_info to memory_info, whose result has no pss field, so reading it raised AttributeError on every sample. The metric is now read defensively, and the verdict is latched: such a machine is detected once instead of being re-probed for every process on every sample.
A smaps file can be empty, which psutil parses to a PSS of zero. That was reported as zero bytes used, which the autoscaler reads as free memory. A PSS of zero now falls back to the RSS of the same process, which memory_full_info() has already read anyway.
A single process denying PSS falls back to RSS just for itself - a denial says nothing about the other processes, so it is not latched.
Every degraded path warns once, so an estimate that misses a subprocess or falls back from PSS to RSS shows up in the log instead of being silently wrong.
LoggerOnce is now thread-safe, since memory metrics are sampled in a worker thread - this PR adds its first caller that runs off the event loop.
Nothing changes when PSS is readable: on a normal Linux process tree the reported current_size is byte-for-byte what master reports, with no warnings and no RSS fallbacks.
Tests cover the denied, exited, zombie, and missing-/proc cases, both PSS fallbacks and their warnings, the latch, and a vanished process not being misreported as a denial.

✍️ Drafted by Claude Code

Copilot

Pull request overview

This PR hardens get_memory_info() (used for autoscaler CPU/memory snapshots) against psutil.AccessDenied in restricted environments, preventing the periodic system-info task from crashing and silently stopping.

Changes:

Catch psutil.AccessDenied when reading the current process’s memory via PSS and fall back to RSS.
Suppress psutil.AccessDenied when aggregating child process memory usage.
Add unit tests covering both the child-skip and current-process RSS fallback behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/crawlee/_utils/system.py	Adds `AccessDenied` handling/fallbacks in memory snapshot collection.
tests/unit/_utils/test_system.py	Adds regression tests for `AccessDenied` scenarios in `get_memory_info()`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

On Linux, get_memory_info reads a process's PSS via memory_full_info, which psutil documents may require elevated privileges. In restricted environments (hardened containers with hidepid/restricted /proc, or an unreadable child subprocess) this raises psutil.AccessDenied, which is not a subclass of psutil.NoSuchProcess and so was not caught by the existing suppress around the child loop; the current-process read was unguarded entirely. The exception propagated out of the recurring system-info task, which has no error handling, silently stopping the autoscaler's CPU/memory snapshots. Suppress AccessDenied alongside NoSuchProcess when summing child memory, and fall back to RSS for the current process when PSS is denied. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

anxkhn · 2026-07-27T13:00:51Z

pushed a revision addressing the latest review and ci feedback.

vdusek · 2026-07-27T13:13:23Z

Same as here #2074 (comment)

@anxkhn Thank you. For future reviews, I'd really appreciate it if you could address all the feedback in a single follow-up commit, rather than splitting the changes across multiple commits, merging them into earlier commits, and force-pushing it. Your current approach makes the changes really hard to review.

vdusek

Thanks for the revision. Both of my earlier comments are correctly addressed - the logger.debug for the PSS->RSS fallback, and the child RSS fallback so an unreadable child is no longer dropped.

However, the change made for the Copilot comment about guarding children() introduces a regression: wrapping the entire loop in suppress means one child dying mid-iteration now aborts the loop and drops all remaining children. I verified this locally - a repro with a dead first child and a live second child worth 40 B yields 140 B on master and 100 B on this branch. CI does not catch it because that path is untested.

Details inline.

✍️ Drafted by Claude Code

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

anxkhn · 2026-07-27T16:52:44Z

@vdusek addressed all new threads, including the child-exit regression test, rebased, and re-requesting review. thank you.

- Move the fallback into the Linux-only `_get_used_memory`; elsewhere it re-ran the call that had just failed. - Cover a missing or zero `pss` field, not just `AccessDenied`. - Warn once about the degraded estimate instead of a debug record per snapshot, and log unlistable children. - Test the real fallback instead of only a patched `_get_used_memory`.

Inspecting a process can also fail with a bare `OSError` - psutil re-raises `FileNotFoundError` for a live process whose `/proc` entry is missing - which escaped the previous `psutil.AccessDenied` handling. Catch it wherever a process is inspected, and make the PSS to RSS fallback report what actually happened: - Warn about PSS only when psutil exposes no `pss` field at all, and latch that verdict instead of re-parsing `smaps` for every process on every sample. A single denied process falls back to RSS just for itself, and a `smaps` file that parses to zero is no longer reported as a missing metric. - Report a child that cannot be measured at all separately from one that exited. - Cap the estimate at the total memory of the machine, so summed RSS cannot pin the autoscaler in a critically overloaded state. - Lock `LoggerOnce`, which is now reached from the metric sampling thread.

vdusek · 2026-07-28T09:32:45Z

@Mantisus could you please check this as well?

…RSS silently

vdusek

LGTM

…memory

Mantisus

LGTM. Just a few nits.

Also, could you update the PR description?

vdusek requested review from Copilot and vdusek July 17, 2026 14:17

Copilot started reviewing on behalf of vdusek July 17, 2026 14:18 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread src/crawlee/_utils/system.py Outdated

vdusek reviewed Jul 20, 2026

View reviewed changes

Comment thread src/crawlee/_utils/system.py Outdated

Comment thread src/crawlee/_utils/system.py Outdated

anxkhn added 2 commits July 27, 2026 18:29

fix(system): preserve child RSS when PSS is denied

cd699de

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

anxkhn force-pushed the fix/memory-info-access-denied branch from b3464f1 to cd699de Compare July 27, 2026 13:00

vdusek reviewed Jul 27, 2026

View reviewed changes

fix(system): continue after child process exits

8d257e5

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

vdusek added 2 commits July 28, 2026 09:53

vdusek changed the title ~~fix(system): Handle psutil.AccessDenied in get_memory_info~~ fix(system): do not fail memory snapshots on inaccessible processes Jul 28, 2026

This comment was marked as outdated.

Sign in to view

vdusek requested a review from Mantisus July 28, 2026 09:32

vdusek added 4 commits July 28, 2026 12:18

fix(system): drop the memory estimate cap and reuse the already read RSS

a83f568

Merge branch 'master' into fix/memory-info-access-denied

2275d63

test: drop the ineffective LoggerOnce race test and cover a zombie child

2d9499b

fix(system): report a per-process PSS denial instead of switching to …

278685d

…RSS silently

vdusek mentioned this pull request Jul 28, 2026

Memory metrics ignore container memory limits (no cgroup awareness) #2095

Open

vdusek approved these changes Jul 28, 2026

View reviewed changes

vdusek added 3 commits July 28, 2026 14:32

Merge branch 'master' into fix/memory-info-access-denied

2f9be42

test: fill shared memory without a multi-gigabyte temporary list

0e526fc

test: run the memory estimation test in isolation and halve its peak …

c849710

…memory

Mantisus approved these changes Jul 28, 2026

View reviewed changes

Comment thread src/crawlee/_utils/system.py Outdated

Comment thread tests/unit/_utils/test_system.py Outdated

address mantisus feedback

86b31c6

vdusek merged commit 5ad4e53 into apify:master Jul 29, 2026
34 checks passed

Uh oh!

Conversation

anxkhn commented Jul 16, 2026 • edited by vdusek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anxkhn commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vdusek commented Jul 27, 2026

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anxkhn commented Jul 27, 2026

Uh oh!

This comment was marked as outdated.

vdusek commented Jul 28, 2026

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Mantisus left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

anxkhn commented Jul 16, 2026 •

edited by vdusek

Loading

anxkhn commented Jul 27, 2026 •

edited

Loading

Mantisus left a comment •

edited

Loading