fix(memory-storage): stop re-adds from duplicating and resetting pending requests by anxkhn · Pull Request #2074 · apify/crawlee-python

anxkhn · 2026-07-14T19:37:18Z

Store pending requests in an OrderedDict keyed by unique key instead of a deque. Repositioning a request to the forefront is now O(1) instead of an O(n) deque.remove() scan, so re-adding K already-pending requests is O(K) instead of O(K*N) — measured 22.4s → 0.004s for 2000 forefront re-adds into a 20k-request queue. The scan only triggered for requests already waiting in the queue, which is what add_requests(..., forefront=True) hits when a handler re-discovers URLs that are still pending.
Fix the same URL being handed out repeatedly. The old scan silently failed once an earlier regular re-add had replaced the registered request object with a differently-valued one, leaving a stale entry behind and appending a second one for the same unique key. Five URLs re-added over three rounds yielded 20 fetches instead of 5 and left pending_request_count at -15. Duplicates are now impossible by construction.
Keep the originally enqueued request on a re-add. The old code replaced the registered request with the incoming duplicate, which on a forefront re-add also replaced the queued one — resetting retry_count (a request could retry forever), label (dispatch to a different handler) and user_data. On a regular re-add it left the two out of sync, so get_request and fetch_next_request disagreed. This restores the documented contract ("Duplicates will be identified but not re-added to the queue") and matches the file-system, SQL and Redis clients, as well as @crawlee/memory-storage in Crawlee for JS.
Make get_request consistent after a reclaim: it now returns the reclaimed object, the same one fetch_next_request hands back.
Write the queue metadata once per batch instead of once per new request, saving N-1 datetime.now() calls on a batch add.

✍️ Drafted by Claude Code

Copilot

Pull request overview

This PR optimizes the in-memory RequestQueue client by eliminating the O(n) deque.remove scan when re-adding already-pending requests with forefront=True, replacing it with an appendleft + “lazy stale entry” skipping approach in fetch_next_request() / is_empty().

Changes:

Reworked MemoryRequestQueueClient.add_batch_of_requests() so forefront re-adds of pending requests no longer scan/remove from the pending deque.
Added stale-entry skipping in fetch_next_request() and front-pruning in is_empty() to support the new “tombstone” behavior.
Added regression tests to guard the performance fix and ordering/dedup semantics.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`src/crawlee/storage_clients/_memory/_request_queue_client.py`	Implements tombstone-based forefront repositioning and stale-entry skipping/pruning logic.
`tests/unit/storage_clients/_memory/test_memory_rq_client.py`	Adds regression tests validating no `deque.remove` usage and preserves order/dedup semantics.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

`MemoryRequestQueueClient.add_batch_of_requests` repositioned an already-pending request to the forefront with `deque.remove`, a linear scan of the whole pending deque. Re-adding a batch of K already-pending requests to a deque of N was O(K*N). Forefront re-adds are a real path: tiered-proxy retries set `request.forefront = True` and re-enqueue while the request is still pending. The forefront reposition now just `appendleft`s the request and leaves the old entry in place. The new object is registered in `_requests_by_unique_key`, superseding the old one, and `fetch_next_request` and `is_empty` skip entries whose object is no longer the one registered for their unique key. Because a forefront reposition always enqueues the live copy ahead of the entry it supersedes, `is_empty` can safely prune stale entries from the front. Only forefront re-adds create such tombstones. A regular re-add of an already-pending request leaves both the deque entry and its registered object untouched, so the identity check never misfires and no request is dropped. Forefront-LIFO / regular-FIFO order and deduplication are unchanged. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

Replace lazy deque tombstones with an ordered mapping so forefront repositioning remains constant-time without stale-entry growth. Keep request updates on regular re-adds, update reclaimed request instances in the index, and leave is_empty side-effect free. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

anxkhn · 2026-07-27T13:04:55Z

pushed a revision addressing the latest review and ci feedback.

vdusek · 2026-07-27T13:12:38Z

@anxkhn Thank you. For future reviews, I'd really appreciate it if you could address all the feedback in a single follow-up commit, rather than splitting the changes across multiple commits, merging them into earlier commits, and force-pushing it. Your current approach makes the changes really hard to review.

anxkhn · 2026-07-27T13:14:34Z

understood, i’ll keep future review fixes in a single follow-up commit without rewriting earlier commits.

vdusek

Thanks. It's much cleaner after the rewrite. Two things left.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

anxkhn · 2026-07-27T16:52:33Z

@vdusek addressed both new threads, rebased, and re-requesting review. thank you.

vdusek · 2026-07-28T09:32:50Z

@Mantisus could you please check this as well?

…sons

…front-reposition

vdusek

LGTM

…front-reposition

Mantisus

That's a great PR, thanks for that. A couple of points from me.

This comment was marked as outdated.

Sign in to view

vdusek requested a review from Copilot July 15, 2026 08:28

Copilot started reviewing on behalf of vdusek July 15, 2026 08:29 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread src/crawlee/storage_clients/_memory/_request_queue_client.py Outdated

vdusek requested changes Jul 15, 2026

View reviewed changes

anxkhn force-pushed the perf/memory-rq-forefront-reposition branch from d2ca2a0 to 02429c4 Compare July 16, 2026 05:56

vdusek requested review from Copilot and vdusek July 17, 2026 14:18

Copilot started reviewing on behalf of vdusek July 17, 2026 14:18 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread tests/unit/storage_clients/_memory/test_memory_rq_client.py Outdated

vdusek reviewed Jul 20, 2026

View reviewed changes

Comment thread src/crawlee/storage_clients/_memory/_request_queue_client.py Outdated

Comment thread src/crawlee/storage_clients/_memory/_request_queue_client.py Outdated

anxkhn added 3 commits July 27, 2026 18:33

test(memory-storage): avoid private queue assertions

009c2dd

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

anxkhn force-pushed the perf/memory-rq-forefront-reposition branch from 02429c4 to 009c2dd Compare July 27, 2026 13:04

This was referenced Jul 27, 2026

feat(dataset): add collect_all_keys option and keep CSV export columns aligned #2079

Merged

fix(system): do not fail memory snapshots on inaccessible processes #2078

Merged

vdusek reviewed Jul 27, 2026

View reviewed changes

Comment thread tests/unit/storage_clients/_memory/test_memory_rq_client.py Outdated

Comment thread src/crawlee/storage_clients/_memory/_request_queue_client.py Outdated

refactor(memory-storage): simplify pending request update

7d110de

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

vdusek added 2 commits July 28, 2026 09:40

style(memory-storage): address review nits

87579e2

fix(memory-storage): keep the enqueued request on re-add

0986a93

This comment was marked as outdated.

Sign in to view

vdusek requested a review from Mantisus July 28, 2026 09:32

vdusek changed the title ~~perf(memory-storage): avoid O(n) deque scan on forefront re-add~~ perf(memory-storage): make forefront request re-add O(1) Jul 28, 2026

test(memory-storage): guard forefront re-add against pairwise compari…

c27dcff

…sons

vdusek added 2 commits July 28, 2026 12:29

Merge remote-tracking branch 'origin/master' into perf/memory-rq-fore…

1103f4c

…front-reposition

test(memory-storage): cover forefront re-add of non-pending requests

4d30667

vdusek changed the title ~~perf(memory-storage): make forefront request re-add O(1)~~ fix(memory-storage): stop re-adds from duplicating and resetting pending requests Jul 28, 2026

vdusek approved these changes Jul 28, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into perf/memory-rq-fore…

214b8e2

…front-reposition

Mantisus suggested changes Jul 28, 2026

View reviewed changes

Comment thread src/crawlee/storage_clients/_memory/_request_queue_client.py Outdated

Comment thread tests/unit/storage_clients/_memory/test_memory_rq_client.py Outdated

address feedback

f594180

vdusek merged commit fae204b into apify:master Jul 29, 2026
34 checks passed

Uh oh!

Conversation

anxkhn commented Jul 14, 2026 • edited by vdusek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anxkhn commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vdusek commented Jul 27, 2026

Uh oh!

anxkhn commented Jul 27, 2026

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

anxkhn commented Jul 27, 2026

Uh oh!

This comment was marked as outdated.

vdusek commented Jul 28, 2026

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Mantisus left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

anxkhn commented Jul 14, 2026 •

edited by vdusek

Loading

anxkhn commented Jul 27, 2026 •

edited

Loading

Mantisus left a comment •

edited

Loading