Skip to content

feat: Add Lambda-Runtime-Invocation-Id support for cross-wiring protection - #214

Merged
maxday merged 3 commits into
mainfrom
feat/invocation-id-cross-wiring
Jul 29, 2026
Merged

feat: Add Lambda-Runtime-Invocation-Id support for cross-wiring protection#214
maxday merged 3 commits into
mainfrom
feat/invocation-id-cross-wiring

Conversation

@vip-amzn

@vip-amzn vip-amzn commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Add Lambda-Runtime-Invocation-Id header support for cross-wiring protection.

The RIC now echoes the invocation ID received from RAPID on /next back on /response and /error, enabling RAPID to detect and reject stale responses from timed-out invocations.

Problem

On Lambda Managed Instances (LMI) and On-Demand (OD), when an invoke times out, the runtime process continues running in the background. If a new invoke arrives with the same requestId, RAPID accepts it. The still-running old invocation eventually posts its response, and RAPID matches it to the new invoke — delivering the wrong response (cross-wiring).

Solution

RAPID sends a unique per-invoke nonce via Lambda-Runtime-Invocation-Id header on /next. The runtime echoes it back on /response and /error. RAPID validates the match before accepting the response.

Changes

  • runtime_client.cpp: Parse Lambda-Runtime-Invocation-Id from /next response, pass to post_success/post_failure via z format (accepts Nonenullptr)
  • lambda_runtime_client.py: Thread invocation_id through to native client
  • bootstrap.py: Pass invocation_id from event request to response/error posting
  • deps/patches/aws-lambda-cpp-add-invocation-id.patch: Patch aws-lambda-cpp to support invocation_id in get_next/do_post
  • deps/patches/aws-lambda-cpp-musl-no-execinfo.patch: Extracted Alpine/musl fix from d3e9632 into a proper patch file so update_deps.sh can reproduce the tarball from scratch (previously baked directly into the tarball without a patch — see d3e9632)
  • Unit tests: 6 new invocation-id specific tests + fix existing assertions for the new parameter

Backward Compatibility

Fully backward compatible in both directions:

  • If RAPID doesn't send the header → RIC doesn't see it → doesn't echo → no behavior change
  • If RIC doesn't echo it (old version) → RAPID skips validation → no behavior change

Testing

  • 95 unit tests pass (includes 6 new invocation-id tests)
  • E2E tested with updated RIE binary (validates invocation-id echo) and stock RIE (backward compat verified)
  • Alpine 3.19 Docker build verified locally (musl/execinfo conditional compilation works)

Comment thread tests/test_bootstrap.py
Comment thread tests/test_lambda_runtime_client.py
@vip-amzn
vip-amzn force-pushed the feat/invocation-id-cross-wiring branch from c14cbf5 to 1fea904 Compare July 28, 2026 10:18
…ction

Echo the invocation ID received from RAPID on /next back on /response
and /error, enabling RAPID to detect and reject stale responses from
timed-out invocations. Fully backward compatible — header only sent
when received.
@vip-amzn
vip-amzn force-pushed the feat/invocation-id-cross-wiring branch from 1fea904 to a5775f7 Compare July 28, 2026 10:23
self.assertEqual(event_request.tenant_id, "")
self.assertEqual(event_request.event_body, response_body)

@patch("awslambdaric.lambda_runtime_client.runtime_client")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All new tests all patch awslambdaric.lambda_runtime_client.runtime_client, so they verify Python-level argument threading, real value, but they stop one layer above the feature. The actual behavior lives in the C++ patch:

if (!invocation_id.empty()) {
    headers = curl_slist_append(headers, (std::string(INVOCATION_ID_HEADER) + ": " + invocation_id).c_str());
}

If someone re-rolls the tarball and drops aws-lambda-cpp-add-invocation-id.patch from update_deps.sh, all tests still pass and the feature silently disappears.

Can you add the following tests?

  1. Round trip: /next returns Lambda-Runtime-Invocation-Id: abc123 → the subsequent /response request carries Lambda-Runtime-Invocation-Id: abc123.
  2. Omission: /next sends no such header → /response carries no such header (not an empty one). This is the backward-compat claim.
  3. Error path: same round trip via /error, since post_failure is a separate C++ overload.

maxday added 2 commits July 29, 2026 15:44
The existing tests all patch runtime_client, so they verify Python-level
argument threading but stop one layer above the behaviour. The header is
emitted by libcurl inside the compiled extension. If
aws-lambda-cpp-add-invocation-id.patch is dropped when the vendored tarball
is re-rolled, every test still passes and the feature silently disappears.

Add tests that drive the built extension against a stub RAPID on a real
socket, asserting on raw received headers: round trip via /response, the
/error path, and omission (no header at all, rather than an empty one).
Verified by removing the curl_slist_append call and rebuilding — the new
tests fail while the existing suite stays green.

The client half runs as a subprocess because post_invocation_result and
post_error hold the GIL across the blocking curl call, so an in-process stub
server thread would never be scheduled to answer the POST.
test: assert the invocation ID reaches the wire
@maxday
maxday self-requested a review July 29, 2026 17:07
@maxday
maxday merged commit 3810eed into main Jul 29, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants