Add propagate-only distributed tracing for Durable Functions

[andystaples]

Summary

• add TaskHubGrpcWorker(..., emit_trace_spans=False) so host integrations can activate inbound W3C trace context without emitting duplicate Durable Task lifecycle spans
• configure azure-functions-durable to use propagate-only tracing and depend on durabletask[opentelemetry]>=1.9.0
• add unit coverage for context parentage, cleanup, default span emission, and Functions worker configuration
• add a real Azure Functions host E2E test backed by an OpenTelemetry Collector

Trace validation

The collector-backed test starts a Durable Functions orchestration through the real Functions host and verifies that:

• the Python user-code span has an exported host span as its parent in the same trace
• the parent is not emitted by the Python durabletask instrumentation scope
• no Python durabletask orchestration, activity, or entity lifecycle span is emitted for the instance

This preserves the existing durabletask/DTS trace shape by default while using the host-owned Durable Functions trace shape in azure-functions-durable.

Closes #179

[Copilot]

Pull request overview

Adds a “propagate-only” distributed tracing mode to the core durabletask worker so host integrations (notably azure-functions-durable) can activate inbound W3C trace context for user code without emitting duplicate Durable Task lifecycle spans already produced by the host.

Changes:

• Added TaskHubGrpcWorker(..., emit_trace_spans=False) and implemented span-suppression + inbound-context activation (propagation) in the worker execution paths.
• Updated azure-functions-durable to depend on durabletask[opentelemetry]>=1.9.0 and to run in propagate-only mode by default.
• Added unit + Azure Functions host E2E coverage validating span parentage and absence of duplicate worker lifecycle spans.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/durabletask/test_tracing.py	Adds unit tests for suppressed span emission and propagate-only orchestration execution behavior.
tests/azure-functions-durable/test_worker_compat.py	Verifies DurableFunctionsWorker configures propagate-only tracing.
tests/azure-functions-durable/e2e/test_distributed_tracing_e2e.py	New E2E validation that user spans correlate to host spans without durabletask lifecycle duplicates.
tests/azure-functions-durable/e2e/conftest.py	Adds a session fixture that launches an OTEL collector and wires OTLP env vars for the tracing app.
tests/azure-functions-durable/e2e/apps/tracing/requirements.txt	Declares app requirements, including the OTLP gRPC exporter for the tracing scenario.
tests/azure-functions-durable/e2e/apps/tracing/local.settings.json	Adds local settings for the tracing test function app.
tests/azure-functions-durable/e2e/apps/tracing/host.json	Enables OpenTelemetry telemetry mode and Durable distributed tracing settings for the E2E app.
tests/azure-functions-durable/e2e/apps/tracing/function_app.py	Implements a minimal DF app that emits a user span during orchestration execution.
tests/azure-functions-durable/e2e/_harness.py	Adds an OtelCollector helper to capture spans for E2E assertions.
pyproject.toml	Bumps core durabletask version to 1.9.0 and confirms opentelemetry extra deps.
noxfile.py	Extends Functions E2E setup to include the tracing app and install the OTLP exporter dependency.
durabletask/worker.py	Adds emit_trace_spans option and applies suppression/propagation behavior in orchestrator/activity/entity execution.
durabletask/internal/tracing.py	Introduces span-emission suppression (ContextVar) and use_trace_context to propagate inbound context without spans.
CHANGELOG.md	Documents the new propagate-only worker tracing option under Unreleased.
azure-functions-durable/pyproject.toml	Updates dependency to durabletask[opentelemetry]>=1.9.0.
azure-functions-durable/CHANGELOG.md	Adds the tracing correlation feature and removes the previous “not yet wired up” limitation.
azure-functions-durable/azure/durable_functions/worker.py	Configures DurableFunctionsWorker to run with emit_trace_spans=False.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

durabletask/worker.py:1220

• When emit_trace_spans is disabled, orch_trace_ctx becomes None, which causes the response to fall back to req.orchestrationTraceContext. That means a non-empty orchestration trace context from the host (e.g., a long-running instance started before switching to propagate-only) will be echoed back indefinitely even though the worker is intentionally not emitting lifecycle spans. To make propagate-only behavior consistent, consider always returning an empty OrchestrationTraceContext when emit_trace_spans=False (and keep the existing fallback only for the emitting case).

            orch_trace_ctx = (
                tracing.build_orchestration_trace_context(
                    start_time_ns, span_id=orch_span_id)
                if self._emit_trace_spans
                else None
            )

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/azure-functions-durable/e2e/_harness.py:200

• get_spans() parses the collector output line-by-line while the collector is concurrently writing to the same file. This can intermittently raise json.JSONDecodeError on a partially-written trailing line and make the E2E test flaky. Consider ignoring undecodable lines (or only the last line) while polling.

        for line in self.output_path.read_text(encoding="utf-8").splitlines():
            if not line.strip():
                continue
            payload = json.loads(line)
            for resource_spans in payload.get("resourceSpans", []):

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

durabletask/internal/tracing.py:246

• use_trace_context() calls otel_context.attach/detach unconditionally. When opentelemetry is not installed (_OTEL_AVAILABLE is False and otel_context is set to None), this will raise at runtime. This can happen if a host sets emit_trace_spans=False without installing the optional OpenTelemetry extra; tracing should remain a no-op in that case.

@contextmanager
def use_trace_context(trace_context: pb.TraceContext | None):
    """Make a protobuf trace context current without creating a span."""
    parent_ctx = extract_trace_context(trace_context)
    if parent_ctx is None:
        yield
        return

    token = otel_context.attach(parent_ctx)

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

durabletask/client.py:1154

• Async schedule_new_orchestration has the same instance-id mismatch: resolved_instance_id is used for span naming, but the request builder is invoked with instance_id=instance_id, which generates a new UUID when instance_id is None. This can make emitted spans refer to a different instance than the one actually started.

                req = build_schedule_new_orchestration_req(
                    orchestrator, input=input, instance_id=instance_id, start_at=start_at,
                    reuse_id_policy=reuse_id_policy, tags=tags,
                    version=version if version else self.default_version,
                    data_converter=self._data_converter)

durabletask/worker.py:1223

• In propagate-only mode (emit_trace_spans=False) the response should not include/persist orchestrationTraceContext, but the current construction always supplies it (falling back to req.orchestrationTraceContext). This can unintentionally keep sending orchestrationTraceContext across dispatches even though lifecycle span emission is disabled.

            res = pb.OrchestratorResponse(
                instanceId=instance_id,

durabletask/client.py:637

• schedule_new_orchestration computes resolved_instance_id for span naming, but then calls build_schedule_new_orchestration_req with instance_id=instance_id. When instance_id is None, the helper generates a different UUID, so emitted create-orchestration spans can reference a different instance ID than the actual request/returned instance ID.

This issue also appears on line 1150 of the same file.

                req = build_schedule_new_orchestration_req(
                    orchestrator, input=input, instance_id=instance_id, start_at=start_at,
                    reuse_id_policy=reuse_id_policy, tags=tags,
                    version=version if version else self.default_version,
                    data_converter=self._data_converter)

tests/azure-functions-durable/e2e/_harness.py:131

• OtelCollector.get_spans() parses the exporter output as JSON-lines (json.loads per line). The collector config writes format: json, which can produce multi-line JSON and break that parser. Switch the fileexporter to jsonl to match the reader.

            "    format: json\n"

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

[berndverst]

Re-reviewed through 335adcc. Both prior findings are resolved: Durable Functions clients now suppress SDK lifecycle spans while preserving ambient trace-context injection, and the E2E guard rejects all durabletask spans on the captured trace. The subsequent scheduled-entity timing change introduces no actionable issue.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

.github/workflows/durabletask-azurefunctions.yml:88

• Using Node.js 21.x and azurite@latest in the E2E workflow reduces reproducibility and can cause sudden CI failures due to upstream changes. Prefer an LTS Node version and a pinned (or major-pinned) Azurite version, similar to how Core Tools is pinned to @4.

      - name: Set up Node.js (needed for Azurite and Functions Core Tools)
        uses: actions/setup-node@v4
        with:
          node-version: "21.x"

      - name: Install Azurite and Azure Functions Core Tools
        run: |
          npm install -g azurite@latest
          npm install -g azure-functions-core-tools@4 --unsafe-perm true

durabletask/worker.py:1220

• In propagate-only mode (emit_trace_spans=False), orch_trace_ctx becomes None, and the response later falls back to echoing req.orchestrationTraceContext. If the host provides a non-empty orchestration trace context from a previous dispatch, propagate-only mode will keep persisting SDK orchestration trace metadata even though lifecycle span emission is disabled. Consider returning an empty pb.OrchestrationTraceContext() here so propagate-only mode always clears the orchestration trace context in responses.

            orch_trace_ctx = (
                tracing.build_orchestration_trace_context(
                    start_time_ns, span_id=orch_span_id)
                if self._emit_trace_spans
                else None
            )

.github/workflows/durabletask.yml:60

• Using Node.js 21.x and azurite@latest makes CI less deterministic and can introduce breakage when Node 21 leaves support or Azurite publishes a breaking change. Prefer an LTS Node version and a pinned (or at least major-pinned) Azurite version to keep E2E/storage tests stable over time.

      - name: Set up Node.js (needed for Azurite)
        uses: actions/setup-node@v4
        with:
          node-version: '21.x'

      - name: Cache npm
        uses: actions/cache@v3
        with:
          path: ~/.npm
          key: ${{ runner.os }}-npm-azurite

      - name: Install Azurite
        run: npm install -g azurite@latest

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

.github/workflows/durabletask-azurefunctions.yml:88

• Installing azurite@latest can introduce flaky/non-reproducible CI because the version can change over time. Pin Azurite to a major or exact version instead.

      - name: Install Azurite and Azure Functions Core Tools
        run: |
          npm install -g azurite@latest
          npm install -g azure-functions-core-tools@4 --unsafe-perm true

.github/workflows/durabletask.yml:60

• Using azurite@latest makes CI non-deterministic (the published version can change and break the workflow without a repo change). Prefer pinning to a major or exact version so builds are reproducible.

      - name: Install Azurite
        run: npm install -g azurite@latest

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.github/workflows/durabletask.yml:60

• Using azurite@latest makes CI non-deterministic and can break builds when Azurite publishes a new release. Pin Azurite to a specific version (or at least a major) so the workflow remains reproducible.

      - name: Install Azurite
        run: npm install -g azurite@latest

[berndverst]

Re-reviewed through 0e260d5, including the Node 22/Azurite workflow updates and the merge with lazy async-client channel creation. Both prior tracing findings remain resolved, merge conflict resolution preserves propagate-only behavior and lazy initialization, and all current checks—including strict Azure Functions E2E—are green.