Prevent stalled Event Hub sends from exhausting memory

[ericavella]

Summary

• bound each Event Hub telemetry queue by buffered bytes so a stalled sender cannot consume all VM memory
• cancel transmissions that exceed 30 seconds, then requeue only while capacity remains and drop excess telemetry safely
• count events as transmitted only after SendAsync succeeds and report bufferedBytes in channel diagnostics
• handle named log events that do not carry an EventContext, preventing EventHubTelemetryLogger.LoggingError null-reference noise
• add unit coverage for byte limits, transmission timeout/requeue, successful accounting, concurrent producers during a blocked send, and null-context logging

Root cause

The ARM64 Graph500 workload emits telemetry quickly enough to expose a failure mode in EventHubTelemetryChannel. If EventHubProducerClient.SendAsync stops returning, the sender thread has already removed its batch and incremented eventsTransmitted. Producers then continue filling a queue sized for 1,000,000 events. On a 16 GiB VM, queued EventData bodies eventually exhaust memory and the VC/guest-agent path disappears before the eight-hour profile can finish.

Separately, named events whose state is not an EventContext reached CreateEventObject with a null context. The logger dereferenced eventContext.Properties, caught the resulting NullReferenceException, and emitted EventHubTelemetryLogger.LoggingError instead of the original event.

This is separate from the earlier Graph500 libfabric fix: the Graph500 child processes now exit successfully, but the telemetry queue could still exhaust the host.

Validation

• VirtualClient.Core.UnitTests: 958 passed, 1 skipped
• EventHub telemetry unit-test slice after the null-context fix: 45 passed
• ARM64 control without Event Hub (f35a8e16-95bc-4477-a8a1-2d387a09234b): completed 3 hours, exit code 0, final RSS ~526 MiB
• unfixed private Event Hub repro (47ba0fcb-4ce4-4e68-b9c5-29c52b898d31): ingress stopped after 74 events; GC heap reached 726 MiB within minutes and was dominated by queued EventData byte arrays
• fixed natural-timeout smoke test (2488e17a-43fc-4b12-bc25-ed19456bd19e): buffer remained bounded and VC exited with code 0
• fresh ARM64 validation (e9ce6b92-7796-474a-ab59-75ff4ff8981a): completed 3 hours against the deliberately stalled Event Hub, exit code 0, RSS remained ~1.9-2.0 GiB with ~13.5 GiB available, then exited naturally

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.