Add Vally evaluation harness for Event Hubs tools by LarryOsterman · Pull Request #3159 · microsoft/mcp

LarryOsterman · 2026-07-24T23:45:50Z

Summary

Adds a first Vally evaluation harness under servers/Azure.Mcp.Server/tests/Vally, covering the �venthubs_eventhub_get and �venthubs_namespace_get tools.

Invoke-VallyEval.ps1: discovery-based runner that walks the directory for evaluations organized by area (namespace subfolder) and tool (.experiment.yaml).
Each experiment runs a shared eval spec (.eval.yaml) as two variants -- baseline (without the Azure MCP server) and treatment (with it) -- to isolate the server's contribution, graded identically via outcome-based LLM-judge graders.
�venthubs/New-EventHubsResources.ps1 / Remove-EventHubsResources.ps1: provisioning scripts for the Event Hubs area (tagged with DeleteAfter, local auth disabled for Safe Secrets Standard compliance).
Updated .vscode/cspell.json with new dictionary terms and CHANGELOG.md with an entry under Other Changes.

This content was migrated from the larryo/first_vally_evaluations branch in the fork larryosterman/mcp.

Invoking Livetests

Copilot submitted PRs are not trustworthy by default. Users with write access to the repo need to validate the contents of this PR before leaving a comment with the text /azp run mcp - pullrequest - live. This will trigger the necessary livetest workflows to complete required validation.

…ports can be generated without running eval

Add servers/Azure.Mcp.Server/tests/Vally with Invoke-VallyEval.ps1 discovery-based runner, README, and first Event Hubs experiment (eventhub-get, namespace-get) with baseline/treatment eval specs and provisioning scripts. Update cspell dictionary and CHANGELOG.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-07-24T23:46:08Z

Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Adds a Vally-based evaluation harness under servers/Azure.Mcp.Server/tests/Vally to measure agent effectiveness for Azure MCP Event Hubs tools using baseline vs treatment experiment variants, including optional per-area Azure provisioning/teardown scripts.

Changes:

Added a discovery-based PowerShell runner (Invoke-VallyEval.ps1) to build azmcp, discover experiments, provision resources per area, run experiments (optionally multiple iterations), and summarize baseline vs treatment outcomes.
Added Event Hubs evaluation specs (*.eval.yaml) and experiments (*.experiment.yaml) for eventhubs_eventhub_get and eventhubs_namespace_get, plus provisioning/teardown scripts using Azure CLI.
Updated repo metadata (CHANGELOG.md, .vscode/cspell.json, and a local .gitignore) to document and support the new harness.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
servers/Azure.Mcp.Server/tests/Vally/README.md	Documents the Vally harness layout, runner usage, and how baseline vs treatment comparisons are interpreted.
servers/Azure.Mcp.Server/tests/Vally/Invoke-VallyEval.ps1	Implements experiment discovery, build/PATH setup, optional provisioning, execution, and results summarization.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/New-EventHubsResources.ps1	Provisions Event Hubs resources used by the eval prompts, with DeleteAfter tagging.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/Remove-EventHubsResources.ps1	Tears down provisioned resources by deleting the resource group.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.experiment.yaml	Defines baseline vs treatment variants for the Event Hub “get” capability.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.eval.yaml	Defines stimuli and outcome-based LLM judge grading for Event Hub “get” scenarios.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/namespace-get.experiment.yaml	Defines baseline vs treatment variants for the namespace “get/list” capability.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/namespace-get.eval.yaml	Defines stimuli and outcome-based LLM judge grading for namespace “get/list” scenarios.
servers/Azure.Mcp.Server/tests/Vally/.gitignore	Ignores local Vally result artifacts directories.
servers/Azure.Mcp.Server/CHANGELOG.md	Adds an “Other Changes” entry documenting the new Vally harness and initial experiments.
.vscode/cspell.json	Adds new dictionary words related to Vally and Event Hubs eval content.

Copilot · 2026-07-27T17:41:22Z

+if ($LASTEXITCODE -ne 0) {
+    # Check for errors (user not logged in, invalid subscription, etc.)
+    Write-Warn "Failed to check if resource group '$ResourceGroup' exists. Verify 'az login' and subscription."
+    exit 1
+}


Fixed in 2c00671: replaced exit 1 with throw so failures propagate as exceptions and the runner can execute cleanup in finally.

Copilot · 2026-07-27T17:41:22Z

+.PARAMETER PreEvalScript
+Optional path to a provisioning script run BEFORE the experiments of an area (e.g.
+./eventhubs/New-EventHubsResources.ps1). Use it to create the Azure resources the
+eval prompts reference. -ResourceGroup and -Subscription are forwarded to it.
+
+If not specified (and -SkipProvisioning is not set), the runner auto-discovers a
+convention-named `New-*Resources.ps1` script in each area directory and uses that.
+
+.PARAMETER PostEvalScript
+Optional path to a teardown script run AFTER an area's experiments (e.g.
+./eventhubs/Remove-EventHubsResources.ps1). It runs in a finally block, so it
+executes even if the run or the pre-eval provisioning fails. -ResourceGroup and
+-Subscription are forwarded to it.
+
+If not specified (and -SkipProvisioning is not set), the runner auto-discovers a
+convention-named `Remove-*Resources.ps1` script in each area directory and uses
+that.


Fixed in 2c00671: removed the duplicated .PARAMETER PreEvalScript and .PARAMETER PostEvalScript blocks from comment-based help.

+|-- eventhubs/                        # An AREA (one subfolder per namespace)
+|   |-- eventhub-get.experiment.yaml      # <tool>.experiment.yaml - defines baseline+treatment variants (required)
+|   |-- eventhub-get.eval.yaml            # <tool>.eval.yaml        - the shared eval spec both variants run (required)
+|   |-- New-EventHubsResources.ps1        # New-*Resources.ps1    - per-area provisioning (optional)
+|   |-- Remove-EventHubsResources.ps1     # Remove-*Resources.ps1 - per-area teardown     (optional)


+The first area, `eventhubs/`, evaluates the **get Event Hub** tool
+(`eventhubs_eventhub_get`). Its `eventhub-get.experiment.yaml` runs the shared
+`eventhub-get.eval.yaml` as two variants that differ only in whether the Azure MCP
+server is present:


…arryOsterman/mcp into larryo/first_vally_evaluations

# Conflicts: # servers/Azure.Mcp.Server/tests/Vally/Invoke-VallyEval.ps1 # servers/Azure.Mcp.Server/tests/Vally/README.md # servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.eval.yaml # servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.experiment.yaml # servers/Azure.Mcp.Server/tests/Vally/eventhubs/namespace-get.eval.yaml # servers/Azure.Mcp.Server/tests/Vally/eventhubs/namespace-get.experiment.yaml

LarryOsterman and others added 8 commits July 8, 2026 14:36

Added initial vally evaluations for eventhubs tools

e38f5b3

Added namespace-get tool evaluations; updated script

54a5b93

Script supports multiple runs; Refactored script so that post-eval re…

3ee2446

…ports can be generated without running eval

Enable execute mode on script

b7b6f0d

PR feedback

6b6bc5d

Added consolidated mode to vally experiments

c7861c1

Merge branch 'microsoft:main' into larryo/first_vally_evaluations

accd162

Copilot AI review requested due to automatic review settings July 24, 2026 23:45

LarryOsterman requested review from a team as code owners July 24, 2026 23:45

LarryOsterman requested review from JasonYeMSFT, RickWinter, chidozieononiwu, g2vinay, saikoumudi, tmeschter and vukelich July 24, 2026 23:45

LarryOsterman marked this pull request as draft July 24, 2026 23:46

Copilot started reviewing on behalf of LarryOsterman July 24, 2026 23:46 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

LarryOsterman and others added 2 commits July 27, 2026 10:22

Merge branch 'larryo/first_vally_evaluations' of https://github.com/L…

141fa53

…arryOsterman/mcp into larryo/first_vally_evaluations

Copilot started work on behalf of LarryOsterman July 27, 2026 17:38 View session

Address Vally harness review feedback

2c00671

Copilot finished work on behalf of LarryOsterman July 27, 2026 17:42

LarryOsterman added 2 commits July 27, 2026 16:55

Use bicep for resource deployments; created skill for new experiments

7e58bcd

Add evaluations for all eventhubs tools

58e565d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Vally evaluation harness for Event Hubs tools - #3159

Add Vally evaluation harness for Event Hubs tools#3159
LarryOsterman wants to merge 13 commits into
mainfrom
larryo/first_vally_evaluations

LarryOsterman commented Jul 24, 2026

Uh oh!

azure-pipelines Bot commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 27, 2026

Uh oh!

Copilot AI Jul 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

LarryOsterman commented Jul 24, 2026

Summary

Invoking Livetests

Uh oh!

azure-pipelines Bot commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants