deepsearch: add evaluator docs - #1826
Open
kalanchan wants to merge 4 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
bobheadxi
reviewed
Jul 24, 2026
Comment on lines
+57
to
+59
| ## Using Evaluator | ||
|
|
||
| Evaluator lets Deep Search write and run Lua scripts over Sourcegraph search results. Deep Search decides when to use Evaluator and writes the script automatically; you do not need to write or run code yourself. |
Member
There was a problem hiding this comment.
Should this just be "using Deep Search"? Plus I don't think we need capital-E evaluator
Contributor
There was a problem hiding this comment.
Yep, I think we're separating evaluator here too much.
Also in general, these docs seem a bit too heavy to me - let's focus on the end user's perspective and not expose internal details IMO.
Contributor
Author
Contributor
There was a problem hiding this comment.
Yeah the way it's done now looks great - a security section is what these folks were looking for IMO.
Contributor
Author
|
@janhartman ready for another round of proof reading! |
janhartman
approved these changes
Jul 30, 2026
eseliger
approved these changes
Jul 31, 2026
eseliger
left a comment
eseliger
left a comment
Member
There was a problem hiding this comment.
curious if we need to put the evaluator so front and center or simply deep search, which btw can use scripting
| - **Repositories** — scope the search to specific repos | ||
| - **Files** — point the agent to particular files | ||
|
|
||
| ## Exhaustive analysis with evaluator |
Member
There was a problem hiding this comment.
Suggested change
| ## Exhaustive analysis with evaluator | |
| ## Exhaustive analysis |
maybe? I feel like you don't need to know, otherwise maybe people get confused that you explicitly need to ask for it?
| - **Cross-referencing** — compare multiple searches, such as files that match one implementation pattern but not another. | ||
| - **Aggregation and reporting** — group findings by repository, language, team, or dependency and calculate totals. | ||
|
|
||
|
|
|
|
||
| ### Security and guardrails | ||
|
|
||
| Evaluator runs scripts generated by Deep Search; it does not execute code from your repositories. The sandbox applies the following controls: |
Member
There was a problem hiding this comment.
Suggested change
| Evaluator runs scripts generated by Deep Search; it does not execute code from your repositories. The sandbox applies the following controls: | |
| Deep Search runs small Lua scripts for analysis, it does not execute code from your repositories. The sandbox applies the following controls: |
|
|
||
| Evaluator runs scripts generated by Deep Search; it does not execute code from your repositories. The sandbox applies the following controls: | ||
|
|
||
| - It has no direct network, system, or general filesystem access. |
Member
There was a problem hiding this comment.
Suggested change
| - It has no direct network, system, or general filesystem access. | |
| - It has no network, system, or general filesystem access. |
|
|
||
| - It has no direct network, system, or general filesystem access. | ||
| - Scripts can only use modules provided by Sourcegraph for search, source file reads, formatting, and generated file outputs. | ||
| - Search and source file operations are read-only and use Sourcegraph's existing APIs and repository permissions. Searches also use the search context selected for the conversation. Evaluator cannot modify repositories. |
Member
There was a problem hiding this comment.
Suggested change
| - Search and source file operations are read-only and use Sourcegraph's existing APIs and repository permissions. Searches also use the search context selected for the conversation. Evaluator cannot modify repositories. | |
| - Search and source file operations are read-only and use Sourcegraph's existing APIs and repository permissions. Searches also use the search context selected for the conversation. Deep Search cannot modify repositories. |
| - It has no direct network, system, or general filesystem access. | ||
| - Scripts can only use modules provided by Sourcegraph for search, source file reads, formatting, and generated file outputs. | ||
| - Search and source file operations are read-only and use Sourcegraph's existing APIs and repository permissions. Searches also use the search context selected for the conversation. Evaluator cannot modify repositories. | ||
| - The generated file module can only write bounded output files managed by Sourcegraph. It does not provide access to the Sourcegraph host filesystem. |
Member
There was a problem hiding this comment.
Suggested change
| - The generated file module can only write bounded output files managed by Sourcegraph. It does not provide access to the Sourcegraph host filesystem. | |
| - Artifacts generated by scripts are stored in object storage. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes https://linear.app/sourcegraph/issue/CU-3011/add-evaluator-section-in-deep-search-docs
adds evaluator section in the deep search docs to explain how it works, guardrails, examples etc.