上篇我们介绍了如何利用OpenEvals提供的预定义提示词结合基于OpenEvals的自动化评估-02LLM-as-a-Judge介绍了基于LLM-as-a-Judge的评估器来评估Agent生成文本的质量我们具体介绍了针对简洁度、正确性、回答相关性和幻觉度这几个评估指标接下来我们介绍剩下的几个包括计划遵循度、惰性和生成代码正确性评估。1. 计划遵循度(PLAN_ADHERENCE)计划遵循度评估用以确定输出是否严格遵循了事先给定的执行计划。将一个复杂任务拆分成若干较小的可执行步骤再进一步采用Todo-List的方式做一步打个勾这是一种常见的Harness手段。在这种场景下我们可以利用计划遵循度评估来检查Agent在执行时有没有脱轨或漏掉某一步骤。计划遵循度评估对应的提示词模板定义在PLAN_ADHERENCE_PROMPT常量中具体模板文本如下。可以看出模板中定义了{inputs}、{plan}和{outputs}三个占位符所以在执行对应评估器时选用通过参数来填充。You are an expert evaluator assessing whether an AI agent followed its declared plan during execution. Your task is to determine whether the agents actions align with its stated plan. Rubric Plan adherence means: - All planned steps are executed in the trace - Steps are performed in the same order as the plan - No additional major actions beyond what was planned - Each step is clearly verifiable in the execution Plan non-adherence includes: - Missing or skipped steps from the plan - Steps executed in a different order than planned - Extra actions or tool calls not mentioned in the plan - Ambiguous trace entries that dont clearly match plan steps - Partial or incomplete execution of planned steps /Rubric Instructions For the execution trace: - Read the agents plan carefully - Review the execution to find corresponding actions for each step - Verify that each planned step appears in the trace - Check that steps are executed in the same order as planned - Identify any actions in the trace not present or unclear in the plan - Make a final judgement on whether the agent followed the plan and output a score /Instructions Reminder You are evaluating plan obedience only, not whether the agent succeeded at the task or produced correct results. A successful outcome with plan deviations receives a low score. When uncertain about whether a trace action matches a plan step, treat it as not followed and assign a low score. /Reminder Now, please grade the following example according to the above instructions: example input {inputs} /input plan {plan} /plan output {outputs} /output /example如下是一个典型的计划遵循度评估例子辅助方法eval与上篇演示的完全一样我们将指定的提示词和用于连接LLMgpt-5.4-mini的ChatOpenAI对象调用create_async_llm_as_judge函数创建了一个作为评估器的SimpleAsyncEvaluator对象。然后将指定的输入、输出、作为评估基准的引用和额外指定的关键字参数作为输入调用此评估器并将评估结果以JSON格式输出。fromopenevalsimportcreate_async_llm_as_judgefromlangchain_openaiimportChatOpenAIfromdotenvimportload_dotenvfromopenevals.prompts.qualityimportPLAN_ADHERENCE_PROMPTimportasyncio,json,functools load_dotenv()judgeChatOpenAI(modelgpt-5.4-mini)asyncdefeval(outputs:str,*,prompt:str,inputs:str|NoneNone,reference_outputs:str|NoneNone,**kwargs):evaluatorcreate_async_llm_as_judge(promptprompt,continuousTrue,judgejudge)resultawaitevaluator(inputsinputs,outputsoutputs,reference_outputsreference_outputs,**kwargs)print(json.dumps(result,ensure_asciiFalse,indent2))asyncdefmain():partialfunctools.partial(eval,promptPLAN_ADHERENCE_PROMPT,inputs 请帮我把如下这段中文产品评论翻译成英文并分析其情感倾向。 该产品具有极好的易用性并且送货快捷。 虽然电池寿命略微短了些但我对该产品依然很满意。 ,plan\ - 步骤一将输入的中文评论翻译为地道的英文。 - 步骤二分析该评论的情感倾向明确标注为[Positive]积极、[Negative]消极或[Neutral]中性。 - 步骤三从中提取出用户最满意的2个核心产品关键词Keywords并在最后以列表形式列出。 )outputs1\ This product is very easy to use, and the delivery speed was extremely fast. However, the battery life is slightly shorter than I expected. Overall, I am very satisfied with this purchase. Sentiment Analysis: The user expresses high satisfaction with the products performance and delivery, despite a minor complaint about the battery. This is a very favorable review. outputs2\ This product is extremely easy to use, and the delivery was fast. Although the battery life is a bit short, I am still very satisfied. Sentiment: [Positive] Keywords: - Easy to use - Fast delivery awaitpartial(outputs1)awaitpartial(outputs2)asyncio.run(main())我们提供的计划很明确地包含三个步骤翻译、情感标准和关键字提取。很明显提供的第二个答案与输入和此计划几乎完全一致但是第一个答案则明显存在问题格式未对齐步骤二失控预设计划要求明确标注为[Positive]、[Negative] 或 [Neutral]。模型虽然写了情感分析但没有采用规定的格式标签严重漏掉步骤步骤三缺失模型完全忘记了执行提取两个核心产品关键词并以列表列出的第三步。这些都体现在输出的评估结果上{key:score,score:0.4,comment:The plan required three steps: translate the Chinese review into natural English, analyze sentiment with an explicit label [Positive]/[Negative]/[Neutral], and extract the top 2 product keywords in a final list. The output clearly performs the translation and gives a sentiment interpretation, but it does not explicitly label the sentiment with one of the required tags. It also completely omits the keyword extraction and final list. Therefore, only part of the plan was followed, with missing and incomplete steps. Thus, the score should be: 0.4.,metadata:null}{key:score,score:1.0,comment:The execution matches the plan well: it translates the Chinese review into natural English, analyzes and clearly labels the sentiment as [Positive], and extracts two core product keywords listed at the end. The steps appear in the same order as planned, and there are no major extra actions beyond the plan. Thus, the score should be: 1.0.,metadata:null}2. 惰性LAZINESS惰性评估Agent是否返回了空白、敷衍或低投入的回复。由于大模型有时会偷懒例如回答此处省略100字、请自己去Google查询或直接给出空白。惰性评估专门用来发现这种消极怠工的行为。惰性评估对应的提示词模板定义在LAZINESS_PROMPT常量中具体模板文本如下。可以看出模板中定义了{inputs}和{outputs}两个占位符所以在执行对应评估器时选用通过参数来填充。值得一提的是由于这是一个负面指标得分越高表明懒惰程度越高。You are an expert conversation evaluator. You will be shown a response from an AI agent. Your task is to assess whether the agent returned a blank, empty, or low-effort response. Rubric A lazy response may include: - A blank or empty output - A one-word or single-sentence response to a request that clearly warrants more detail - A response that deflects, refuses, or says I cant help with that without a meaningful attempt - A response that acknowledges the request but provides no substantive content (e.g. Thats a great question!) - Filler text or placeholder content that does not address the users request - A response that is clearly copy-pasted or templated without being tailored to the request A non-lazy response may include: - A response that directly addresses the users request with appropriate depth and detail - A response that makes a genuine attempt even if imperfect - A concise response to a simple request (brevity alone is not laziness) /Rubric Instructions For the agent response: 1. Assess whether the response makes a genuine, substantive attempt to address the request 2. Consider whether the level of effort is appropriate given the complexity of the request 3. Assign TRUE if the response is lazy (blank, empty, or low-effort), FALSE if it is a genuine attempt /Instructions Please grade the following example according to the above instructions: example input {inputs} /input output {outputs} /output /example在如下这个针对惰性评估的演示程序中输入明确要求提供10条不同的用户信息但是第一个输出只提供了三条省去的七条记录换成了一个简单的文字说明//… 此处省略第3到第9个用户的数据格式与上方相同…。只有第二条老老实实地按照要求提供了10条不同的虚拟用户信息。评估结果对惰性程度做出了公正的判定前者得分0.8后者得分0完全没有偷懒。asyncdefmain():partialfunctools.partial(eval,promptLAZINESS_PROMPT,inputs\ 我需要测试我的电商系统请帮我生成一组结构完整的JSON格式测试数据。 里面必须包含**10条**不同的虚拟用户信息字段包括id, name, email, age, register_date。 )outputs1\ [ { id: 1, name: User_1, email: user1example.com, age: 25, register_date: 2026-01-01 }, { id: 2, name: User_2, email: user2example.com, age: 30, register_date: 2026-01-02 }, // ... 此处省略第3到第9个用户的数据格式与上方相同 ... { id: 10, name: User_10, email: user10example.com, age: 28, register_date: 2026-01-10 } ] outputs2\ json [ {id: 1, name: User_1, email: user1example.com, age: 25, register_date: 2026-01-01}, {id: 2, name: User_2, email: user2example.com, age: 30, register_date: 2026-01-02}, {id: 3, name: User_3, email: user3example.com, age: 22, register_date: 2026-01-03}, {id: 4, name: User_4, email: user4example.com, age: 35, register_date: 2026-01-04}, {id: 5, name: User_5, email: user5example.com, age: 28, register_date: 2026-01-05}, {id: 6, name: User_6, email: user6example.com, age: 41, register_date: 2026-01-06}, {id: 7, name: User_7, email: user7example.com, age: 19, register_date: 2026-01-07}, {id: 8, name: User_8, email: user8example.com, age: 31, register_date: 2026-01-08}, {id: 9, name: User_9, email: user9example.com, age: 26, register_date: 2026-01-09}, {id: 10, name: User_10, email: user10example.com, age: 45, register_date: 2026-01-10} ] awaitpartial(outputs1)awaitpartial(outputs2)输出{key:score,score:0.8,comment:The response is not blank, but it is low-effort because it only provides 1st and 10th entries and then uses a placeholder comment to omit the other 8 required users. The user explicitly asked for a complete JSON dataset with 10 distinct virtual users and all required fields, so this does not fully satisfy the request and appears templated/incomplete. Thus, the score should be: 0.8.,metadata:null}{key:score,score:0.0,comment:The response directly fulfills the users request by providing a structured JSON array with exactly 10 distinct virtual user records and the required fields: id, name, email, age, and register_date. It is substantive, tailored to the prompt, and not filler or deflection. Thus, the score should be: 0.0.,metadata:null}3. 代码正确性CODE_CORRECTNESS毫无疑问Coding Agent已经成为Agent最大的应用领域所以针对Agent生成代码的正确性的评估就显得尤为重要。代码正确性评估根据问题描述/规范说明书评估生成代码的正确性。它不依赖参考答案直接根据提示词里的需求描述让人工或评测大模型去静态分析生成的代码是否符合需求。代码正确性评估对应的提示词模板定义在CODE_CORRECTNESS_PROMPT常量中具体模板文本如下。可以看出模板中定义了{inputs}和{outputs}两个占位符所以在执行对应评估器时选用通过参数来填充。You are an expert code reviewer evaluating code for correctness. Your task is to assign a score based on the following rubric: Rubric A correct code solution: - Solves the problem completely as specified in the input - Should contain only valid code without any additional text - Handles all edge cases appropriately - Contains absolutely no bugs or logical errors - Uses efficient and appropriate algorithms/data structures - Follows language-specific best practices - Has correct syntax and would compile/run without errors When scoring, you should penalize: - Logical errors or bugs that would cause incorrect behavior - Missing edge case handling - Overly inefficient implementations when better approaches exist - Incomplete solutions that dont address all requirements - Syntax errors that would prevent compilation/execution - Security vulnerabilities or unsafe practices - Additional text that is not code /Rubric Instructions - Carefully analyze both the output code and the initial input query - Meticulously check for functional correctness and completeness - Focus on whether the code would work correctly rather than style preferences /Instructions Reminder The goal is to evaluate whether the code correctly solves the given problem. /Reminder input {inputs} /input output {outputs} /output在如下的演示的针对代码正确性评估的程序中我们针对同一个需求在一个静态类ArticelReader中定义一个远程读取指定文章内容的ReadAsync方法。我们针对生成的两端C#代码实施代码正确性评估。第二段代码在第一段代码的基础上进行了如下的改进添加了一个必要的CancellationToken类型的参数验证的输入参数articelId的有效性在一个using块中创建HttpClient确保该对象及时释放在得到响应后对状态码实施有效性验证很显然第二段代码的质量更高在针对正确性评估应该得到更高的分数输出的评估结果也体现了这一点。asyncdefmain():partialfunctools.partial(eval,promptCODE_CORRECTNESS_PROMPT,inputs 资源站点https://example.com发布了很多纯文本的文章每篇文章对应的URL为https://example.com/articel/{articelId}。 使用C#定义了一个静态ArticelReader类并在其中定义标准的异步方法ReadAsync方法读取指定*articelId*标识的文章内容。 )outputs1 csharp public static class ArticelReader { public async static Taskstring ReadAsync(string articelId) { var response await new HttpClient().GetAsync($https://example.com/articel/{articelId}); return await response.Content.ReadAsStringAsync(); } } outputs2 csharp public static class ArticelReader { public async static Taskstring ReadAsync(string articelId, CancellationToken cancellationToken) { if (string.IsNullOrWhiteSpace(articelId)) { throw new ArgumentException(Specified articelId is null or white space., nameof(articelId)); } using var httpClient new HttpClient(); using var response await httpClient.GetAsync( $https://example.com/articel/{Uri.EscapeDataString(articelId)}, cancellationToken); response.EnsureSuccessStatusCode(); return await response.Content.ReadAsStringAsync(cancellationToken); } } awaitpartial(outputs1)awaitpartial(outputs2)输出{key:score,score:0.72,comment:The code is syntactically valid C# and does implement an asynchronous ReadAsync method that fetches content from the specified URL pattern and returns the response body as a string. However, it has important shortcomings relative to a robust correct solution: it creates a new HttpClient instance on every call, which is an anti-pattern and can lead to socket/resource exhaustion; it does not dispose the HttpClient or the HttpResponseMessage; it does not check HTTP status codes, so failures may be returned as misleading content or empty/error pages without exception handling; and it assumes the URL path spelling exactly as given without validating input. The problem statement is minimal, so the basic functionality is present, but these correctness and resource-management issues prevent a perfect score. Thus, the score should be: 0.72.,metadata:null}{key:score,score:0.9,comment:The provided C# code is syntactically valid and implements an asynchronous ReadAsync method that validates input, constructs the expected URL, sends an HTTP GET request, ensures success status, and returns the response body as a string. It uses proper async/await patterns and cancellation token support. However, there is a likely issue with using using var response await httpClient.GetAsync(...) because HttpResponseMessage is disposable and disposed after the method returns, but the content is read before disposal so this is fine. A more substantive concern is that creating a new HttpClient per call is generally inefficient and can cause socket exhaustion in real applications, though it may still work correctly for isolated use. The main risk is that the prompt only asks to define a standard async method for reading article content, and this code does satisfy that. There are no obvious logical bugs in the implementation for the stated task. Thus, the score should be: 0.9.,metadata:null}第二段生成的代码被判定为0.9分因为它觉得每次创建新的HttpClient不够有效如果此方法被频繁调用可能会造成客户端可用端口被耗尽的风险。但是对于输入的需求来说如果要将针对HttpClient的调用封装在方法内部也只有只能这么做。4. 为代码正确性评估提供自定义标准上述的代码正确性评估有一个特点完全根据LLM自己针对编程语言的理解来进行评估。但在很多情况下LLM自己的理解未必正确此时就需要利用reference_outputs提供自定义的评估基准。此时就需要通过另一个常量CODE_CORRECTNESS_PROMPT_WITH_REFERENCE_OUTPUTS提供的提示词模板具体的模板文本如下。它在额外添加{reference_outputs}占位符并添加了采用提供的引用作为评估标准的措辞。You are an expert code reviewer evaluating code for correctness. Your task is to assign a score based on the following rubric: Rubric A correct code solution: - Solves the problem completely as specified in the input - Should contain only valid code without any additional text - Handles all edge cases appropriately - Contains absolutely no bugs or logical errors - Uses efficient and appropriate algorithms/data structures - Follows language-specific best practices - Has correct syntax and would compile/run without errors When scoring, you should penalize: - Logical errors or bugs that would cause incorrect behavior - Missing edge case handling - Overly inefficient implementations when better approaches exist - Incomplete solutions that dont address all requirements - Syntax errors that would prevent compilation/execution - Security vulnerabilities or unsafe practices - Additional text that is not code /Rubric Instructions - Carefully analyze both the output code and the initial input query - Meticulously check for functional correctness and completeness - Focus on whether the code would work correctly rather than style preferences - Compare the output with the reference output to verify correctness - The reference output represents the expected behavior or result - Code that produces results matching the reference output should be scored higher - Consider edge cases where the code might produce correct results for the given examples but fail in other scenarios /Instructions Reminder The goal is to evaluate whether the code correctly solves the given problem and produces output that matches the reference. /Reminder input {inputs} /input output {outputs} /output reference_output {reference_outputs} /reference_output在上面的例子中我们在一个Utility方法内部创建HttpClient实现远程调用被罚分我们看看如果我们利用提供的reference_outputs让评估忽略这一点看看会不会得到更高的分数。为此我们只需要按照如下的方式改变创建偏函数的方式为它提供一个reference_outputs参数。第二段代码得分从原来的0.90提升到0.95。asyncdefmain():partialfunctools.partial(eval,promptCODE_CORRECTNESS_PROMPT_WITH_REFERENCE_OUTPUTS,inputs 资源站点https://example.com发布了很多纯文本的文章每篇文章对应的URL为https://example.com/articel/{articelId}。 使用C#定义了一个静态ArticelReader类并在其中定义标准的异步方法ReadAsync方法读取指定*articelId*标识的文章内容。,reference_outputs 在一个静态的Utility方法中隐藏传输细节只能在内部创建HttpClient并及时将它释放。 这种做法虽然可能导致socket exhaustion的风险但也是无奈之举请在评估中忽略这一点。)输出{key:score,score:0.95,comment:The provided output is valid C# code and implements an async static ReadAsync method on the specified ArticelReader class. It validates the articelId argument, constructs the expected URL, uses HttpClient internally, and returns the response content as a string. It also correctly accepts a CancellationToken and propagates it to the HTTP call and content read. The code appears syntactically correct and functionally aligned with the prompt’s intent. Minor concerns remain: the prompt/reference emphasizes hiding transport details in a utility method and creating/disposal of HttpClient is acceptable, but the solution does not explicitly specify request options beyond GET and may rely on framework versions where ReadAsStringAsync(CancellationToken) availability can vary. These are minor compatibility concerns rather than clear bugs. Thus, the score should be: 0.95.,metadata:null}