o i#@sdZddlmZddlZddlZddlZddlZddlZddl Z ddl m Z m Z m Z mZddlmZmZddlmZddlmZmZddlmZdd lmZdd lmZmZmZmZmZm Z ddl!Z!ddl"Z"ddl#Z#dd l!m$Z$dd l%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-ddl.m/Z/ddl0m1Z1m2Z2ddl3m4Z4ddl5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;ddlZ>m?Z?ddl@mAZAmBZBddlCmDZDddlEmFZFddlGmHZHddlImJZJddlKmLZLddlMmNZNddlOmPZPddlQmRZRmSZSmTZTmUZUerddlVmWZWejXdkr dd lYmZZZneZZZd!Z[e"j\d"d#Z]e9d$ed%Z^ e9d&ed%Z_ e9d'ed%Z` d(Za d)Zb d*Zce)edeSZee)edeUZfe)eTZgGd+d,d,e&ee^e_e`fd-d.ZhGd/d0d0e&ee^e_e`fd-d.Zied1d2Gd3d4d4ee^e_e`fZjGd5d6d6e&ee^e_e`fd-d7d8Zkdzd{d@dAZleGdBdCdCZm d|d}dKdLZnd~dTdUZoe)e epeAfZqddYdZZred[d\dd%ZsddadbZtddedfZuddkdlZvddpdqZwddxdyZxdS)aDataset management for pydantic evals. This module provides functionality for creating, loading, saving, and evaluating datasets of test cases. Each case must have inputs, and can optionally have a name, expected output, metadata, and case-specific evaluators. Datasets can be loaded from and saved to YAML or JSON files, and can be evaluated against a task function to produce an evaluation report. ) annotationsN) AwaitableCallableMappingSequence)AsyncExitStack nullcontext) ContextVar) dataclassfield)iscoroutinefunction)Path) TYPE_CHECKINGAnyGenericLiteralUnioncast) to_thread) BaseModel ConfigDictField TypeAdapterValidationErrormodel_serializer) _typing_extra)to_json)SerializationInfoSerializerFunctionWrapHandler)Progress) NotRequiredSelf TypedDictTypeVar)get_event_loop)get_unwrapped_function_name logfire_spantask_group_gather)EvaluationResult Evaluator run_evaluator)DEFAULT_EVALUATORS)EvaluatorContext)EvaluatorFailure) EvaluatorSpec)SpanTree)context_subtree)EvaluationReport ReportCaseReportCaseAggregateReportCaseFailure) RetryConfig) )ExceptionGroup)CaseDatasetincrement_eval_metricset_eval_attributezpydantic-evals) otel_scopeInputsT)defaultOutputT MetadataTz./test_cases.yamlz./{stem}_schema.jsonz # yaml-language-server: $schema=c@sPeZdZUdZdZded<ded<dZded<dZd ed <ee d Z d ed <dS) _CaseModelzBInternal model for a case, used for serialization/deserialization.N str | Nonenamer@inputsMetadataT | NonemetadataOutputT | Noneexpected_outputdefault_factorylist[EvaluatorSpec] evaluators) __name__ __module__ __qualname____doc__rF__annotations__rIrKrlistrOrVrV\/var/www/html/karishye-ai-python/venv/lib/python3.10/site-packages/pydantic_evals/dataset.pyrDVs    rDforbidextrac@sLeZdZUdZedddZded<dZded<ded <eed Z d ed <dS) _DatasetModelzEInternal model for a dataset, used for serialization/deserialization.N$schema)rAaliasrEjson_schema_pathrFz-list[_CaseModel[InputsT, OutputT, MetadataT]]casesrLrNrO) rPrQrRrSrr^rTrFrUrOrVrVrVrWr[`s  r[F)initc@sleZdZUdZded< ded< dZded< dZd ed < eed Z d ed < ddddddddZ dS)r;aLA single row of a [`Dataset`][pydantic_evals.Dataset]. Each case represents a single test scenario with inputs to test. A case may optionally specify a name, expected outputs to compare against, and arbitrary metadata. Cases can also have their own specific evaluators which are run in addition to dataset-level evaluators. Example: ```python from pydantic_evals import Case case = Case( name='Simple addition', inputs={'a': 1, 'b': 2}, expected_output=3, metadata={'description': 'Tests basic addition'}, ) ``` rErFr@rGNrHrIrJrKrL,list[Evaluator[InputsT, OutputT, MetadataT]]rOrVrFrIrKrO2tuple[Evaluator[InputsT, OutputT, MetadataT], ...]cCs&||_||_||_||_t||_dS)a/Initialize a new test case. Args: name: Optional name for the case. If not provided, a generic name will be assigned when added to a dataset. inputs: The inputs to the task being evaluated. metadata: Optional metadata for the case, which can be used by evaluators. expected_output: Optional expected output of the task, used for comparison in evaluators. evaluators: Tuple of evaluators specific to this case. These are in addition to any dataset-level evaluators. N)rFrGrIrKrUrO)selfrFrGrIrKrOrVrVrW__init__s z Case.__init__) rFrErGr@rIrHrKrJrOrc) rPrQrRrSrTrIrKr rUrOrerVrVrVrWr;js"   r;cseZdZUdZdZded< ded< gZded< dd d ddfd dZ     deddddfdd Z     deddddfd!d"Z dddd d#dgd+d,Z dhdid0d1Z e e jdjd3d4Ze  dkdld dmdd?dndDdEZe dodd?dpdHdIZe dqdrdLdMZded fdsdPdQZe dodtdRdSZe dodudTdUZe e jdvdWdXZe dwdZd[Zed\d]dxdbdcZZS)yr<uA dataset of test [cases][pydantic_evals.Case]. Datasets allow you to organize a collection of test cases and evaluate them against a task function. They can be loaded from and saved to YAML or JSON files, and can have dataset-level evaluators that apply to all cases. Example: ```python # Create a dataset with two test cases from dataclasses import dataclass from pydantic_evals import Case, Dataset from pydantic_evals.evaluators import Evaluator, EvaluatorContext @dataclass class ExactMatch(Evaluator): def evaluate(self, ctx: EvaluatorContext) -> bool: return ctx.output == ctx.expected_output dataset = Dataset( cases=[ Case(name='test1', inputs={'text': 'Hello'}, expected_output='HELLO'), Case(name='test2', inputs={'text': 'World'}, expected_output='WORLD'), ], evaluators=[ExactMatch()], ) # Evaluate the dataset against a task function async def uppercase(inputs: dict) -> str: return inputs['text'].upper() async def main(): report = await dataset.evaluate(uppercase) report.print() ''' Evaluation Summary: uppercase ┏━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓ ┃ Case ID ┃ Assertions ┃ Duration ┃ ┡━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩ │ test1 │ ✔ │ 10ms │ ├──────────┼────────────┼──────────┤ │ test2 │ ✔ │ 10ms │ ├──────────┼────────────┼──────────┤ │ Averages │ 100.0% ✔ │ 10ms │ └──────────┴────────────┴──────────┘ ''' ``` NrErFz'list[Case[InputsT, OutputT, MetadataT]]r_rarOrV)rFrO+Sequence[Case[InputsT, OutputT, MetadataT]]0Sequence[Evaluator[InputsT, OutputT, MetadataT]]cs`tt}|D]}|jdurq|j|vrtd|j||jqtj||t|ddS)a-Initialize a new dataset with test cases and optional evaluators. Args: name: Optional name for the dataset. cases: Sequence of test cases to include in the dataset. evaluators: Optional sequence of evaluators to apply to all cases in the dataset. NDuplicate case name: )rFr_rO)setstrrF ValueErroraddsuperrerU)rdrFr_rO case_namescase __class__rVrWres    zDataset.__init__T) task_namerItaskFCallable[[InputsT], Awaitable[OutputT]] | Callable[[InputsT], OutputT]max_concurrency int | Noneprogressbool retry_taskRetryConfig | Noneretry_evaluatorsrrrIdict[str, Any] | Nonereturn-EvaluationReport[InputsT, OutputT, MetadataT]c s|pt}|p |}tj} |rtnd|dur t|ntddi} |dur/|| d<t d||jtjd| } pEt rSj d|| dnddfd d | j } durnd} d}n | j d} | j d}tfddtjdDIdH}g}g}|D]}t|tr||q||qt|||||| d}dtji}|dur||d<|}dur||d<|jdur| d|j| d|Wdn1swYWd|SWd|S1swY|S)aUEvaluates the test cases in the dataset using the given task. This method runs the task on each case in the dataset, applies evaluators, and collects results into a report. Cases are run concurrently, limited by `max_concurrency` if specified. Args: task: The task to evaluate. This should be a callable that takes the inputs of the case and returns the output. name: The name of the experiment being run, this is used to identify the experiment in the report. If omitted, the task_name will be used; if that is not specified, the name of the task function is used. max_concurrency: The maximum number of concurrent evaluations of the task to allow. If None, all cases will be evaluated concurrently. progress: Whether to show a progress bar for the evaluation. Defaults to `True`. retry_task: Optional retry configuration for the task execution. retry_evaluators: Optional retry configuration for evaluator execution. task_name: Optional override to the name of the task being executed, otherwise the name of the task function will be used. metadata: Optional dict of experiment metadata. Returns: A report containing the results of the evaluation. Ngen_ai.operation.name experimentrIevaluate {name})rFrr dataset_namen_casesz Evaluating )totalro!Case[InputsT, OutputT, MetadataT]report_case_namerjc sv4IdH't||jIdH}r"dur"jdd|WdIdHS1IdHs4wYdS)Nr%)advance)_run_task_and_evaluatorsrOupdate)rorresult)limiter progress_barr{ryrdrstask_idrVrW _handle_case;s 0z&Dataset.evaluate.._handle_case032x016xcs"g|] \}}||ffdd qS)cs||jp d|S)NCase rF)roirrVrWLsz-Dataset.evaluate...rV).0rrorrVrW Ksz$Dataset.evaluate..r%)rFr_failuresexperiment_metadataspan_idtrace_idraveragesassertion_pass_ratezlogfire.experiment.metadata)r)rorrrj)r&lenr_ranyio Semaphorerr'rFradd_taskcontextrrr( enumerate isinstancer4appendr3r assertions set_attribute)rdrsrFrurwryr{rrrI total_casesextra_attributes eval_spanrrrcases_and_failuresr_ritemreportfull_experiment_metadatarrV)rrrr{ryrdrsrrWevaluates~ "            (9 99zDataset.evaluatec Cs"t|j||||||||dS)aEvaluates the test cases in the dataset using the given task. This is a synchronous wrapper around [`evaluate`][pydantic_evals.Dataset.evaluate] provided for convenience. Args: task: The task to evaluate. This should be a callable that takes the inputs of the case and returns the output. name: The name of the experiment being run, this is used to identify the experiment in the report. If omitted, the task_name will be used; if that is not specified, the name of the task function is used. max_concurrency: The maximum number of concurrent evaluations of the task to allow. If None, all cases will be evaluated concurrently. progress: Whether to show a progress bar for the evaluation. Defaults to `True`. retry_task: Optional retry configuration for the task execution. retry_evaluators: Optional retry configuration for evaluator execution. task_name: Optional override to the name of the task being executed, otherwise the name of the task function will be used. metadata: Optional dict of experiment metadata. Returns: A report containing the results of the evaluation. )rFrurwryr{rrrI)r$run_until_completer) rdrsrFrurwryr{rrrIrVrVrW evaluate_syncis!zDataset.evaluate_syncrbrGr@rHrKrJrcNonecCsN|dd|jDvrtd|ttttf|||||d}|j|dS)adAdds a case to the dataset. This is a convenience method for creating a [`Case`][pydantic_evals.Case] and adding it to the dataset. Args: name: Optional name for the case. If not provided, a generic name will be assigned. inputs: The inputs to the task being evaluated. metadata: Optional metadata for the case, which can be used by evaluators. expected_output: The expected output of the task, used for comparison in evaluators. evaluators: Tuple of evaluators specific to this case, in addition to dataset-level evaluators. cSsh|]}|jqSrVr)rrorVrVrW sz#Dataset.add_case..rh)rFrGrIrKrON)r_rkr;r@rBrCr)rdrFrGrIrKrOrorVrVrWadd_cases zDataset.add_case evaluator&Evaluator[InputsT, OutputT, MetadataT] specific_casecCsZ|dur |j|dSd}|jD]}|j|kr |j|d}q|s+td|ddS)aAdds an evaluator to the dataset or a specific case. Args: evaluator: The evaluator to add. specific_case: If provided, the evaluator will only be added to the case with this name. If None, the evaluator will be added to all cases in the dataset. Raises: ValueError: If `specific_case` is provided but no case with that name exists in the dataset. NFTrz not found in the dataset)rOrr_rFrk)rdrraddedrorVrVrW add_evaluators   zDataset.add_evaluator4tuple[type[InputsT], type[OutputT], type[MetadataT]]cCsb|jD]}t|di}t|ddpt|dd}dkr!|Sqtd|dttttfS)zGet the type parameters for the Dataset class. Returns: A tuple of (InputsT, OutputT, MetadataT) types. __pydantic_generic_metadata__argsrV__args__r8z/Could not determine the generic parameters for z; using `Any` for each. You should explicitly set the generic parameters via `Dataset[MyInputs, MyOutput, MyMetadata]` when serializing or deserializing.)__mro__getattrrgetwarningswarn UserWarningr)clscrIrrVrVrW_paramss  $  zDataset._paramspath Path | strfmtLiteral['yaml', 'json'] | Nonecustom_evaluator_types6Sequence[type[Evaluator[InputsT, OutputT, MetadataT]]]r!c Csnt|}|||}t|}z |j||||jdWSty6}zt|d|jd|d|d}~ww)aLoad a dataset from a file. Args: path: Path to the file to load. fmt: Format of the file. If None, the format will be inferred from the file extension. Must be either 'yaml' or 'json'. custom_evaluator_types: Custom evaluator classes to use when deserializing the dataset. These are additional evaluators beyond the default ones. Returns: A new Dataset instance loaded from the file. Raises: ValidationError: If the file cannot be parsed as a valid dataset. ValueError: If the format cannot be inferred from the file extension. )rr default_namez2 contains data that does not match the schema for z: .N)r _infer_fmt read_text from_textstemrrkrP)rrrrrawerVrVrW from_files  zDataset.from_fileyamlrcontentsrjLiteral['yaml', 'json']rcCsB|dkrt|}|j|||dS|}||}||||S)a|Load a dataset from a string. Args: contents: The string content to parse. fmt: Format of the content. Must be either 'yaml' or 'json'. custom_evaluator_types: Custom evaluator classes to use when deserializing the dataset. These are additional evaluators beyond the default ones. default_name: Default name of the dataset, to be used if not specified in the serialized contents. Returns: A new Dataset instance parsed from the string. Raises: ValidationError: If the content cannot be parsed as a valid dataset. rr)r safe_load from_dict_serialization_typemodel_validate_json_from_dataset_model)rrrrrloadeddataset_model_type dataset_modelrVrVrWrs   zDataset.from_textdatadict[str, Any]cCs |}||}||||S)a<Load a dataset from a dictionary. Args: data: Dictionary representation of the dataset. custom_evaluator_types: Custom evaluator classes to use when deserializing the dataset. These are additional evaluators beyond the default ones. default_name: Default name of the dataset, to be used if not specified in the data. Returns: A new Dataset instance created from the dictionary. Raises: ValidationError: If the dictionary cannot be converted to a valid dataset. )rmodel_validater)rrrrrrrVrVrWr(s zDataset.from_dictr*_DatasetModel[InputsT, OutputT, MetadataT]c CsLt|}g}g}g}|jD]'}zt|d|} Wnty.} z || WYd} ~ q d} ~ ww|| q |jD]I} g} | jD](}z t|| j|} Wntya} z || WYd} ~ q?d} ~ ww| | q?ttt t f| j| j | j | j d} | | _|| q8|rtt|d|dd||j|d}|jdur||_||_|S)aCreate a Dataset from a _DatasetModel. Args: dataset_model: The _DatasetModel to convert. custom_evaluator_types: Custom evaluator classes to register for deserialization. default_name: Default name of the dataset, to be used if the value is `None` in the provided model. Returns: A new Dataset instance created from the _DatasetModel. N)rFrGrIrKz* error(s) loading evaluators from registryr8)rFr_) _get_registryrO_load_evaluator_from_registryrkrr_rFr;r@rBrCrGrIrKr:r)rrrrregistryr_errorsdataset_evaluatorsspecdataset_evaluatorrrowrOrrrVrVrWrBsN            zDataset._from_dataset_model schema_pathPath | str | Nonec Cs t|}|||}d}|durCt|trt|j|jd}|s,t|}|j|}n||r9tt ||}nt|}| ||ddi}|dkrp|j dd|d}t j |dd }|rit|} | d |}||dS||d <|jd d|d } || d dS)aSave the dataset to a file. Args: path: Path to save the dataset to. fmt: Format to use. If None, the format will be inferred from the file extension. Must be either 'yaml' or 'json'. schema_path: Path to save the JSON schema to. If None, no schema will be saved. Can be a string template with {stem} which will be replaced with the dataset filename stem. custom_evaluator_types: Custom evaluator classes to include in the schema. N)ruse_short_formTrjson)modeby_aliasrF) sort_keys r\)indentrr)r rrrjformatr is_absoluteparentis_relative_to_get_relative_path_reference _save_schema model_dumprdump_YAML_SCHEMA_LINE_PREFIX write_textmodel_dump_json) rdrrrr schema_refr dumped_datacontentyaml_language_server_line json_datarVrVrWto_fileys0      zDataset.to_filecst|}g|D]\}t|}|ddi}t|jD]"}| |j t |j |j ur=t||j ||j <q#||j ||j <q#dfdd }t|d ksU|s\tt|d krr|\}|d |int|d kr|\}|d |it|d kr|d |} |d| iq |\} } } Gfdddtdd} Gfdddtdd}|}ddi|dd<|S)ajGenerate a JSON schema for this dataset type, including evaluator details. This is useful for generating a schema that can be used to validate YAML-format dataset files. Args: custom_evaluator_types: Custom evaluator classes to include in the schema. Returns: A dictionary representing the JSON schema. r}Ncls_name_prefixrjfieldsrrcs*t|d|}tddd}||_|S)N_rXTrZarbitrary_types_allowed)r"r__pydantic_config__)r r tdconfigrrVrW_make_typed_dicts zCDataset.model_json_schema_with_evaluators.._make_typed_dictrr%short_evaluatorevaluator_paramsrcsNeZdZUdZded<ded<dZded<dZded <r%gZd ed <dSdS) z7Dataset.model_json_schema_with_evaluators..CaseNrErFin_typerGzmeta_type | NonerIzout_type | NonerK*list[Union[tuple(evaluator_schema_types)]]rO)rPrQrRrFrTrIrKrOrVevaluator_schema_typesrVrWr;s   r;rXrYcs6eZdZUdZded<ded<rgZded<dSdS)z:Dataset.model_json_schema_with_evaluators..DatasetNrErFz list[Case]r_rrO)rPrQrRrFrTrOrVrrVrWr<s r<typestring propertiesr\)r rjr rr}r)ritemsrget_function_type_hintspopinspect signature parametersvalues setdefaultrFrrAemptyr rrrrrmodel_json_schema)rrrevaluator_class type_hintsrequired_type_hintsprtype_hint_type params_tdrout_type meta_typer;r< json_schemarV)rrFrW!model_json_schema_with_evaluatorss<         z)Dataset.model_json_schema_with_evaluatorscCsLt|}||}t|ddd}|r||kr$||dSdS)zSave the JSON schema for this dataset type to a file. Args: path: Path to save the schema to. custom_evaluator_types: Custom evaluator classes to include in the schema. r)rrN)r r.rdecodeexistsrr)rrrr-schema_contentrVrVrWrs  zDataset._save_schema0type[_DatasetModel[InputsT, OutputT, MetadataT]]cCs|\}}}t|||fS)zGet the serialization type for this dataset class. Returns: A _DatasetModel type with the same generic parameters as this Dataset class. )rr[)r input_type output_type metadata_typerVrVrWrszDataset._serialization_typer cCs@|dur|S|j}|dvrdS|dkrdStd|jd)ahInfer the format to use for a file based on its extension. Args: path: The path to infer the format for. fmt: The explicitly provided format, if any. Returns: The inferred format ('yaml' or 'json'). Raises: ValueError: If the format cannot be inferred from the file extension. N>.yml.yamlrz.jsonrz$Could not infer format for filename z/. Use the `fmt` argument to specify the format.)suffixlowerrkrF)rrrr8rVrVrWrs  zDataset._infer_fmtwrap)rnxtrinforcCsHttttfdB|j}t|tr |d}r d|i||BS||S)zAdd the JSON schema path to the serialized output. See for context, that seems to be the nearest there is to a spec for this. Nr\)rdictrjrrrr)rdr;r<rschemarVrVrW_add_json_schema)szDataset._add_json_schema)rFrEr_rfrOrg)NNTNN)rsrtrFrErurvrwrxryrzr{rzrrrErIr|r}r~) rFrErGr@rIrHrKrJrOrcr}rN)rrrrEr}r)r}r)NrV)rrrrrrr}r!)rrV) rrjrrrrrrEr}r!)rV)rrrrrrEr}r!)rVN)rrrrrrEr}r!)rrrrrrrr)rrr}r)rrrr)r}r2)rr rrr}r)r;rr<rr}r)rPrQrRrSrFrTrOrerrrr classmethod functoolscacherrrrrDEFAULT_SCHEMA_PATH_TEMPLATErr.rrrrr? __classcell__rVrVrprWr<s  2   j 1# 9 /L r<Tr targetr source_prefixrjr}cCsV|s|}zt|t||WSty*t||j|ddYSw)atGet a relative path reference from source to target. Recursively resolve a relative path to target from source, adding '..' as needed. This is useful for creating a relative path reference from a source file to a target file. Args: target: The target path to reference. source: The source path to reference from. _prefix: Internal prefix used during recursion. Returns: A Path object representing the relative path from source to target. Example: If source is '/a/b/c.py' and target is '/a/d/e.py', the relative path reference would be '../../d/e.py'. z../)rI)rresolver relative_torkrr)rGrHrIrVrVrWr7s rc@sXeZdZUdZededZded<ededZded<dddZ dddZ dddZ dS)_TaskRunz>Internal class to track metrics and attributes for a task run.F)r`rMr attributeszdict[str, int | float]metricsrFrjvalue int | floatr}rcC||j|<dS)zRecord a metric value. Args: name: The name of the metric. value: The value of the metric. N)rNrdrFrOrVrVrW record_metric\z_TaskRun.record_metricamountcCs:|j|d}||}|dkr|dkrdS|||dS)zIncrement a metric value. Args: name: The name of the metric. amount: The amount to increment by. Note: If the current value is 0 and the increment amount is 0, no metric will be recorded. rN)rNrrS)rdrFrU current_valueincremented_valuerVrVrWincrement_metrices  z_TaskRun.increment_metricrcCrQ)zRecord an attribute value. Args: name: The name of the attribute. value: The value of the attribute. N)rMrRrVrVrWrecord_attributeurTz_TaskRun.record_attributeN)rFrjrOrPr}rrFrjrUrPr}rrFrjrOrr}r) rPrQrRrSr r=rMrTrNrSrXrYrVrVrVrWrLUs   rLrs1Callable[[InputsT], Awaitable[OutputT] | OutputT]rorretryrz-EvaluatorContext[InputsT, OutputT, MetadataT]c sfdd}|rddlm}|di||}|IdH\}}}}t|trv|D]K} | jD]C\} } | dkrD| dkrD|dd q1t| ttBsLq1| d krW|d | q1| d rf|| d | q1| d rt|| d | q1q*t t t tfjjjj||||j|jd S)anRun a task on a case and return the context for evaluators. Args: task: The task to run. case: The case to run the task on. retry: The retry config to use. Returns: An EvaluatorContext containing the inputs, actual output, expected output, and metadata. Raises: Exception: Any exception raised by the task. c st}tdurtdt|}zctdtdB}t/}t }t r5t t j IdH}n t t tj IdH}t |}Wdn1sRwYWdn1sawYt||}||||fWt|St|w)NzCA task run has already been entered. Task runs should not be nestedzexecute {task})rs)rL_CURRENT_TASK_RUNr RuntimeErrorrir'r&r2time perf_counterr rrBrGrrun_sync_get_span_durationreset) task_run_token task_span span_tree_t0 task_output_fallback_duration duration_rorsrVrW _run_onces.   z_run_task.._run_oncer)r]Nrchatrequestsr%zoperation.costcostzgen_ai.usage.details.z gen_ai.usage.) rFrGrIrKoutputduration _span_treerMrNrV)pydantic_ai.retriesr]rr1rMrrXintfloat startswith removeprefixr.r@rBrCrFrGrIrKrN) rsror]rotenacity_retrytask_run task_outputrt span_treenodekvrVrnrW _run_tasks@     rrtrrraryr{XReportCase[InputsT, OutputT, MetadataT] | ReportCaseFailure[InputsT, OutputT, MetadataT]csd}d}ztdt|||j|j|jd}|j} | dur'| jd}| jd}t} t |||IdH| dj | dj | dj | d j|j|} g} g} | rtfd d | DIdH}|D]}t|try| |ql| |qlt| \}}}| d t|| d t|| dt|Wdn1swYt| }ttttf||j|j|jj j j|||j t||||| dWSty }z$ttttf||j|j|jt|j d|t!"||dWYd}~Sd}~ww)aRun a task on a case and evaluate the results. Args: task: The task to run. case: The case to run the task on. report_case_name: The name to use for this case in the report. dataset_evaluators: Evaluators from the dataset to apply to this case. retry_task: The retry config to use for running the task. retry_evaluators: The retry config to use for running the evaluators. Returns: A ReportCase containing the evaluation results. Nzcase: {case_name})rr case_namerGrIrKrrrs task_durationrNrMcsg|] }|ffdd qS)cs t|Sr@r+)evr{scoring_contextrVrWrs z5_run_task_and_evaluators...rV)rrrrVrWrsz,_run_task_and_evaluators..rscoreslabels)rFrGrIrKrsrNrMrrrrtotal_durationrrevaluator_failures: )rFrGrIrK error_messageerror_stacktracerr)#r'r&rGrIrKrrrrarrrsrtrNrMrOr(rr/rextend _group_evaluator_outputs_by_type_evaluation_results_adapter dump_pythonr4r@rBrCrd Exceptionr6rrP traceback format_exc)rsrorrryr{rr case_spanrrjrOevaluator_outputsrevaluator_outputs_by_taskoutputsrrrrlexcrVrrWrs        &  revaluation_resultsSequence[EvaluationResult]ttuple[dict[str, EvaluationResult[bool]], dict[str, EvaluationResult[int | float]], dict[str, EvaluationResult[str]]]c Csi}i}i}tt}|D]Q}|j}||vr5d}|d||vr.|d7}|d||vs!|d|}|||t}rF|||<q |tt} rS| ||<q |t} r^| ||<q |||fS)a8Group evaluator outputs by their result type. Args: evaluation_results: Sequence of evaluation results to group. Returns: A tuple of dictionaries mapping evaluator names to their results, grouped by result type: (success_evaluations, metric_evaluations, string_evaluations) rr r%)rirjrFrldowncastrxrwrx) rrrr seen_nameserrFr8 assertionscorelabelrVrVrWr0s*     rz_TaskRun | Noner_rFrOrrcC$t}|dur|||dSdS)zSet an attribute on the current task run. Args: name: The name of the attribute. value: The value of the attribute. N)r_rrY)rFrO current_caserVrVrWr>Yr>rUrPcCr)zIncrement a metric on the current task run. Args: name: The name of the metric. amount: The amount to increment by. N)r_rrX)rFrUrrVrVrWr=err=spanlogfire_api.LogfireSpanfallbackrxc Cs.z |j|jdWSttfy|YSw)a0Calculate the duration of a span in seconds. We prefer to obtain the duration from a span for the sake of consistency with observability and to make the values more reliable during testing. However, if the span is not available (e.g. when using logfire_api without logfire installed), we fall back to the provided duration. Args: span: The span to calculate the duration for. fallback: The fallback duration to use if unable to obtain the duration from the span. Returns: The duration of the span in seconds. iʚ;)end_time start_timeAttributeError TypeError)rrrVrVrWrdqs rdrr:Mapping[str, type[Evaluator[InputsT, OutputT, MetadataT]]]cCsi}|D]/}t|tstd|dd|jvr td|d|}||vr/td||||<qtD] }|||q6|S)zCreate a registry of evaluator types from default and custom evaluators. Args: custom_evaluator_types: Additional evaluator classes to include in the registry. Returns: A mapping from evaluator names to evaluator classes. zBAll custom evaluator classes must be subclasses of Evaluator, but z is not__dataclass_fields__zFAll custom evaluator classes must be decorated with `@dataclass`, but z Duplicate evaluator class name: ) issubclassr*rk__dict__get_serialization_namer-r")rrr%rFrVrVrWrs"      rrrrErr0rc Cs||j}|durtd|jdt|dz ||ji|jWStyJ}z|dur5d|nd}td|jd|d ||d}~ww) aLoad an evaluator from the registry based on a specification. Args: registry: Mapping from evaluator names to evaluator classes. case_name: Name of the case this evaluator will be used for, or None for dataset-level evaluators. spec: Specification of the evaluator to load. Returns: An initialized evaluator instance. Raises: ValueError: If the evaluator name is not found in the registry. Nz Evaluator zA is not in the provided `custom_evaluator_types`. Valid choices: zr. If you are trying to use a custom evaluator, you must include its type in the `custom_evaluator_types` argument.zcase datasetz Failed to instantiate evaluator z for r)rrFrkrUkeysrkwargsr)rrrr%r case_detailrVrVrWrs r)rF)rGr rHr rIrjr}r r@)rsr\rorr]rzr}r^)rsrtrorrrjrraryrzr{rzr}r)rrr}rr[rZ)rrrrxr}rx)rrr}r)rrrrErr0r}r)yrS __future__r _annotationsrBrsysrarrcollections.abcrrrr contextlibrr contextvarsr dataclassesr r r pathlibr typingrrrrrrr logfire_apirrpydanticrrrrrrpydantic._internalr pydantic_corerpydantic_core.core_schemarr rich.progressrtyping_extensionsr r!r"r#pydantic_evals._utilsr$_utilsr&r'r(rOr)r*evaluators._run_evaluatorr,evaluators.commonr-evaluators.contextr.evaluators.evaluatorr/evaluators.specr0otelr1otel._context_subtreer2 reportingr3r4r5r6rvr7 version_infoexceptiongroupr:__all__Logfire_logfirer@rBrCDEFAULT_DATASET_PATHrDrrU_REPORT_CASES_ADAPTER_REPORT_CASE_FAILURES_ADAPTER_REPORT_CASE_AGGREGATE_ADAPTERrDr[r;r<rrLrrrjrrr_r>r=rdrrrVrVrVrWs                             "@ , P^ &  "