o i,@sddlmZddlZddlmZmZddlmZmZddl m Z m Z m Z ddl mZmZmZddlmZmZddlmZdd lmZdd lmZmZdd lmZd d lmZddlmZddl m!Z!dZ"e#e$Be%Be&BZ' e GdddZ(e'e(Bee&e'e(BfBZ) ede'ddZ* edZ+e Gdddee*Z,e GdddZ-ededdZ. ededdZ/ ededdZ0 Gd d!d!eZ1e d"d#Gd$d%d%ee.e/e0fe1d&Z2dS)') annotationsN)ABCMetaabstractmethod) AwaitableMapping)MISSING dataclassfields)AnyGenericcast) ConfigDictmodel_serializer)to_jsonable_python)SerializationInfo)TypeVar deprecated)_utils)get_event_loop)EvaluatorContext) EvaluatorSpec)EvaluationReasonEvaluationResultEvaluationScalar EvaluatorEvaluatorFailureEvaluatorOutputrc@s&eZdZUdZded<dZded<dS)ra7The result of running an evaluator with an optional explanation. Contains a scalar value and an optional "reason" explaining the value. Args: value: The scalar result of the evaluation (boolean, integer, float, or string). reason: An optional explanation of the evaluation result. rvalueN str | Nonereason)__name__ __module__ __qualname____doc____annotations__r!r'r'i/var/www/html/karishye-ai-python/venv/lib/python3.10/site-packages/pydantic_evals/evaluators/evaluator.pyr(s  rEvaluationScalarTT)default covariantTc@s<eZdZUdZded<ded<ded<ded <dddZdS)rayThe details of an individual evaluation result. Contains the name, value, reason, and source evaluator for a single evaluation. Args: name: The name of the evaluation. value: The scalar result of the evaluation. reason: An optional explanation of the evaluation result. source: The spec of the evaluator that produced this result. strnamer)rr r!rsource value_typestype[T]returnEvaluationResult[T] | NonecGsB|D]}t|j|rt|jtr|turqttt|SqdS)a#Attempt to downcast this result to a more specific type. Args: *value_types: The types to check the value against. Returns: A downcast version of this result if the value is an instance of one of the given types, otherwise None. N) isinstancerboolr rr,)selfr0 value_typer'r'r(downcastTs zEvaluationResult.downcastN)r0r1r2r3)r"r#r$r%r&r8r'r'r'r(rBs  rc@s2eZdZUdZded<ded<ded<ded<dS) rzARepresents a failure raised during the execution of an evaluator.r-r. error_messageerror_stacktracerr/N)r"r#r$r%r&r'r'r'r(rhs  rInputsT)r* contravariantOutputT MetadataTcs"eZdZdZd fd d ZZS) _StrictABCMetazXAn ABC-like metaclass that goes further and disallows even defining abstract subclasses.r.r-basestuple[type, ...] namespacedict[str, Any]kwargsr c sltj||||fi|}tdd|jddD}|r4|jr4ddd|jD}t|d||S)Ncss|]}t|tVqdS)N)r4r?).0cr'r'r( sz)_StrictABCMeta.__new__..rz, cSsg|]}|qSr'r')rEmr'r'r( sz*_StrictABCMeta.__new__..z& must implement all abstract methods: )super__new__any__mro____abstractmethods__join TypeError)mclsr.r@rBrDresultis_proper_subclassabstractmethods __class__r'r(rKs  z_StrictABCMeta.__new__)r.r-r@rArBrCrDr )r"r#r$r%rK __classcell__r'r'rUr(r?}sr?F)reprc@seZdZdZeddZed%ddZeedd%d d Z d%d d Z e d&ddZ d'ddZ d'ddZeddd(ddZd)dd Zd*d"d#ZejZd$S)+ra7Base class for all evaluators. Evaluators can assess the performance of a task in a variety of ways, as a function of the EvaluatorContext. Subclasses must implement the `evaluate` method. Note it can be defined with either `def` or `async def`. Example: ```python from dataclasses import dataclass from pydantic_evals.evaluators import Evaluator, EvaluatorContext @dataclass class ExactMatch(Evaluator): def evaluate(self, ctx: EvaluatorContext) -> bool: return ctx.output == ctx.expected_output ``` T)arbitrary_types_allowedr2r-cCs|jS)zReturn the 'name' of this Evaluator to use during serialization. Returns: The name of the Evaluator, which is typically the class name. )r"clsr'r'r(get_serialization_namesz Evaluator.get_serialization_name>`name` has been renamed, use `get_serialization_name` instead.cCs|S)r])r\rZr'r'r(r.szEvaluator.namecCs"t|dd}t|tr |S|S)aLReturn the default name to use in reports for the output of this evaluator. By default, if the evaluator has an attribute called `evaluation_name` of type string, that will be used. Otherwise, the serialization name of the evaluator (which is usually the class name) will be used. This can be overridden to get a more descriptive name in evaluation reports, e.g. using instance information. Note that evaluators that return a mapping of results will always use the keys of that mapping as the names of the associated evaluation results. evaluation_nameN)getattrr4r-r\)r6r^r'r'r(get_default_evaluation_names z%Evaluator.get_default_evaluation_namectx-EvaluatorContext[InputsT, OutputT, MetadataT],EvaluatorOutput | Awaitable[EvaluatorOutput]cCstd)azEvaluate the task output in the given context. This is the main evaluation method that subclasses must implement. It can be either synchronous or asynchronous, returning either an EvaluatorOutput directly or an Awaitable[EvaluatorOutput]. Args: ctx: The context containing the inputs, outputs, and metadata for evaluation. Returns: The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping of evaluation names to either of those. Can be returned either synchronously or as an awaitable for asynchronous evaluation. zYou must implement `evaluate`.)NotImplementedError)r6rar'r'r(evaluateszEvaluator.evaluatercCs*||}t|rt|Stt|S)aRun the evaluator synchronously, handling both sync and async implementations. This method ensures synchronous execution by running any async evaluate implementation to completion using run_until_complete. Args: ctx: The context containing the inputs, outputs, and metadata for evaluation. Returns: The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping of evaluation names to either of those. )reinspect iscoroutinerrun_until_completer rr6raoutputr'r'r( evaluate_syncs   zEvaluator.evaluate_synccs*||}t|r|IdHStt|S)aRun the evaluator asynchronously, handling both sync and async implementations. This method ensures asynchronous execution by properly awaiting any async evaluate implementation. For synchronous implementations, it returns the result directly. Args: ctx: The context containing the inputs, outputs, and metadata for evaluation. Returns: The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping of evaluation names to either of those. N)rerfrgr rrir'r'r(evaluate_asyncs     zEvaluator.evaluate_asyncplain)modeinforr cCst||jddS)zSerialize this Evaluator to a JSON-serializable form. Returns: A JSON-serializable representation of this evaluator as an EvaluatorSpec. T)contextserialize_unknown)ras_specrp)r6ror'r'r( serializes zEvaluator.serializercCsN|}t|dkr d}nt|dkrtt|f}n|}t||dS)Nrr)r. arguments)build_serialization_argumentslennextitervaluesrr\)r6 raw_argumentsrtr'r'r(rr s  zEvaluator.as_specrCcCsZi}t|D]$}t||j}|jtur||jkrq|jtur%||kr%q|||j<q|S)a[Build the arguments for serialization. Evaluators are serialized for inclusion as the "source" in an `EvaluationResult`. If you want to modify how the evaluator is serialized for that or other purposes, you can override this method. Returns: A dictionary of arguments to be used during serialization. )r r_r.r*rdefault_factory)r6rzfieldrr'r'r(rus       z'Evaluator.build_serialization_argumentsN)r2r-)rarbr2rc)rarbr2r)rorr2r )r2r)r2rC)r"r#r$r%r __pydantic_config__ classmethodr\rr.r`rrerkrlrrsrrrurdataclasses_no_defaults_repr__repr__r'r'r'r(rs$        r) metaclass)3 __future__rrfabcrrcollections.abcrr dataclassesrrr typingr r r pydanticr r pydantic_corerpydantic_core.core_schemartyping_extensionsrr pydantic_airrrprspecr__all__r5intfloatr-rrrr)r,rrr;r=r>r?rr'r'r'r(sJ        %  $