o i@sdZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZdd lmZdd lmZdd lmZd Ze d edZ e dedZ e dedZ ddddddd&d$d%ZdS)'zUtilities for generating example datasets for pydantic_evals. This module provides functions for generating sample datasets for testing and examples, using LLMs to create realistic test data with proper structure. ) annotations)Sequence)Path)Any)ValidationError)TypeVar)Agentmodels)strip_markdown_fences)Dataset) Evaluator)generate_datasetInputsT)defaultOutputT MetadataTNz openai:gpt-4o)pathcustom_evaluator_typesmodel n_examplesextra_instructions dataset_type*type[Dataset[InputsT, OutputT, MetadataT]]rPath | str | Noner6Sequence[type[Evaluator[InputsT, OutputT, MetadataT]]]r$models.Model | models.KnownModelNamerintr str | Nonereturn$Dataset[InputsT, OutputT, MetadataT]c s||}t|d|d|dtdd}||pdIdH}t|j} z |j| d|d }WntyD} z td |j| d} ~ ww|durP|j ||d |S) a1Use an LLM to generate a dataset of test cases, each consisting of input, expected output, and metadata. This function creates a properly structured dataset with the specified input, output, and metadata types. It uses an LLM to attempt to generate realistic test cases that conform to the types' schemas. Args: path: Optional path to save the generated dataset. If provided, the dataset will be saved to this location. dataset_type: The type of dataset to generate, with the desired input, output, and metadata types. custom_evaluator_types: Optional sequence of custom evaluator classes to include in the schema. model: The Pydantic AI model to use for generation. Defaults to 'gpt-4o'. n_examples: Number of examples to generate. Defaults to 3. extra_instructions: Optional additional instructions to provide to the LLM. Returns: A properly structured Dataset object with generated test cases. Raises: ValidationError: If the LLM's response cannot be parsed as a valid dataset. z@Generate an object that is in compliance with this JSON schema: z Include z example cases. You must not include any characters in your response before the opening { of the JSON object, or after the closing }.) system_prompt output_typeretrieszPlease generate the object.Njson)fmtrzRaw response from model: )r) !model_json_schema_with_evaluatorsrstrrunr output from_textrprintto_file) rrrrrr output_schemaagentresultr+err_/var/www/html/karishye-ai-python/venv/lib/python3.10/site-packages/pydantic_evals/generation.pyr !s,  r )rrrrrrrrrrrrr r!)__doc__ __future__rcollections.abcrpathlibrtypingrpydanticrtyping_extensionsr pydantic_airr pydantic_ai._utilsr pydantic_evalsr #pydantic_evals.evaluators.evaluatorr __all__rrrr rrrr3s0