Skip to content

Conversation

@jsong468
Copy link
Contributor

@jsong468 jsong468 commented Dec 9, 2025

Description

This draft adds underlying_model param and environment variable which differs from model_name which is often actually the deployment name (i.e. Azure endpoints). The underlying_model will give better insight into the actual model used rather than the deployment and will be useful for identifying scoring configurations and their associated evaluation metrics (since deployment names can be custom made).

underlying_model can be passed in explicitly as a param in the constructor or set via environment variable (e.g. OPENAI_CHAT_UNDERLYING_MODEL)

UPDATE: We are not implementing method to get the underlying model name programmatically in this PR.
This also adds a method get_underlying_model_async to get the underlying model name programmatically for certain OpenAI targets that return the model in the response (by sending a very simple prompt, I couldn't get GETs to work...). (The reason why it is not done automatically upon get_identifier is because it seems like overkill to make the method async throughout the codebase. We also can't call it upon instantiation because async inits aren't a thing). So, calling this method is manual, and the preferred way would be through environment variable or passed value, which would flow into the identifier.

Tests and Documentation

Unit tests added

@jbolor21
Copy link
Contributor

jbolor21 commented Dec 9, 2025

great work Justin, I think this makes a lot of sense how the deployment name could be arbitrary but the actual type of deployment is useful esp for metrics!

@jsong468 jsong468 changed the title [DRAFT] FEAT: adding underlying_model for target identification FEAT: adding underlying_model for target identification Dec 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants