Improve validation in structured generation columns

[andreatgretel]

Is your feature request related to a problem? Please describe.

We would like to store a Pydantic model inside the LLMStructuredColumnConfig, since it allows us to better validate the output. However, column configs need to be serializable. Because of that, we end up converting that to a JSON schema, and using gsonschema to do the validation.

Unfortunately, validating using the JSON schema is not as good as using the Pydantic model. Recently, for instance, we had an issue where models would generate either "price": "12.34" (converted to string) or "price": 12.34 (converted to float), both of which passed validation (price being a Decimal), and later would have issues writing to Parquet. See #171 for more details.

Describe the solution you'd like

It would be nice to be able to serialize the Pydantic model keeping features such as complex types, validators etc. One possibility would be creating our own serialization.

Describe alternatives you've considered

Alternative solutions include adding more info to the prompt, depending on the type; and standardizing types later, before writing to Parquet.

Additional context

See notebook attached (by @nabinchha) for one possibility on how to ser/de Pydantic models.

test_serialization_pydantic_core_schema (1).ipynb

[eric-tramel]

This is a dark path. I thought about trying to serialize the Pydantic object itself, but it isn't really feasible, rebuilding Pydantic classes from a JSONSchema or from a serialization isn't really supported behavior for Pydantic. Also, one cannot reliably infer a Pydantic BaseModel type from a JSONSchema since they are not 1-to-1 interoperable -- pydantic implements some things that are not expressed in JSONSchema. Likewise, JSONSchema has some advanced features which are a tangle to try to implement as fixed a Pydantic class.

In the client-server model of things, the serialization gets hard. Because if you want to serialize the Pydantic model as a python object, now we have the exact same kinds of problems that we've had before in distributed systems of yesteryear @andreatgretel (e.g. cloudpickle and python deps). Also, it creates a security problem for the client-server model of the hosted NDD model.

I'd sooner suggest dropping Decimal usage than walk this path 😓 Or, baring that, one can produce a custom serialization or type interpolation at time of writing the Parquet.

[andreatgretel]

Ack! We were discussing this for some time this afternoon, and it's clear that creating our own serialization would cause some issues... Good to know you gave it some thought already.

We wouldn't of course pickle or anything like that. We discussed doing some basic serialization which might be just enough to provide better schema validation. The idea would be essentially having a sort of JSONSchema which is actually 1-to-1, at least for basic types, see attached notebook for an example. Unclear if this is actually possible and what are its limitations, though.

The Decimal issue itself is solved (we added some additional validation inside gsonschema in #171), this would be mostly for other types that JSON can't handle very well.