Column info for invalid column number#8
Conversation
|
This is a good idea, I'm sure extra warnings would be helpful. That said, the code could be a bit cleaner if you used sets: schema_columns = set(self.get_column_names())
df_columns = set(df.columns)
add_schema_columns = schema_columns - df_columns
add_df_columns = df_columns - schema_columnsAlso could you please add at least one test that test that your warning is correct in |
…s are shown in output message of validation warning
|
I have added a unit test for the warning. Thanks for the hint. |
| df = pd.DataFrame.from_dict({'a': [1, 2, 3]}) | ||
|
|
||
| out = self.schema.validate(df, columns=['a', 'b']) | ||
| assert out[0].message == 'The column b exists in the schema but not in the data frame' |
There was a problem hiding this comment.
This should be a self.assert() method (https://docs.python.org/3.7/library/unittest.html#unittest.TestCase.debug)
| # should raise a PanSchArgumentError | ||
| self.assertRaises(PanSchArgumentError, self.schema.validate, df, columns=['c']) | ||
|
|
||
| def test_column_not_present_shown(self): |
There was a problem hiding this comment.
Please add a docstring that explains this test (can be one sentence)
| ) | ||
| return errors | ||
|
|
||
| errors.append( |
There was a problem hiding this comment.
Why do you have a second ValidationWarning added here? You already have one error if add_schema_columns and one error if add_df_columns. If there is an error with the number of columns this will produce two ValidationWarning objects, when it should produce one.
| schema_columns = set(self.get_column_names()) | ||
| df_columns = set(df.columns) | ||
|
|
||
| add_schema_columns = [col for col in schema_columns if col not in df_columns] |
There was a problem hiding this comment.
Use set operations here, (https://docs.python.org/3/library/stdtypes.html#set) e.g. add_schema_columns = schema_columns - df_columns
2 participants
I have updated the ValidationError message in case the number of columns for the schema and the date frame do not match to include the names of the columns which are additionally present in the schema and/or data frame. This should be more informative for the user.