spark_auto_mapper.automappers.automapper_base¶
Module Contents¶
Classes¶
Abstract Base class for AutoMappers |
- class spark_auto_mapper.automappers.automapper_base.AutoMapperBase¶
Abstract Base class for AutoMappers
- abstract transform_with_data_frame(self, df, source_df, keys)¶
Internal function called by base class to transform the data frame
- Parameters
df (pyspark.sql.DataFrame) – destination data frame
source_df (Optional[pyspark.sql.DataFrame]) – source data frame
keys (List[str]) – key columns
- Return type
pyspark.sql.DataFrame
:return data frame after the transform
- abstract get_column_specs(self, source_df)¶
Gets column specs (Spark expressions)
- Parameters
source_df (Optional[pyspark.sql.DataFrame]) – source data frame
- Returns
dictionary of column name, column expression
- Return type
Dict[str, pyspark.sql.Column]
- check_schema(self, parent_column, source_df)¶
Checks the schema
- Parameters
parent_column (Optional[str]) – parent column
source_df (Optional[pyspark.sql.DataFrame]) – source data frame
- Returns
result of checking schema
- Return type
Optional[spark_auto_mapper.automappers.check_schema_result.CheckSchemaResult]