spark_auto_mapper.automappers.container

Module Contents

Classes

AutoMapperContainer

Abstract Base class for AutoMappers
class spark_auto_mapper.automappers.container.AutoMapperContainer

Bases: spark_auto_mapper.automappers.automapper_base.AutoMapperBase

Abstract Base class for AutoMappers

generate_mappers(self, mappers_dict, column_schema, include_null_properties, skip_schema_validation, skip_if_columns_null_or_empty)
Parameters

  • mappers_dict (Dict[str, spark_auto_mapper.type_definitions.defined_types.AutoMapperAnyDataType]) –

  • column_schema (Dict[str, pyspark.sql.types.StructField]) –

  • include_null_properties (bool) –

  • skip_schema_validation (List[str]) –

  • skip_if_columns_null_or_empty (Optional[List[str]]) –

Return type

None

transform_with_data_frame(self, df, source_df, keys)

Internal function called by base class to transform the data frame

Parameters

  • df (pyspark.sql.DataFrame) – destination data frame

  • source_df (Optional[pyspark.sql.DataFrame]) – source data frame

  • keys (List[str]) – key columns

Return type

pyspark.sql.DataFrame

:return data frame after the transform

get_column_specs(self, source_df)

Gets column specs (Spark expressions)

Parameters

source_df (Optional[pyspark.sql.DataFrame]) – source data frame

Returns

dictionary of column name, column expression

Return type

Dict[str, pyspark.sql.Column]