spark_auto_mapper.data_types.if_column_exists

Module Contents

Classes

AutoMapperIfColumnExistsType

Allows for columns to be defined based in which a source column may not exist. If the optional source column does
class spark_auto_mapper.data_types.if_column_exists.AutoMapperIfColumnExistsType(column, if_exists, if_not_exists)

Bases: spark_auto_mapper.data_types.data_type_base.AutoMapperDataTypeBase, Generic[_TAutoMapperDataType]

Allows for columns to be defined based in which a source column may not exist. If the optional source column does not exist, the “default” column definition is used instead.

Parameters

  • column (spark_auto_mapper.type_definitions.wrapper_types.AutoMapperColumnOrColumnLikeType) –

  • if_exists (Optional[_TAutoMapperDataType]) –

  • if_not_exists (Optional[_TAutoMapperDataType]) –

get_column_spec(self, source_df, current_column)

Gets the column spec for this automapper data type

Parameters

  • source_df (Optional[pyspark.sql.DataFrame]) – source data frame in case the automapper type needs that data to decide what to do

  • current_column (Optional[pyspark.sql.Column]) – (Optional) this is set when we are inside an array

Return type

pyspark.sql.Column