spark_pipeline_framework.proxy_generator.python_proxy_base

Module Contents

Classes

PythonProxyBase

Abstract class for transformers that transform one dataset into another.

class spark_pipeline_framework.proxy_generator.python_proxy_base.PythonProxyBase(name: Optional[str] = None, parameters: Optional[Dict[str, Any]] = None, progress_logger: Optional[spark_pipeline_framework.progress_logger.progress_logger.ProgressLogger] = None, verify_count_remains_same: bool = False)

Bases: spark_pipeline_framework.transformers.framework_transformer.v1.framework_transformer.FrameworkTransformer

Abstract class for transformers that transform one dataset into another.

New in version 1.3.0.

_transform(self, df: pyspark.sql.dataframe.DataFrame) → pyspark.sql.dataframe.DataFrame

Transforms the input dataset.

datasetpyspark.sql.DataFrame

input dataset.

pyspark.sql.DataFrame

transformed dataset

setName(self, value: str) → PythonProxyBase

getName(self) → str

setProgressLogger(self, value: spark_pipeline_framework.progress_logger.progress_logger.ProgressLogger) → PythonProxyBase

getProgressLogger(self) → Optional[spark_pipeline_framework.progress_logger.progress_logger.ProgressLogger]