spark_pipeline_framework.pipelines.framework_param_pipeline¶
Module Contents¶
Classes¶
Abstract class for transformers that transform one dataset into another. |
- class spark_pipeline_framework.pipelines.framework_param_pipeline.FrameworkParamPipeline(parameters: Dict[str, Any], progress_logger: spark_pipeline_framework.progress_logger.progress_logger.ProgressLogger)¶
Bases:
spark_pipeline_framework.pipelines.framework_pipeline.FrameworkPipelineAbstract class for transformers that transform one dataset into another.
New in version 1.3.0.
- fit(self, df: pyspark.sql.dataframe.DataFrame, response: Dict[str, Any]) FrameworkParamPipeline¶
- _transform(self, df: pyspark.sql.dataframe.DataFrame, response: Dict[str, Any]) Any¶
Transforms the input dataset.
- dataset
pyspark.sql.DataFrame input dataset.
pyspark.sql.DataFrametransformed dataset
- dataset
- transform(self, dataset: pyspark.sql.dataframe.DataFrame, response: Dict[str, Any], params: Dict[Any, Any] = None)¶
Transforms the input dataset with optional parameters.
New in version 1.3.0.
- dataset
pyspark.sql.DataFrame input dataset
- paramsdict, optional
an optional param map that overrides embedded params.
pyspark.sql.DataFrametransformed dataset
- dataset
- create_steps(self, my_list: Union[List[pyspark.ml.base.Transformer], List[spark_pipeline_framework.transformers.framework_transformer.v1.framework_transformer.FrameworkTransformer], List[spark_pipeline_framework.transformers.framework_param_transformer.v1.framework_param_transformer.FrameworkParamTransformer], List[Union[pyspark.ml.base.Transformer, List[pyspark.ml.base.Transformer]]], List[Union[spark_pipeline_framework.transformers.framework_transformer.v1.framework_transformer.FrameworkTransformer, List[spark_pipeline_framework.transformers.framework_transformer.v1.framework_transformer.FrameworkTransformer]]], List[Union[spark_pipeline_framework.transformers.framework_param_transformer.v1.framework_param_transformer.FrameworkParamTransformer, List[spark_pipeline_framework.transformers.framework_param_transformer.v1.framework_param_transformer.FrameworkParamTransformer]]]]) List[Union[pyspark.ml.base.Transformer, spark_pipeline_framework.transformers.framework_param_transformer.v1.framework_param_transformer.FrameworkParamTransformer]]¶