spark_pipeline_framework.utilities.file_helpers

Module Contents

Functions

get_absolute_paths(filepath: Union[str, List[str], pathlib.Path]) → List[str]	Abstracts handling of paths so we can use paths on both k8s in AWS as well as local
isfile(path: str) → bool
listdir(path: str) → List[str]

spark_pipeline_framework.utilities.file_helpers.get_absolute_paths(filepath: Union[str, List[str], pathlib.Path]) → List[str]

Abstracts handling of paths so we can use paths on both k8s in AWS as well as local

Parameters

filepath – the path or paths to format appropriately

Returns

a list of paths optimized for local or k8s usages

spark_pipeline_framework.utilities.file_helpers.isfile(path: str) → bool

spark_pipeline_framework.utilities.file_helpers.listdir(path: str) → List[str]