stages

Type Members

1. class Cacher extends Transformer with Wrappable with DefaultParamsWritable

   

2. class ClassBalancer extends Estimator[ClassBalancerModel] with DefaultParamsWritable with HasInputCol with HasOutputCol

   

   An estimator that calculates the weights for balancing a dataset.

   An estimator that calculates the weights for balancing a dataset. For example, if the negative class is half the size of the positive class, the weights will be 2 for rows with negative classes and 1 for rows with positive classes. these weights can be used in weighted classifiers and regressors to correct for heavily skewed datasets. The inputCol should be the labels of the classes, and the output col will be the requisite weights.

3. class ClassBalancerModel extends Model[ClassBalancerModel] with ConstructorWritable[ClassBalancerModel]

   

4. class DropColumns extends Transformer with Wrappable with DefaultParamsWritable

   

   DropColumns takes a dataframe and a list of columns to drop as input and returns a dataframe comprised of only those columns not listed in the input list.

   DropColumns takes a dataframe and a list of columns to drop as input and returns a dataframe comprised of only those columns not listed in the input list.

5. class DynamicBufferedBatcher[T] extends Iterator[List[T]]

   

6. class DynamicMiniBatchTransformer extends Transformer with MiniBatchBase

   

7. class EnsembleByKey extends Transformer with Wrappable with DefaultParamsWritable

   

8. class Explode extends Transformer with HasInputCol with HasOutputCol with Wrappable with DefaultParamsWritable

   

9. class FixedBatcher[T] extends Iterator[List[T]]

   

10. class FixedBufferedBatcher[T] extends Iterator[List[T]]

    

11. class FixedMiniBatchTransformer extends Transformer with MiniBatchBase with HasBatchSize

    

12. class FlattenBatch extends Transformer with Wrappable with DefaultParamsWritable

    

13. trait HasBatchSize extends Params

    

14. trait HasMiniBatcher extends Params

    

15. class Lambda extends Transformer with Wrappable with ComplexParamsWritable

    

16. trait MiniBatchBase extends Transformer with DefaultParamsWritable with Wrappable

    

17. class MultiColumnAdapter extends Estimator[PipelineModel] with Wrappable with ComplexParamsWritable

    

    The MultiColumnAdapter takes a unary pipeline stage and a list of input output column pairs and applies the pipeline stage to each input column after being fit

    The MultiColumnAdapter takes a unary pipeline stage and a list of input output column pairs and applies the pipeline stage to each input column after being fit

18. class RenameColumn extends Transformer with Wrappable with DefaultParamsWritable with HasInputCol with HasOutputCol

    

    RenameColumn takes a dataframe with an input and an output column name and returns a dataframe comprised of the original columns with the input column renamed as the output column name.

    RenameColumn takes a dataframe with an input and an output column name and returns a dataframe comprised of the original columns with the input column renamed as the output column name.

19. class Repartition extends Transformer with Wrappable with DefaultParamsWritable

    

    Partitions the dataset into n partitions

20. class SelectColumns extends Transformer with Wrappable with DefaultParamsWritable

    

    SelectColumns takes a dataframe and a list of columns to select as input and returns a dataframe comprised of only those columns listed in the input list.

    SelectColumns takes a dataframe and a list of columns to select as input and returns a dataframe comprised of only those columns listed in the input list.

    The columns to be selected is a list of column names

21. class StratifiedRepartition extends Transformer with Wrappable with DefaultParamsWritable with HasLabelCol with HasSeed

    

    StratifiedRepartition repartitions the DataFrame such that each label is selected in each partition.

    StratifiedRepartition repartitions the DataFrame such that each label is selected in each partition. This may be necessary in some cases such as in LightGBM multiclass classification, where it is necessary for at least one instance of each label to be present on each partition.

22. class SummarizeData extends Transformer with SummarizeDataParams

    

    Compute summary statistics for the dataset.

    Compute summary statistics for the dataset. The following statistics are computed: - counts - basic - sample - percentiles - errorThreshold - error threshold for quantiles

23. trait SummarizeDataParams extends Wrappable with DefaultParamsWritable

    

24. class TextPreprocessor extends Transformer with HasInputCol with HasOutputCol with Wrappable with ComplexParamsWritable

    

    TextPreprocessor takes a dataframe and a dictionary that maps (text -> replacement text), scans each cell in the input col and replaces all substring matches with the corresponding value.

    TextPreprocessor takes a dataframe and a dictionary that maps (text -> replacement text), scans each cell in the input col and replaces all substring matches with the corresponding value. Priority is given to longer keys and from left to right.

25. class TimeIntervalBatcher[T] extends Iterator[List[T]]

    

26. class TimeIntervalMiniBatchTransformer extends Transformer with MiniBatchBase

    

27. class Timer extends Estimator[TimerModel] with TimerParams with ComplexParamsWritable

    

28. class TimerModel extends Model[TimerModel] with TimerParams with ConstructorWritable[TimerModel]

    

29. trait TimerParams extends Wrappable

    

30. class Trie extends Serializable

    

31. class UDFTransformer extends Transformer with Wrappable with ComplexParamsWritable with HasInputCol with HasInputCols with HasOutputCol

    

    UDFTransformer takes as input input column, output column, and a UserDefinedFunction returns a dataframe comprised of the original columns with the output column as the result of the udf applied to the input column

    UDFTransformer takes as input input column, output column, and a UserDefinedFunction returns a dataframe comprised of the original columns with the output column as the result of the udf applied to the input column

    Annotations

    @InternalWrapper()

32. class UnicodeNormalize extends Transformer with HasInputCol with HasOutputCol with Wrappable with ComplexParamsWritable

    

    UnicodeNormalize takes a dataframe and normalizes the unicode representation.

    UnicodeNormalize takes a dataframe and normalizes the unicode representation.