synapse.ml.lightgbm package

Submodules

synapse.ml.lightgbm.LightGBMClassificationModel module

class synapse.ml.lightgbm.LightGBMClassificationModel.LightGBMClassificationModel(java_obj=None, actualNumClasses=None, featuresCol='features', featuresShapCol='', labelCol='label', leafPredictionCol='', lightGBMBooster=None, numIterations=- 1, predictDisableShapeCheck=False, predictionCol='prediction', probabilityCol='probability', rawPredictionCol='rawPrediction', startIteration=0, thresholds=None)

Bases: LightGBMModelMixin, _LightGBMClassificationModel

getBoosterNumClasses()

Get the number of classes from the booster.

Returns:

The number of classes.

static loadNativeModelFromFile(filename)

Load the model from a native LightGBM text file.

static loadNativeModelFromString(model)

Load the model from a native LightGBM model string.

synapse.ml.lightgbm.LightGBMClassifier module

class synapse.ml.lightgbm.LightGBMClassifier.LightGBMClassifier(java_obj=None, baggingFraction=1.0, baggingFreq=0, baggingSeed=3, binSampleCount=200000, boostFromAverage=True, boostingType='gbdt', catSmooth=10.0, categoricalSlotIndexes=[], categoricalSlotNames=[], catl2=10.0, chunkSize=10000, dataRandomSeed=1, dataTransferMode='streaming', defaultListenPort=12400, deterministic=False, driverListenPort=0, dropRate=0.1, dropSeed=4, earlyStoppingRound=0, executionMode=None, extraSeed=6, featureFraction=1.0, featureFractionByNode=None, featureFractionSeed=2, featuresCol='features', featuresShapCol='', fobj=None, improvementTolerance=0.0, initScoreCol=None, isEnableSparse=True, isProvideTrainingMetric=False, isUnbalance=False, labelCol='label', lambdaL1=0.0, lambdaL2=0.0, leafPredictionCol='', learningRate=0.1, matrixType='auto', maxBin=255, maxBinByFeature=[], maxCatThreshold=32, maxCatToOnehot=4, maxDeltaStep=0.0, maxDepth=- 1, maxDrop=50, maxNumClasses=100, maxStreamingOMPThreads=16, metric='', microBatchSize=100, minDataInLeaf=20, minDataPerBin=3, minDataPerGroup=100, minGainToSplit=0.0, minSumHessianInLeaf=0.001, modelString='', monotoneConstraints=[], monotoneConstraintsMethod='basic', monotonePenalty=0.0, negBaggingFraction=1.0, numBatches=0, numIterations=100, numLeaves=31, numTasks=0, numThreads=0, objective='binary', objectiveSeed=5, otherRate=0.1, parallelism='data_parallel', passThroughArgs='', posBaggingFraction=1.0, predictDisableShapeCheck=False, predictionCol='prediction', probabilityCol='probability', rawPredictionCol='rawPrediction', referenceDataset=None, repartitionByGroupingColumn=True, samplingMode='subset', samplingSubsetSize=1000000, seed=None, skipDrop=0.5, slotNames=[], thresholds=None, timeout=1200.0, topK=20, topRate=0.2, uniformDrop=False, useBarrierExecutionMode=False, useMissing=True, useSingleDatasetMode=True, validationIndicatorCol=None, verbosity=- 1, weightCol=None, xGBoostDartMode=False, zeroAsMissing=False)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaEstimator

Parameters:

• baggingFraction (float) – Bagging fraction

• baggingFreq (int) – Bagging frequency

• baggingSeed (int) – Bagging seed

• binSampleCount (int) – Number of samples considered at computing histogram bins

• boostFromAverage (bool) – Adjusts initial score to the mean of labels for faster convergence

• boostingType (str) – Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling).

• catSmooth (float) – this can reduce the effect of noises in categorical features, especially for categories with few data

• categoricalSlotIndexes (list) – List of categorical column indexes, the slot index in the features column

• categoricalSlotNames (list) – List of categorical column slot names, the slot name in the features column

• catl2 (float) – L2 regularization in categorical split

• chunkSize (int) – Advanced parameter to specify the chunk size for copying Java data to native. If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.

• dataRandomSeed (int) – Random seed for sampling data to construct histogram bins.

• dataTransferMode (str) – Specify how SynapseML transfers data from Spark to LightGBM. Values can be streaming, bulk. Default is bulk, which is the legacy mode.

• defaultListenPort (int) – The default listen port on executors, used for testing

• deterministic (bool) – Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters. Note: setting this to true may slow down training. To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true

• driverListenPort (int) – The listen port on a driver. Default value is 0 (random)

• dropRate (float) – Dropout rate: a fraction of previous trees to drop during the dropout

• dropSeed (int) – Random seed to choose dropping models. Only used in dart.

• earlyStoppingRound (int) – Early stopping round

• executionMode (str) – Deprecated. Please use dataTransferMode.

• extraSeed (int) – Random seed for selecting threshold when extra_trees is true

• featureFraction (float) – Feature fraction

• featureFractionByNode (float) – Feature fraction by node

• featureFractionSeed (int) – Feature fraction seed

• featuresCol (str) – features column name

• featuresShapCol (str) – Output SHAP vector column name after prediction containing the feature contribution values

• fobj (object) – Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).

• improvementTolerance (float) – Tolerance to consider improvement in metric

• initScoreCol (str) – The name of the initial score column, used for continued training

• isEnableSparse (bool) – Used to enable/disable sparse optimization

• isProvideTrainingMetric (bool) – Whether output metric result over training dataset.

• isUnbalance (bool) – Set to true if training data is unbalanced in binary classification scenario

• labelCol (str) – label column name

• lambdaL1 (float) – L1 regularization

• lambdaL2 (float) – L2 regularization

• leafPredictionCol (str) – Predicted leaf indices’s column name

• learningRate (float) – Learning rate or shrinkage rate

• matrixType (str) – Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense. Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.

• maxBin (int) – Max bin

• maxBinByFeature (list) – Max number of bins for each feature

• maxCatThreshold (int) – limit number of split points considered for categorical features

• maxCatToOnehot (int) – when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used

• maxDeltaStep (float) – Used to limit the max output of tree leaves

• maxDepth (int) – Max depth

• maxDrop (int) – Max number of dropped trees during one boosting iteration

• maxNumClasses (int) – Number of max classes to infer numClass in multi-class classification.

• maxStreamingOMPThreads (int) – Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it’s best to set a fixed value.

• metric (str) – Metrics to be evaluated on the evaluation data. Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv.

• microBatchSize (int) – Specify how many elements are sent in a streaming micro-batch.

• minDataInLeaf (int) – Minimal number of data in one leaf. Can be used to deal with over-fitting.

• minDataPerBin (int) – Minimal number of data inside one bin

• minDataPerGroup (int) – minimal number of data per categorical group

• minGainToSplit (float) – The minimal gain to perform split

• minSumHessianInLeaf (float) – Minimal sum hessian in one leaf

• modelString (str) – LightGBM model to retrain

• monotoneConstraints (list) – used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.

• monotoneConstraintsMethod (str) – Monotone constraints method. basic, intermediate, or advanced.

• monotonePenalty (float) – A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.

• negBaggingFraction (float) – Negative Bagging fraction

• numBatches (int) – If greater than 0, splits data into separate batches during training

• numIterations (int) – Number of iterations, LightGBM constructs num_class * num_iterations trees

• numLeaves (int) – Number of leaves

• numTasks (int) – Advanced parameter to specify the number of tasks. SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.

• numThreads (int) – Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.

• objective (str) – The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova.

• objectiveSeed (int) – Random seed for objectives, if random process is needed. Currently used only for rank_xendcg objective.

• otherRate (float) – The retain ratio of small gradient data. Only used in goss.

• parallelism (str) – Tree learner parallelism, can be set to data_parallel or voting_parallel

• passThroughArgs (str) – Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true

• posBaggingFraction (float) – Positive Bagging fraction

• predictDisableShapeCheck (bool) – control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data

• predictionCol (str) – prediction column name

• probabilityCol (str) – Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities

• rawPredictionCol (str) – raw prediction (a.k.a. confidence) column name

• referenceDataset (list) – The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().

• repartitionByGroupingColumn (bool) – Repartition training data according to grouping column, on by default.

• samplingMode (str) – Data sampling for streaming mode. Sampled data is used to define bins. ‘global’: sample from all data, ‘subset’: sample from first N rows, or ‘fixed’: Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.

• samplingSubsetSize (int) – Specify subset size N for the sampling mode ‘subset’. ‘binSampleCount’ rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.

• seed (int) – Main seed, used to generate other seeds

• skipDrop (float) – Probability of skipping the dropout procedure during a boosting iteration

• slotNames (list) – List of slot names in the features column

• thresholds (list) – Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0 excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class’s threshold

• timeout (float) – Timeout in seconds

• topK (int) – The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0

• topRate (float) – The retain ratio of large gradient data. Only used in goss.

• uniformDrop (bool) – Set this to true to use uniform drop in dart mode

• useBarrierExecutionMode (bool) – Barrier execution mode which uses a barrier stage, off by default.

• useMissing (bool) – Set this to false to disable the special handle of missing value

• useSingleDatasetMode (bool) – Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.

• validationIndicatorCol (str) – Indicates whether the row is for training or validation

• verbosity (int) – Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug

• weightCol (str) – The name of the weight column

• xGBoostDartMode (bool) – Set this to true to use xgboost dart mode

• zeroAsMissing (bool) – Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values

baggingFraction = Param(parent='undefined', name='baggingFraction', doc='Bagging fraction')

baggingFreq = Param(parent='undefined', name='baggingFreq', doc='Bagging frequency')

baggingSeed = Param(parent='undefined', name='baggingSeed', doc='Bagging seed')

binSampleCount = Param(parent='undefined', name='binSampleCount', doc='Number of samples considered at computing histogram bins')

boostFromAverage = Param(parent='undefined', name='boostFromAverage', doc='Adjusts initial score to the mean of labels for faster convergence')

boostingType = Param(parent='undefined', name='boostingType', doc='Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling). ')

catSmooth = Param(parent='undefined', name='catSmooth', doc='this can reduce the effect of noises in categorical features, especially for categories with few data')

categoricalSlotIndexes = Param(parent='undefined', name='categoricalSlotIndexes', doc='List of categorical column indexes, the slot index in the features column')

categoricalSlotNames = Param(parent='undefined', name='categoricalSlotNames', doc='List of categorical column slot names, the slot name in the features column')

catl2 = Param(parent='undefined', name='catl2', doc='L2 regularization in categorical split')

chunkSize = Param(parent='undefined', name='chunkSize', doc='Advanced parameter to specify the chunk size for copying Java data to native.  If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.')

dataRandomSeed = Param(parent='undefined', name='dataRandomSeed', doc='Random seed for sampling data to construct histogram bins.')

dataTransferMode = Param(parent='undefined', name='dataTransferMode', doc='Specify how SynapseML transfers data from Spark to LightGBM.  Values can be streaming, bulk. Default is bulk, which is the legacy mode.')

defaultListenPort = Param(parent='undefined', name='defaultListenPort', doc='The default listen port on executors, used for testing')

deterministic = Param(parent='undefined', name='deterministic', doc='Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters.  Note: setting this to true may slow down training.  To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true')

driverListenPort = Param(parent='undefined', name='driverListenPort', doc='The listen port on a driver. Default value is 0 (random)')

dropRate = Param(parent='undefined', name='dropRate', doc='Dropout rate: a fraction of previous trees to drop during the dropout')

dropSeed = Param(parent='undefined', name='dropSeed', doc='Random seed to choose dropping models. Only used in dart.')

earlyStoppingRound = Param(parent='undefined', name='earlyStoppingRound', doc='Early stopping round')

executionMode = Param(parent='undefined', name='executionMode', doc='Deprecated. Please use dataTransferMode.')

extraSeed = Param(parent='undefined', name='extraSeed', doc='Random seed for selecting threshold when extra_trees is true')

featureFraction = Param(parent='undefined', name='featureFraction', doc='Feature fraction')

featureFractionByNode = Param(parent='undefined', name='featureFractionByNode', doc='Feature fraction by node')

featureFractionSeed = Param(parent='undefined', name='featureFractionSeed', doc='Feature fraction seed')

featuresCol = Param(parent='undefined', name='featuresCol', doc='features column name')

featuresShapCol = Param(parent='undefined', name='featuresShapCol', doc='Output SHAP vector column name after prediction containing the feature contribution values')

fobj = Param(parent='undefined', name='fobj', doc='Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).')

getBaggingFraction()

Returns:

Bagging fraction

Return type:

baggingFraction

getBaggingFreq()

Returns:

Bagging frequency

Return type:

baggingFreq

getBaggingSeed()

Returns:

Bagging seed

Return type:

baggingSeed

getBinSampleCount()

Returns:

Number of samples considered at computing histogram bins

Return type:

binSampleCount

getBoostFromAverage()

Returns:

Adjusts initial score to the mean of labels for faster convergence

Return type:

boostFromAverage

getBoostingType()

Returns:

Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling).

Return type:

boostingType

getCatSmooth()

Returns:

this can reduce the effect of noises in categorical features, especially for categories with few data

Return type:

catSmooth

getCategoricalSlotIndexes()

Returns:

List of categorical column indexes, the slot index in the features column

Return type:

categoricalSlotIndexes

getCategoricalSlotNames()

Returns:

List of categorical column slot names, the slot name in the features column

Return type:

categoricalSlotNames

getCatl2()

Returns:

L2 regularization in categorical split

Return type:

catl2

getChunkSize()

Returns:

Advanced parameter to specify the chunk size for copying Java data to native. If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.

Return type:

chunkSize

getDataRandomSeed()

Returns:

Random seed for sampling data to construct histogram bins.

Return type:

dataRandomSeed

getDataTransferMode()

Returns:

Specify how SynapseML transfers data from Spark to LightGBM. Values can be streaming, bulk. Default is bulk, which is the legacy mode.

Return type:

dataTransferMode

getDefaultListenPort()

Returns:

The default listen port on executors, used for testing

Return type:

defaultListenPort

getDeterministic()

Returns:

Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters. Note: setting this to true may slow down training. To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true

Return type:

deterministic

getDriverListenPort()

Returns:

The listen port on a driver. Default value is 0 (random)

Return type:

driverListenPort

getDropRate()

Returns:

Dropout rate: a fraction of previous trees to drop during the dropout

Return type:

dropRate

getDropSeed()

Returns:

Random seed to choose dropping models. Only used in dart.

Return type:

dropSeed

getEarlyStoppingRound()

Returns:

Early stopping round

Return type:

earlyStoppingRound

getExecutionMode()

Returns:

Deprecated. Please use dataTransferMode.

Return type:

executionMode

getExtraSeed()

Returns:

Random seed for selecting threshold when extra_trees is true

Return type:

extraSeed

getFeatureFraction()

Returns:

Feature fraction

Return type:

featureFraction

getFeatureFractionByNode()

Returns:

Feature fraction by node

Return type:

featureFractionByNode

getFeatureFractionSeed()

Returns:

Feature fraction seed

Return type:

featureFractionSeed

getFeaturesCol()

Returns:

features column name

Return type:

featuresCol

getFeaturesShapCol()

Returns:

Output SHAP vector column name after prediction containing the feature contribution values

Return type:

featuresShapCol

getFobj()

Returns:

Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).

Return type:

fobj

getImprovementTolerance()

Returns:

Tolerance to consider improvement in metric

Return type:

improvementTolerance

getInitScoreCol()

Returns:

The name of the initial score column, used for continued training

Return type:

initScoreCol

getIsEnableSparse()

Returns:

Used to enable/disable sparse optimization

Return type:

isEnableSparse

getIsProvideTrainingMetric()

Returns:

Whether output metric result over training dataset.

Return type:

isProvideTrainingMetric

getIsUnbalance()

Returns:

Set to true if training data is unbalanced in binary classification scenario

Return type:

isUnbalance

static getJavaPackage()

Returns package name String.

getLabelCol()

Returns:

label column name

Return type:

labelCol

getLambdaL1()

Returns:

L1 regularization

Return type:

lambdaL1

getLambdaL2()

Returns:

L2 regularization

Return type:

lambdaL2

getLeafPredictionCol()

Returns:

Predicted leaf indices’s column name

Return type:

leafPredictionCol

getLearningRate()

Returns:

Learning rate or shrinkage rate

Return type:

learningRate

getMatrixType()

Returns:

Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense. Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.

Return type:

matrixType

getMaxBin()

Returns:

Max bin

Return type:

maxBin

getMaxBinByFeature()

Returns:

Max number of bins for each feature

Return type:

maxBinByFeature

getMaxCatThreshold()

Returns:

limit number of split points considered for categorical features

Return type:

maxCatThreshold

getMaxCatToOnehot()

Returns:

when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used

Return type:

maxCatToOnehot

getMaxDeltaStep()

Returns:

Used to limit the max output of tree leaves

Return type:

maxDeltaStep

getMaxDepth()

Returns:

Max depth

Return type:

maxDepth

getMaxDrop()

Returns:

Max number of dropped trees during one boosting iteration

Return type:

maxDrop

getMaxNumClasses()

Returns:

Number of max classes to infer numClass in multi-class classification.

Return type:

maxNumClasses

getMaxStreamingOMPThreads()

Returns:

Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it’s best to set a fixed value.

Return type:

maxStreamingOMPThreads

getMetric()

Returns:

Metrics to be evaluated on the evaluation data. Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv.

Return type:

metric

getMicroBatchSize()

Returns:

Specify how many elements are sent in a streaming micro-batch.

Return type:

microBatchSize

getMinDataInLeaf()

Returns:

Minimal number of data in one leaf. Can be used to deal with over-fitting.

Return type:

minDataInLeaf

getMinDataPerBin()

Returns:

Minimal number of data inside one bin

Return type:

minDataPerBin

getMinDataPerGroup()

Returns:

minimal number of data per categorical group

Return type:

minDataPerGroup

getMinGainToSplit()

Returns:

The minimal gain to perform split

Return type:

minGainToSplit

getMinSumHessianInLeaf()

Returns:

Minimal sum hessian in one leaf

Return type:

minSumHessianInLeaf

getModelString()

Returns:

LightGBM model to retrain

Return type:

modelString

getMonotoneConstraints()

Returns:

used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.

Return type:

monotoneConstraints

getMonotoneConstraintsMethod()

Returns:

Monotone constraints method. basic, intermediate, or advanced.

Return type:

monotoneConstraintsMethod

getMonotonePenalty()

Returns:

A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.

Return type:

monotonePenalty

getNegBaggingFraction()

Returns:

Negative Bagging fraction

Return type:

negBaggingFraction

getNumBatches()

Returns:

If greater than 0, splits data into separate batches during training

Return type:

numBatches

getNumIterations()

Returns:

Number of iterations, LightGBM constructs num_class * num_iterations trees

Return type:

numIterations

getNumLeaves()

Returns:

Number of leaves

Return type:

numLeaves

getNumTasks()

Returns:

Advanced parameter to specify the number of tasks. SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.

Return type:

numTasks

getNumThreads()

Returns:

Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.

Return type:

numThreads

getObjective()

Returns:

The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova.

Return type:

objective

getObjectiveSeed()

Returns:

Random seed for objectives, if random process is needed. Currently used only for rank_xendcg objective.

Return type:

objectiveSeed

getOtherRate()

Returns:

The retain ratio of small gradient data. Only used in goss.

Return type:

otherRate

getParallelism()

Returns:

Tree learner parallelism, can be set to data_parallel or voting_parallel

Return type:

parallelism

getPassThroughArgs()

Returns:

Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true

Return type:

passThroughArgs

getPosBaggingFraction()

Returns:

Positive Bagging fraction

Return type:

posBaggingFraction

getPredictDisableShapeCheck()

Returns:

control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data

Return type:

predictDisableShapeCheck

getPredictionCol()

Returns:

prediction column name

Return type:

predictionCol

getProbabilityCol()

Returns:

Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities

Return type:

probabilityCol

getRawPredictionCol()

Returns:

raw prediction (a.k.a. confidence) column name

Return type:

rawPredictionCol

getReferenceDataset()

Returns:

The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().

Return type:

referenceDataset

getRepartitionByGroupingColumn()

Returns:

Repartition training data according to grouping column, on by default.

Return type:

repartitionByGroupingColumn

getSamplingMode()

Returns:

Data sampling for streaming mode. Sampled data is used to define bins. ‘global’: sample from all data, ‘subset’: sample from first N rows, or ‘fixed’: Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.

Return type:

samplingMode

getSamplingSubsetSize()

Returns:

Specify subset size N for the sampling mode ‘subset’. ‘binSampleCount’ rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.

Return type:

samplingSubsetSize

getSeed()

Returns:

Main seed, used to generate other seeds

Return type:

seed

getSkipDrop()

Returns:

Probability of skipping the dropout procedure during a boosting iteration

Return type:

skipDrop

getSlotNames()

Returns:

List of slot names in the features column

Return type:

slotNames

getThresholds()

Returns:

Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0 excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class’s threshold

Return type:

thresholds

getTimeout()

Returns:

Timeout in seconds

Return type:

timeout

getTopK()

Returns:

The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0

Return type:

topK

getTopRate()

Returns:

The retain ratio of large gradient data. Only used in goss.

Return type:

topRate

getUniformDrop()

Returns:

Set this to true to use uniform drop in dart mode

Return type:

uniformDrop

getUseBarrierExecutionMode()

Returns:

Barrier execution mode which uses a barrier stage, off by default.

Return type:

useBarrierExecutionMode

getUseMissing()

Returns:

Set this to false to disable the special handle of missing value

Return type:

useMissing

getUseSingleDatasetMode()

Returns:

Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.

Return type:

useSingleDatasetMode

getValidationIndicatorCol()

Returns:

Indicates whether the row is for training or validation

Return type:

validationIndicatorCol

getVerbosity()

Returns:

Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug

Return type:

verbosity

getWeightCol()

Returns:

The name of the weight column

Return type:

weightCol

getXGBoostDartMode()

Returns:

Set this to true to use xgboost dart mode

Return type:

xGBoostDartMode

getZeroAsMissing()

Returns:

Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values

Return type:

zeroAsMissing

improvementTolerance = Param(parent='undefined', name='improvementTolerance', doc='Tolerance to consider improvement in metric')

initScoreCol = Param(parent='undefined', name='initScoreCol', doc='The name of the initial score column, used for continued training')

isEnableSparse = Param(parent='undefined', name='isEnableSparse', doc='Used to enable/disable sparse optimization')

isProvideTrainingMetric = Param(parent='undefined', name='isProvideTrainingMetric', doc='Whether output metric result over training dataset.')

isUnbalance = Param(parent='undefined', name='isUnbalance', doc='Set to true if training data is unbalanced in binary classification scenario')

labelCol = Param(parent='undefined', name='labelCol', doc='label column name')

lambdaL1 = Param(parent='undefined', name='lambdaL1', doc='L1 regularization')

lambdaL2 = Param(parent='undefined', name='lambdaL2', doc='L2 regularization')

leafPredictionCol = Param(parent='undefined', name='leafPredictionCol', doc="Predicted leaf indices's column name")

learningRate = Param(parent='undefined', name='learningRate', doc='Learning rate or shrinkage rate')

matrixType = Param(parent='undefined', name='matrixType', doc='Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense.  Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.')

maxBin = Param(parent='undefined', name='maxBin', doc='Max bin')

maxBinByFeature = Param(parent='undefined', name='maxBinByFeature', doc='Max number of bins for each feature')

maxCatThreshold = Param(parent='undefined', name='maxCatThreshold', doc='limit number of split points considered for categorical features')

maxCatToOnehot = Param(parent='undefined', name='maxCatToOnehot', doc='when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used')

maxDeltaStep = Param(parent='undefined', name='maxDeltaStep', doc='Used to limit the max output of tree leaves')

maxDepth = Param(parent='undefined', name='maxDepth', doc='Max depth')

maxDrop = Param(parent='undefined', name='maxDrop', doc='Max number of dropped trees during one boosting iteration')

maxNumClasses = Param(parent='undefined', name='maxNumClasses', doc='Number of max classes to infer numClass in multi-class classification.')

maxStreamingOMPThreads = Param(parent='undefined', name='maxStreamingOMPThreads', doc="Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it's best to set a fixed value.")

metric = Param(parent='undefined', name='metric', doc='Metrics to be evaluated on the evaluation data.  Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv. ')

microBatchSize = Param(parent='undefined', name='microBatchSize', doc='Specify how many elements are sent in a streaming micro-batch.')

minDataInLeaf = Param(parent='undefined', name='minDataInLeaf', doc='Minimal number of data in one leaf. Can be used to deal with over-fitting.')

minDataPerBin = Param(parent='undefined', name='minDataPerBin', doc='Minimal number of data inside one bin')

minDataPerGroup = Param(parent='undefined', name='minDataPerGroup', doc='minimal number of data per categorical group')

minGainToSplit = Param(parent='undefined', name='minGainToSplit', doc='The minimal gain to perform split')

minSumHessianInLeaf = Param(parent='undefined', name='minSumHessianInLeaf', doc='Minimal sum hessian in one leaf')

modelString = Param(parent='undefined', name='modelString', doc='LightGBM model to retrain')

monotoneConstraints = Param(parent='undefined', name='monotoneConstraints', doc='used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.')

monotoneConstraintsMethod = Param(parent='undefined', name='monotoneConstraintsMethod', doc='Monotone constraints method. basic, intermediate, or advanced.')

monotonePenalty = Param(parent='undefined', name='monotonePenalty', doc='A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.')

negBaggingFraction = Param(parent='undefined', name='negBaggingFraction', doc='Negative Bagging fraction')

numBatches = Param(parent='undefined', name='numBatches', doc='If greater than 0, splits data into separate batches during training')

numIterations = Param(parent='undefined', name='numIterations', doc='Number of iterations, LightGBM constructs num_class * num_iterations trees')

numLeaves = Param(parent='undefined', name='numLeaves', doc='Number of leaves')

numTasks = Param(parent='undefined', name='numTasks', doc='Advanced parameter to specify the number of tasks.  SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.')

numThreads = Param(parent='undefined', name='numThreads', doc='Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.')

objective = Param(parent='undefined', name='objective', doc='The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova. ')

objectiveSeed = Param(parent='undefined', name='objectiveSeed', doc='Random seed for objectives, if random process is needed.  Currently used only for rank_xendcg objective.')

otherRate = Param(parent='undefined', name='otherRate', doc='The retain ratio of small gradient data. Only used in goss.')

parallelism = Param(parent='undefined', name='parallelism', doc='Tree learner parallelism, can be set to data_parallel or voting_parallel')

passThroughArgs = Param(parent='undefined', name='passThroughArgs', doc='Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true')

posBaggingFraction = Param(parent='undefined', name='posBaggingFraction', doc='Positive Bagging fraction')

predictDisableShapeCheck = Param(parent='undefined', name='predictDisableShapeCheck', doc='control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data')

predictionCol = Param(parent='undefined', name='predictionCol', doc='prediction column name')

probabilityCol = Param(parent='undefined', name='probabilityCol', doc='Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities')

rawPredictionCol = Param(parent='undefined', name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column name')

classmethod read()

Returns an MLReader instance for this class.

referenceDataset = Param(parent='undefined', name='referenceDataset', doc='The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().')

repartitionByGroupingColumn = Param(parent='undefined', name='repartitionByGroupingColumn', doc='Repartition training data according to grouping column, on by default.')

samplingMode = Param(parent='undefined', name='samplingMode', doc="Data sampling for streaming mode. Sampled data is used to define bins. 'global': sample from all data, 'subset': sample from first N rows, or 'fixed': Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.")

samplingSubsetSize = Param(parent='undefined', name='samplingSubsetSize', doc="Specify subset size N for the sampling mode 'subset'. 'binSampleCount' rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.")

seed = Param(parent='undefined', name='seed', doc='Main seed, used to generate other seeds')

setBaggingFraction(value)

Parameters:

baggingFraction – Bagging fraction

setBaggingFreq(value)

Parameters:

baggingFreq – Bagging frequency

setBaggingSeed(value)

Parameters:

baggingSeed – Bagging seed

setBinSampleCount(value)

Parameters:

binSampleCount – Number of samples considered at computing histogram bins

setBoostFromAverage(value)

Parameters:

boostFromAverage – Adjusts initial score to the mean of labels for faster convergence

setBoostingType(value)

Parameters:

boostingType – Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling).

setCatSmooth(value)

Parameters:

catSmooth – this can reduce the effect of noises in categorical features, especially for categories with few data

setCategoricalSlotIndexes(value)

Parameters:

categoricalSlotIndexes – List of categorical column indexes, the slot index in the features column

setCategoricalSlotNames(value)

Parameters:

categoricalSlotNames – List of categorical column slot names, the slot name in the features column

setCatl2(value)

Parameters:

catl2 – L2 regularization in categorical split

setChunkSize(value)

Parameters:

chunkSize – Advanced parameter to specify the chunk size for copying Java data to native. If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.

setDataRandomSeed(value)

Parameters:

dataRandomSeed – Random seed for sampling data to construct histogram bins.

setDataTransferMode(value)

Parameters:

dataTransferMode – Specify how SynapseML transfers data from Spark to LightGBM. Values can be streaming, bulk. Default is bulk, which is the legacy mode.

setDefaultListenPort(value)

Parameters:

defaultListenPort – The default listen port on executors, used for testing

setDeterministic(value)

Parameters:

deterministic – Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters. Note: setting this to true may slow down training. To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true

setDriverListenPort(value)

Parameters:

driverListenPort – The listen port on a driver. Default value is 0 (random)

setDropRate(value)

Parameters:

dropRate – Dropout rate: a fraction of previous trees to drop during the dropout

setDropSeed(value)

Parameters:

dropSeed – Random seed to choose dropping models. Only used in dart.

setEarlyStoppingRound(value)

Parameters:

earlyStoppingRound – Early stopping round

setExecutionMode(value)

Parameters:

executionMode – Deprecated. Please use dataTransferMode.

setExtraSeed(value)

Parameters:

extraSeed – Random seed for selecting threshold when extra_trees is true

setFeatureFraction(value)

Parameters:

featureFraction – Feature fraction

setFeatureFractionByNode(value)

Parameters:

featureFractionByNode – Feature fraction by node

setFeatureFractionSeed(value)

Parameters:

featureFractionSeed – Feature fraction seed

setFeaturesCol(value)

Parameters:

featuresCol – features column name

setFeaturesShapCol(value)

Parameters:

featuresShapCol – Output SHAP vector column name after prediction containing the feature contribution values

setFobj(value)

Parameters:

fobj – Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).

setImprovementTolerance(value)

Parameters:

improvementTolerance – Tolerance to consider improvement in metric

setInitScoreCol(value)

Parameters:

initScoreCol – The name of the initial score column, used for continued training

setIsEnableSparse(value)

Parameters:

isEnableSparse – Used to enable/disable sparse optimization

setIsProvideTrainingMetric(value)

Parameters:

isProvideTrainingMetric – Whether output metric result over training dataset.

setIsUnbalance(value)

Parameters:

isUnbalance – Set to true if training data is unbalanced in binary classification scenario

setLabelCol(value)

Parameters:

labelCol – label column name

setLambdaL1(value)

Parameters:

lambdaL1 – L1 regularization

setLambdaL2(value)

Parameters:

lambdaL2 – L2 regularization

setLeafPredictionCol(value)

Parameters:

leafPredictionCol – Predicted leaf indices’s column name

setLearningRate(value)

Parameters:

learningRate – Learning rate or shrinkage rate

setMatrixType(value)

Parameters:

matrixType – Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense. Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.

setMaxBin(value)

Parameters:

maxBin – Max bin

setMaxBinByFeature(value)

Parameters:

maxBinByFeature – Max number of bins for each feature

setMaxCatThreshold(value)

Parameters:

maxCatThreshold – limit number of split points considered for categorical features

setMaxCatToOnehot(value)

Parameters:

maxCatToOnehot – when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used

setMaxDeltaStep(value)

Parameters:

maxDeltaStep – Used to limit the max output of tree leaves

setMaxDepth(value)

Parameters:

maxDepth – Max depth

setMaxDrop(value)

Parameters:

maxDrop – Max number of dropped trees during one boosting iteration

setMaxNumClasses(value)

Parameters:

maxNumClasses – Number of max classes to infer numClass in multi-class classification.

setMaxStreamingOMPThreads(value)

Parameters:

maxStreamingOMPThreads – Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it’s best to set a fixed value.

setMetric(value)

Parameters:

metric – Metrics to be evaluated on the evaluation data. Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv.

setMicroBatchSize(value)

Parameters:

microBatchSize – Specify how many elements are sent in a streaming micro-batch.

setMinDataInLeaf(value)

Parameters:

minDataInLeaf – Minimal number of data in one leaf. Can be used to deal with over-fitting.

setMinDataPerBin(value)

Parameters:

minDataPerBin – Minimal number of data inside one bin

setMinDataPerGroup(value)

Parameters:

minDataPerGroup – minimal number of data per categorical group

setMinGainToSplit(value)

Parameters:

minGainToSplit – The minimal gain to perform split

setMinSumHessianInLeaf(value)

Parameters:

minSumHessianInLeaf – Minimal sum hessian in one leaf

setModelString(value)

Parameters:

modelString – LightGBM model to retrain

setMonotoneConstraints(value)

Parameters:

monotoneConstraints – used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.

setMonotoneConstraintsMethod(value)

Parameters:

monotoneConstraintsMethod – Monotone constraints method. basic, intermediate, or advanced.

setMonotonePenalty(value)

Parameters:

monotonePenalty – A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.

setNegBaggingFraction(value)

Parameters:

negBaggingFraction – Negative Bagging fraction

setNumBatches(value)

Parameters:

numBatches – If greater than 0, splits data into separate batches during training

setNumIterations(value)

Parameters:

numIterations – Number of iterations, LightGBM constructs num_class * num_iterations trees

setNumLeaves(value)

Parameters:

numLeaves – Number of leaves

setNumTasks(value)

Parameters:

numTasks – Advanced parameter to specify the number of tasks. SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.

setNumThreads(value)

Parameters:

numThreads – Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.

setObjective(value)

Parameters:

objective – The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova.

setObjectiveSeed(value)

Parameters:

objectiveSeed – Random seed for objectives, if random process is needed. Currently used only for rank_xendcg objective.

setOtherRate(value)

Parameters:

otherRate – The retain ratio of small gradient data. Only used in goss.

setParallelism(value)

Parameters:

parallelism – Tree learner parallelism, can be set to data_parallel or voting_parallel

setParams(baggingFraction=1.0, baggingFreq=0, baggingSeed=3, binSampleCount=200000, boostFromAverage=True, boostingType='gbdt', catSmooth=10.0, categoricalSlotIndexes=[], categoricalSlotNames=[], catl2=10.0, chunkSize=10000, dataRandomSeed=1, dataTransferMode='streaming', defaultListenPort=12400, deterministic=False, driverListenPort=0, dropRate=0.1, dropSeed=4, earlyStoppingRound=0, executionMode=None, extraSeed=6, featureFraction=1.0, featureFractionByNode=None, featureFractionSeed=2, featuresCol='features', featuresShapCol='', fobj=None, improvementTolerance=0.0, initScoreCol=None, isEnableSparse=True, isProvideTrainingMetric=False, isUnbalance=False, labelCol='label', lambdaL1=0.0, lambdaL2=0.0, leafPredictionCol='', learningRate=0.1, matrixType='auto', maxBin=255, maxBinByFeature=[], maxCatThreshold=32, maxCatToOnehot=4, maxDeltaStep=0.0, maxDepth=- 1, maxDrop=50, maxNumClasses=100, maxStreamingOMPThreads=16, metric='', microBatchSize=100, minDataInLeaf=20, minDataPerBin=3, minDataPerGroup=100, minGainToSplit=0.0, minSumHessianInLeaf=0.001, modelString='', monotoneConstraints=[], monotoneConstraintsMethod='basic', monotonePenalty=0.0, negBaggingFraction=1.0, numBatches=0, numIterations=100, numLeaves=31, numTasks=0, numThreads=0, objective='binary', objectiveSeed=5, otherRate=0.1, parallelism='data_parallel', passThroughArgs='', posBaggingFraction=1.0, predictDisableShapeCheck=False, predictionCol='prediction', probabilityCol='probability', rawPredictionCol='rawPrediction', referenceDataset=None, repartitionByGroupingColumn=True, samplingMode='subset', samplingSubsetSize=1000000, seed=None, skipDrop=0.5, slotNames=[], thresholds=None, timeout=1200.0, topK=20, topRate=0.2, uniformDrop=False, useBarrierExecutionMode=False, useMissing=True, useSingleDatasetMode=True, validationIndicatorCol=None, verbosity=- 1, weightCol=None, xGBoostDartMode=False, zeroAsMissing=False)

Set the (keyword only) parameters

setPassThroughArgs(value)

Parameters:

passThroughArgs – Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true

setPosBaggingFraction(value)

Parameters:

posBaggingFraction – Positive Bagging fraction

setPredictDisableShapeCheck(value)

Parameters:

predictDisableShapeCheck – control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data

setPredictionCol(value)

Parameters:

predictionCol – prediction column name

setProbabilityCol(value)

Parameters:

probabilityCol – Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities

setRawPredictionCol(value)

Parameters:

rawPredictionCol – raw prediction (a.k.a. confidence) column name

setReferenceDataset(value)

Parameters:

referenceDataset – The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().

setRepartitionByGroupingColumn(value)

Parameters:

repartitionByGroupingColumn – Repartition training data according to grouping column, on by default.

setSamplingMode(value)

Parameters:

samplingMode – Data sampling for streaming mode. Sampled data is used to define bins. ‘global’: sample from all data, ‘subset’: sample from first N rows, or ‘fixed’: Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.

setSamplingSubsetSize(value)

Parameters:

samplingSubsetSize – Specify subset size N for the sampling mode ‘subset’. ‘binSampleCount’ rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.

setSeed(value)

Parameters:

seed – Main seed, used to generate other seeds

setSkipDrop(value)

Parameters:

skipDrop – Probability of skipping the dropout procedure during a boosting iteration

setSlotNames(value)

Parameters:

slotNames – List of slot names in the features column

setThresholds(value)

Parameters:

thresholds – Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0 excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class’s threshold

setTimeout(value)

Parameters:

timeout – Timeout in seconds

setTopK(value)

Parameters:

topK – The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0

setTopRate(value)

Parameters:

topRate – The retain ratio of large gradient data. Only used in goss.

setUniformDrop(value)

Parameters:

uniformDrop – Set this to true to use uniform drop in dart mode

setUseBarrierExecutionMode(value)

Parameters:

useBarrierExecutionMode – Barrier execution mode which uses a barrier stage, off by default.

setUseMissing(value)

Parameters:

useMissing – Set this to false to disable the special handle of missing value

setUseSingleDatasetMode(value)

Parameters:

useSingleDatasetMode – Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.

setValidationIndicatorCol(value)

Parameters:

validationIndicatorCol – Indicates whether the row is for training or validation

setVerbosity(value)

Parameters:

verbosity – Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug

setWeightCol(value)

Parameters:

weightCol – The name of the weight column

setXGBoostDartMode(value)

Parameters:

xGBoostDartMode – Set this to true to use xgboost dart mode

setZeroAsMissing(value)

Parameters:

zeroAsMissing – Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values

skipDrop = Param(parent='undefined', name='skipDrop', doc='Probability of skipping the dropout procedure during a boosting iteration')

slotNames = Param(parent='undefined', name='slotNames', doc='List of slot names in the features column')

thresholds = Param(parent='undefined', name='thresholds', doc="Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0 excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold")

timeout = Param(parent='undefined', name='timeout', doc='Timeout in seconds')

topK = Param(parent='undefined', name='topK', doc='The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0')

topRate = Param(parent='undefined', name='topRate', doc='The retain ratio of large gradient data. Only used in goss.')

uniformDrop = Param(parent='undefined', name='uniformDrop', doc='Set this to true to use uniform drop in dart mode')

useBarrierExecutionMode = Param(parent='undefined', name='useBarrierExecutionMode', doc='Barrier execution mode which uses a barrier stage, off by default.')

useMissing = Param(parent='undefined', name='useMissing', doc='Set this to false to disable the special handle of missing value')

useSingleDatasetMode = Param(parent='undefined', name='useSingleDatasetMode', doc='Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.')

validationIndicatorCol = Param(parent='undefined', name='validationIndicatorCol', doc='Indicates whether the row is for training or validation')

verbosity = Param(parent='undefined', name='verbosity', doc='Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug')

weightCol = Param(parent='undefined', name='weightCol', doc='The name of the weight column')

xGBoostDartMode = Param(parent='undefined', name='xGBoostDartMode', doc='Set this to true to use xgboost dart mode')

zeroAsMissing = Param(parent='undefined', name='zeroAsMissing', doc='Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values')

synapse.ml.lightgbm.LightGBMRanker module

class synapse.ml.lightgbm.LightGBMRanker.LightGBMRanker(java_obj=None, baggingFraction=1.0, baggingFreq=0, baggingSeed=3, binSampleCount=200000, boostFromAverage=True, boostingType='gbdt', catSmooth=10.0, categoricalSlotIndexes=[], categoricalSlotNames=[], catl2=10.0, chunkSize=10000, dataRandomSeed=1, dataTransferMode='streaming', defaultListenPort=12400, deterministic=False, driverListenPort=0, dropRate=0.1, dropSeed=4, earlyStoppingRound=0, evalAt=[1, 2, 3, 4, 5], executionMode=None, extraSeed=6, featureFraction=1.0, featureFractionByNode=None, featureFractionSeed=2, featuresCol='features', featuresShapCol='', fobj=None, groupCol=None, improvementTolerance=0.0, initScoreCol=None, isEnableSparse=True, isProvideTrainingMetric=False, labelCol='label', labelGain=[], lambdaL1=0.0, lambdaL2=0.0, leafPredictionCol='', learningRate=0.1, matrixType='auto', maxBin=255, maxBinByFeature=[], maxCatThreshold=32, maxCatToOnehot=4, maxDeltaStep=0.0, maxDepth=- 1, maxDrop=50, maxNumClasses=100, maxPosition=20, maxStreamingOMPThreads=16, metric='', microBatchSize=100, minDataInLeaf=20, minDataPerBin=3, minDataPerGroup=100, minGainToSplit=0.0, minSumHessianInLeaf=0.001, modelString='', monotoneConstraints=[], monotoneConstraintsMethod='basic', monotonePenalty=0.0, negBaggingFraction=1.0, numBatches=0, numIterations=100, numLeaves=31, numTasks=0, numThreads=0, objective='lambdarank', objectiveSeed=5, otherRate=0.1, parallelism='data_parallel', passThroughArgs='', posBaggingFraction=1.0, predictDisableShapeCheck=False, predictionCol='prediction', referenceDataset=None, repartitionByGroupingColumn=True, samplingMode='subset', samplingSubsetSize=1000000, seed=None, skipDrop=0.5, slotNames=[], timeout=1200.0, topK=20, topRate=0.2, uniformDrop=False, useBarrierExecutionMode=False, useMissing=True, useSingleDatasetMode=True, validationIndicatorCol=None, verbosity=- 1, weightCol=None, xGBoostDartMode=False, zeroAsMissing=False)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaEstimator

Parameters:

• baggingFraction (float) – Bagging fraction

• baggingFreq (int) – Bagging frequency

• baggingSeed (int) – Bagging seed

• binSampleCount (int) – Number of samples considered at computing histogram bins

• boostFromAverage (bool) – Adjusts initial score to the mean of labels for faster convergence

• boostingType (str) – Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling).

• catSmooth (float) – this can reduce the effect of noises in categorical features, especially for categories with few data

• categoricalSlotIndexes (list) – List of categorical column indexes, the slot index in the features column

• categoricalSlotNames (list) – List of categorical column slot names, the slot name in the features column

• catl2 (float) – L2 regularization in categorical split

• chunkSize (int) – Advanced parameter to specify the chunk size for copying Java data to native. If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.

• dataRandomSeed (int) – Random seed for sampling data to construct histogram bins.

• dataTransferMode (str) – Specify how SynapseML transfers data from Spark to LightGBM. Values can be streaming, bulk. Default is bulk, which is the legacy mode.

• defaultListenPort (int) – The default listen port on executors, used for testing

• deterministic (bool) – Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters. Note: setting this to true may slow down training. To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true

• driverListenPort (int) – The listen port on a driver. Default value is 0 (random)

• dropRate (float) – Dropout rate: a fraction of previous trees to drop during the dropout

• dropSeed (int) – Random seed to choose dropping models. Only used in dart.

• earlyStoppingRound (int) – Early stopping round

• evalAt (list) – NDCG and MAP evaluation positions, separated by comma

• executionMode (str) – Deprecated. Please use dataTransferMode.

• extraSeed (int) – Random seed for selecting threshold when extra_trees is true

• featureFraction (float) – Feature fraction

• featureFractionByNode (float) – Feature fraction by node

• featureFractionSeed (int) – Feature fraction seed

• featuresCol (str) – features column name

• featuresShapCol (str) – Output SHAP vector column name after prediction containing the feature contribution values

• fobj (object) – Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).

• groupCol (str) – The name of the group column

• improvementTolerance (float) – Tolerance to consider improvement in metric

• initScoreCol (str) – The name of the initial score column, used for continued training

• isEnableSparse (bool) – Used to enable/disable sparse optimization

• isProvideTrainingMetric (bool) – Whether output metric result over training dataset.

• labelCol (str) – label column name

• labelGain (list) – graded relevance for each label in NDCG

• lambdaL1 (float) – L1 regularization

• lambdaL2 (float) – L2 regularization

• leafPredictionCol (str) – Predicted leaf indices’s column name

• learningRate (float) – Learning rate or shrinkage rate

• matrixType (str) – Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense. Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.

• maxBin (int) – Max bin

• maxBinByFeature (list) – Max number of bins for each feature

• maxCatThreshold (int) – limit number of split points considered for categorical features

• maxCatToOnehot (int) – when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used

• maxDeltaStep (float) – Used to limit the max output of tree leaves

• maxDepth (int) – Max depth

• maxDrop (int) – Max number of dropped trees during one boosting iteration

• maxNumClasses (int) – Number of max classes to infer numClass in multi-class classification.

• maxPosition (int) – optimized NDCG at this position

• maxStreamingOMPThreads (int) – Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it’s best to set a fixed value.

• metric (str) – Metrics to be evaluated on the evaluation data. Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv.

• microBatchSize (int) – Specify how many elements are sent in a streaming micro-batch.

• minDataInLeaf (int) – Minimal number of data in one leaf. Can be used to deal with over-fitting.

• minDataPerBin (int) – Minimal number of data inside one bin

• minDataPerGroup (int) – minimal number of data per categorical group

• minGainToSplit (float) – The minimal gain to perform split

• minSumHessianInLeaf (float) – Minimal sum hessian in one leaf

• modelString (str) – LightGBM model to retrain

• monotoneConstraints (list) – used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.

• monotoneConstraintsMethod (str) – Monotone constraints method. basic, intermediate, or advanced.

• monotonePenalty (float) – A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.

• negBaggingFraction (float) – Negative Bagging fraction

• numBatches (int) – If greater than 0, splits data into separate batches during training

• numIterations (int) – Number of iterations, LightGBM constructs num_class * num_iterations trees

• numLeaves (int) – Number of leaves

• numTasks (int) – Advanced parameter to specify the number of tasks. SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.

• numThreads (int) – Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.

• objective (str) – The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova.

• objectiveSeed (int) – Random seed for objectives, if random process is needed. Currently used only for rank_xendcg objective.

• otherRate (float) – The retain ratio of small gradient data. Only used in goss.

• parallelism (str) – Tree learner parallelism, can be set to data_parallel or voting_parallel

• passThroughArgs (str) – Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true

• posBaggingFraction (float) – Positive Bagging fraction

• predictDisableShapeCheck (bool) – control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data

• predictionCol (str) – prediction column name

• referenceDataset (list) – The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().

• repartitionByGroupingColumn (bool) – Repartition training data according to grouping column, on by default.

• samplingMode (str) – Data sampling for streaming mode. Sampled data is used to define bins. ‘global’: sample from all data, ‘subset’: sample from first N rows, or ‘fixed’: Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.

• samplingSubsetSize (int) – Specify subset size N for the sampling mode ‘subset’. ‘binSampleCount’ rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.

• seed (int) – Main seed, used to generate other seeds

• skipDrop (float) – Probability of skipping the dropout procedure during a boosting iteration

• slotNames (list) – List of slot names in the features column

• timeout (float) – Timeout in seconds

• topK (int) – The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0

• topRate (float) – The retain ratio of large gradient data. Only used in goss.

• uniformDrop (bool) – Set this to true to use uniform drop in dart mode

• useBarrierExecutionMode (bool) – Barrier execution mode which uses a barrier stage, off by default.

• useMissing (bool) – Set this to false to disable the special handle of missing value

• useSingleDatasetMode (bool) – Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.

• validationIndicatorCol (str) – Indicates whether the row is for training or validation

• verbosity (int) – Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug

• weightCol (str) – The name of the weight column

• xGBoostDartMode (bool) – Set this to true to use xgboost dart mode

• zeroAsMissing (bool) – Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values

baggingFraction = Param(parent='undefined', name='baggingFraction', doc='Bagging fraction')

baggingFreq = Param(parent='undefined', name='baggingFreq', doc='Bagging frequency')

baggingSeed = Param(parent='undefined', name='baggingSeed', doc='Bagging seed')

binSampleCount = Param(parent='undefined', name='binSampleCount', doc='Number of samples considered at computing histogram bins')

boostFromAverage = Param(parent='undefined', name='boostFromAverage', doc='Adjusts initial score to the mean of labels for faster convergence')

boostingType = Param(parent='undefined', name='boostingType', doc='Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling). ')

catSmooth = Param(parent='undefined', name='catSmooth', doc='this can reduce the effect of noises in categorical features, especially for categories with few data')

categoricalSlotIndexes = Param(parent='undefined', name='categoricalSlotIndexes', doc='List of categorical column indexes, the slot index in the features column')

categoricalSlotNames = Param(parent='undefined', name='categoricalSlotNames', doc='List of categorical column slot names, the slot name in the features column')

catl2 = Param(parent='undefined', name='catl2', doc='L2 regularization in categorical split')

chunkSize = Param(parent='undefined', name='chunkSize', doc='Advanced parameter to specify the chunk size for copying Java data to native.  If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.')

dataRandomSeed = Param(parent='undefined', name='dataRandomSeed', doc='Random seed for sampling data to construct histogram bins.')

dataTransferMode = Param(parent='undefined', name='dataTransferMode', doc='Specify how SynapseML transfers data from Spark to LightGBM.  Values can be streaming, bulk. Default is bulk, which is the legacy mode.')

defaultListenPort = Param(parent='undefined', name='defaultListenPort', doc='The default listen port on executors, used for testing')

deterministic = Param(parent='undefined', name='deterministic', doc='Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters.  Note: setting this to true may slow down training.  To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true')

driverListenPort = Param(parent='undefined', name='driverListenPort', doc='The listen port on a driver. Default value is 0 (random)')

dropRate = Param(parent='undefined', name='dropRate', doc='Dropout rate: a fraction of previous trees to drop during the dropout')

dropSeed = Param(parent='undefined', name='dropSeed', doc='Random seed to choose dropping models. Only used in dart.')

earlyStoppingRound = Param(parent='undefined', name='earlyStoppingRound', doc='Early stopping round')

evalAt = Param(parent='undefined', name='evalAt', doc='NDCG and MAP evaluation positions, separated by comma')

executionMode = Param(parent='undefined', name='executionMode', doc='Deprecated. Please use dataTransferMode.')

extraSeed = Param(parent='undefined', name='extraSeed', doc='Random seed for selecting threshold when extra_trees is true')

featureFraction = Param(parent='undefined', name='featureFraction', doc='Feature fraction')

featureFractionByNode = Param(parent='undefined', name='featureFractionByNode', doc='Feature fraction by node')

featureFractionSeed = Param(parent='undefined', name='featureFractionSeed', doc='Feature fraction seed')

featuresCol = Param(parent='undefined', name='featuresCol', doc='features column name')

featuresShapCol = Param(parent='undefined', name='featuresShapCol', doc='Output SHAP vector column name after prediction containing the feature contribution values')

fobj = Param(parent='undefined', name='fobj', doc='Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).')

getBaggingFraction()

Returns:

Bagging fraction

Return type:

baggingFraction

getBaggingFreq()

Returns:

Bagging frequency

Return type:

baggingFreq

getBaggingSeed()

Returns:

Bagging seed

Return type:

baggingSeed

getBinSampleCount()

Returns:

Number of samples considered at computing histogram bins

Return type:

binSampleCount

getBoostFromAverage()

Returns:

Adjusts initial score to the mean of labels for faster convergence

Return type:

boostFromAverage

getBoostingType()

Returns:

Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling).

Return type:

boostingType

getCatSmooth()

Returns:

this can reduce the effect of noises in categorical features, especially for categories with few data

Return type:

catSmooth

getCategoricalSlotIndexes()

Returns:

List of categorical column indexes, the slot index in the features column

Return type:

categoricalSlotIndexes

getCategoricalSlotNames()

Returns:

List of categorical column slot names, the slot name in the features column

Return type:

categoricalSlotNames

getCatl2()

Returns:

L2 regularization in categorical split

Return type:

catl2

getChunkSize()

Returns:

Advanced parameter to specify the chunk size for copying Java data to native. If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.

Return type:

chunkSize

getDataRandomSeed()

Returns:

Random seed for sampling data to construct histogram bins.

Return type:

dataRandomSeed

getDataTransferMode()

Returns:

Specify how SynapseML transfers data from Spark to LightGBM. Values can be streaming, bulk. Default is bulk, which is the legacy mode.

Return type:

dataTransferMode

getDefaultListenPort()

Returns:

The default listen port on executors, used for testing

Return type:

defaultListenPort

getDeterministic()

Returns:

Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters. Note: setting this to true may slow down training. To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true

Return type:

deterministic

getDriverListenPort()

Returns:

The listen port on a driver. Default value is 0 (random)

Return type:

driverListenPort

getDropRate()

Returns:

Dropout rate: a fraction of previous trees to drop during the dropout

Return type:

dropRate

getDropSeed()

Returns:

Random seed to choose dropping models. Only used in dart.

Return type:

dropSeed

getEarlyStoppingRound()

Returns:

Early stopping round

Return type:

earlyStoppingRound

getEvalAt()

Returns:

NDCG and MAP evaluation positions, separated by comma

Return type:

evalAt

getExecutionMode()

Returns:

Deprecated. Please use dataTransferMode.

Return type:

executionMode

getExtraSeed()

Returns:

Random seed for selecting threshold when extra_trees is true

Return type:

extraSeed

getFeatureFraction()

Returns:

Feature fraction

Return type:

featureFraction

getFeatureFractionByNode()

Returns:

Feature fraction by node

Return type:

featureFractionByNode

getFeatureFractionSeed()

Returns:

Feature fraction seed

Return type:

featureFractionSeed

getFeaturesCol()

Returns:

features column name

Return type:

featuresCol

getFeaturesShapCol()

Returns:

Output SHAP vector column name after prediction containing the feature contribution values

Return type:

featuresShapCol

getFobj()

Returns:

Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).

Return type:

fobj

getGroupCol()

Returns:

The name of the group column

Return type:

groupCol

getImprovementTolerance()

Returns:

Tolerance to consider improvement in metric

Return type:

improvementTolerance

getInitScoreCol()

Returns:

The name of the initial score column, used for continued training

Return type:

initScoreCol

getIsEnableSparse()

Returns:

Used to enable/disable sparse optimization

Return type:

isEnableSparse

getIsProvideTrainingMetric()

Returns:

Whether output metric result over training dataset.

Return type:

isProvideTrainingMetric

static getJavaPackage()

Returns package name String.

getLabelCol()

Returns:

label column name

Return type:

labelCol

getLabelGain()

Returns:

graded relevance for each label in NDCG

Return type:

labelGain

getLambdaL1()

Returns:

L1 regularization

Return type:

lambdaL1

getLambdaL2()

Returns:

L2 regularization

Return type:

lambdaL2

getLeafPredictionCol()

Returns:

Predicted leaf indices’s column name

Return type:

leafPredictionCol

getLearningRate()

Returns:

Learning rate or shrinkage rate

Return type:

learningRate

getMatrixType()

Returns:

Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense. Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.

Return type:

matrixType

getMaxBin()

Returns:

Max bin

Return type:

maxBin

getMaxBinByFeature()

Returns:

Max number of bins for each feature

Return type:

maxBinByFeature

getMaxCatThreshold()

Returns:

limit number of split points considered for categorical features

Return type:

maxCatThreshold

getMaxCatToOnehot()

Returns:

when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used

Return type:

maxCatToOnehot

getMaxDeltaStep()

Returns:

Used to limit the max output of tree leaves

Return type:

maxDeltaStep

getMaxDepth()

Returns:

Max depth

Return type:

maxDepth

getMaxDrop()

Returns:

Max number of dropped trees during one boosting iteration

Return type:

maxDrop

getMaxNumClasses()

Returns:

Number of max classes to infer numClass in multi-class classification.

Return type:

maxNumClasses

getMaxPosition()

Returns:

optimized NDCG at this position

Return type:

maxPosition

getMaxStreamingOMPThreads()

Returns:

Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it’s best to set a fixed value.

Return type:

maxStreamingOMPThreads

getMetric()

Returns:

Metrics to be evaluated on the evaluation data. Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv.

Return type:

metric

getMicroBatchSize()

Returns:

Specify how many elements are sent in a streaming micro-batch.

Return type:

microBatchSize

getMinDataInLeaf()

Returns:

Minimal number of data in one leaf. Can be used to deal with over-fitting.

Return type:

minDataInLeaf

getMinDataPerBin()

Returns:

Minimal number of data inside one bin

Return type:

minDataPerBin

getMinDataPerGroup()

Returns:

minimal number of data per categorical group

Return type:

minDataPerGroup

getMinGainToSplit()

Returns:

The minimal gain to perform split

Return type:

minGainToSplit

getMinSumHessianInLeaf()

Returns:

Minimal sum hessian in one leaf

Return type:

minSumHessianInLeaf

getModelString()

Returns:

LightGBM model to retrain

Return type:

modelString

getMonotoneConstraints()

Returns:

used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.

Return type:

monotoneConstraints

getMonotoneConstraintsMethod()

Returns:

Monotone constraints method. basic, intermediate, or advanced.

Return type:

monotoneConstraintsMethod

getMonotonePenalty()

Returns:

A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.

Return type:

monotonePenalty

getNegBaggingFraction()

Returns:

Negative Bagging fraction

Return type:

negBaggingFraction

getNumBatches()

Returns:

If greater than 0, splits data into separate batches during training

Return type:

numBatches

getNumIterations()

Returns:

Number of iterations, LightGBM constructs num_class * num_iterations trees

Return type:

numIterations

getNumLeaves()

Returns:

Number of leaves

Return type:

numLeaves

getNumTasks()

Returns:

Advanced parameter to specify the number of tasks. SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.

Return type:

numTasks

getNumThreads()

Returns:

Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.

Return type:

numThreads

getObjective()

Returns:

The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova.

Return type:

objective

getObjectiveSeed()

Returns:

Random seed for objectives, if random process is needed. Currently used only for rank_xendcg objective.

Return type:

objectiveSeed

getOtherRate()

Returns:

The retain ratio of small gradient data. Only used in goss.

Return type:

otherRate

getParallelism()

Returns:

Tree learner parallelism, can be set to data_parallel or voting_parallel

Return type:

parallelism

getPassThroughArgs()

Returns:

Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true

Return type:

passThroughArgs

getPosBaggingFraction()

Returns:

Positive Bagging fraction

Return type:

posBaggingFraction

getPredictDisableShapeCheck()

Returns:

control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data

Return type:

predictDisableShapeCheck

getPredictionCol()

Returns:

prediction column name

Return type:

predictionCol

getReferenceDataset()

Returns:

The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().

Return type:

referenceDataset

getRepartitionByGroupingColumn()

Returns:

Repartition training data according to grouping column, on by default.

Return type:

repartitionByGroupingColumn

getSamplingMode()

Returns:

Data sampling for streaming mode. Sampled data is used to define bins. ‘global’: sample from all data, ‘subset’: sample from first N rows, or ‘fixed’: Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.

Return type:

samplingMode

getSamplingSubsetSize()

Returns:

Specify subset size N for the sampling mode ‘subset’. ‘binSampleCount’ rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.

Return type:

samplingSubsetSize

getSeed()

Returns:

Main seed, used to generate other seeds

Return type:

seed

getSkipDrop()

Returns:

Probability of skipping the dropout procedure during a boosting iteration

Return type:

skipDrop

getSlotNames()

Returns:

List of slot names in the features column

Return type:

slotNames

getTimeout()

Returns:

Timeout in seconds

Return type:

timeout

getTopK()

Returns:

The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0

Return type:

topK

getTopRate()

Returns:

The retain ratio of large gradient data. Only used in goss.

Return type:

topRate

getUniformDrop()

Returns:

Set this to true to use uniform drop in dart mode

Return type:

uniformDrop

getUseBarrierExecutionMode()

Returns:

Barrier execution mode which uses a barrier stage, off by default.

Return type:

useBarrierExecutionMode

getUseMissing()

Returns:

Set this to false to disable the special handle of missing value

Return type:

useMissing

getUseSingleDatasetMode()

Returns:

Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.

Return type:

useSingleDatasetMode

getValidationIndicatorCol()

Returns:

Indicates whether the row is for training or validation

Return type:

validationIndicatorCol

getVerbosity()

Returns:

Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug

Return type:

verbosity

getWeightCol()

Returns:

The name of the weight column

Return type:

weightCol

getXGBoostDartMode()

Returns:

Set this to true to use xgboost dart mode

Return type:

xGBoostDartMode

getZeroAsMissing()

Returns:

Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values

Return type:

zeroAsMissing

groupCol = Param(parent='undefined', name='groupCol', doc='The name of the group column')

improvementTolerance = Param(parent='undefined', name='improvementTolerance', doc='Tolerance to consider improvement in metric')

initScoreCol = Param(parent='undefined', name='initScoreCol', doc='The name of the initial score column, used for continued training')

isEnableSparse = Param(parent='undefined', name='isEnableSparse', doc='Used to enable/disable sparse optimization')

isProvideTrainingMetric = Param(parent='undefined', name='isProvideTrainingMetric', doc='Whether output metric result over training dataset.')

labelCol = Param(parent='undefined', name='labelCol', doc='label column name')

labelGain = Param(parent='undefined', name='labelGain', doc='graded relevance for each label in NDCG')

lambdaL1 = Param(parent='undefined', name='lambdaL1', doc='L1 regularization')

lambdaL2 = Param(parent='undefined', name='lambdaL2', doc='L2 regularization')

leafPredictionCol = Param(parent='undefined', name='leafPredictionCol', doc="Predicted leaf indices's column name")

learningRate = Param(parent='undefined', name='learningRate', doc='Learning rate or shrinkage rate')

matrixType = Param(parent='undefined', name='matrixType', doc='Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense.  Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.')

maxBin = Param(parent='undefined', name='maxBin', doc='Max bin')

maxBinByFeature = Param(parent='undefined', name='maxBinByFeature', doc='Max number of bins for each feature')

maxCatThreshold = Param(parent='undefined', name='maxCatThreshold', doc='limit number of split points considered for categorical features')

maxCatToOnehot = Param(parent='undefined', name='maxCatToOnehot', doc='when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used')

maxDeltaStep = Param(parent='undefined', name='maxDeltaStep', doc='Used to limit the max output of tree leaves')

maxDepth = Param(parent='undefined', name='maxDepth', doc='Max depth')

maxDrop = Param(parent='undefined', name='maxDrop', doc='Max number of dropped trees during one boosting iteration')

maxNumClasses = Param(parent='undefined', name='maxNumClasses', doc='Number of max classes to infer numClass in multi-class classification.')

maxPosition = Param(parent='undefined', name='maxPosition', doc='optimized NDCG at this position')

maxStreamingOMPThreads = Param(parent='undefined', name='maxStreamingOMPThreads', doc="Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it's best to set a fixed value.")

metric = Param(parent='undefined', name='metric', doc='Metrics to be evaluated on the evaluation data.  Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv. ')

microBatchSize = Param(parent='undefined', name='microBatchSize', doc='Specify how many elements are sent in a streaming micro-batch.')

minDataInLeaf = Param(parent='undefined', name='minDataInLeaf', doc='Minimal number of data in one leaf. Can be used to deal with over-fitting.')

minDataPerBin = Param(parent='undefined', name='minDataPerBin', doc='Minimal number of data inside one bin')

minDataPerGroup = Param(parent='undefined', name='minDataPerGroup', doc='minimal number of data per categorical group')

minGainToSplit = Param(parent='undefined', name='minGainToSplit', doc='The minimal gain to perform split')

minSumHessianInLeaf = Param(parent='undefined', name='minSumHessianInLeaf', doc='Minimal sum hessian in one leaf')

modelString = Param(parent='undefined', name='modelString', doc='LightGBM model to retrain')

monotoneConstraints = Param(parent='undefined', name='monotoneConstraints', doc='used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.')

monotoneConstraintsMethod = Param(parent='undefined', name='monotoneConstraintsMethod', doc='Monotone constraints method. basic, intermediate, or advanced.')

monotonePenalty = Param(parent='undefined', name='monotonePenalty', doc='A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.')

negBaggingFraction = Param(parent='undefined', name='negBaggingFraction', doc='Negative Bagging fraction')

numBatches = Param(parent='undefined', name='numBatches', doc='If greater than 0, splits data into separate batches during training')

numIterations = Param(parent='undefined', name='numIterations', doc='Number of iterations, LightGBM constructs num_class * num_iterations trees')

numLeaves = Param(parent='undefined', name='numLeaves', doc='Number of leaves')

numTasks = Param(parent='undefined', name='numTasks', doc='Advanced parameter to specify the number of tasks.  SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.')

numThreads = Param(parent='undefined', name='numThreads', doc='Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.')

objective = Param(parent='undefined', name='objective', doc='The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova. ')

objectiveSeed = Param(parent='undefined', name='objectiveSeed', doc='Random seed for objectives, if random process is needed.  Currently used only for rank_xendcg objective.')

otherRate = Param(parent='undefined', name='otherRate', doc='The retain ratio of small gradient data. Only used in goss.')

parallelism = Param(parent='undefined', name='parallelism', doc='Tree learner parallelism, can be set to data_parallel or voting_parallel')

passThroughArgs = Param(parent='undefined', name='passThroughArgs', doc='Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true')

posBaggingFraction = Param(parent='undefined', name='posBaggingFraction', doc='Positive Bagging fraction')

predictDisableShapeCheck = Param(parent='undefined', name='predictDisableShapeCheck', doc='control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data')

predictionCol = Param(parent='undefined', name='predictionCol', doc='prediction column name')

classmethod read()

Returns an MLReader instance for this class.

referenceDataset = Param(parent='undefined', name='referenceDataset', doc='The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().')

repartitionByGroupingColumn = Param(parent='undefined', name='repartitionByGroupingColumn', doc='Repartition training data according to grouping column, on by default.')

samplingMode = Param(parent='undefined', name='samplingMode', doc="Data sampling for streaming mode. Sampled data is used to define bins. 'global': sample from all data, 'subset': sample from first N rows, or 'fixed': Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.")

samplingSubsetSize = Param(parent='undefined', name='samplingSubsetSize', doc="Specify subset size N for the sampling mode 'subset'. 'binSampleCount' rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.")

seed = Param(parent='undefined', name='seed', doc='Main seed, used to generate other seeds')

setBaggingFraction(value)

Parameters:

baggingFraction – Bagging fraction

setBaggingFreq(value)

Parameters:

baggingFreq – Bagging frequency

setBaggingSeed(value)

Parameters:

baggingSeed – Bagging seed

setBinSampleCount(value)

Parameters:

binSampleCount – Number of samples considered at computing histogram bins

setBoostFromAverage(value)

Parameters:

boostFromAverage – Adjusts initial score to the mean of labels for faster convergence

setBoostingType(value)

Parameters:

boostingType – Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling).

setCatSmooth(value)

Parameters:

catSmooth – this can reduce the effect of noises in categorical features, especially for categories with few data

setCategoricalSlotIndexes(value)

Parameters:

categoricalSlotIndexes – List of categorical column indexes, the slot index in the features column

setCategoricalSlotNames(value)

Parameters:

categoricalSlotNames – List of categorical column slot names, the slot name in the features column

setCatl2(value)

Parameters:

catl2 – L2 regularization in categorical split

setChunkSize(value)

Parameters:

chunkSize – Advanced parameter to specify the chunk size for copying Java data to native. If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.

setDataRandomSeed(value)

Parameters:

dataRandomSeed – Random seed for sampling data to construct histogram bins.

setDataTransferMode(value)

Parameters:

dataTransferMode – Specify how SynapseML transfers data from Spark to LightGBM. Values can be streaming, bulk. Default is bulk, which is the legacy mode.

setDefaultListenPort(value)

Parameters:

defaultListenPort – The default listen port on executors, used for testing

setDeterministic(value)

Parameters:

deterministic – Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters. Note: setting this to true may slow down training. To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true

setDriverListenPort(value)

Parameters:

driverListenPort – The listen port on a driver. Default value is 0 (random)

setDropRate(value)

Parameters:

dropRate – Dropout rate: a fraction of previous trees to drop during the dropout

setDropSeed(value)

Parameters:

dropSeed – Random seed to choose dropping models. Only used in dart.

setEarlyStoppingRound(value)

Parameters:

earlyStoppingRound – Early stopping round

setEvalAt(value)

Parameters:

evalAt – NDCG and MAP evaluation positions, separated by comma

setExecutionMode(value)

Parameters:

executionMode – Deprecated. Please use dataTransferMode.

setExtraSeed(value)

Parameters:

extraSeed – Random seed for selecting threshold when extra_trees is true

setFeatureFraction(value)

Parameters:

featureFraction – Feature fraction

setFeatureFractionByNode(value)

Parameters:

featureFractionByNode – Feature fraction by node

setFeatureFractionSeed(value)

Parameters:

featureFractionSeed – Feature fraction seed

setFeaturesCol(value)

Parameters:

featuresCol – features column name

setFeaturesShapCol(value)

Parameters:

featuresShapCol – Output SHAP vector column name after prediction containing the feature contribution values

setFobj(value)

Parameters:

fobj – Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).

setGroupCol(value)

Parameters:

groupCol – The name of the group column

setImprovementTolerance(value)

Parameters:

improvementTolerance – Tolerance to consider improvement in metric

setInitScoreCol(value)

Parameters:

initScoreCol – The name of the initial score column, used for continued training

setIsEnableSparse(value)

Parameters:

isEnableSparse – Used to enable/disable sparse optimization

setIsProvideTrainingMetric(value)

Parameters:

isProvideTrainingMetric – Whether output metric result over training dataset.

setLabelCol(value)

Parameters:

labelCol – label column name

setLabelGain(value)

Parameters:

labelGain – graded relevance for each label in NDCG

setLambdaL1(value)

Parameters:

lambdaL1 – L1 regularization

setLambdaL2(value)

Parameters:

lambdaL2 – L2 regularization

setLeafPredictionCol(value)

Parameters:

leafPredictionCol – Predicted leaf indices’s column name

setLearningRate(value)

Parameters:

learningRate – Learning rate or shrinkage rate

setMatrixType(value)

Parameters:

matrixType – Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense. Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.

setMaxBin(value)

Parameters:

maxBin – Max bin

setMaxBinByFeature(value)

Parameters:

maxBinByFeature – Max number of bins for each feature

setMaxCatThreshold(value)

Parameters:

maxCatThreshold – limit number of split points considered for categorical features

setMaxCatToOnehot(value)

Parameters:

maxCatToOnehot – when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used

setMaxDeltaStep(value)

Parameters:

maxDeltaStep – Used to limit the max output of tree leaves

setMaxDepth(value)

Parameters:

maxDepth – Max depth

setMaxDrop(value)

Parameters:

maxDrop – Max number of dropped trees during one boosting iteration

setMaxNumClasses(value)

Parameters:

maxNumClasses – Number of max classes to infer numClass in multi-class classification.

setMaxPosition(value)

Parameters:

maxPosition – optimized NDCG at this position

setMaxStreamingOMPThreads(value)

Parameters:

maxStreamingOMPThreads – Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it’s best to set a fixed value.

setMetric(value)

Parameters:

metric – Metrics to be evaluated on the evaluation data. Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv.

setMicroBatchSize(value)

Parameters:

microBatchSize – Specify how many elements are sent in a streaming micro-batch.

setMinDataInLeaf(value)

Parameters:

minDataInLeaf – Minimal number of data in one leaf. Can be used to deal with over-fitting.

setMinDataPerBin(value)

Parameters:

minDataPerBin – Minimal number of data inside one bin

setMinDataPerGroup(value)

Parameters:

minDataPerGroup – minimal number of data per categorical group

setMinGainToSplit(value)

Parameters:

minGainToSplit – The minimal gain to perform split

setMinSumHessianInLeaf(value)

Parameters:

minSumHessianInLeaf – Minimal sum hessian in one leaf

setModelString(value)

Parameters:

modelString – LightGBM model to retrain

setMonotoneConstraints(value)

Parameters:

monotoneConstraints – used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.

setMonotoneConstraintsMethod(value)

Parameters:

monotoneConstraintsMethod – Monotone constraints method. basic, intermediate, or advanced.

setMonotonePenalty(value)

Parameters:

monotonePenalty – A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.

setNegBaggingFraction(value)

Parameters:

negBaggingFraction – Negative Bagging fraction

setNumBatches(value)

Parameters:

numBatches – If greater than 0, splits data into separate batches during training

setNumIterations(value)

Parameters:

numIterations – Number of iterations, LightGBM constructs num_class * num_iterations trees

setNumLeaves(value)

Parameters:

numLeaves – Number of leaves

setNumTasks(value)

Parameters:

numTasks – Advanced parameter to specify the number of tasks. SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.

setNumThreads(value)

Parameters:

numThreads – Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.

setObjective(value)

Parameters:

objective – The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova.

setObjectiveSeed(value)

Parameters:

objectiveSeed – Random seed for objectives, if random process is needed. Currently used only for rank_xendcg objective.

setOtherRate(value)

Parameters:

otherRate – The retain ratio of small gradient data. Only used in goss.

setParallelism(value)

Parameters:

parallelism – Tree learner parallelism, can be set to data_parallel or voting_parallel

setParams(baggingFraction=1.0, baggingFreq=0, baggingSeed=3, binSampleCount=200000, boostFromAverage=True, boostingType='gbdt', catSmooth=10.0, categoricalSlotIndexes=[], categoricalSlotNames=[], catl2=10.0, chunkSize=10000, dataRandomSeed=1, dataTransferMode='streaming', defaultListenPort=12400, deterministic=False, driverListenPort=0, dropRate=0.1, dropSeed=4, earlyStoppingRound=0, evalAt=[1, 2, 3, 4, 5], executionMode=None, extraSeed=6, featureFraction=1.0, featureFractionByNode=None, featureFractionSeed=2, featuresCol='features', featuresShapCol='', fobj=None, groupCol=None, improvementTolerance=0.0, initScoreCol=None, isEnableSparse=True, isProvideTrainingMetric=False, labelCol='label', labelGain=[], lambdaL1=0.0, lambdaL2=0.0, leafPredictionCol='', learningRate=0.1, matrixType='auto', maxBin=255, maxBinByFeature=[], maxCatThreshold=32, maxCatToOnehot=4, maxDeltaStep=0.0, maxDepth=- 1, maxDrop=50, maxNumClasses=100, maxPosition=20, maxStreamingOMPThreads=16, metric='', microBatchSize=100, minDataInLeaf=20, minDataPerBin=3, minDataPerGroup=100, minGainToSplit=0.0, minSumHessianInLeaf=0.001, modelString='', monotoneConstraints=[], monotoneConstraintsMethod='basic', monotonePenalty=0.0, negBaggingFraction=1.0, numBatches=0, numIterations=100, numLeaves=31, numTasks=0, numThreads=0, objective='lambdarank', objectiveSeed=5, otherRate=0.1, parallelism='data_parallel', passThroughArgs='', posBaggingFraction=1.0, predictDisableShapeCheck=False, predictionCol='prediction', referenceDataset=None, repartitionByGroupingColumn=True, samplingMode='subset', samplingSubsetSize=1000000, seed=None, skipDrop=0.5, slotNames=[], timeout=1200.0, topK=20, topRate=0.2, uniformDrop=False, useBarrierExecutionMode=False, useMissing=True, useSingleDatasetMode=True, validationIndicatorCol=None, verbosity=- 1, weightCol=None, xGBoostDartMode=False, zeroAsMissing=False)

Set the (keyword only) parameters

setPassThroughArgs(value)

Parameters:

passThroughArgs – Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true

setPosBaggingFraction(value)

Parameters:

posBaggingFraction – Positive Bagging fraction

setPredictDisableShapeCheck(value)

Parameters:

predictDisableShapeCheck – control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data

setPredictionCol(value)

Parameters:

predictionCol – prediction column name

setReferenceDataset(value)

Parameters:

referenceDataset – The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().

setRepartitionByGroupingColumn(value)

Parameters:

repartitionByGroupingColumn – Repartition training data according to grouping column, on by default.

setSamplingMode(value)

Parameters:

samplingMode – Data sampling for streaming mode. Sampled data is used to define bins. ‘global’: sample from all data, ‘subset’: sample from first N rows, or ‘fixed’: Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.

setSamplingSubsetSize(value)

Parameters:

samplingSubsetSize – Specify subset size N for the sampling mode ‘subset’. ‘binSampleCount’ rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.

setSeed(value)

Parameters:

seed – Main seed, used to generate other seeds

setSkipDrop(value)

Parameters:

skipDrop – Probability of skipping the dropout procedure during a boosting iteration

setSlotNames(value)

Parameters:

slotNames – List of slot names in the features column

setTimeout(value)

Parameters:

timeout – Timeout in seconds

setTopK(value)

Parameters:

topK – The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0

setTopRate(value)

Parameters:

topRate – The retain ratio of large gradient data. Only used in goss.

setUniformDrop(value)

Parameters:

uniformDrop – Set this to true to use uniform drop in dart mode

setUseBarrierExecutionMode(value)

Parameters:

useBarrierExecutionMode – Barrier execution mode which uses a barrier stage, off by default.

setUseMissing(value)

Parameters:

useMissing – Set this to false to disable the special handle of missing value

setUseSingleDatasetMode(value)

Parameters:

useSingleDatasetMode – Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.

setValidationIndicatorCol(value)

Parameters:

validationIndicatorCol – Indicates whether the row is for training or validation

setVerbosity(value)

Parameters:

verbosity – Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug

setWeightCol(value)

Parameters:

weightCol – The name of the weight column

setXGBoostDartMode(value)

Parameters:

xGBoostDartMode – Set this to true to use xgboost dart mode

setZeroAsMissing(value)

Parameters:

zeroAsMissing – Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values

skipDrop = Param(parent='undefined', name='skipDrop', doc='Probability of skipping the dropout procedure during a boosting iteration')

slotNames = Param(parent='undefined', name='slotNames', doc='List of slot names in the features column')

timeout = Param(parent='undefined', name='timeout', doc='Timeout in seconds')

topK = Param(parent='undefined', name='topK', doc='The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0')

topRate = Param(parent='undefined', name='topRate', doc='The retain ratio of large gradient data. Only used in goss.')

uniformDrop = Param(parent='undefined', name='uniformDrop', doc='Set this to true to use uniform drop in dart mode')

useBarrierExecutionMode = Param(parent='undefined', name='useBarrierExecutionMode', doc='Barrier execution mode which uses a barrier stage, off by default.')

useMissing = Param(parent='undefined', name='useMissing', doc='Set this to false to disable the special handle of missing value')

useSingleDatasetMode = Param(parent='undefined', name='useSingleDatasetMode', doc='Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.')

validationIndicatorCol = Param(parent='undefined', name='validationIndicatorCol', doc='Indicates whether the row is for training or validation')

verbosity = Param(parent='undefined', name='verbosity', doc='Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug')

weightCol = Param(parent='undefined', name='weightCol', doc='The name of the weight column')

xGBoostDartMode = Param(parent='undefined', name='xGBoostDartMode', doc='Set this to true to use xgboost dart mode')

zeroAsMissing = Param(parent='undefined', name='zeroAsMissing', doc='Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values')

synapse.ml.lightgbm.LightGBMRankerModel module

class synapse.ml.lightgbm.LightGBMRankerModel.LightGBMRankerModel(java_obj=None, featuresCol='features', featuresShapCol='', labelCol='label', leafPredictionCol='', lightGBMBooster=None, numIterations=- 1, predictDisableShapeCheck=False, predictionCol='prediction', startIteration=0)

Bases: LightGBMModelMixin, _LightGBMRankerModel

getBoosterNumClasses()

Get the number of classes from the booster.

Returns:

The number of classes.

static loadNativeModelFromFile(filename)

Load the model from a native LightGBM text file.

static loadNativeModelFromString(model)

Load the model from a native LightGBM model string.

synapse.ml.lightgbm.LightGBMRegressionModel module

class synapse.ml.lightgbm.LightGBMRegressionModel.LightGBMRegressionModel(java_obj=None, featuresCol='features', featuresShapCol='', labelCol='label', leafPredictionCol='', lightGBMBooster=None, numIterations=- 1, predictDisableShapeCheck=False, predictionCol='prediction', startIteration=0)

Bases: LightGBMModelMixin, _LightGBMRegressionModel

static loadNativeModelFromFile(filename)

Load the model from a native LightGBM text file.

static loadNativeModelFromString(model)

Load the model from a native LightGBM model string.

synapse.ml.lightgbm.LightGBMRegressor module

class synapse.ml.lightgbm.LightGBMRegressor.LightGBMRegressor(java_obj=None, alpha=0.9, baggingFraction=1.0, baggingFreq=0, baggingSeed=3, binSampleCount=200000, boostFromAverage=True, boostingType='gbdt', catSmooth=10.0, categoricalSlotIndexes=[], categoricalSlotNames=[], catl2=10.0, chunkSize=10000, dataRandomSeed=1, dataTransferMode='streaming', defaultListenPort=12400, deterministic=False, driverListenPort=0, dropRate=0.1, dropSeed=4, earlyStoppingRound=0, executionMode=None, extraSeed=6, featureFraction=1.0, featureFractionByNode=None, featureFractionSeed=2, featuresCol='features', featuresShapCol='', fobj=None, improvementTolerance=0.0, initScoreCol=None, isEnableSparse=True, isProvideTrainingMetric=False, labelCol='label', lambdaL1=0.0, lambdaL2=0.0, leafPredictionCol='', learningRate=0.1, matrixType='auto', maxBin=255, maxBinByFeature=[], maxCatThreshold=32, maxCatToOnehot=4, maxDeltaStep=0.0, maxDepth=- 1, maxDrop=50, maxNumClasses=100, maxStreamingOMPThreads=16, metric='', microBatchSize=100, minDataInLeaf=20, minDataPerBin=3, minDataPerGroup=100, minGainToSplit=0.0, minSumHessianInLeaf=0.001, modelString='', monotoneConstraints=[], monotoneConstraintsMethod='basic', monotonePenalty=0.0, negBaggingFraction=1.0, numBatches=0, numIterations=100, numLeaves=31, numTasks=0, numThreads=0, objective='regression', objectiveSeed=5, otherRate=0.1, parallelism='data_parallel', passThroughArgs='', posBaggingFraction=1.0, predictDisableShapeCheck=False, predictionCol='prediction', referenceDataset=None, repartitionByGroupingColumn=True, samplingMode='subset', samplingSubsetSize=1000000, seed=None, skipDrop=0.5, slotNames=[], timeout=1200.0, topK=20, topRate=0.2, tweedieVariancePower=1.5, uniformDrop=False, useBarrierExecutionMode=False, useMissing=True, useSingleDatasetMode=True, validationIndicatorCol=None, verbosity=- 1, weightCol=None, xGBoostDartMode=False, zeroAsMissing=False)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaEstimator

Parameters:

• alpha (float) – parameter for Huber loss and Quantile regression

• baggingFraction (float) – Bagging fraction

• baggingFreq (int) – Bagging frequency

• baggingSeed (int) – Bagging seed

• binSampleCount (int) – Number of samples considered at computing histogram bins

• boostFromAverage (bool) – Adjusts initial score to the mean of labels for faster convergence

• boostingType (str) – Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling).

• catSmooth (float) – this can reduce the effect of noises in categorical features, especially for categories with few data

• categoricalSlotIndexes (list) – List of categorical column indexes, the slot index in the features column

• categoricalSlotNames (list) – List of categorical column slot names, the slot name in the features column

• catl2 (float) – L2 regularization in categorical split

• chunkSize (int) – Advanced parameter to specify the chunk size for copying Java data to native. If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.

• dataRandomSeed (int) – Random seed for sampling data to construct histogram bins.

• dataTransferMode (str) – Specify how SynapseML transfers data from Spark to LightGBM. Values can be streaming, bulk. Default is bulk, which is the legacy mode.

• defaultListenPort (int) – The default listen port on executors, used for testing

• deterministic (bool) – Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters. Note: setting this to true may slow down training. To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true

• driverListenPort (int) – The listen port on a driver. Default value is 0 (random)

• dropRate (float) – Dropout rate: a fraction of previous trees to drop during the dropout

• dropSeed (int) – Random seed to choose dropping models. Only used in dart.

• earlyStoppingRound (int) – Early stopping round

• executionMode (str) – Deprecated. Please use dataTransferMode.

• extraSeed (int) – Random seed for selecting threshold when extra_trees is true

• featureFraction (float) – Feature fraction

• featureFractionByNode (float) – Feature fraction by node

• featureFractionSeed (int) – Feature fraction seed

• featuresCol (str) – features column name

• featuresShapCol (str) – Output SHAP vector column name after prediction containing the feature contribution values

• fobj (object) – Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).

• improvementTolerance (float) – Tolerance to consider improvement in metric

• initScoreCol (str) – The name of the initial score column, used for continued training

• isEnableSparse (bool) – Used to enable/disable sparse optimization

• isProvideTrainingMetric (bool) – Whether output metric result over training dataset.

• labelCol (str) – label column name

• lambdaL1 (float) – L1 regularization

• lambdaL2 (float) – L2 regularization

• leafPredictionCol (str) – Predicted leaf indices’s column name

• learningRate (float) – Learning rate or shrinkage rate

• matrixType (str) – Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense. Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.

• maxBin (int) – Max bin

• maxBinByFeature (list) – Max number of bins for each feature

• maxCatThreshold (int) – limit number of split points considered for categorical features

• maxCatToOnehot (int) – when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used

• maxDeltaStep (float) – Used to limit the max output of tree leaves

• maxDepth (int) – Max depth

• maxDrop (int) – Max number of dropped trees during one boosting iteration

• maxNumClasses (int) – Number of max classes to infer numClass in multi-class classification.

• maxStreamingOMPThreads (int) – Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it’s best to set a fixed value.

• metric (str) – Metrics to be evaluated on the evaluation data. Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv.

• microBatchSize (int) – Specify how many elements are sent in a streaming micro-batch.

• minDataInLeaf (int) – Minimal number of data in one leaf. Can be used to deal with over-fitting.

• minDataPerBin (int) – Minimal number of data inside one bin

• minDataPerGroup (int) – minimal number of data per categorical group

• minGainToSplit (float) – The minimal gain to perform split

• minSumHessianInLeaf (float) – Minimal sum hessian in one leaf

• modelString (str) – LightGBM model to retrain

• monotoneConstraints (list) – used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.

• monotoneConstraintsMethod (str) – Monotone constraints method. basic, intermediate, or advanced.

• monotonePenalty (float) – A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.

• negBaggingFraction (float) – Negative Bagging fraction

• numBatches (int) – If greater than 0, splits data into separate batches during training

• numIterations (int) – Number of iterations, LightGBM constructs num_class * num_iterations trees

• numLeaves (int) – Number of leaves

• numTasks (int) – Advanced parameter to specify the number of tasks. SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.

• numThreads (int) – Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.

• objective (str) – The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova.

• objectiveSeed (int) – Random seed for objectives, if random process is needed. Currently used only for rank_xendcg objective.

• otherRate (float) – The retain ratio of small gradient data. Only used in goss.

• parallelism (str) – Tree learner parallelism, can be set to data_parallel or voting_parallel

• passThroughArgs (str) – Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true

• posBaggingFraction (float) – Positive Bagging fraction

• predictDisableShapeCheck (bool) – control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data

• predictionCol (str) – prediction column name

• referenceDataset (list) – The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().

• repartitionByGroupingColumn (bool) – Repartition training data according to grouping column, on by default.

• samplingMode (str) – Data sampling for streaming mode. Sampled data is used to define bins. ‘global’: sample from all data, ‘subset’: sample from first N rows, or ‘fixed’: Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.

• samplingSubsetSize (int) – Specify subset size N for the sampling mode ‘subset’. ‘binSampleCount’ rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.

• seed (int) – Main seed, used to generate other seeds

• skipDrop (float) – Probability of skipping the dropout procedure during a boosting iteration

• slotNames (list) – List of slot names in the features column

• timeout (float) – Timeout in seconds

• topK (int) – The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0

• topRate (float) – The retain ratio of large gradient data. Only used in goss.

• tweedieVariancePower (float) – control the variance of tweedie distribution, must be between 1 and 2

• uniformDrop (bool) – Set this to true to use uniform drop in dart mode

• useBarrierExecutionMode (bool) – Barrier execution mode which uses a barrier stage, off by default.

• useMissing (bool) – Set this to false to disable the special handle of missing value

• useSingleDatasetMode (bool) – Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.

• validationIndicatorCol (str) – Indicates whether the row is for training or validation

• verbosity (int) – Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug

• weightCol (str) – The name of the weight column

• xGBoostDartMode (bool) – Set this to true to use xgboost dart mode

• zeroAsMissing (bool) – Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values

alpha = Param(parent='undefined', name='alpha', doc='parameter for Huber loss and Quantile regression')

baggingFraction = Param(parent='undefined', name='baggingFraction', doc='Bagging fraction')

baggingFreq = Param(parent='undefined', name='baggingFreq', doc='Bagging frequency')

baggingSeed = Param(parent='undefined', name='baggingSeed', doc='Bagging seed')

binSampleCount = Param(parent='undefined', name='binSampleCount', doc='Number of samples considered at computing histogram bins')

boostFromAverage = Param(parent='undefined', name='boostFromAverage', doc='Adjusts initial score to the mean of labels for faster convergence')

boostingType = Param(parent='undefined', name='boostingType', doc='Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling). ')

catSmooth = Param(parent='undefined', name='catSmooth', doc='this can reduce the effect of noises in categorical features, especially for categories with few data')

categoricalSlotIndexes = Param(parent='undefined', name='categoricalSlotIndexes', doc='List of categorical column indexes, the slot index in the features column')

categoricalSlotNames = Param(parent='undefined', name='categoricalSlotNames', doc='List of categorical column slot names, the slot name in the features column')

catl2 = Param(parent='undefined', name='catl2', doc='L2 regularization in categorical split')

chunkSize = Param(parent='undefined', name='chunkSize', doc='Advanced parameter to specify the chunk size for copying Java data to native.  If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.')

dataRandomSeed = Param(parent='undefined', name='dataRandomSeed', doc='Random seed for sampling data to construct histogram bins.')

dataTransferMode = Param(parent='undefined', name='dataTransferMode', doc='Specify how SynapseML transfers data from Spark to LightGBM.  Values can be streaming, bulk. Default is bulk, which is the legacy mode.')

defaultListenPort = Param(parent='undefined', name='defaultListenPort', doc='The default listen port on executors, used for testing')

deterministic = Param(parent='undefined', name='deterministic', doc='Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters.  Note: setting this to true may slow down training.  To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true')

driverListenPort = Param(parent='undefined', name='driverListenPort', doc='The listen port on a driver. Default value is 0 (random)')

dropRate = Param(parent='undefined', name='dropRate', doc='Dropout rate: a fraction of previous trees to drop during the dropout')

dropSeed = Param(parent='undefined', name='dropSeed', doc='Random seed to choose dropping models. Only used in dart.')

earlyStoppingRound = Param(parent='undefined', name='earlyStoppingRound', doc='Early stopping round')

executionMode = Param(parent='undefined', name='executionMode', doc='Deprecated. Please use dataTransferMode.')

extraSeed = Param(parent='undefined', name='extraSeed', doc='Random seed for selecting threshold when extra_trees is true')

featureFraction = Param(parent='undefined', name='featureFraction', doc='Feature fraction')

featureFractionByNode = Param(parent='undefined', name='featureFractionByNode', doc='Feature fraction by node')

featureFractionSeed = Param(parent='undefined', name='featureFractionSeed', doc='Feature fraction seed')

featuresCol = Param(parent='undefined', name='featuresCol', doc='features column name')

featuresShapCol = Param(parent='undefined', name='featuresShapCol', doc='Output SHAP vector column name after prediction containing the feature contribution values')

fobj = Param(parent='undefined', name='fobj', doc='Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).')

getAlpha()

Returns:

parameter for Huber loss and Quantile regression

Return type:

alpha

getBaggingFraction()

Returns:

Bagging fraction

Return type:

baggingFraction

getBaggingFreq()

Returns:

Bagging frequency

Return type:

baggingFreq

getBaggingSeed()

Returns:

Bagging seed

Return type:

baggingSeed

getBinSampleCount()

Returns:

Number of samples considered at computing histogram bins

Return type:

binSampleCount

getBoostFromAverage()

Returns:

Adjusts initial score to the mean of labels for faster convergence

Return type:

boostFromAverage

getBoostingType()

Returns:

Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling).

Return type:

boostingType

getCatSmooth()

Returns:

this can reduce the effect of noises in categorical features, especially for categories with few data

Return type:

catSmooth

getCategoricalSlotIndexes()

Returns:

List of categorical column indexes, the slot index in the features column

Return type:

categoricalSlotIndexes

getCategoricalSlotNames()

Returns:

List of categorical column slot names, the slot name in the features column

Return type:

categoricalSlotNames

getCatl2()

Returns:

L2 regularization in categorical split

Return type:

catl2

getChunkSize()

Returns:

Advanced parameter to specify the chunk size for copying Java data to native. If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.

Return type:

chunkSize

getDataRandomSeed()

Returns:

Random seed for sampling data to construct histogram bins.

Return type:

dataRandomSeed

getDataTransferMode()

Returns:

Specify how SynapseML transfers data from Spark to LightGBM. Values can be streaming, bulk. Default is bulk, which is the legacy mode.

Return type:

dataTransferMode

getDefaultListenPort()

Returns:

The default listen port on executors, used for testing

Return type:

defaultListenPort

getDeterministic()

Returns:

Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters. Note: setting this to true may slow down training. To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true

Return type:

deterministic

getDriverListenPort()

Returns:

The listen port on a driver. Default value is 0 (random)

Return type:

driverListenPort

getDropRate()

Returns:

Dropout rate: a fraction of previous trees to drop during the dropout

Return type:

dropRate

getDropSeed()

Returns:

Random seed to choose dropping models. Only used in dart.

Return type:

dropSeed

getEarlyStoppingRound()

Returns:

Early stopping round

Return type:

earlyStoppingRound

getExecutionMode()

Returns:

Deprecated. Please use dataTransferMode.

Return type:

executionMode

getExtraSeed()

Returns:

Random seed for selecting threshold when extra_trees is true

Return type:

extraSeed

getFeatureFraction()

Returns:

Feature fraction

Return type:

featureFraction

getFeatureFractionByNode()

Returns:

Feature fraction by node

Return type:

featureFractionByNode

getFeatureFractionSeed()

Returns:

Feature fraction seed

Return type:

featureFractionSeed

getFeaturesCol()

Returns:

features column name

Return type:

featuresCol

getFeaturesShapCol()

Returns:

Output SHAP vector column name after prediction containing the feature contribution values

Return type:

featuresShapCol

getFobj()

Returns:

Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).

Return type:

fobj

getImprovementTolerance()

Returns:

Tolerance to consider improvement in metric

Return type:

improvementTolerance

getInitScoreCol()

Returns:

The name of the initial score column, used for continued training

Return type:

initScoreCol

getIsEnableSparse()

Returns:

Used to enable/disable sparse optimization

Return type:

isEnableSparse

getIsProvideTrainingMetric()

Returns:

Whether output metric result over training dataset.

Return type:

isProvideTrainingMetric

static getJavaPackage()

Returns package name String.

getLabelCol()

Returns:

label column name

Return type:

labelCol

getLambdaL1()

Returns:

L1 regularization

Return type:

lambdaL1

getLambdaL2()

Returns:

L2 regularization

Return type:

lambdaL2

getLeafPredictionCol()

Returns:

Predicted leaf indices’s column name

Return type:

leafPredictionCol

getLearningRate()

Returns:

Learning rate or shrinkage rate

Return type:

learningRate

getMatrixType()

Returns:

Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense. Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.

Return type:

matrixType

getMaxBin()

Returns:

Max bin

Return type:

maxBin

getMaxBinByFeature()

Returns:

Max number of bins for each feature

Return type:

maxBinByFeature

getMaxCatThreshold()

Returns:

limit number of split points considered for categorical features

Return type:

maxCatThreshold

getMaxCatToOnehot()

Returns:

when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used

Return type:

maxCatToOnehot

getMaxDeltaStep()

Returns:

Used to limit the max output of tree leaves

Return type:

maxDeltaStep

getMaxDepth()

Returns:

Max depth

Return type:

maxDepth

getMaxDrop()

Returns:

Max number of dropped trees during one boosting iteration

Return type:

maxDrop

getMaxNumClasses()

Returns:

Number of max classes to infer numClass in multi-class classification.

Return type:

maxNumClasses

getMaxStreamingOMPThreads()

Returns:

Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it’s best to set a fixed value.

Return type:

maxStreamingOMPThreads

getMetric()

Returns:

Metrics to be evaluated on the evaluation data. Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv.

Return type:

metric

getMicroBatchSize()

Returns:

Specify how many elements are sent in a streaming micro-batch.

Return type:

microBatchSize

getMinDataInLeaf()

Returns:

Minimal number of data in one leaf. Can be used to deal with over-fitting.

Return type:

minDataInLeaf

getMinDataPerBin()

Returns:

Minimal number of data inside one bin

Return type:

minDataPerBin

getMinDataPerGroup()

Returns:

minimal number of data per categorical group

Return type:

minDataPerGroup

getMinGainToSplit()

Returns:

The minimal gain to perform split

Return type:

minGainToSplit

getMinSumHessianInLeaf()

Returns:

Minimal sum hessian in one leaf

Return type:

minSumHessianInLeaf

getModelString()

Returns:

LightGBM model to retrain

Return type:

modelString

getMonotoneConstraints()

Returns:

used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.

Return type:

monotoneConstraints

getMonotoneConstraintsMethod()

Returns:

Monotone constraints method. basic, intermediate, or advanced.

Return type:

monotoneConstraintsMethod

getMonotonePenalty()

Returns:

A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.

Return type:

monotonePenalty

getNegBaggingFraction()

Returns:

Negative Bagging fraction

Return type:

negBaggingFraction

getNumBatches()

Returns:

If greater than 0, splits data into separate batches during training

Return type:

numBatches

getNumIterations()

Returns:

Number of iterations, LightGBM constructs num_class * num_iterations trees

Return type:

numIterations

getNumLeaves()

Returns:

Number of leaves

Return type:

numLeaves

getNumTasks()

Returns:

Advanced parameter to specify the number of tasks. SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.

Return type:

numTasks

getNumThreads()

Returns:

Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.

Return type:

numThreads

getObjective()

Returns:

The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova.

Return type:

objective

getObjectiveSeed()

Returns:

Random seed for objectives, if random process is needed. Currently used only for rank_xendcg objective.

Return type:

objectiveSeed

getOtherRate()

Returns:

The retain ratio of small gradient data. Only used in goss.

Return type:

otherRate

getParallelism()

Returns:

Tree learner parallelism, can be set to data_parallel or voting_parallel

Return type:

parallelism

getPassThroughArgs()

Returns:

Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true

Return type:

passThroughArgs

getPosBaggingFraction()

Returns:

Positive Bagging fraction

Return type:

posBaggingFraction

getPredictDisableShapeCheck()

Returns:

control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data

Return type:

predictDisableShapeCheck

getPredictionCol()

Returns:

prediction column name

Return type:

predictionCol

getReferenceDataset()

Returns:

The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().

Return type:

referenceDataset

getRepartitionByGroupingColumn()

Returns:

Repartition training data according to grouping column, on by default.

Return type:

repartitionByGroupingColumn

getSamplingMode()

Returns:

Data sampling for streaming mode. Sampled data is used to define bins. ‘global’: sample from all data, ‘subset’: sample from first N rows, or ‘fixed’: Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.

Return type:

samplingMode

getSamplingSubsetSize()

Returns:

Specify subset size N for the sampling mode ‘subset’. ‘binSampleCount’ rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.

Return type:

samplingSubsetSize

getSeed()

Returns:

Main seed, used to generate other seeds

Return type:

seed

getSkipDrop()

Returns:

Probability of skipping the dropout procedure during a boosting iteration

Return type:

skipDrop

getSlotNames()

Returns:

List of slot names in the features column

Return type:

slotNames

getTimeout()

Returns:

Timeout in seconds

Return type:

timeout

getTopK()

Returns:

The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0

Return type:

topK

getTopRate()

Returns:

The retain ratio of large gradient data. Only used in goss.

Return type:

topRate

getTweedieVariancePower()

Returns:

control the variance of tweedie distribution, must be between 1 and 2

Return type:

tweedieVariancePower

getUniformDrop()

Returns:

Set this to true to use uniform drop in dart mode

Return type:

uniformDrop

getUseBarrierExecutionMode()

Returns:

Barrier execution mode which uses a barrier stage, off by default.

Return type:

useBarrierExecutionMode

getUseMissing()

Returns:

Set this to false to disable the special handle of missing value

Return type:

useMissing

getUseSingleDatasetMode()

Returns:

Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.

Return type:

useSingleDatasetMode

getValidationIndicatorCol()

Returns:

Indicates whether the row is for training or validation

Return type:

validationIndicatorCol

getVerbosity()

Returns:

Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug

Return type:

verbosity

getWeightCol()

Returns:

The name of the weight column

Return type:

weightCol

getXGBoostDartMode()

Returns:

Set this to true to use xgboost dart mode

Return type:

xGBoostDartMode

getZeroAsMissing()

Returns:

Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values

Return type:

zeroAsMissing

improvementTolerance = Param(parent='undefined', name='improvementTolerance', doc='Tolerance to consider improvement in metric')

initScoreCol = Param(parent='undefined', name='initScoreCol', doc='The name of the initial score column, used for continued training')

isEnableSparse = Param(parent='undefined', name='isEnableSparse', doc='Used to enable/disable sparse optimization')

isProvideTrainingMetric = Param(parent='undefined', name='isProvideTrainingMetric', doc='Whether output metric result over training dataset.')

labelCol = Param(parent='undefined', name='labelCol', doc='label column name')

lambdaL1 = Param(parent='undefined', name='lambdaL1', doc='L1 regularization')

lambdaL2 = Param(parent='undefined', name='lambdaL2', doc='L2 regularization')

leafPredictionCol = Param(parent='undefined', name='leafPredictionCol', doc="Predicted leaf indices's column name")

learningRate = Param(parent='undefined', name='learningRate', doc='Learning rate or shrinkage rate')

matrixType = Param(parent='undefined', name='matrixType', doc='Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense.  Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.')

maxBin = Param(parent='undefined', name='maxBin', doc='Max bin')

maxBinByFeature = Param(parent='undefined', name='maxBinByFeature', doc='Max number of bins for each feature')

maxCatThreshold = Param(parent='undefined', name='maxCatThreshold', doc='limit number of split points considered for categorical features')

maxCatToOnehot = Param(parent='undefined', name='maxCatToOnehot', doc='when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used')

maxDeltaStep = Param(parent='undefined', name='maxDeltaStep', doc='Used to limit the max output of tree leaves')

maxDepth = Param(parent='undefined', name='maxDepth', doc='Max depth')

maxDrop = Param(parent='undefined', name='maxDrop', doc='Max number of dropped trees during one boosting iteration')

maxNumClasses = Param(parent='undefined', name='maxNumClasses', doc='Number of max classes to infer numClass in multi-class classification.')

maxStreamingOMPThreads = Param(parent='undefined', name='maxStreamingOMPThreads', doc="Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it's best to set a fixed value.")

metric = Param(parent='undefined', name='metric', doc='Metrics to be evaluated on the evaluation data.  Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv. ')

microBatchSize = Param(parent='undefined', name='microBatchSize', doc='Specify how many elements are sent in a streaming micro-batch.')

minDataInLeaf = Param(parent='undefined', name='minDataInLeaf', doc='Minimal number of data in one leaf. Can be used to deal with over-fitting.')

minDataPerBin = Param(parent='undefined', name='minDataPerBin', doc='Minimal number of data inside one bin')

minDataPerGroup = Param(parent='undefined', name='minDataPerGroup', doc='minimal number of data per categorical group')

minGainToSplit = Param(parent='undefined', name='minGainToSplit', doc='The minimal gain to perform split')

minSumHessianInLeaf = Param(parent='undefined', name='minSumHessianInLeaf', doc='Minimal sum hessian in one leaf')

modelString = Param(parent='undefined', name='modelString', doc='LightGBM model to retrain')

monotoneConstraints = Param(parent='undefined', name='monotoneConstraints', doc='used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.')

monotoneConstraintsMethod = Param(parent='undefined', name='monotoneConstraintsMethod', doc='Monotone constraints method. basic, intermediate, or advanced.')

monotonePenalty = Param(parent='undefined', name='monotonePenalty', doc='A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.')

negBaggingFraction = Param(parent='undefined', name='negBaggingFraction', doc='Negative Bagging fraction')

numBatches = Param(parent='undefined', name='numBatches', doc='If greater than 0, splits data into separate batches during training')

numIterations = Param(parent='undefined', name='numIterations', doc='Number of iterations, LightGBM constructs num_class * num_iterations trees')

numLeaves = Param(parent='undefined', name='numLeaves', doc='Number of leaves')

numTasks = Param(parent='undefined', name='numTasks', doc='Advanced parameter to specify the number of tasks.  SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.')

numThreads = Param(parent='undefined', name='numThreads', doc='Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.')

objective = Param(parent='undefined', name='objective', doc='The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova. ')

objectiveSeed = Param(parent='undefined', name='objectiveSeed', doc='Random seed for objectives, if random process is needed.  Currently used only for rank_xendcg objective.')

otherRate = Param(parent='undefined', name='otherRate', doc='The retain ratio of small gradient data. Only used in goss.')

parallelism = Param(parent='undefined', name='parallelism', doc='Tree learner parallelism, can be set to data_parallel or voting_parallel')

passThroughArgs = Param(parent='undefined', name='passThroughArgs', doc='Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true')

posBaggingFraction = Param(parent='undefined', name='posBaggingFraction', doc='Positive Bagging fraction')

predictDisableShapeCheck = Param(parent='undefined', name='predictDisableShapeCheck', doc='control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data')

predictionCol = Param(parent='undefined', name='predictionCol', doc='prediction column name')

classmethod read()

Returns an MLReader instance for this class.

referenceDataset = Param(parent='undefined', name='referenceDataset', doc='The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().')

repartitionByGroupingColumn = Param(parent='undefined', name='repartitionByGroupingColumn', doc='Repartition training data according to grouping column, on by default.')

samplingMode = Param(parent='undefined', name='samplingMode', doc="Data sampling for streaming mode. Sampled data is used to define bins. 'global': sample from all data, 'subset': sample from first N rows, or 'fixed': Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.")

samplingSubsetSize = Param(parent='undefined', name='samplingSubsetSize', doc="Specify subset size N for the sampling mode 'subset'. 'binSampleCount' rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.")

seed = Param(parent='undefined', name='seed', doc='Main seed, used to generate other seeds')

setAlpha(value)

Parameters:

alpha – parameter for Huber loss and Quantile regression

setBaggingFraction(value)

Parameters:

baggingFraction – Bagging fraction

setBaggingFreq(value)

Parameters:

baggingFreq – Bagging frequency

setBaggingSeed(value)

Parameters:

baggingSeed – Bagging seed

setBinSampleCount(value)

Parameters:

binSampleCount – Number of samples considered at computing histogram bins

setBoostFromAverage(value)

Parameters:

boostFromAverage – Adjusts initial score to the mean of labels for faster convergence

setBoostingType(value)

Parameters:

boostingType – Default gbdt = traditional Gradient Boosting Decision Tree. Options are: gbdt, gbrt, rf (Random Forest), random_forest, dart (Dropouts meet Multiple Additive Regression Trees), goss (Gradient-based One-Side Sampling).

setCatSmooth(value)

Parameters:

catSmooth – this can reduce the effect of noises in categorical features, especially for categories with few data

setCategoricalSlotIndexes(value)

Parameters:

categoricalSlotIndexes – List of categorical column indexes, the slot index in the features column

setCategoricalSlotNames(value)

Parameters:

categoricalSlotNames – List of categorical column slot names, the slot name in the features column

setCatl2(value)

Parameters:

catl2 – L2 regularization in categorical split

setChunkSize(value)

Parameters:

chunkSize – Advanced parameter to specify the chunk size for copying Java data to native. If set too high, memory may be wasted, but if set too low, performance may be reduced during data copy.If dataset size is known beforehand, set to the number of rows in the dataset.

setDataRandomSeed(value)

Parameters:

dataRandomSeed – Random seed for sampling data to construct histogram bins.

setDataTransferMode(value)

Parameters:

dataTransferMode – Specify how SynapseML transfers data from Spark to LightGBM. Values can be streaming, bulk. Default is bulk, which is the legacy mode.

setDefaultListenPort(value)

Parameters:

defaultListenPort – The default listen port on executors, used for testing

setDeterministic(value)

Parameters:

deterministic – Used only with cpu devide type. Setting this to true should ensure stable results when using the same data and the same parameters. Note: setting this to true may slow down training. To avoid potential instability due to numerical issues, please set force_col_wise=true or force_row_wise=true when setting deterministic=true

setDriverListenPort(value)

Parameters:

driverListenPort – The listen port on a driver. Default value is 0 (random)

setDropRate(value)

Parameters:

dropRate – Dropout rate: a fraction of previous trees to drop during the dropout

setDropSeed(value)

Parameters:

dropSeed – Random seed to choose dropping models. Only used in dart.

setEarlyStoppingRound(value)

Parameters:

earlyStoppingRound – Early stopping round

setExecutionMode(value)

Parameters:

executionMode – Deprecated. Please use dataTransferMode.

setExtraSeed(value)

Parameters:

extraSeed – Random seed for selecting threshold when extra_trees is true

setFeatureFraction(value)

Parameters:

featureFraction – Feature fraction

setFeatureFractionByNode(value)

Parameters:

featureFractionByNode – Feature fraction by node

setFeatureFractionSeed(value)

Parameters:

featureFractionSeed – Feature fraction seed

setFeaturesCol(value)

Parameters:

featuresCol – features column name

setFeaturesShapCol(value)

Parameters:

featuresShapCol – Output SHAP vector column name after prediction containing the feature contribution values

setFobj(value)

Parameters:

fobj – Customized objective function. Should accept two parameters: preds, train_data, and return (grad, hess).

setImprovementTolerance(value)

Parameters:

improvementTolerance – Tolerance to consider improvement in metric

setInitScoreCol(value)

Parameters:

initScoreCol – The name of the initial score column, used for continued training

setIsEnableSparse(value)

Parameters:

isEnableSparse – Used to enable/disable sparse optimization

setIsProvideTrainingMetric(value)

Parameters:

isProvideTrainingMetric – Whether output metric result over training dataset.

setLabelCol(value)

Parameters:

labelCol – label column name

setLambdaL1(value)

Parameters:

lambdaL1 – L1 regularization

setLambdaL2(value)

Parameters:

lambdaL2 – L2 regularization

setLeafPredictionCol(value)

Parameters:

leafPredictionCol – Predicted leaf indices’s column name

setLearningRate(value)

Parameters:

learningRate – Learning rate or shrinkage rate

setMatrixType(value)

Parameters:

matrixType – Advanced parameter to specify whether the native lightgbm matrix constructed should be sparse or dense. Values can be auto, sparse or dense. Default value is auto, which samples first ten rows to determine type.

setMaxBin(value)

Parameters:

maxBin – Max bin

setMaxBinByFeature(value)

Parameters:

maxBinByFeature – Max number of bins for each feature

setMaxCatThreshold(value)

Parameters:

maxCatThreshold – limit number of split points considered for categorical features

setMaxCatToOnehot(value)

Parameters:

maxCatToOnehot – when number of categories of one feature smaller than or equal to this, one-vs-other split algorithm will be used

setMaxDeltaStep(value)

Parameters:

maxDeltaStep – Used to limit the max output of tree leaves

setMaxDepth(value)

Parameters:

maxDepth – Max depth

setMaxDrop(value)

Parameters:

maxDrop – Max number of dropped trees during one boosting iteration

setMaxNumClasses(value)

Parameters:

maxNumClasses – Number of max classes to infer numClass in multi-class classification.

setMaxStreamingOMPThreads(value)

Parameters:

maxStreamingOMPThreads – Maximum number of OpenMP threads used by a LightGBM thread. Used only for thread-safe buffer allocation. Use -1 to use OpenMP default, but in a Spark environment it’s best to set a fixed value.

setMetric(value)

Parameters:

metric – Metrics to be evaluated on the evaluation data. Options are: empty string or not specified means that metric corresponding to specified objective will be used (this is possible only for pre-defined objective functions, otherwise no evaluation metric will be added). None (string, not a None value) means that no metric will be registered, aliases: na, null, custom. l1, absolute loss, aliases: mean_absolute_error, mae, regression_l1. l2, square loss, aliases: mean_squared_error, mse, regression_l2, regression. rmse, root square loss, aliases: root_mean_squared_error, l2_root. quantile, Quantile regression. mape, MAPE loss, aliases: mean_absolute_percentage_error. huber, Huber loss. fair, Fair loss. poisson, negative log-likelihood for Poisson regression. gamma, negative log-likelihood for Gamma regression. gamma_deviance, residual deviance for Gamma regression. tweedie, negative log-likelihood for Tweedie regression. ndcg, NDCG, aliases: lambdarank. map, MAP, aliases: mean_average_precision. auc, AUC. binary_logloss, log loss, aliases: binary. binary_error, for one sample: 0 for correct classification, 1 for error classification. multi_logloss, log loss for multi-class classification, aliases: multiclass, softmax, multiclassova, multiclass_ova, ova, ovr. multi_error, error rate for multi-class classification. cross_entropy, cross-entropy (with optional linear weights), aliases: xentropy. cross_entropy_lambda, intensity-weighted cross-entropy, aliases: xentlambda. kullback_leibler, Kullback-Leibler divergence, aliases: kldiv.

setMicroBatchSize(value)

Parameters:

microBatchSize – Specify how many elements are sent in a streaming micro-batch.

setMinDataInLeaf(value)

Parameters:

minDataInLeaf – Minimal number of data in one leaf. Can be used to deal with over-fitting.

setMinDataPerBin(value)

Parameters:

minDataPerBin – Minimal number of data inside one bin

setMinDataPerGroup(value)

Parameters:

minDataPerGroup – minimal number of data per categorical group

setMinGainToSplit(value)

Parameters:

minGainToSplit – The minimal gain to perform split

setMinSumHessianInLeaf(value)

Parameters:

minSumHessianInLeaf – Minimal sum hessian in one leaf

setModelString(value)

Parameters:

modelString – LightGBM model to retrain

setMonotoneConstraints(value)

Parameters:

monotoneConstraints – used for constraints of monotonic features. 1 means increasing, -1 means decreasing, 0 means non-constraint. Specify all features in order.

setMonotoneConstraintsMethod(value)

Parameters:

monotoneConstraintsMethod – Monotone constraints method. basic, intermediate, or advanced.

setMonotonePenalty(value)

Parameters:

monotonePenalty – A penalization parameter X forbids any monotone splits on the first X (rounded down) level(s) of the tree.

setNegBaggingFraction(value)

Parameters:

negBaggingFraction – Negative Bagging fraction

setNumBatches(value)

Parameters:

numBatches – If greater than 0, splits data into separate batches during training

setNumIterations(value)

Parameters:

numIterations – Number of iterations, LightGBM constructs num_class * num_iterations trees

setNumLeaves(value)

Parameters:

numLeaves – Number of leaves

setNumTasks(value)

Parameters:

numTasks – Advanced parameter to specify the number of tasks. SynapseML tries to guess this based on cluster configuration, but this parameter can be used to override.

setNumThreads(value)

Parameters:

numThreads – Number of threads per executor for LightGBM. For the best speed, set this to the number of real CPU cores.

setObjective(value)

Parameters:

objective – The Objective. For regression applications, this can be: regression_l2, regression_l1, huber, fair, poisson, quantile, mape, gamma or tweedie. For classification applications, this can be: binary, multiclass, or multiclassova.

setObjectiveSeed(value)

Parameters:

objectiveSeed – Random seed for objectives, if random process is needed. Currently used only for rank_xendcg objective.

setOtherRate(value)

Parameters:

otherRate – The retain ratio of small gradient data. Only used in goss.

setParallelism(value)

Parameters:

parallelism – Tree learner parallelism, can be set to data_parallel or voting_parallel

setParams(alpha=0.9, baggingFraction=1.0, baggingFreq=0, baggingSeed=3, binSampleCount=200000, boostFromAverage=True, boostingType='gbdt', catSmooth=10.0, categoricalSlotIndexes=[], categoricalSlotNames=[], catl2=10.0, chunkSize=10000, dataRandomSeed=1, dataTransferMode='streaming', defaultListenPort=12400, deterministic=False, driverListenPort=0, dropRate=0.1, dropSeed=4, earlyStoppingRound=0, executionMode=None, extraSeed=6, featureFraction=1.0, featureFractionByNode=None, featureFractionSeed=2, featuresCol='features', featuresShapCol='', fobj=None, improvementTolerance=0.0, initScoreCol=None, isEnableSparse=True, isProvideTrainingMetric=False, labelCol='label', lambdaL1=0.0, lambdaL2=0.0, leafPredictionCol='', learningRate=0.1, matrixType='auto', maxBin=255, maxBinByFeature=[], maxCatThreshold=32, maxCatToOnehot=4, maxDeltaStep=0.0, maxDepth=- 1, maxDrop=50, maxNumClasses=100, maxStreamingOMPThreads=16, metric='', microBatchSize=100, minDataInLeaf=20, minDataPerBin=3, minDataPerGroup=100, minGainToSplit=0.0, minSumHessianInLeaf=0.001, modelString='', monotoneConstraints=[], monotoneConstraintsMethod='basic', monotonePenalty=0.0, negBaggingFraction=1.0, numBatches=0, numIterations=100, numLeaves=31, numTasks=0, numThreads=0, objective='regression', objectiveSeed=5, otherRate=0.1, parallelism='data_parallel', passThroughArgs='', posBaggingFraction=1.0, predictDisableShapeCheck=False, predictionCol='prediction', referenceDataset=None, repartitionByGroupingColumn=True, samplingMode='subset', samplingSubsetSize=1000000, seed=None, skipDrop=0.5, slotNames=[], timeout=1200.0, topK=20, topRate=0.2, tweedieVariancePower=1.5, uniformDrop=False, useBarrierExecutionMode=False, useMissing=True, useSingleDatasetMode=True, validationIndicatorCol=None, verbosity=- 1, weightCol=None, xGBoostDartMode=False, zeroAsMissing=False)

Set the (keyword only) parameters

setPassThroughArgs(value)

Parameters:

passThroughArgs – Direct string to pass through to LightGBM library (appended with other explicitly set params). Will override any parameters given with explicit setters. Can include multiple parameters in one string. e.g., force_row_wise=true

setPosBaggingFraction(value)

Parameters:

posBaggingFraction – Positive Bagging fraction

setPredictDisableShapeCheck(value)

Parameters:

predictDisableShapeCheck – control whether or not LightGBM raises an error when you try to predict on data with a different number of features than the training data

setPredictionCol(value)

Parameters:

predictionCol – prediction column name

setReferenceDataset(value)

Parameters:

referenceDataset – The reference Dataset that was used for the fit. If using samplingMode=custom, this must be set before fit().

setRepartitionByGroupingColumn(value)

Parameters:

repartitionByGroupingColumn – Repartition training data according to grouping column, on by default.

setSamplingMode(value)

Parameters:

samplingMode – Data sampling for streaming mode. Sampled data is used to define bins. ‘global’: sample from all data, ‘subset’: sample from first N rows, or ‘fixed’: Take first N rows as sample.Values can be global, subset, or fixed. Default is subset.

setSamplingSubsetSize(value)

Parameters:

samplingSubsetSize – Specify subset size N for the sampling mode ‘subset’. ‘binSampleCount’ rows will be chosen from the first N values of the dataset. Subset can be used when rows are expected to be random and data is huge.

setSeed(value)

Parameters:

seed – Main seed, used to generate other seeds

setSkipDrop(value)

Parameters:

skipDrop – Probability of skipping the dropout procedure during a boosting iteration

setSlotNames(value)

Parameters:

slotNames – List of slot names in the features column

setTimeout(value)

Parameters:

timeout – Timeout in seconds

setTopK(value)

Parameters:

topK – The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0

setTopRate(value)

Parameters:

topRate – The retain ratio of large gradient data. Only used in goss.

setTweedieVariancePower(value)

Parameters:

tweedieVariancePower – control the variance of tweedie distribution, must be between 1 and 2

setUniformDrop(value)

Parameters:

uniformDrop – Set this to true to use uniform drop in dart mode

setUseBarrierExecutionMode(value)

Parameters:

useBarrierExecutionMode – Barrier execution mode which uses a barrier stage, off by default.

setUseMissing(value)

Parameters:

useMissing – Set this to false to disable the special handle of missing value

setUseSingleDatasetMode(value)

Parameters:

useSingleDatasetMode – Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.

setValidationIndicatorCol(value)

Parameters:

validationIndicatorCol – Indicates whether the row is for training or validation

setVerbosity(value)

Parameters:

verbosity – Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug

setWeightCol(value)

Parameters:

weightCol – The name of the weight column

setXGBoostDartMode(value)

Parameters:

xGBoostDartMode – Set this to true to use xgboost dart mode

setZeroAsMissing(value)

Parameters:

zeroAsMissing – Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values

skipDrop = Param(parent='undefined', name='skipDrop', doc='Probability of skipping the dropout procedure during a boosting iteration')

slotNames = Param(parent='undefined', name='slotNames', doc='List of slot names in the features column')

timeout = Param(parent='undefined', name='timeout', doc='Timeout in seconds')

topK = Param(parent='undefined', name='topK', doc='The top_k value used in Voting parallel, set this to larger value for more accurate result, but it will slow down the training speed. It should be greater than 0')

topRate = Param(parent='undefined', name='topRate', doc='The retain ratio of large gradient data. Only used in goss.')

tweedieVariancePower = Param(parent='undefined', name='tweedieVariancePower', doc='control the variance of tweedie distribution, must be between 1 and 2')

uniformDrop = Param(parent='undefined', name='uniformDrop', doc='Set this to true to use uniform drop in dart mode')

useBarrierExecutionMode = Param(parent='undefined', name='useBarrierExecutionMode', doc='Barrier execution mode which uses a barrier stage, off by default.')

useMissing = Param(parent='undefined', name='useMissing', doc='Set this to false to disable the special handle of missing value')

useSingleDatasetMode = Param(parent='undefined', name='useSingleDatasetMode', doc='Use single dataset execution mode to create a single native dataset per executor (singleton) to reduce memory and communication overhead.')

validationIndicatorCol = Param(parent='undefined', name='validationIndicatorCol', doc='Indicates whether the row is for training or validation')

verbosity = Param(parent='undefined', name='verbosity', doc='Verbosity where lt 0 is Fatal, eq 0 is Error, eq 1 is Info, gt 1 is Debug')

weightCol = Param(parent='undefined', name='weightCol', doc='The name of the weight column')

xGBoostDartMode = Param(parent='undefined', name='xGBoostDartMode', doc='Set this to true to use xgboost dart mode')

zeroAsMissing = Param(parent='undefined', name='zeroAsMissing', doc='Set to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices). Set to false to use na for representing missing values')

synapse.ml.lightgbm.mixin module

class synapse.ml.lightgbm.mixin.LightGBMModelMixin

Bases: object

getBoosterBestIteration()

Get the best iteration from the booster.

Returns:

The best iteration, if early stopping was triggered.

getBoosterNumFeatures()

Get the number of features from the booster.

Returns:

The number of features.

getBoosterNumTotalIterations()

Get the total number of iterations trained.

Returns:

The total number of iterations trained.

getBoosterNumTotalModel()

Get the total number of models trained.

Returns:

The total number of models.

getFeatureImportances(importance_type='split')

Get the feature importances as a list. The importance_type can be “split” or “gain”.

getFeatureShaps(vector)

Get the local shap feature importances.

getNativeModel()

Get the native model serialized representation as a string.

saveNativeModel(filename, overwrite=True)

Save the booster as string format to a local or WASB remote location.

setPredictDisableShapeCheck(value=None)

Set shape check or not when predict.

Module contents

SynapseML is an ecosystem of tools aimed towards expanding the distributed computing framework Apache Spark in several new directions. SynapseML adds many deep learning and data science tools to the Spark ecosystem, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK), LightGBM and OpenCV. These tools enable powerful and highly-scalable predictive and analytical models for a variety of datasources.

SynapseML also brings new networking capabilities to the Spark Ecosystem. With the HTTP on Spark project, users can embed any web service into their SparkML models. In this vein, SynapseML provides easy to use SparkML transformers for a wide variety of Microsoft Cognitive Services. For production grade deployment, the Spark Serving project enables high throughput, sub-millisecond latency web services, backed by your Spark cluster.

SynapseML requires Scala 2.12, Spark 3.0+, and Python 3.6+.