synapse.ml.stages package

Submodules

synapse.ml.stages.Cacher module

class synapse.ml.stages.Cacher.Cacher(java_obj=None, disable=False)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

disable (bool) – Whether or disable caching (so that you can turn it off during evaluation)

disable = Param(parent='undefined', name='disable', doc='Whether or disable caching (so that you can turn it off during evaluation)')

getDisable()

Returns:

Whether or disable caching (so that you can turn it off during evaluation)

Return type:

disable

static getJavaPackage()

Returns package name String.

classmethod read()

Returns an MLReader instance for this class.

setDisable(value)

Parameters:

disable – Whether or disable caching (so that you can turn it off during evaluation)

setParams(disable=False)

Set the (keyword only) parameters

synapse.ml.stages.ClassBalancer module

class synapse.ml.stages.ClassBalancer.ClassBalancer(java_obj=None, broadcastJoin=True, inputCol=None, outputCol='weight')

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaEstimator

Parameters:

• broadcastJoin (bool) – Whether to broadcast the class to weight mapping to the worker

• inputCol (str) – The name of the input column

• outputCol (str) – The name of the output column

broadcastJoin = Param(parent='undefined', name='broadcastJoin', doc='Whether to broadcast the class to weight mapping to the worker')

getBroadcastJoin()

Returns:

Whether to broadcast the class to weight mapping to the worker

Return type:

broadcastJoin

getInputCol()

Returns:

The name of the input column

Return type:

inputCol

static getJavaPackage()

Returns package name String.

getOutputCol()

Returns:

The name of the output column

Return type:

outputCol

inputCol = Param(parent='undefined', name='inputCol', doc='The name of the input column')

outputCol = Param(parent='undefined', name='outputCol', doc='The name of the output column')

classmethod read()

Returns an MLReader instance for this class.

setBroadcastJoin(value)

Parameters:

broadcastJoin – Whether to broadcast the class to weight mapping to the worker

setInputCol(value)

Parameters:

inputCol – The name of the input column

setOutputCol(value)

Parameters:

outputCol – The name of the output column

setParams(broadcastJoin=True, inputCol=None, outputCol='weight')

Set the (keyword only) parameters

synapse.ml.stages.ClassBalancerModel module

class synapse.ml.stages.ClassBalancerModel.ClassBalancerModel(java_obj=None, broadcastJoin=None, inputCol=None, outputCol=None, weights=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaModel

Parameters:

• broadcastJoin (bool) – whether to broadcast join

• inputCol (str) – The name of the input column

• outputCol (str) – The name of the output column

• weights (object) – the dataframe of weights

broadcastJoin = Param(parent='undefined', name='broadcastJoin', doc='whether to broadcast join')

getBroadcastJoin()

Returns:

whether to broadcast join

Return type:

broadcastJoin

getInputCol()

Returns:

The name of the input column

Return type:

inputCol

static getJavaPackage()

Returns package name String.

getOutputCol()

Returns:

The name of the output column

Return type:

outputCol

getWeights()

Returns:

the dataframe of weights

Return type:

weights

inputCol = Param(parent='undefined', name='inputCol', doc='The name of the input column')

outputCol = Param(parent='undefined', name='outputCol', doc='The name of the output column')

classmethod read()

Returns an MLReader instance for this class.

setBroadcastJoin(value)

Parameters:

broadcastJoin – whether to broadcast join

setInputCol(value)

Parameters:

inputCol – The name of the input column

setOutputCol(value)

Parameters:

outputCol – The name of the output column

setParams(broadcastJoin=None, inputCol=None, outputCol=None, weights=None)

Set the (keyword only) parameters

setWeights(value)

Parameters:

weights – the dataframe of weights

weights = Param(parent='undefined', name='weights', doc='the dataframe of weights')

synapse.ml.stages.DropColumns module

class synapse.ml.stages.DropColumns.DropColumns(java_obj=None, cols=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

cols (list) – Comma separated list of column names

cols = Param(parent='undefined', name='cols', doc='Comma separated list of column names')

getCols()

Returns:

Comma separated list of column names

Return type:

cols

static getJavaPackage()

Returns package name String.

classmethod read()

Returns an MLReader instance for this class.

setCols(value)

Parameters:

cols – Comma separated list of column names

setParams(cols=None)

Set the (keyword only) parameters

synapse.ml.stages.DynamicMiniBatchTransformer module

class synapse.ml.stages.DynamicMiniBatchTransformer.DynamicMiniBatchTransformer(java_obj=None, maxBatchSize=2147483647)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

maxBatchSize (int) – The max size of the buffer

static getJavaPackage()

Returns package name String.

getMaxBatchSize()

Returns:

The max size of the buffer

Return type:

maxBatchSize

maxBatchSize = Param(parent='undefined', name='maxBatchSize', doc='The max size of the buffer')

classmethod read()

Returns an MLReader instance for this class.

setMaxBatchSize(value)

Parameters:

maxBatchSize – The max size of the buffer

setParams(maxBatchSize=2147483647)

Set the (keyword only) parameters

synapse.ml.stages.EnsembleByKey module

class synapse.ml.stages.EnsembleByKey.EnsembleByKey(java_obj=None, colNames=None, collapseGroup=True, cols=None, keys=None, strategy='mean', vectorDims=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• colNames (list) – Names of the result of each col

• collapseGroup (bool) – Whether to collapse all items in group to one entry

• cols (list) – Cols to ensemble

• keys (list) – Keys to group by

• strategy (str) – How to ensemble the scores, ex: mean

• vectorDims (dict) – the dimensions of any vector columns, used to avoid materialization

colNames = Param(parent='undefined', name='colNames', doc='Names of the result of each col')

collapseGroup = Param(parent='undefined', name='collapseGroup', doc='Whether to collapse all items in group to one entry')

cols = Param(parent='undefined', name='cols', doc='Cols to ensemble')

getColNames()

Returns:

Names of the result of each col

Return type:

colNames

getCollapseGroup()

Returns:

Whether to collapse all items in group to one entry

Return type:

collapseGroup

getCols()

Returns:

Cols to ensemble

Return type:

cols

static getJavaPackage()

Returns package name String.

getKeys()

Returns:

Keys to group by

Return type:

keys

getStrategy()

Returns:

How to ensemble the scores, ex: mean

Return type:

strategy

getVectorDims()

Returns:

the dimensions of any vector columns, used to avoid materialization

Return type:

vectorDims

keys = Param(parent='undefined', name='keys', doc='Keys to group by')

classmethod read()

Returns an MLReader instance for this class.

setColNames(value)

Parameters:

colNames – Names of the result of each col

setCollapseGroup(value)

Parameters:

collapseGroup – Whether to collapse all items in group to one entry

setCols(value)

Parameters:

cols – Cols to ensemble

setKeys(value)

Parameters:

keys – Keys to group by

setParams(colNames=None, collapseGroup=True, cols=None, keys=None, strategy='mean', vectorDims=None)

Set the (keyword only) parameters

setStrategy(value)

Parameters:

strategy – How to ensemble the scores, ex: mean

setVectorDims(value)

Parameters:

vectorDims – the dimensions of any vector columns, used to avoid materialization

strategy = Param(parent='undefined', name='strategy', doc='How to ensemble the scores, ex: mean')

vectorDims = Param(parent='undefined', name='vectorDims', doc='the dimensions of any vector columns, used to avoid materialization')

synapse.ml.stages.Explode module

class synapse.ml.stages.Explode.Explode(java_obj=None, inputCol=None, outputCol='Explode_71b15d6e55a5_output')

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• inputCol (str) – The name of the input column

• outputCol (str) – The name of the output column

getInputCol()

Returns:

The name of the input column

Return type:

inputCol

static getJavaPackage()

Returns package name String.

getOutputCol()

Returns:

The name of the output column

Return type:

outputCol

inputCol = Param(parent='undefined', name='inputCol', doc='The name of the input column')

outputCol = Param(parent='undefined', name='outputCol', doc='The name of the output column')

classmethod read()

Returns an MLReader instance for this class.

setInputCol(value)

Parameters:

inputCol – The name of the input column

setOutputCol(value)

Parameters:

outputCol – The name of the output column

setParams(inputCol=None, outputCol='Explode_71b15d6e55a5_output')

Set the (keyword only) parameters

synapse.ml.stages.FixedMiniBatchTransformer module

class synapse.ml.stages.FixedMiniBatchTransformer.FixedMiniBatchTransformer(java_obj=None, batchSize=None, buffered=False, maxBufferSize=2147483647)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• batchSize (int) – The max size of the buffer

• buffered (bool) – Whether or not to buffer batches in memory

• maxBufferSize (int) – The max size of the buffer

batchSize = Param(parent='undefined', name='batchSize', doc='The max size of the buffer')

buffered = Param(parent='undefined', name='buffered', doc='Whether or not to buffer batches in memory')

getBatchSize()

Returns:

The max size of the buffer

Return type:

batchSize

getBuffered()

Returns:

Whether or not to buffer batches in memory

Return type:

buffered

static getJavaPackage()

Returns package name String.

getMaxBufferSize()

Returns:

The max size of the buffer

Return type:

maxBufferSize

maxBufferSize = Param(parent='undefined', name='maxBufferSize', doc='The max size of the buffer')

classmethod read()

Returns an MLReader instance for this class.

setBatchSize(value)

Parameters:

batchSize – The max size of the buffer

setBuffered(value)

Parameters:

buffered – Whether or not to buffer batches in memory

setMaxBufferSize(value)

Parameters:

maxBufferSize – The max size of the buffer

setParams(batchSize=None, buffered=False, maxBufferSize=2147483647)

Set the (keyword only) parameters

synapse.ml.stages.FlattenBatch module

class synapse.ml.stages.FlattenBatch.FlattenBatch(java_obj=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Args:

static getJavaPackage()

Returns package name String.

classmethod read()

Returns an MLReader instance for this class.

setParams()

Set the (keyword only) parameters

synapse.ml.stages.Lambda module

class synapse.ml.stages.Lambda.Lambda(java_obj=None, transformFunc=None, transformSchemaFunc=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• transformFunc (object) – holder for dataframe function

• transformSchemaFunc (object) – the output schema after the transformation

static getJavaPackage()

Returns package name String.

getTransformFunc()

Returns:

holder for dataframe function

Return type:

transformFunc

getTransformSchemaFunc()

Returns:

the output schema after the transformation

Return type:

transformSchemaFunc

classmethod read()

Returns an MLReader instance for this class.

setParams(transformFunc=None, transformSchemaFunc=None)

Set the (keyword only) parameters

setTransformFunc(value)

Parameters:

transformFunc – holder for dataframe function

setTransformSchemaFunc(value)

Parameters:

transformSchemaFunc – the output schema after the transformation

transformFunc = Param(parent='undefined', name='transformFunc', doc='holder for dataframe function')

transformSchemaFunc = Param(parent='undefined', name='transformSchemaFunc', doc='the output schema after the transformation')

synapse.ml.stages.MultiColumnAdapter module

class synapse.ml.stages.MultiColumnAdapter.MultiColumnAdapter(java_obj=None, baseStage=None, inputCols=None, outputCols=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaEstimator

Parameters:

• baseStage (object) – base pipeline stage to apply to every column

• inputCols (list) – list of column names encoded as a string

• outputCols (list) – list of column names encoded as a string

baseStage = Param(parent='undefined', name='baseStage', doc='base pipeline stage to apply to every column')

getBaseStage()

Returns:

base pipeline stage to apply to every column

Return type:

baseStage

getInputCols()

Returns:

list of column names encoded as a string

Return type:

inputCols

static getJavaPackage()

Returns package name String.

getOutputCols()

Returns:

list of column names encoded as a string

Return type:

outputCols

inputCols = Param(parent='undefined', name='inputCols', doc='list of column names encoded as a string')

outputCols = Param(parent='undefined', name='outputCols', doc='list of column names encoded as a string')

classmethod read()

Returns an MLReader instance for this class.

setBaseStage(value)

Parameters:

baseStage – base pipeline stage to apply to every column

setInputCols(value)

Parameters:

inputCols – list of column names encoded as a string

setOutputCols(value)

Parameters:

outputCols – list of column names encoded as a string

setParams(baseStage=None, inputCols=None, outputCols=None)

Set the (keyword only) parameters

synapse.ml.stages.PartitionConsolidator module

class synapse.ml.stages.PartitionConsolidator.PartitionConsolidator(java_obj=None, concurrency=1, concurrentTimeout=None, inputCol=None, outputCol=None, timeout=60.0)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• concurrency (int) – max number of concurrent calls

• concurrentTimeout (float) – max number seconds to wait on futures if concurrency >= 1

• inputCol (str) – The name of the input column

• outputCol (str) – The name of the output column

• timeout (float) – number of seconds to wait before closing the connection

concurrency = Param(parent='undefined', name='concurrency', doc='max number of concurrent calls')

concurrentTimeout = Param(parent='undefined', name='concurrentTimeout', doc='max number seconds to wait on futures if concurrency >= 1')

getConcurrency()

Returns:

max number of concurrent calls

Return type:

concurrency

getConcurrentTimeout()

Returns:

max number seconds to wait on futures if concurrency >= 1

Return type:

concurrentTimeout

getInputCol()

Returns:

The name of the input column

Return type:

inputCol

static getJavaPackage()

Returns package name String.

getOutputCol()

Returns:

The name of the output column

Return type:

outputCol

getTimeout()

Returns:

number of seconds to wait before closing the connection

Return type:

timeout

inputCol = Param(parent='undefined', name='inputCol', doc='The name of the input column')

outputCol = Param(parent='undefined', name='outputCol', doc='The name of the output column')

classmethod read()

Returns an MLReader instance for this class.

setConcurrency(value)

Parameters:

concurrency – max number of concurrent calls

setConcurrentTimeout(value)

Parameters:

concurrentTimeout – max number seconds to wait on futures if concurrency >= 1

setInputCol(value)

Parameters:

inputCol – The name of the input column

setOutputCol(value)

Parameters:

outputCol – The name of the output column

setParams(concurrency=1, concurrentTimeout=None, inputCol=None, outputCol=None, timeout=60.0)

Set the (keyword only) parameters

setTimeout(value)

Parameters:

timeout – number of seconds to wait before closing the connection

timeout = Param(parent='undefined', name='timeout', doc='number of seconds to wait before closing the connection')

synapse.ml.stages.RenameColumn module

class synapse.ml.stages.RenameColumn.RenameColumn(java_obj=None, inputCol=None, outputCol=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• inputCol (str) – The name of the input column

• outputCol (str) – The name of the output column

getInputCol()

Returns:

The name of the input column

Return type:

inputCol

static getJavaPackage()

Returns package name String.

getOutputCol()

Returns:

The name of the output column

Return type:

outputCol

inputCol = Param(parent='undefined', name='inputCol', doc='The name of the input column')

outputCol = Param(parent='undefined', name='outputCol', doc='The name of the output column')

classmethod read()

Returns an MLReader instance for this class.

setInputCol(value)

Parameters:

inputCol – The name of the input column

setOutputCol(value)

Parameters:

outputCol – The name of the output column

setParams(inputCol=None, outputCol=None)

Set the (keyword only) parameters

synapse.ml.stages.Repartition module

class synapse.ml.stages.Repartition.Repartition(java_obj=None, disable=False, n=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• disable (bool) – Whether to disable repartitioning (so that one can turn it off for evaluation)

• n (int) – Number of partitions

disable = Param(parent='undefined', name='disable', doc='Whether to disable repartitioning (so that one can turn it off for evaluation)')

getDisable()

Returns:

Whether to disable repartitioning (so that one can turn it off for evaluation)

Return type:

disable

static getJavaPackage()

Returns package name String.

getN()

Returns:

Number of partitions

Return type:

n

n = Param(parent='undefined', name='n', doc='Number of partitions')

classmethod read()

Returns an MLReader instance for this class.

setDisable(value)

Parameters:

disable – Whether to disable repartitioning (so that one can turn it off for evaluation)

setN(value)

Parameters:

n – Number of partitions

setParams(disable=False, n=None)

Set the (keyword only) parameters

synapse.ml.stages.SelectColumns module

class synapse.ml.stages.SelectColumns.SelectColumns(java_obj=None, cols=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

cols (list) – Comma separated list of selected column names

cols = Param(parent='undefined', name='cols', doc='Comma separated list of selected column names')

getCols()

Returns:

Comma separated list of selected column names

Return type:

cols

static getJavaPackage()

Returns package name String.

classmethod read()

Returns an MLReader instance for this class.

setCols(value)

Parameters:

cols – Comma separated list of selected column names

setParams(cols=None)

Set the (keyword only) parameters

synapse.ml.stages.StratifiedRepartition module

class synapse.ml.stages.StratifiedRepartition.StratifiedRepartition(java_obj=None, labelCol=None, mode='mixed', seed=1518410069)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• labelCol (str) – The name of the label column

• mode (str) – Specify equal to repartition with replacement across all labels, specify original to keep the ratios in the original dataset, or specify mixed to use a heuristic

• seed (long) – random seed

static getJavaPackage()

Returns package name String.

getLabelCol()

Returns:

The name of the label column

Return type:

labelCol

getMode()

Returns:

Specify equal to repartition with replacement across all labels, specify original to keep the ratios in the original dataset, or specify mixed to use a heuristic

Return type:

mode

getSeed()

Returns:

random seed

Return type:

seed

labelCol = Param(parent='undefined', name='labelCol', doc='The name of the label column')

mode = Param(parent='undefined', name='mode', doc='Specify equal to repartition with replacement across all labels, specify original to keep the ratios in the original dataset, or specify mixed to use a heuristic')

classmethod read()

Returns an MLReader instance for this class.

seed = Param(parent='undefined', name='seed', doc='random seed')

setLabelCol(value)

Parameters:

labelCol – The name of the label column

setMode(value)

Parameters:

mode – Specify equal to repartition with replacement across all labels, specify original to keep the ratios in the original dataset, or specify mixed to use a heuristic

setParams(labelCol=None, mode='mixed', seed=1518410069)

Set the (keyword only) parameters

setSeed(value)

Parameters:

seed – random seed

synapse.ml.stages.SummarizeData module

class synapse.ml.stages.SummarizeData.SummarizeData(java_obj=None, basic=True, counts=True, errorThreshold=0.0, percentiles=True, sample=True)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• basic (bool) – Compute basic statistics

• counts (bool) – Compute count statistics

• errorThreshold (float) – Threshold for quantiles - 0 is exact

• percentiles (bool) – Compute percentiles

• sample (bool) – Compute sample statistics

basic = Param(parent='undefined', name='basic', doc='Compute basic statistics')

counts = Param(parent='undefined', name='counts', doc='Compute count statistics')

errorThreshold = Param(parent='undefined', name='errorThreshold', doc='Threshold for quantiles - 0 is exact')

getBasic()

Returns:

Compute basic statistics

Return type:

basic

getCounts()

Returns:

Compute count statistics

Return type:

counts

getErrorThreshold()

Returns:

Threshold for quantiles - 0 is exact

Return type:

errorThreshold

static getJavaPackage()

Returns package name String.

getPercentiles()

Returns:

Compute percentiles

Return type:

percentiles

getSample()

Returns:

Compute sample statistics

Return type:

sample

percentiles = Param(parent='undefined', name='percentiles', doc='Compute percentiles')

classmethod read()

Returns an MLReader instance for this class.

sample = Param(parent='undefined', name='sample', doc='Compute sample statistics')

setBasic(value)

Parameters:

basic – Compute basic statistics

setCounts(value)

Parameters:

counts – Compute count statistics

setErrorThreshold(value)

Parameters:

errorThreshold – Threshold for quantiles - 0 is exact

setParams(basic=True, counts=True, errorThreshold=0.0, percentiles=True, sample=True)

Set the (keyword only) parameters

setPercentiles(value)

Parameters:

percentiles – Compute percentiles

setSample(value)

Parameters:

sample – Compute sample statistics

synapse.ml.stages.TextPreprocessor module

class synapse.ml.stages.TextPreprocessor.TextPreprocessor(java_obj=None, inputCol=None, map=None, normFunc=None, outputCol=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• inputCol (str) – The name of the input column

• map (dict) – Map of substring match to replacement

• normFunc (str) – Name of normalization function to apply

• outputCol (str) – The name of the output column

getInputCol()

Returns:

The name of the input column

Return type:

inputCol

static getJavaPackage()

Returns package name String.

getMap()

Returns:

Map of substring match to replacement

Return type:

map

getNormFunc()

Returns:

Name of normalization function to apply

Return type:

normFunc

getOutputCol()

Returns:

The name of the output column

Return type:

outputCol

inputCol = Param(parent='undefined', name='inputCol', doc='The name of the input column')

map = Param(parent='undefined', name='map', doc='Map of substring match to replacement')

normFunc = Param(parent='undefined', name='normFunc', doc='Name of normalization function to apply')

outputCol = Param(parent='undefined', name='outputCol', doc='The name of the output column')

classmethod read()

Returns an MLReader instance for this class.

setInputCol(value)

Parameters:

inputCol – The name of the input column

setMap(value)

Parameters:

map – Map of substring match to replacement

setNormFunc(value)

Parameters:

normFunc – Name of normalization function to apply

setOutputCol(value)

Parameters:

outputCol – The name of the output column

setParams(inputCol=None, map=None, normFunc=None, outputCol=None)

Set the (keyword only) parameters

synapse.ml.stages.TimeIntervalMiniBatchTransformer module

class synapse.ml.stages.TimeIntervalMiniBatchTransformer.TimeIntervalMiniBatchTransformer(java_obj=None, maxBatchSize=2147483647, millisToWait=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• maxBatchSize (int) – The max size of the buffer

• millisToWait (int) – The time to wait before constructing a batch

static getJavaPackage()

Returns package name String.

getMaxBatchSize()

Returns:

The max size of the buffer

Return type:

maxBatchSize

getMillisToWait()

Returns:

The time to wait before constructing a batch

Return type:

millisToWait

maxBatchSize = Param(parent='undefined', name='maxBatchSize', doc='The max size of the buffer')

millisToWait = Param(parent='undefined', name='millisToWait', doc='The time to wait before constructing a batch')

classmethod read()

Returns an MLReader instance for this class.

setMaxBatchSize(value)

Parameters:

maxBatchSize – The max size of the buffer

setMillisToWait(value)

Parameters:

millisToWait – The time to wait before constructing a batch

setParams(maxBatchSize=2147483647, millisToWait=None)

Set the (keyword only) parameters

synapse.ml.stages.Timer module

class synapse.ml.stages.Timer.Timer(java_obj=None, disableMaterialization=True, logToScala=True, stage=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaEstimator

Parameters:

• disableMaterialization (bool) – Whether to disable timing (so that one can turn it off for evaluation)

• logToScala (bool) – Whether to output the time to the scala console

• stage (object) – The stage to time

disableMaterialization = Param(parent='undefined', name='disableMaterialization', doc='Whether to disable timing (so that one can turn it off for evaluation)')

getDisableMaterialization()

Returns:

Whether to disable timing (so that one can turn it off for evaluation)

Return type:

disableMaterialization

static getJavaPackage()

Returns package name String.

getLogToScala()

Returns:

Whether to output the time to the scala console

Return type:

logToScala

getStage()

Returns:

The stage to time

Return type:

stage

logToScala = Param(parent='undefined', name='logToScala', doc='Whether to output the time to the scala console')

classmethod read()

Returns an MLReader instance for this class.

setDisableMaterialization(value)

Parameters:

disableMaterialization – Whether to disable timing (so that one can turn it off for evaluation)

setLogToScala(value)

Parameters:

logToScala – Whether to output the time to the scala console

setParams(disableMaterialization=True, logToScala=True, stage=None)

Set the (keyword only) parameters

setStage(value)

Parameters:

stage – The stage to time

stage = Param(parent='undefined', name='stage', doc='The stage to time')

synapse.ml.stages.TimerModel module

class synapse.ml.stages.TimerModel.TimerModel(java_obj=None, disableMaterialization=True, logToScala=True, stage=None, transformer=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaModel

Parameters:

• disableMaterialization (bool) – Whether to disable timing (so that one can turn it off for evaluation)

• logToScala (bool) – Whether to output the time to the scala console

• stage (object) – The stage to time

• transformer (object) – inner model to time

disableMaterialization = Param(parent='undefined', name='disableMaterialization', doc='Whether to disable timing (so that one can turn it off for evaluation)')

getDisableMaterialization()

Returns:

Whether to disable timing (so that one can turn it off for evaluation)

Return type:

disableMaterialization

static getJavaPackage()

Returns package name String.

getLogToScala()

Returns:

Whether to output the time to the scala console

Return type:

logToScala

getStage()

Returns:

The stage to time

Return type:

stage

getTransformer()

Returns:

inner model to time

Return type:

transformer

logToScala = Param(parent='undefined', name='logToScala', doc='Whether to output the time to the scala console')

classmethod read()

Returns an MLReader instance for this class.

setDisableMaterialization(value)

Parameters:

disableMaterialization – Whether to disable timing (so that one can turn it off for evaluation)

setLogToScala(value)

Parameters:

logToScala – Whether to output the time to the scala console

setParams(disableMaterialization=True, logToScala=True, stage=None, transformer=None)

Set the (keyword only) parameters

setStage(value)

Parameters:

stage – The stage to time

setTransformer(value)

Parameters:

transformer – inner model to time

stage = Param(parent='undefined', name='stage', doc='The stage to time')

transformer = Param(parent='undefined', name='transformer', doc='inner model to time')

synapse.ml.stages.UDFTransformer module

class synapse.ml.stages.UDFTransformer.UDFTransformer(inputCol=None, inputCols=None, outputCol=None, udf=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• inputCol (str) – The name of the input column (default: )

• outputCol (str) – The name of the output column

• udf (object) – User Defined Python Function to be applied to the DF input col

• udfScala (object) – User Defined Function to be applied to the DF input col

getInputCol()

Returns:

The name of the input column (default: )

Return type:

str

getInputCols()

Returns:

The name of the input column (default: )

Return type:

str

static getJavaPackage()

Returns package name String.

getOutputCol()

Returns:

The name of the output column

Return type:

str

getUDF()

classmethod read()

Returns an MLReader instance for this class.

setInputCol(value)

Parameters:

inputCol (str) – The name of the input column (default: )

setInputCols(value)

Parameters:

inputCols (list) – The names of the input columns (default: )

setOutputCol(value)

Parameters:

outputCol (str) – The name of the output column

setUDF(udf)

synapse.ml.stages.UnicodeNormalize module

class synapse.ml.stages.UnicodeNormalize.UnicodeNormalize(java_obj=None, form=None, inputCol=None, lower=None, outputCol=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• form (str) – Unicode normalization form: NFC, NFD, NFKC, NFKD

• inputCol (str) – The name of the input column

• lower (bool) – Lowercase text

• outputCol (str) – The name of the output column

form = Param(parent='undefined', name='form', doc='Unicode normalization form: NFC, NFD, NFKC, NFKD')

getForm()

Returns:

Unicode normalization form: NFC, NFD, NFKC, NFKD

Return type:

form

getInputCol()

Returns:

The name of the input column

Return type:

inputCol

static getJavaPackage()

Returns package name String.

getLower()

Returns:

Lowercase text

Return type:

lower

getOutputCol()

Returns:

The name of the output column

Return type:

outputCol

inputCol = Param(parent='undefined', name='inputCol', doc='The name of the input column')

lower = Param(parent='undefined', name='lower', doc='Lowercase text')

outputCol = Param(parent='undefined', name='outputCol', doc='The name of the output column')

classmethod read()

Returns an MLReader instance for this class.

setForm(value)

Parameters:

form – Unicode normalization form: NFC, NFD, NFKC, NFKD

setInputCol(value)

Parameters:

inputCol – The name of the input column

setLower(value)

Parameters:

lower – Lowercase text

setOutputCol(value)

Parameters:

outputCol – The name of the output column

setParams(form=None, inputCol=None, lower=None, outputCol=None)

Set the (keyword only) parameters

Module contents

SynapseML is an ecosystem of tools aimed towards expanding the distributed computing framework Apache Spark in several new directions. SynapseML adds many deep learning and data science tools to the Spark ecosystem, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK), LightGBM and OpenCV. These tools enable powerful and highly-scalable predictive and analytical models for a variety of datasources.

SynapseML also brings new networking capabilities to the Spark Ecosystem. With the HTTP on Spark project, users can embed any web service into their SparkML models. In this vein, SynapseML provides easy to use SparkML transformers for a wide variety of Microsoft Cognitive Services. For production grade deployment, the Spark Serving project enables high throughput, sub-millisecond latency web services, backed by your Spark cluster.

SynapseML requires Scala 2.12, Spark 3.0+, and Python 3.6+.