synapse.ml.downloader package

Submodules

synapse.ml.downloader.ModelDownloader module

class synapse.ml.downloader.ModelDownloader.ModelDownloader(sparkSession, localPath, serverURL='https://mmlspark.azureedge.net/datasets/CNTKModels/')

Bases: object

A class for downloading CNTK pretrained models in python. To download all models use the downloadModels function. To browse models from the microsoft server please use remoteModels.

Parameters:

• sparkSession (SparkSession) – A spark session for interfacing between python and java

• localPath (str) – The folder to save models to

• serverURL (str) – The location of the model Server, beware this default can change!

downloadByName(name)

Downloads a named model

Parameters:

name (str) – The name of the model

downloadModel(model)

Download a model

Parameters:

model (object) – The model to be downloaded

Returns:

model schema

Return type:

object

downloadModels(models=None)

Download models

Parameters:

models – The models to be downloaded

Returns:

list of models downloaded

Return type:

list

localModels()

Downloads models stored locally on the filesystem

remoteModels()

Downloads models stored remotely.

class synapse.ml.downloader.ModelDownloader.ModelSchema(name, dataset, modelType, uri, hash, size, inputNode, numLayers, layerNames)

Bases: object

An object that represents a model.

Parameters:

• name (str) – Name of the model

• dataset (DataFrame) – Dataset it was trained on

• modelType (str) – Domain that the model operates on

• uri (str) – The location of the model’s bytes

• hash (str) – The sha256 hash of the models bytes

• size (int) – the size of the model in bytes

• inputNode (int) – the node which represents the input

• numLayers (int) – the number of layers of the model

• layerNames (array) – the names of nodes that represent layers in the network

static fromJava(jobj)

toJava(sparkSession)

Module contents

SynapseML is an ecosystem of tools aimed towards expanding the distributed computing framework Apache Spark in several new directions. SynapseML adds many deep learning and data science tools to the Spark ecosystem, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK), LightGBM and OpenCV. These tools enable powerful and highly-scalable predictive and analytical models for a variety of datasources.

SynapseML also brings new networking capabilities to the Spark Ecosystem. With the HTTP on Spark project, users can embed any web service into their SparkML models. In this vein, SynapseML provides easy to use SparkML transformers for a wide variety of Microsoft Cognitive Services. For production grade deployment, the Spark Serving project enables high throughput, sub-millisecond latency web services, backed by your Spark cluster.

SynapseML requires Scala 2.12, Spark 3.0+, and Python 3.6+.