mmlspark.io.binary package¶

Submodules¶

mmlspark.io.binary.BinaryFileReader module¶

mmlspark.io.binary.BinaryFileReader.BinaryFileFields = ['path', 'bytes']¶: Names of Binary File Schema field names.

mmlspark.io.binary.BinaryFileReader.BinaryFileSchema = StructType(List(StructField(path,StringType,true),StructField(bytes,BinaryType,true)))¶

Schema for Binary Files.

Schema records consist of BinaryFileFields name, Type, and ??: path bytes

mmlspark.io.binary.BinaryFileReader.isBinaryFile(df, column)[source]¶

Returns True if the column contains binary files

Parameters

df (DataFrame) – The DataFrame to be processed
column (bool) – The name of the column being inspected

Returns

True if the colum is a binary files column

Return type

bool

mmlspark.io.binary.BinaryFileReader.readBinaryFiles(self, path, recursive=False, sampleRatio=1.0, inspectZip=True, seed=0)[source]¶

Reads the directory of binary files from the local or remote (WASB) source This function is attached to SparkSession class.

Example

>>> spark.readBinaryFiles(path, recursive, sampleRatio = 1.0, inspectZip = True)

Parameters

path (str) – Path to the file directory
recursive (b (double) – Fraction of the files loaded into the dataframe

Returns

DataFrame with a single column “value”; see binaryFileSchema for details

Return type

DataFrame

mmlspark.io.binary.BinaryFileReader.streamBinaryFiles(self, path, sampleRatio=1.0, inspectZip=True, seed=0)[source]¶

Streams the directory of binary files from the local or remote (WASB) source This function is attached to SparkSession class.

Example

>>> spark.streamBinaryFiles(path, sampleRatio = 1.0, inspectZip = True)

Parameters: path (str) – Path to the file directory
Returns: DataFrame with a single column “value”; see binaryFileSchema for details
Return type: DataFrame

Module contents¶

MicrosoftML is a library of Python classes to interface with the Microsoft scala APIs to utilize Apache Spark to create distibuted machine learning models.

MicrosoftML simplifies training and scoring classifiers and regressors, as well as facilitating the creation of models using the CNTK library, images, and text.