mmlspark.cyber.utils package

Submodules

mmlspark.cyber.utils.spark_utils module

class mmlspark.cyber.utils.spark_utils.DataFrameUtils

Bases: object

Extension methods over Spark DataFrame

static get_spark_session(df: pyspark.sql.dataframe.DataFrame) → pyspark.sql.session.SparkSession

get the associated Spark session

Parameters

df (DataFrame) – the dataframe of which we want to get its Spark session

static make_empty(df: pyspark.sql.dataframe.DataFrame) → pyspark.sql.dataframe.DataFrame

make an empty dataframe with the same schema

Parameters

• the dataframe whose schema we wish to use (df) –

• an empty dataframe (Returns) –

• ------- –

static zip_with_index(df: pyspark.sql.dataframe.DataFrame, start_index: int = 0, col_name: str = 'rowId', partition_col: Union[List[str], str] = [], order_by_col: Union[List[str], str] = []) → pyspark.sql.dataframe.DataFrame

add an index to the given dataframe

Parameters

• df (dataframe) – the dataframe to add the index to

• start_index (int) – the value to start the count from

• col_name (str) – the name of the index column which will be added as last column in the output data frame

• partition_col (Union[List[str], str]) – optional column name or list of columns names that define a partitioning to assign indices independently to, e.g., assign sequential indices separately to each distinct tenant

• order_by_col (Union[List[str], str]) – optional order by column name or list of columns that are used for sorting the data frame or partitions before indexing

class mmlspark.cyber.utils.spark_utils.ExplainBuilder

Bases: object

static build(explainable: Any, **kwargs)

static copy_params(from_explainable: Any, to_explainable: Any)

static get_method(the_explainable, the_method_name)

static get_methods(the_explainable)

Module contents

MicrosoftML is a library of Python classes to interface with the Microsoft scala APIs to utilize Apache Spark to create distibuted machine learning models.

MicrosoftML simplifies training and scoring classifiers and regressors, as well as facilitating the creation of models using the CNTK library, images, and text.