mmlspark.cyber.utils package


mmlspark.cyber.utils.spark_utils module

class mmlspark.cyber.utils.spark_utils.DataFrameUtils[source]

Bases: object

Extension methods over Spark DataFrame

static get_spark_session(df: pyspark.sql.dataframe.DataFrame) → pyspark.sql.session.SparkSession[source]

get the associated Spark session


df (DataFrame) – the dataframe of which we want to get its Spark session

static make_empty(df: pyspark.sql.dataframe.DataFrame) → pyspark.sql.dataframe.DataFrame[source]

make an empty dataframe with the same schema

  • the dataframe whose schema we wish to use (df) –

  • an empty dataframe (Returns) –

  • -------

static zip_with_index(df: pyspark.sql.dataframe.DataFrame, start_index: int = 0, col_name: str = 'rowId', partition_col: Union[List[str], str] = [], order_by_col: Union[List[str], str] = []) → pyspark.sql.dataframe.DataFrame[source]

add an index to the given dataframe

  • df (dataframe) – the dataframe to add the index to

  • start_index (int) – the value to start the count from

  • col_name (str) – the name of the index column which will be added as last column in the output data frame

  • partition_col (Union[List[str], str]) – optional column name or list of columns names that define a partitioning to assign indices independently to, e.g., assign sequential indices separately to each distinct tenant

  • order_by_col (Union[List[str], str]) – optional order by column name or list of columns that are used for sorting the data frame or partitions before indexing

class mmlspark.cyber.utils.spark_utils.ExplainBuilder[source]

Bases: object

static build(explainable: Any, **kwargs)[source]
static copy_params(from_explainable: Any, to_explainable: Any)[source]
static get_method(the_explainable, the_method_name)[source]
static get_methods(the_explainable)[source]

Module contents

MicrosoftML is a library of Python classes to interface with the Microsoft scala APIs to utilize Apache Spark to create distibuted machine learning models.

MicrosoftML simplifies training and scoring classifiers and regressors, as well as facilitating the creation of models using the CNTK library, images, and text.