synapse.ml.core.logging package

Submodules

synapse.ml.core.logging.SynapseMLLogger module

class synapse.ml.core.logging.SynapseMLLogger.CustomFormatter(fmt=None, datefmt=None, style='%', validate=True)

Bases: logging.Formatter

format(record)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class synapse.ml.core.logging.SynapseMLLogger.SynapseMLLogger(library_name: Optional[str] = None, library_version: Optional[str] = None, uid: Optional[str] = None, log_level: int = 20)

Bases: object

classmethod get_column_number(args, kwargs)

classmethod get_error_fields(e: Exception) → Dict[str, str]

classmethod get_hadoop_conf_entries()

get_required_log_fields(uid: str, class_name: str, method: str)

log_class(feature_name: str)

log_fit()

log_message(message: str)

log_transform()

log_verb()

classmethod safe_get_spark_context() → pyspark.context.SparkContext

Module contents

SynapseML is an ecosystem of tools aimed towards expanding the distributed computing framework Apache Spark in several new directions. SynapseML adds many deep learning and data science tools to the Spark ecosystem, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK), LightGBM and OpenCV. These tools enable powerful and highly-scalable predictive and analytical models for a variety of datasources.

SynapseML also brings new networking capabilities to the Spark Ecosystem. With the HTTP on Spark project, users can embed any web service into their SparkML models. In this vein, SynapseML provides easy to use SparkML transformers for a wide variety of Microsoft Cognitive Services. For production grade deployment, the Spark Serving project enables high throughput, sub-millisecond latency web services, backed by your Spark cluster.

SynapseML requires Scala 2.12, Spark 3.0+, and Python 3.6+.