synapse.ml.cognitive.langchain package

Submodules

synapse.ml.cognitive.langchain.LangchainTransform module

This file defines the Langchain transformation in SynapseML. To use this transformation, one needs to first define a chain, then set that chain as a parameter for the LangchainTransformer. Also needs to specify the inputColumn and outputColumn. Then this transformer will perform the operation defined in the Langchain chain to transform the input Column and save it to the OutputColumn. Example Usage:

>>> transformer = LangchainTransformer()
...                       .setInputCol("input_column_name")
...                       .setOutputCol("output_column_name")
...                       .setChain(pre_defined_chain)
...                       .setSubscriptionKey(OPENAI_API_KEY)
...                       .setUrl(baseURL)
>>> transformer.transform(sentenceDataFrame)

If the chain does not have memory, you can also save and load the Langchain Transformer. The saving of chains with memory is currently not supported in Langchain, so we can’t save transformers with that kind of chains Example Usage:

>>> transformer.save(path)
>>> loaded_transformer = LangchainTransformer.load(path)

class synapse.ml.cognitive.langchain.LangchainTransform.LangchainTransformer(inputCol=None, outputCol=None, chain=None, subscriptionKey=None, url=None, apiVersion='2022-12-01')

Bases: pyspark.ml.base.Transformer, pyspark.ml.param.shared.HasInputCol, pyspark.ml.param.shared.HasOutputCol, pyspark.ml.util.DefaultParamsReadable, pyspark.ml.util.DefaultParamsWritable

getApiVersion()

getChain()

getSubscriptionKey()

getUrl()

classmethod read() → LangchainTransformerParamsReader[RL]

Returns a LangchainTransformerParamsReader instance for this class.

setApiVersion(value: str)

setChain(value)

setInputCol(value: str)

Sets the value of inputCol.

setOutputCol(value: str)

Sets the value of outputCol.

setParams(inputCol=None, outputCol=None, chain=None, subscriptionKey=None, url=None, apiVersion='2022-12-01')

setSubscriptionKey(value: str)

set the openAI api key

setUrl(value: str)

write() → synapse.ml.cognitive.langchain.LangchainTransform.LangchainTransformerParamsWriter

Returns a DefaultParamsWriter instance for this class.

class synapse.ml.cognitive.langchain.LangchainTransform.LangchainTransformerParamsReader(cls)

Bases: pyspark.ml.util.DefaultParamsReader

load(path: str) → synapse.ml.cognitive.langchain.LangchainTransform.RL

Load the ML instance from the input path.

class synapse.ml.cognitive.langchain.LangchainTransform.LangchainTransformerParamsWriter(instance)

Bases: pyspark.ml.util.DefaultParamsWriter

saveImpl(path: str) → None

save() handles overwriting and then calls this method. Subclasses should override this method to implement the actual saving of the instance.

Module contents

SynapseML is an ecosystem of tools aimed towards expanding the distributed computing framework Apache Spark in several new directions. SynapseML adds many deep learning and data science tools to the Spark ecosystem, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK), LightGBM and OpenCV. These tools enable powerful and highly-scalable predictive and analytical models for a variety of datasources.

SynapseML also brings new networking capabilities to the Spark Ecosystem. With the HTTP on Spark project, users can embed any web service into their SparkML models. In this vein, SynapseML provides easy to use SparkML transformers for a wide variety of Microsoft Cognitive Services. For production grade deployment, the Spark Serving project enables high throughput, sub-millisecond latency web services, backed by your Spark cluster.

SynapseML requires Scala 2.12, Spark 3.0+, and Python 3.6+.