synapse.ml.services.search package

Submodules

synapse.ml.services.search.AddDocuments module

class synapse.ml.services.search.AddDocuments.AddDocuments(java_obj=None, AADToken=None, AADTokenCol=None, CustomAuthHeader=None, CustomAuthHeaderCol=None, actionCol='@search.action', batchSize=100, concurrency=1, concurrentTimeout=None, customHeaders=None, customHeadersCol=None, customUrlRoot=None, errorCol='AddDocuments_4ceb5fa20ed1_error', handler=None, indexName=None, outputCol='AddDocuments_4ceb5fa20ed1_output', serviceName=None, subscriptionKey=None, subscriptionKeyCol=None, timeout=60.0, url=None)

Bases: ComplexParamsMixin, JavaMLReadable, JavaMLWritable, JavaTransformer

Parameters:

• AADToken (object) – AAD Token used for authentication

• CustomAuthHeader (object) – A Custom Value for Authorization Header

• actionCol (str) – You can combine actions, such as an upload and a delete, in the same batch. upload: An upload action is similar to an ‘upsert’ where the document will be inserted if it is new and updated/replaced if it exists. Note that all fields are replaced in the update case. merge: Merge updates an existing document with the specified fields. If the document doesn’t exist, the merge will fail. Any field you specify in a merge will replace the existing field in the document. This includes fields of type Collection(Edm.String). For example, if the document contains a field ‘tags’ with value [‘budget’] and you execute a merge with value [‘economy’, ‘pool’] for ‘tags’, the final value of the ‘tags’ field will be [‘economy’, ‘pool’]. It will not be [‘budget’, ‘economy’, ‘pool’]. mergeOrUpload: This action behaves like merge if a document with the given key already exists in the index. If the document does not exist, it behaves like upload with a new document. delete: Delete removes the specified document from the index. Note that any field you specify in a delete operation, other than the key field, will be ignored. If you want to remove an individual field from a document, use merge instead and simply set the field explicitly to null.

• batchSize (int) – The max size of the buffer

• concurrency (int) – max number of concurrent calls

• concurrentTimeout (float) – max number seconds to wait on futures if concurrency >= 1

• customHeaders (object) – Map of Custom Header Key-Value Tuples.

• customUrlRoot (str) – The custom URL root for the service. This will not append OpenAI specific model path completions (i.e. /chat/completions) to the URL.

• errorCol (str) – column to hold http errors

• handler (object) – Which strategy to use when handling requests

• indexName (str) –

• outputCol (str) – The name of the output column

• serviceName (str) –

• subscriptionKey (object) – the API key to use

• timeout (float) – number of seconds to wait before closing the connection

• url (str) – Url of the service

AADToken = Param(parent='undefined', name='AADToken', doc='ServiceParam: AAD Token used for authentication')

CustomAuthHeader = Param(parent='undefined', name='CustomAuthHeader', doc='ServiceParam: A Custom Value for Authorization Header')

actionCol = Param(parent='undefined', name='actionCol', doc=" You can combine actions, such as an upload and a delete, in the same batch.  upload: An upload action is similar to an 'upsert' where the document will be inserted if it is new and updated/replaced if it exists. Note that all fields are replaced in the update case.  merge: Merge updates an existing document with the specified fields. If the document doesn't exist, the merge will fail. Any field you specify in a merge will replace the existing field in the document. This includes fields of type Collection(Edm.String). For example, if the document contains a field 'tags' with value ['budget'] and you execute a merge with value ['economy', 'pool'] for 'tags', the final value of the 'tags' field will be ['economy', 'pool'].  It will not be ['budget', 'economy', 'pool'].  mergeOrUpload: This action behaves like merge if a document  with the given key already exists in the index.  If the document does not exist, it behaves like upload with a new document.  delete: Delete removes the specified document from the index.  Note that any field you specify in a delete operation,  other than the key field, will be ignored. If you want to   remove an individual field from a document, use merge   instead and simply set the field explicitly to null.     ")

batchSize = Param(parent='undefined', name='batchSize', doc='The max size of the buffer')

concurrency = Param(parent='undefined', name='concurrency', doc='max number of concurrent calls')

concurrentTimeout = Param(parent='undefined', name='concurrentTimeout', doc='max number seconds to wait on futures if concurrency >= 1')

customHeaders = Param(parent='undefined', name='customHeaders', doc='ServiceParam: Map of Custom Header Key-Value Tuples.')

customUrlRoot = Param(parent='undefined', name='customUrlRoot', doc='The custom URL root for the service. This will not append OpenAI specific model path completions (i.e. /chat/completions) to the URL.')

errorCol = Param(parent='undefined', name='errorCol', doc='column to hold http errors')

getAADToken()

Returns:

AAD Token used for authentication

Return type:

AADToken

getActionCol()

Returns:

You can combine actions, such as an upload and a delete, in the same batch. upload: An upload action is similar to an ‘upsert’ where the document will be inserted if it is new and updated/replaced if it exists. Note that all fields are replaced in the update case. merge: Merge updates an existing document with the specified fields. If the document doesn’t exist, the merge will fail. Any field you specify in a merge will replace the existing field in the document. This includes fields of type Collection(Edm.String). For example, if the document contains a field ‘tags’ with value [‘budget’] and you execute a merge with value [‘economy’, ‘pool’] for ‘tags’, the final value of the ‘tags’ field will be [‘economy’, ‘pool’]. It will not be [‘budget’, ‘economy’, ‘pool’]. mergeOrUpload: This action behaves like merge if a document with the given key already exists in the index. If the document does not exist, it behaves like upload with a new document. delete: Delete removes the specified document from the index. Note that any field you specify in a delete operation, other than the key field, will be ignored. If you want to remove an individual field from a document, use merge instead and simply set the field explicitly to null.

Return type:

actionCol

getBatchSize()

Returns:

The max size of the buffer

Return type:

batchSize

getConcurrency()

Returns:

max number of concurrent calls

Return type:

concurrency

getConcurrentTimeout()

Returns:

max number seconds to wait on futures if concurrency >= 1

Return type:

concurrentTimeout

getCustomAuthHeader()

Returns:

A Custom Value for Authorization Header

Return type:

CustomAuthHeader

getCustomHeaders()

Returns:

Map of Custom Header Key-Value Tuples.

Return type:

customHeaders

getCustomUrlRoot()

Returns:

The custom URL root for the service. This will not append OpenAI specific model path completions (i.e. /chat/completions) to the URL.

Return type:

customUrlRoot

getErrorCol()

Returns:

column to hold http errors

Return type:

errorCol

getHandler()

Returns:

Which strategy to use when handling requests

Return type:

handler

getIndexName()

Return type:

indexName

static getJavaPackage()

Returns package name String.

getOutputCol()

Returns:

The name of the output column

Return type:

outputCol

getServiceName()

Return type:

serviceName

getSubscriptionKey()

Returns:

the API key to use

Return type:

subscriptionKey

getTimeout()

Returns:

number of seconds to wait before closing the connection

Return type:

timeout

getUrl()

Returns:

Url of the service

Return type:

url

handler = Param(parent='undefined', name='handler', doc='Which strategy to use when handling requests')

indexName = Param(parent='undefined', name='indexName', doc='')

outputCol = Param(parent='undefined', name='outputCol', doc='The name of the output column')

classmethod read()

Returns an MLReader instance for this class.

serviceName = Param(parent='undefined', name='serviceName', doc='')

setAADToken(value)

Parameters:

AADToken – AAD Token used for authentication

setAADTokenCol(value)

Parameters:

AADToken – AAD Token used for authentication

setActionCol(value)

Parameters:

actionCol – You can combine actions, such as an upload and a delete, in the same batch. upload: An upload action is similar to an ‘upsert’ where the document will be inserted if it is new and updated/replaced if it exists. Note that all fields are replaced in the update case. merge: Merge updates an existing document with the specified fields. If the document doesn’t exist, the merge will fail. Any field you specify in a merge will replace the existing field in the document. This includes fields of type Collection(Edm.String). For example, if the document contains a field ‘tags’ with value [‘budget’] and you execute a merge with value [‘economy’, ‘pool’] for ‘tags’, the final value of the ‘tags’ field will be [‘economy’, ‘pool’]. It will not be [‘budget’, ‘economy’, ‘pool’]. mergeOrUpload: This action behaves like merge if a document with the given key already exists in the index. If the document does not exist, it behaves like upload with a new document. delete: Delete removes the specified document from the index. Note that any field you specify in a delete operation, other than the key field, will be ignored. If you want to remove an individual field from a document, use merge instead and simply set the field explicitly to null.

setBatchSize(value)

Parameters:

batchSize – The max size of the buffer

setConcurrency(value)

Parameters:

concurrency – max number of concurrent calls

setConcurrentTimeout(value)

Parameters:

concurrentTimeout – max number seconds to wait on futures if concurrency >= 1

setCustomAuthHeader(value)

Parameters:

CustomAuthHeader – A Custom Value for Authorization Header

setCustomAuthHeaderCol(value)

Parameters:

CustomAuthHeader – A Custom Value for Authorization Header

setCustomHeaders(value)

Parameters:

customHeaders – Map of Custom Header Key-Value Tuples.

setCustomHeadersCol(value)

Parameters:

customHeaders – Map of Custom Header Key-Value Tuples.

setCustomServiceName(value)

setCustomUrlRoot(value)

Parameters:

customUrlRoot – The custom URL root for the service. This will not append OpenAI specific model path completions (i.e. /chat/completions) to the URL.

setDefaultInternalEndpoint(value)

setEndpoint(value)

setErrorCol(value)

Parameters:

errorCol – column to hold http errors

setHandler(value)

Parameters:

handler – Which strategy to use when handling requests

setIndexName(value)

Parameters:

indexName –

setOutputCol(value)

Parameters:

outputCol – The name of the output column

setParams(AADToken=None, AADTokenCol=None, CustomAuthHeader=None, CustomAuthHeaderCol=None, actionCol='@search.action', batchSize=100, concurrency=1, concurrentTimeout=None, customHeaders=None, customHeadersCol=None, customUrlRoot=None, errorCol='AddDocuments_4ceb5fa20ed1_error', handler=None, indexName=None, outputCol='AddDocuments_4ceb5fa20ed1_output', serviceName=None, subscriptionKey=None, subscriptionKeyCol=None, timeout=60.0, url=None)

Set the (keyword only) parameters

setServiceName(value)

Parameters:

serviceName –

setSubscriptionKey(value)

Parameters:

subscriptionKey – the API key to use

setSubscriptionKeyCol(value)

Parameters:

subscriptionKey – the API key to use

setTimeout(value)

Parameters:

timeout – number of seconds to wait before closing the connection

setUrl(value)

Parameters:

url – Url of the service

subscriptionKey = Param(parent='undefined', name='subscriptionKey', doc='ServiceParam: the API key to use')

timeout = Param(parent='undefined', name='timeout', doc='number of seconds to wait before closing the connection')

url = Param(parent='undefined', name='url', doc='Url of the service')

synapse.ml.services.search.AzureSearchWriter module

synapse.ml.services.search.AzureSearchWriter.streamToAzureSearch(df, **options)

synapse.ml.services.search.AzureSearchWriter.writeToAzureSearch(df, **options)

Module contents

SynapseML is an ecosystem of tools aimed towards expanding the distributed computing framework Apache Spark in several new directions. SynapseML adds many deep learning and data science tools to the Spark ecosystem, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK), LightGBM and OpenCV. These tools enable powerful and highly-scalable predictive and analytical models for a variety of datasources.

SynapseML also brings new networking capabilities to the Spark Ecosystem. With the HTTP on Spark project, users can embed any web service into their SparkML models. In this vein, SynapseML provides easy to use SparkML transformers for a wide variety of Microsoft Cognitive Services. For production grade deployment, the Spark Serving project enables high throughput, sub-millisecond latency web services, backed by your Spark cluster.

SynapseML requires Scala 2.12, Spark 3.0+, and Python 3.6+.