Get number of default cores from sparkSession(required) or master(optional) for 1 executor.
Get number of default cores from sparkSession(required) or master(optional) for 1 executor.
The current spark session. If master parameter is not set, the master in the spark session is used.
This param is needed for unittest. If set, the function return the value for it. if not set, basically, master in spark (SparkSession) is used.
The number of default cores per executor based on master.
Returns a list of executor id and host.
Returns a list of executor id and host.
The dataset containing the current spark session.
List of executors as an array of (id,host).
Returns the number of executors * number of tasks.
Returns the number of executors * number of tasks.
The dataset containing the current spark session.
The number of tasks per executor.
The number of executors * number of tasks.
Get number of tasks from dummy dataset for 1 executor.
Get number of tasks from dummy dataset for 1 executor. Note: all executors have same number of cores, and this is more reliable than getting value from conf.
The dataset containing the current spark session.
The number of tasks per executor.