For an array of numeric columns and an array of desired quantiles, returns a matrix of approximate values for each column at each of the desired quantiles. For example, result(0)(1) contains the approximate value for column cols(0) at quantile percentile(1).

This function uses the t-Digest algorithm.

For example, the following code:

import session.implicits._
val df = Seq((0.1, 0.5), (0.2, 0.6), (0.3, 0.7)).toDF("a", "b")
val res = double2.stat.approxQuantile(Array("a", "b"), Array(0, 0.1, 0.6))

prints out the following result:

res: Array(Array(Some(0.05), Some(0.15000000000000002), Some(0.25)),
           Array(Some(0.45), Some(0.55), Some(0.6499999999999999)))

For a specified numeric column and an array of desired quantiles, returns an approximate value for the column at each of the desired quantiles.

This function uses the t-Digest algorithm.

For example, the following code:

import session.implicits._
val df = Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 0).toDF("a")
val res = df.stat.approxQuantile("a", Array(0, 0.1, 0.4, 0.6, 1))

prints out the following result:

res: Array(Some(-0.5), Some(0.5), Some(3.5), Some(5.5), Some(9.5))

Calculates the correlation coefficient for non-null pairs in two numeric columns.

For example, the following code:

import session.implicits._
val df = Seq((0.1, 0.5), (0.2, 0.6), (0.3, 0.7)).toDF("a", "b")
double res = df.stat.corr("a", "b").get

prints out the following result:

res: 0.9999999999999991

Calculates the sample covariance for non-null pairs in two numeric columns.

For example, the following code:

import session.implicits._
val df = Seq((0.1, 0.5), (0.2, 0.6), (0.3, 0.7)).toDF("a", "b")
double res = df.stat.cov("a", "b").get

prints out the following result:

res: 0.010000000000000037

Computes a pair-wise frequency table (a contingency table) for the specified columns. The method returns a DataFrame containing this table.

In the returned contingency table:

• The first column of each row contains the distinct values of col1 .
• The name of the first column is the name of col1 .
• The rest of the column names are the distinct values of col2 .
• The counts are returned as Longs.
• For pairs that have no occurrences, the contingency table contains 0 as the count.

Note: The number of distinct values in col2 should not exceed 1000.

For example, the following code:

import session.implicits._
val df = Seq((1, 1), (1, 2), (2, 1), (2, 1), (2, 3), (3, 2), (3, 3)).toDF("key", "value")
val ct = df.stat.crosstab("key", "value")
ct.show()

prints out the following result:

---------------------------------------------------------------------------------------------
|"KEY"  |"CAST(1 AS NUMBER(38,0))"  |"CAST(2 AS NUMBER(38,0))"  |"CAST(3 AS NUMBER(38,0))"  |
---------------------------------------------------------------------------------------------
|1      |1                          |1                          |0                          |
|2      |2                          |0                          |1                          |
|3      |0                          |1                          |1                          |
---------------------------------------------------------------------------------------------

Returns a DataFrame containing a stratified sample without replacement, based on a Map that specifies the fraction for each stratum.

For example, the following code:

import session.implicits._
val df = Seq(("Bob", 17), ("Alice", 10), ("Nico", 8), ("Bob", 12)).toDF("name", "age")
val fractions = Map("Bob" -> 0.5, "Nico" -> 1.0)
df.stat.sampleBy("name", fractions).show()

prints out the following result:

------------------
|"NAME"  |"AGE"  |
------------------
|Bob     |17     |
|Nico    |8      |
------------------

Returns a DataFrame containing a stratified sample without replacement, based on a Map that specifies the fraction for each stratum.

For example, the following code:

import session.implicits._
val df = Seq(("Bob", 17), ("Alice", 10), ("Nico", 8), ("Bob", 12)).toDF("name", "age")
val fractions = Map("Bob" -> 0.5, "Nico" -> 1.0)
df.stat.sampleBy(col("name"), fractions).show()

prints out the following result:

------------------
|"NAME"  |"AGE"  |
------------------
|Bob     |17     |
|Nico    |8      |
------------------