Rank Association Profile

Overview

The Rank Association Profile bucket measures how well your model’s scores rank-order examples relative to their outcomes or business value. It focuses on correlation between scores and targets, rather than only on thresholded classification.

This is especially useful when:

You primarily care about ordering (e.g., who to review first, which lead to call)
Decisions are based on top-K segments rather than a single threshold
Targets may be continuous (e.g., revenue, loss amount) even if the model is classification-based

Metrics

Let score be the model’s prediction (probability/score) and target be the outcome (binary or continuous).

spearman_rho
Spearman rank correlation between score and target, computed per time bucket:

Rank scores → rank_score
Rank targets → rank_target
Compute Pearson correlation between the ranks:

spearman_rho = corr(rank_score, rank_target)

kendall_tau
Kendall’s τ rank correlation between score and target, representing concordant vs discordant pairs:

kendall_tau = (number_concordant_pairs − number_discordant_pairs) / total_pairs

In practice, Kendall τ is computed via a built-in function where available (e.g., corr_kendall) or approximated via sampling.

Data Requirements

{{score_col}} – model score or probability
{{target_col}} – outcome (binary label, continuous value, or ordinal target)
{{timestamp_col}} – event time

Optionally, include subgroup or model version columns for segmented profiles.

Base Metric SQL — Spearman and Kendall per Day

WITH
  base_data AS (
    SELECT
      time_bucket (INTERVAL '5 minutes', {{timestamp_col}}) AS bucket,
      {{x_col}}::float AS x,
      {{y_col}}::float AS y
    FROM
      {{dataset}}
    WHERE
      {{x_col}} IS NOT NULL
      AND {{y_col}} IS NOT NULL
  ),
  -- Spearman: Pearson correlation on ranks
  spearman_stats AS (
    SELECT
      r.bucket AS bucket,
      COUNT(*)::float AS n,
      AVG(r.rx) AS mean_rx,
      AVG(r.ry) AS mean_ry,
      AVG(r.rx * r.ry) AS mean_rxry,
      AVG(r.rx * r.rx) AS mean_rx2,
      AVG(r.ry * r.ry) AS mean_ry2
    FROM
      (
        SELECT
          bucket AS bucket,
          RANK() OVER (
            PARTITION BY
              bucket
            ORDER BY
              x
          )::float AS rx,
          RANK() OVER (
            PARTITION BY
              bucket
            ORDER BY
              y
          )::float AS ry
        FROM
          base_data
      ) AS r
    GROUP BY
      r.bucket
  ),
  -- Kendall: pairwise concordant / discordant counts
  kendall_pairs AS (
    SELECT
      b1.bucket AS bucket,
      CASE
        WHEN (
          b1.x < b2.x
          AND b1.y < b2.y
        )
        OR (
          b1.x > b2.x
          AND b1.y > b2.y
        ) THEN 1
        ELSE 0
      END AS concordant,
      CASE
        WHEN (
          b1.x < b2.x
          AND b1.y > b2.y
        )
        OR (
          b1.x > b2.x
          AND b1.y < b2.y
        ) THEN 1
        ELSE 0
      END AS discordant
    FROM
      (
        SELECT
          bucket,
          x,
          y,
          ROW_NUMBER() OVER (
            PARTITION BY
              bucket
            ORDER BY
              {{timestamp_col}},
              x,
              y
          ) AS row_id
        FROM
          base_data
      ) AS b1
      JOIN (
        SELECT
          bucket,
          x,
          y,
          ROW_NUMBER() OVER (
            PARTITION BY
              bucket
            ORDER BY
              {{timestamp_col}},
              x,
              y
          ) AS row_id
        FROM
          base_data
      ) AS b2 ON b1.bucket = b2.bucket
      AND b1.row_id < b2.row_id
  ),
  kendall_stats AS (
    SELECT
      bucket AS bucket,
      SUM(concordant)::float AS c,
      SUM(discordant)::float AS d
    FROM
      kendall_pairs
    GROUP BY
      bucket
  )
SELECT
  s.bucket AS bucket,
  -- Spearman's rho (never NULL)
  CASE
    WHEN s.n < 2 THEN 0
    ELSE COALESCE(
      (s.mean_rxry - s.mean_rx * s.mean_ry) / NULLIF(
        sqrt(
          (s.mean_rx2 - s.mean_rx * s.mean_rx) * (s.mean_ry2 - s.mean_ry * s.mean_ry)
        ),
        0
      ),
      0
    )
  END AS spearman_rho,
  -- Kendall's tau (never NULL)
  CASE
    WHEN COALESCE(k.c + k.d, 0) = 0 THEN 0
    ELSE COALESCE((k.c - k.d) / NULLIF(k.c + k.d, 0), 0)
  END AS kendall_tau
FROM
  spearman_stats AS s
  LEFT JOIN kendall_stats AS k ON s.bucket = k.bucket
ORDER BY
  s.bucket;

If your SQL engine does not support Kendall τ directly, you can compute it offline or via a custom function and then load it into Arthur as a time series metric.

Plots

Plot 16 — Spearman Over Time

Uses:

spearman_rho

SELECT
    day,
    spearman_rho
FROM {{bucket_7_rank_association_metrics}}
ORDER BY day;

What this shows
This plot tracks how strongly score rankings align with target rankings over time.

How to interpret it

Higher spearman_rho means that as scores increase, targets generally increase too (good ordering).
Drops in Spearman often show that the model’s ranking power is weakening, even if thresholded metrics (like accuracy) look stable.
Particularly useful in prioritization use cases (e.g., lead scoring, collections).

Plot 17 — Kendall Tau Over Time

Uses:

kendall_tau

SELECT
    day,
    kendall_tau
FROM {{bucket_7_rank_association_metrics}}
ORDER BY day;

What this shows
This plot shows Kendall’s τ, which measures pairwise agreement in ranking between scores and targets.

How to interpret it

Unlike Spearman, Kendall focuses on how many pairs are ordered correctly vs incorrectly.
It can be more robust to outliers, so divergences between Spearman and Kendall might highlight unusual target distributions.
If your DB doesn’t provide Kendall natively, you can still reserve this plot for offline-computed metrics.

Plot 18 — Spearman + Kendall Combined

Uses:

spearman_rho
kendall_tau

SELECT
    day,
    spearman_rho,
    kendall_tau
FROM {{bucket_7_rank_association_metrics}}
ORDER BY day;

What this shows
This combined plot overlays both Spearman and Kendall rank correlations.

How to interpret it

When both move together, you have a consistent picture of ranking quality.
If they diverge—e.g., Spearman stable but Kendall falling—it may indicate that ranking issues are concentrated in specific regions or pairs.
This is a powerful diagnostic for evaluating whether a new model actually improves ordering, not just threshold metrics.

Use Cases

Lead scoring and sales prioritization
Collections and recovery models
Any pipeline where analysts work the top-ranked items first
Evaluating whether a new model actually improves ranking quality