Guides
Start typing to search…

Curve-Based Discrimination

Uses ROC, AUC, KS, Gini, and related curves to measure how well the model separates positives from negatives across the full score distribution.

Overview

The Curve-Based Discrimination bucket measures how well your model’s scores separate positive from negative outcomes across the full score range, using classic evaluation curves.

This bucket is implemented for binary classification. Multiclass models can be evaluated via one-vs-rest or macro-averaged strategies by defining separate metrics per class.

It uses:

  • Receiver Operating Characteristic (ROC) and AUC-ROC
  • Gini coefficient
  • KS (Kolmogorov–Smirnov) statistic and score

Metrics

All metrics are derived from (fpr, tpr) pairs computed across a grid of thresholds on the positive-class score.

auc_roc
Area under the ROC curve, approximated via the trapezoidal rule:

For adjacent ROC points (fpr_i, tpr_i) and (fpr_{i+1}, tpr_{i+1}):

ΔAUC_i = (fpr_{i+1} − fpr_i) × (tpr_{i+1} + tpr_i) / 2
auc_roc = Σ_i ΔAUC_i

gini_coefficient
Linear transform of AUC commonly used in risk/scoring domains:

gini_coefficient = 2 × auc_roc − 1

ks_statistic
Maximum separation between the cumulative distributions of scores for positives vs negatives:

ks_statistic = max_threshold | CDF_positive(score ≥ t) − CDF_negative(score ≥ t) |

ks_score
The threshold value t* at which ks_statistic is attained. This is often used as a candidate operating point.

Data Requirements

  • {{label_col}} – binary ground truth (0 for negative, 1 for positive)
  • {{score_col}} – probability or score for the positive class (continuous)
  • {{timestamp_col}} – event timestamp

Base Metric SQL — ROC Components

WITH base AS (
    SELECT
        {{timestamp_col}} AS event_ts,
        {{label_col}}    AS label,
        {{score_col}}    AS score
    FROM {{dataset}}
),
grid AS (
    SELECT generate_series(0.0, 1.0, 0.01) AS threshold
),
scored AS (
    SELECT
        time_bucket(INTERVAL '5 minutes', event_ts) AS ts,
        g.threshold,
        label,
        score,
        CASE WHEN score >= g.threshold THEN 1 ELSE 0 END AS pred_pos
    FROM base
    CROSS JOIN grid g
)
SELECT
    ts     																											AS ts,
    threshold																										AS threshold,
    SUM(CASE WHEN label = 1 THEN 1 ELSE 0 END)                  AS actual_pos,
    SUM(CASE WHEN label = 0 THEN 1 ELSE 0 END)                  AS actual_neg,
    SUM(CASE WHEN pred_pos = 1 AND label = 1 THEN 1 ELSE 0 END) AS tp,
    SUM(CASE WHEN pred_pos = 1 AND label = 0 THEN 1 ELSE 0 END) AS fp
FROM scored
GROUP BY ts, threshold
ORDER BY ts, threshold;

You can materialize this as {{bucket_5_curve_metrics_daily}}.

Computing AUC-ROC, Gini, and KS

AUC-ROC (Binary Only)

WITH roc_points AS (
    SELECT
        day,
        threshold,
        tpr,
        fpr,
        LAG(fpr) OVER (PARTITION BY day ORDER BY fpr) AS prev_fpr,
        LAG(tpr) OVER (PARTITION BY day ORDER BY fpr) AS prev_tpr
    FROM {{bucket_5_curve_metrics_daily}}
)
SELECT
    day,
    SUM(
        COALESCE((fpr - prev_fpr) * (tpr + prev_tpr) / 2.0, 0.0)
    ) AS auc_roc
FROM roc_points
GROUP BY day;

Gini Coefficient

SELECT
    day,
    auc_roc,
    2 * auc_roc - 1 AS gini_coefficient
FROM {{bucket_5_auc_metrics}};

KS Statistic and Score

WITH roc_points AS (
    SELECT
        day,
        threshold,
        tpr,
        fpr
    FROM {{bucket_5_curve_metrics_daily}}
),
cum AS (
    SELECT
        day,
        threshold,
        SUM(tpr) OVER (PARTITION BY day ORDER BY threshold DESC) AS cum_tpr,
        SUM(fpr) OVER (PARTITION BY day ORDER BY threshold DESC) AS cum_fpr
    FROM roc_points
)
SELECT
    day,
    MAX(ABS(cum_tpr - cum_fpr)) AS ks_statistic,
    FIRST_VALUE(threshold) OVER (
        PARTITION BY day
        ORDER BY ABS(cum_tpr - cum_fpr) DESC
    ) AS ks_score
FROM cum
GROUP BY day;

Plots

Plot 13 — AUC + Gini Over Time

Uses:

  • auc_roc
  • gini_coefficient
SELECT
    day,
    auc_roc,
    gini_coefficient
FROM {{bucket_5_auc_gini_metrics}}
ORDER BY day;

What this shows
This plot tracks both AUC and its corresponding Gini coefficient over time, providing two familiar views of discrimination strength.

How to interpret it

  • Sustained drops in auc_roc or gini_coefficient usually indicate that the model’s ability to separate positives from negatives has degraded.
  • If AUC remains stable while business KPIs worsen, the issue may be threshold selection or data mix, not raw score discrimination.
  • Gini is often the metric business stakeholders and regulators expect in credit/risk settings, so having both on one chart helps bridge ML and business views.

Plot 14 — KS Statistic Over Time

Uses:

  • ks_statistic
  • ks_score
SELECT
    day,
    ks_statistic,
    ks_score
FROM {{bucket_5_ks_metrics}}
ORDER BY day;

What this shows
This plot tracks the maximum separation between positive and negative score distributions (ks_statistic) and where that separation occurs in score space (ks_score).

How to interpret it

  • Higher ks_statistic values indicate stronger separation; many credit models target KS in a specific band (e.g., 0.3–0.5).
  • Changes in ks_score tell you where in the score range the separation is happening—if it drifts, your best cut-off point may be moving.
  • If KS drops while AUC is flat, the problem may be in specific parts of the distribution (e.g., tail behavior) rather than global ranking.

Plot 15A — Combined AUC + KS

Uses:

  • auc_roc
  • ks_statistic
  • ks_score
SELECT
    a.day,
    a.auc_roc,
    k.ks_statistic,
    k.ks_score
FROM {{bucket_5_auc_gini_metrics}} a
JOIN {{bucket_5_ks_metrics}} k USING (day)
ORDER BY a.day;

What this shows
This plot overlays global ranking quality (AUC) with maximum local separation (KS) on the same time axis.

How to interpret it

  • When AUC and KS move together, the model’s ranking power is consistently changing.
  • Divergence (e.g., AUC flat, KS moving) suggests that certain score regions become more/less separative even though global ranking is unchanged.
  • This is an excellent high-level monitoring view for risk committees and model governance reviews.

Plot 15B — KS-Only Variant (optional)

Uses:

  • ks_statistic
SELECT
    day,
    ks_statistic
FROM {{bucket_5_ks_metrics}}
ORDER BY day;

What this shows
A lightweight KS-only time series focusing on separation strength.

How to interpret it

  • Useful when you want a single scalar to monitor drift in discrimination.
  • Can be wired into simple guardrails (e.g., “alert if KS falls below 0.25”).

Alternative SQL

WITH scored AS (
  SELECT
    time_bucket(INTERVAL '5 minutes', {{timestamp_col}}) AS bucket,
    {{ground_truth}} AS label,
    {{score_col}} AS score
  FROM
    {{dataset}}
),

-- Total positives/negatives per bucket (for guards & normalization)
bucket_totals AS (
  SELECT
    bucket,
    SUM(CASE WHEN label = 1 THEN 1 ELSE 0 END)::float AS num_pos,
    SUM(CASE WHEN label = 0 THEN 1 ELSE 0 END)::float AS num_neg
  FROM
    scored
  GROUP BY
    bucket
),

-- Use each distinct score in a bucket as a threshold candidate
thresholds AS (
  SELECT DISTINCT
    bucket,
    score AS threshold
  FROM
    scored
),

-- Confusion matrix per (bucket, threshold)
metrics_per_threshold AS (
  SELECT
    t.bucket,
    t.threshold,
    SUM(
      CASE
        WHEN s.label = 1 AND s.score >= t.threshold THEN 1
        ELSE 0
      END
    )::float AS tp,
    SUM(
      CASE
        WHEN s.label = 0 AND s.score >= t.threshold THEN 1
        ELSE 0
      END
    )::float AS fp,
    SUM(
      CASE
        WHEN s.label = 0 AND s.score < t.threshold THEN 1
        ELSE 0
      END
    )::float AS tn,
    SUM(
      CASE
        WHEN s.label = 1 AND s.score < t.threshold THEN 1
        ELSE 0
      END
    )::float AS fn
  FROM
    thresholds t
    JOIN scored s
      ON s.bucket = t.bucket
  GROUP BY
    t.bucket,
    t.threshold
),

-- ROC points: TPR vs FPR for each threshold
roc_points AS (
  SELECT
    m.bucket,
    m.threshold,
    COALESCE(m.tp / NULLIF(m.tp + m.fn, 0), 0) AS tpr,
    COALESCE(m.fp / NULLIF(m.fp + m.tn, 0), 0) AS fpr
  FROM
    metrics_per_threshold m
),

-- Add (0,0) and (1,1) endpoints for each bucket
roc_with_endpoints AS (
  SELECT
    rp.bucket,
    0::float AS fpr,
    0::float AS tpr
  FROM
    roc_points rp
  GROUP BY
    rp.bucket

  UNION ALL

  SELECT
    rp.bucket,
    rp.fpr,
    rp.tpr
  FROM
    roc_points rp

  UNION ALL

  SELECT
    rp.bucket,
    1::float AS fpr,
    1::float AS tpr
  FROM
    roc_points rp
  GROUP BY
    rp.bucket
),

-- Order ROC points by FPR per bucket
roc_ordered AS (
  SELECT
    bucket,
    fpr,
    tpr,
    ROW_NUMBER() OVER (PARTITION BY bucket ORDER BY fpr) AS rn
  FROM
    roc_with_endpoints
),

-- Pair each point with the next one (for trapezoids)
roc_pairs AS (
  SELECT
    r.bucket,
    r.fpr AS fpr_curr,
    r.tpr AS tpr_curr,
    LEAD(r.fpr) OVER (PARTITION BY r.bucket ORDER BY r.rn) AS fpr_next,
    LEAD(r.tpr) OVER (PARTITION BY r.bucket ORDER BY r.rn) AS tpr_next
  FROM
    roc_ordered r
),

-- AUC via trapezoidal rule: sum over (ΔFPR * (TPR_i + TPR_{i+1}) / 2)
auc_per_bucket AS (
  SELECT
    bucket,
    COALESCE(
      SUM(
        CASE
          WHEN fpr_next IS NULL THEN 0
          ELSE (fpr_next - fpr_curr) * (tpr_next + tpr_curr) / 2.0
        END
      ),
      0
    ) AS auc_roc
  FROM
    roc_pairs
  GROUP BY
    bucket
),

-- KS statistic: max |TPR - FPR| over ROC curve
ks_per_bucket AS (
  SELECT
    bucket,
    MAX(ABS(tpr - fpr)) A

Updated 1 day ago