Estimate project
← All posts

Understanding the concordance index

This article explains the Concordance index, also known as the C-index.

6 min read
Machine Learningmedtech

What is the C-index for?

Biomedical machine learning frequently confronts us with the task of survival analysis, whose goal is to predict when an event – for example, disease recurrence – will happen. While it is used extensively in medicine, survival analysis also applies to monitoring machine failures or to the insurance industry. Predictive algorithms could be evaluated with L1 or L2 distance, but this is not always the most suitable metric.

Imagine that only some patients experience disease recurrence, while others might experience it later – they had not had it when the evaluation took place, and we cannot tell whether it will ever happen to them. In such a case, ground truth times exist only for patients with biochemical recurrence, yet the patients without it still tell us something about how good our algorithm is. The metric usually applied in this situation is the Concordance index, also called the C-index. It takes every case into account, both positive and negative, treating them as censored cases.

The basic concordance index covered in this article is Harrell’s concordance index, first introduced by Frank Harell in 1982 in “Evaluating the Yield of Medical Tests“. It is the most intuitive variant of the c-index; however, Harrell’s concordance index has been shown to be overly optimistic as the share of censored data grows, and it is of limited use when we care mainly about a particular time window – for example, only about outcomes within 4 years. The first problem can be tackled, for instance, with the concordance index from right-censored survival, and the second by extending the AUC ROC to a Time-dependent Area under the ROC.

The idea behind the c-index

You can view the concordance index as a measure of how well patients are ordered by when the event occurs. It quantifies a predictor’s ability to rank subjects by estimating what fraction of all comparable pairs in the dataset are ordered correctly. A positive case counts as correctly sorted if every case (positive or negative), examined one by one, that outlives the case under investigation was in fact predicted to outlive it. When computing the C-index, cases are checked in pairs, and the result is the ratio of pairs satisfying the condition to all pairs.

The C-index ranges from 0.5, corresponding to a completely random algorithm, to 1.0, which means perfect predictions. Note that the metric does not assess how precise the predicted time is, but whether patients were ordered correctly. Nor does it directly assess how well patients were split into those with and without disease recurrence. It does, however, partly account for this, since negative cases outlive positive ones. In addition, the C-index deals with the uncertainty surrounding negative cases.

A negative case may turn out to be a false negative, because the disease may recur later even though it had not at the time of evaluation. The C-index sorts cases by their last follow-up time and censors negative cases. As a result, the first comparable item in a pair comparison is always a positive case, and a positive case can only be compared with negative cases that have a longer follow-up time (negative cases with a shorter follow-up time are skipped) and with other positive cases whose time to event is longer.

Figure 1. Sort patients by ground truth time in ascending order and iterate in ascending direction. Compare every case with the cases that come after it, skipping negative cases. Increment the nominator for concordant pairs (those predicted in the right order). Use the count of all compared pairs as the denominator.

Example

An algorithm is always easier to grasp with an example.

{
   "results": [
     {
       "case_id": "case_0",
       "case_id_gt_time": 1.35,
       "case_id_gt_event": 0,
       "case_id_prediction_years_to_recurrence": 1.48
     },
     {
       "case_id": "case_1",
       "case_id_gt_time": 11.89,
       "case_id_gt_event": 1,
       "case_id_prediction_years_to_recurrence": 3.52
     },
     {
       "case_id": "case_2",
       "case_id_gt_time": 19.17,
       "case_id_gt_event": 0,
       "case_id_prediction_years_to_recurrence": 5.52
     }
   ],
   "aggregates": {
     "c_index": 1.0
   }
 }

The trained algorithm produced the results shown above. There are two negative cases (case_0 and case_2) and a single positive one (case_1). For the negative cases, the time of the last negative check is given (1.35y and 19.17y), and for the positive case, the time to recurrence (11.89y). The algorithm was applied to every case, and case_id_prediction_years_to_recurrence is reported. Years to recurrence carry no real meaning for negative cases, but to count as correctly predicted, they should exceed the positive case’s time to recurrence.

Looking at these results, we can spot something counterintuitive about the c-index score. It equals 1.0 (the best possible value), even though the negative case_0 was predicted to have a shorter case_id_prediction_years_to_recurrence than the positive case_1. This is reasonable, though, because case_0 may still become positive by the time it reaches the gt_time of case_1. Since this is unknown, we can’t penalize the algorithm for failing to predict that case_0 will overlive case_1.

The algorithm step by step:

  1. Sort every case by ground truth time in ascending order: [case_0, case_1, case_2].
  2. Go through the sorted cases from the first element onward, skipping negative elements (censored cases). Since the first element of the list is negative, we jump directly to the second one.
  3. Always compare elements moving in the ascending direction of the list. The next element is case_1, so we compare case_1 with the remaining elements. With 3 cases in the list, the only one left is case_2. We check whether case_1 has a shorter predicted time than case_2. It does, so we increment nominator += 1. With a longer list, we would go on comparing case_1 with case_3, case_4, … and so forth, adding one to the nominator for every pair that satisfies the condition. Since there are no further elements, we leave the loop.
  4. Finally, the nominator is divided by the total number of evaluated pairs. In this example, the nominator and dominator both equal 1 (there is just one comparable pair, and it satisfies the condition), so the c-index equals 1.

Visual example 1. An illustration of how the concordance index is computed.

Implementation

The c-index is easy to apply thanks to the function provided in the sksurv library.

from sksurv.metrics import concordance_index_censored

(
    cindex, 
    concordant_pairs, 
    discordant_pairs, 
    tied_risk, 
    tied_time,
) = concordance_index_censored(
    event_indicator=groundtruth_events,
    event_time=groundtruth_times,
    estimate=predicted_event_probabilities,
)

It is evident that Harrell’s concordance index generalizes to binary classification, where the probability that an event occurs is inversely correlated with the time until it occurs. For instance, if our risk model is any good, patients who developed the disease sooner should receive higher risk scores. Put differently, of two patients, the one with the higher risk score should get the disease earlier.

Conclusions

The C-index makes it possible to evaluate algorithms that predict survival. Its key advantage is the ability to deal with censored data – for example, patients whose disease has not recurred yet, or patients for whom following up on the event of interest is impossible because another event occurred. The C-index does have some drawbacks, though. It says nothing about the size of the error: how far the predicted values deviate from the ground truth values stays unknown.