Assessing Biases in the Evaluation of Classification Assays for HIV Infection Recency

Identifying recent HIV infection cases has important public health and clinical implications. It is essential for estimating incidence rates to monitor epidemic trends and evaluate the effectiveness of interventions. Detecting recent cases is also important for HIV prevention given the crucial role that recently infected individuals play in disease transmission, and because early treatment onset can improve the clinical outlook of patients while reducing transmission risk. Critical to this enterprise is the development and proper assessment of accurate classification assays that, based on cross-sectional samples of viral sequences, help determine infection recency status. 

In this work we assess some of the biases present in the evaluation of HIV recency classification algorithms that rely on measures of within-host viral diversity. Particularly, we examine how the time since infection (TSI) distribution of the infected subjects from which viral samples are drawn affect performance metrics (e.g., area under the ROC curve, sensitivity, specificity, accuracy and precision), potentially leading to misguided conclusions about the efficacy of classification assays. 

By comparing the performance of a given HIV recency assay using six different TSI distributions (four simulated TSI distributions representing different epidemic scenarios, and two empirical TSI distributions), we show that conclusions about the overall efficacy of the assay depend critically on properties of the TSI distribution. Moreover, we demonstrate that an assay with high overall classification accuracy, mainly due to properly sorting members of the well-represented groups in the validation dataset, can still perform notoriously poorly when sorting members of the less represented groups. This is an inherent issue of classification and diagnostics procedures that is often underappreciated. Thus, this work underscores the importance of acknowledging and properly addressing evaluation biases when proposing new HIV recency assays.

Below:  Empirical time since infection distributions of two available datasets.

On the left, D228 represents 228 samples (from 42 subjects) with recent infection of subtype C in Botswana from 2004 to 2008. Subjects were followed longitudinally for no more than 755 days [17]. On the right, D561 represents a meta database (freely available at Los Alamos HIV public database; accessed August 2014) of 561 samples (from 462 subjects) with subtype B and C. The maximum TSI is 8888 days.

Below: Hypothetical Time Since Infection distributions.

These distributions have a Beta distribution kernel with parameters aand b. They are meant to represent different epidemic scenarios (akin to those in [18]). The blue line represents the case of an “emerging” epidemic; the orange line a “waning” epidemic; the green line a “stable” epidemic; and the black line an epidemic that has been partially controlled for a period of time, but has recently resurged. The latter scenario is arguably the least likely to be found in reality, and we have included it to mimic the properties of the TSI distribution of D561 in Fig 1.

Below: (left) Expected diversity evolution for different model behaviors, as determined by parameter h in Eq (3). (right) Diversity values of model f(t) with h = 500 for two standard deviation values: u = 0.05 (purple) and u = 0.2 (red). The profiles of real data commonly relate more to the case of u= 0.2.

Full article at: http://goo.gl/RnVmZc

By: 

Oscar Patterson-Lomba, Marcello Pagano

Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, United States of America

Julia W. Wu

Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, United States of America

  

More at:  https://twitter.com/hiv_insight