HIV Articles  
Back 
 
 
D.A.D. Study: 2nd EDITORIAL, Challenges in Using Observational Studies to Evaluate Adverse Effects of Treatment
 
 
  NEJM April 2007
Michael D. Hughes, Ph.D., and Paige L. Williams, Ph.D.
Dr. Hughes is a professor and Dr. Williams a senior lecturer in the Center for Biostatistics in AIDS Research at the Harvard School of Public Health, Boston.
 
In this issue of the Journal, Friis-Moller and colleagues (pages 1723-1735) report on results from a prospective observational study involving more than 23,000 patients infected with HIV. The study, called the Data Collection on Adverse Effects of Anti-HIV Drugs (DAD) trial, identified a possible increased risk of myocardial infarction associated with exposure to protease inhibitors but not to nonnucleoside reverse-transcriptase inhibitors. Because randomized studies are rarely well powered for evaluating adverse effects of treatment, well-designed observational studies are important.1 However, such studies raise complex questions concerning both the potential confounding of risk associations and the mechanisms by which treatment might cause an adverse effect.
 
Studies that aim to evaluate the relationship between cumulative exposure to treatment for a chronic disease and the risk of an adverse event face the fundamental problem that with each additional year of treatment, other possibly relevant temporal factors - such as the patient's age, the duration of illness, the calendar period of observation, and the duration of follow-up - also increase by a year. The difficulties in separating competing temporal effects of age, period of observation, and cohort are well known in vital-status studies.2
 
A simple example illustrates how the issue might arise in studies examining treatment as a risk factor for an adverse event. Consider a scenario in which all patients begin taking drug X at the time of diagnosis and continue taking it for 4 years, after which they all switch to drug Y for 4 years. Assume that there is no true association between the risk of the adverse event and the time since diagnosis. Assume also that the risk of the event increases proportionately by 20% with each year of cumulative exposure to drug X, that this risk is not reversible when treatment is discontinued, and that the risk also increases by 20% with each subsequent year of exposure to drug Y independently of previous exposure to drug X (see diagram).
 
The effect of exposure to the two drugs results in an apparent association with the time since diagnosis in which the risk increases by 20% per year throughout the 8 years of follow-up. For a sufficiently large study, a statistical model that included the time since diagnosis as a predictor of risk would identify this "induced" association. Also, a model that included the cumulative exposures to both drug X and drug Y, but not time since diagnosis, would correctly identify the true associations with both drugs. It is not possible, however, to fit a model that simultaneously includes time since diagnosis and the cumulative exposures to drugs X and Y because these three temporal variables are collinear: the time since diagnosis is exactly equal to the sum of the cumulative exposures to drugs X and Y at every time point (e.g., at 6 years since diagnosis, the sum is 4 years for drug X and 2 years for drug Y). Thus, the true effects of the two drugs cannot be separated from the induced association with time from diagnosis.
 
The inability to disentangle true associations and induced associations when predictor variables are collinear, or strongly related, becomes more complex as the number of temporal factors increases (e.g., if advancing age or the cumulative exposure to a third drug that was used in combination with both drugs X and Y was also a risk factor). Furthermore, it is easy to be misled by models that can be fitted. For example, a model that included time since diagnosis and cumulative exposure to either drug X or drug Y (but not both) would find no drug effect because the variable for time since diagnosis would capture the induced association, leaving no apparent additional risk associated with the one drug included in the model.
 
A second major problem in analyzing observational studies of risk factors for adverse events is that these factors may also influence the choice of treatment, causing what is sometimes referred to as channeling bias.3 Confounding might occur, for example, if patients who had a greater risk of myocardial infarction because of adverse lipid levels were preferentially given nonnucleoside reverse-transcriptase inhibitors rather than protease inhibitors. There are various statistical methods for addressing this problem, including multivariable adjustment for risk factors for myocardial infarction that might affect the treatment selection, an approach investigators used in the DAD study. However, the role of risk factors in influencing the choice of first-line antiretroviral therapy or the timing of a switch to a different therapy most likely changed over calendar time, as more drugs and additional data regarding treatment-related risk factors became available. Thus, simple adjustment for risk-factor levels before the initiation of antiretroviral therapy may not entirely address potential confounding.
 
A second approach used in the DAD study to address possible channeling bias was to evaluate risk in terms of cumulative exposure to a treatment. After discontinuation of the treatment, the cumulative exposure is held constant. For example, in the diagram, cumulative exposure to drug X increases from 0 to 4 years and is then held constant at 4 years as time since diagnosis increases from 4 to 8 years. The use of this method implies that the level of risk of myocardial infarction per year of cumulative exposure to treatment is not reduced after the discontinuation of treatment. However, after therapy is stopped, lipid levels may improve to such an extent that a reduction in the risk of myocardial infarction might be anticipated, even if the possibility of some nonreversible atherosclerosis is taken into account.
 
Confounding might also have arisen because of the use of lipid-lowering therapies for dyslipidemia associated with antiretroviral therapy. If all patients who had lipid abnormalities after they began therapy with protease inhibitors subsequently started taking lipid-lowering drugs, the overall risk of myocardial infarction among recipients of protease inhibitors would probably be lower than if lipid-lowering therapies were not used. This practice would attenuate the association between exposure to protease inhibitors and the risk of myocardial infarction. Although such attenuation might suggest that the analysis ought to adjust for the use of lipid-lowering agents, the effects of such an adjustment on the quantification of risk are not necessarily predictable, because the lipid levels on which a physician would base the decision to use lipid-lowering agents are also affected by the use of protease inhibitors. Moreover, since multiple confounding variables may be operating in different directions, the quantification of risk associations becomes inherently difficult. Therefore, there is considerable advantage in demonstrating how risk associations vary as different confounding variables are considered, but the extent to which such a demonstration can be provided in a journal article is limited by space constraints.
 
The DAD study also reported on an analysis of the extent to which the increased risk of myocardial infarction associated with particular treatments can be explained by a hypothesized mechanism (i.e., altered lipid levels). The general approach, based on concepts that were developed for validating surrogate end points, is to show that adding the mechanistic marker or markers into a statistical model that relates the risk of an adverse event to treatment reduces, or ideally eliminates, the original risk association. In other words, the risk ratio per year of exposure should be reduced toward 1 when markers are also included in the model.
 
Despite the usefulness of this approach in evaluating mechanisms, its conceptual framework has limitations, particularly when treatments might have both adverse and beneficial effects on the event of interest,4 as is plausible if HIV infection might lead to cardiovascular disease. Many other factors also affect the approach, including imprecision in and the frequency of the measurement of markers. These factors tend to attenuate the extent of the treatment effect that can be explained by a marker and hence might suggest, perhaps inappropriately, that other mechanisms of action exist.
 
Clearly, there are major challenges in attributing the risk of adverse events to exposure to specific treatments. The DAD study represents a successful collaboration among various cohort-study investigators to provide a standardized evaluation of the risk of adverse events associated with specific classes of antiretroviral therapy; it should thus serve as a model for other studies. Nevertheless, important analytic complexities arise when one attempts to attribute the risk of adverse events to a specific treatment, particularly if the rarity of the adverse event limits the statistical power for investigating the complex effects of potential confounding variables and for exploring the mechanisms by which treatment might cause the event. Replication of findings in different studies, ideally using a variety of analytic methods, is important. Such replication seems to have occurred in part in the case of the DAD study. Recently reported results from another study suggest a similar association between the risk of cardiovascular events and treatment with protease inhibitors, with the caveat that cumulative exposure may have a reduced relevance after patients interrupt all antiretroviral therapy.5
 
Dr. Hughes reports being a paid member of the data and safety monitoring committee for Boehringer Ingelheim, Tibotec, and Virionyx. No other potential conflict of interest relevant to this article was reported.
 
 
 
 
  icon paper stack View Older Articles   Back to Top   www.natap.org