HIV Articles  
Low Bone Mineral Density, Renal Dysfunction, and Fracture Risk in HIV Infection: A Cross-Sectional Study
  The Journal of Infectious Diseases Dec 1 2009;200:1746-1754
Alexandra Calmy,1,2,5 Christoph A. Fux,6 Richard Norris,1,2 Nathalie Vallier,5 Cécile Delhumeau,5 Katherine Samaras,3 Karl Hesse,1,2 Bernard Hirschel,5 David A. Cooper,1,2,4 and Andrew Carr1,2
1HIV, Immunology and Infectious Diseases Unit and 2Centre for Applied Medical Research, St Vincent's Hospital, 3Garvan Institute for Medical Research, and 4National Centre in HIV Epidemiology and Clinical Research, University of New South Wales, Sydney, Australia; 5HIV Unit, Geneva University Hospital, Geneva, and 6University Clinic for Infectious Diseases and University of Berne, Switzerland
"Taken together, these observations argue for using a FRAX score computed without BMD only as a screening tool in all HIV-positive patients with no indications for DXA scanning. Given our findings and published data, BMD measurement may be appropriate for HIV-positive postmenopausal women and men >50 years of age, all HIV-positive patients with documented hypogonadism, and bPI and/or tenofovir recipients."...."The use of a tool such as the WHO FRAX tool warrants further validation studies in HIV-infected patients."

Background. Reduced bone mineral density (BMD) is common in adults infected with human immunodeficiency virus (HIV). The role of proximal renal tubular dysfunction (PRTD) and alterations in bone metabolism in HIV-related low BMD are incompletely understood.
Methods. We quantified BMD (dual-energy x-ray absorptiometry), blood and urinary markers of bone metabolism and renal function, and risk factors for low BMD (hip or spine T score, -1 or less) in an ambulatory care setting. We determined factors associated with low BMD and calculated 10-year fracture risks using the World Health Organization FRAX equation.
Results. We studied 153 adults (98% men; median age, 48 years; median body mass index, 24.5; 67 [44%] were receiving tenofovir, 81 [53%] were receiving a boosted protease inhibitor [PI]). Sixty-five participants (42%) had low BMD, and 11 (7%) had PRTD. PI therapy was associated with low BMD in multivariable analysis (odds ratio, 2.69; 95% confidence interval, 1.09-6.63). Tenofovir use was associated with increased osteoblast and osteoclast activity (p<.002). The mean estimated 10-year risks were 1.2% for hip fracture and 5.4% for any major osteoporotic fracture.
Conclusions. In this mostly male population, low BMD was significantly associated with PI therapy. Tenofovir recipients showed evidence of increased bone turnover. Measurement of BMD and estimation of fracture risk may be warranted in treated HIV-infected adults.
Low bone mineral density (BMD), including premature osteopenia and osteoporosis, is common in persons infected with human immunodeficiency virus (HIV) [1-5]. A review of cross-sectional studies found that HIV-infected adults had a 6.4-fold increased odds ratio (OR) of osteopenia and a 3.7-fold increased OR of osteoporosis compared with uninfected controls [1]. "Classic" risk factors identified were low body mass index, weight loss, corticosteroid use, and smoking, together with the duration of HIV infection [2, 6].
Dynamics in BMD depend on the balance between osteolytic activities of osteoclasts and regenerative activities of osteoblasts. Reductions in BMD directly correlate with the risk of bone fractures. Every reduction of 1 standard deviation (SD) in vertebral BMD, for example, resulted in a 2-fold increased risk of vertebral fracture [7]. The mechanism of low BMD in HIV-infected adults is uncertain, as are the relative effects of classic risk factors, HIV itself, and specific antiretroviral therapies on BMD and fracture risk in treated HIV-infected patients.
Particular drugs and drug classes have been associated with low BMD. Tenofovir (TDF) is a nucleotide analogue reverse-transcriptase inhibitor shown to reduce BMD [8, 9]. Low BMD has been attributed to the use of HIV protease inhibitors (PIs), but other antiretrovirals have also been implicated [10, 11]. Studies published to date have not evaluated the roles of specific antiretrovirals in the context of a broader examination of classic risk factors.
Tenofovir can induce proximal renal tubular dysfunction (PRTD). This specific reabsorption defect of glomerular filtration products can result in excessive renal phosphate, uric acid, and bicarbonate losses, as well as proteinuria and glucosuria, particularly in patients with preexisting nephropathy [12]. PRTD might promote loss of BMD through renal phosphate wasting [13].
The risk of a fracture in an individual patient not only depends on BMD but is also associated with numerous other factors, including age, sex, alcohol use, and smoking. The World Health Organization (WHO) recently issued the FRAX equation to calculate the 10-year risk of fracture based on key risk factors [14]. We hypothesized that the comparison between FRAX- and BMD-derived fracture risks may provide better insight into the significance of low BMD seen in the context of long-term antiretroviral treatment (ART) and may help identify HIV-infected patients at greater risk of fracture at any given BMD value.
In the current study, we determined the prevalence of low BMD and its relationship with numerous potential risk factors, including PRTD, tenofovir, and PI therapy, in a cohort of HIV-infected adults receiving combined antiretroviral treatment. We also estimated the 10-year fracture risk.
In agreement with other studies, we found low BMD to be common in HIV-infected men, with 47% of patients having WHO-defined osteopenia or osteoporosis [1]. Use of a bPI was independently associated with low BMD. Few studies have looked at BMD after adjustment for HIV-independent risk factors, HIV-related parameters, and antiretroviral treatment characteristics. A meta-analysis of 12 cross-sectional studies calculated a pooled OR for low BMD of 1.57 for PI-treated versus PI-untreated patients [17]. However, concomitant disease and treatment variables were not evaluated. Recently, osteopenia was found to be more common in premenopausal HIV-infected women receiving PI-based therapy (17%) than in premenopausal, uninfected women (7%) [18]. In the Aquitaine cohort an association was reported between BMD and nadir CD4+ cell count in women, but, again, no adjustment for known risk factors was performed [19].
In contrast, we assessed the risk for low BMD adjusted for a large range of classic, HIV-related, and antiretroviral treatment variables, including PRTD. Only bPIs and low testosterone remained significantly associated with low BMD in multivariable analysis, suggesting a causative role for PIs in the pathogenesis of osteopenia in patients with stable, mostly virologically suppressed HIV disease. With <10% of our study population having severe immunosuppression (CD4+ cell count, <200 cells/µL), and 83% having suppressed viremia, it is difficult to conclude that HIV replication and disease severity are important contributors to the observed low BMD in our population. PI-associated loss of BMD may be due to altered 1,25-dihydroxyvitamin D3 production [20] or stimulated osteoclast or inhibited osteoblast activity [21, 22]. Although low BMD is also found in untreated HIV-infected individuals, a randomized prospective study demonstrated greater BMD loss with continuous antiretroviral treatment than with intermittent antiretroviral treatment [23]. A recent study showed a median decrease of 4.1% in lumbar spine BMD and 2.7% in hip BMD 48 weeks after antiretroviral treatment initiation, with greater decrements in the PI-treated patients [24]. Another study examined the change in total BMD in HIV-infected persons randomized to efavirenz or lopinavir-ritonavir. There was an average 2.4% loss in BMD during a period of 96 weeks, regardless of treatment. Interestingly, the switch to bPI monotherapy after 24 weeks was not associated with improvement in BMD [25].
The effect of PIs on BMD has been controversial. Earlier cross-sectional studies suggesting a negative PI effect were not adjusted for HIV-independent and HIV-related confounding factors. Moreover, individual PIs may have differential effects on BMD. Our results reinforce the possible role of bPI in reductions in BMD [26].
Although T and Z scores for both hip and spine were consistently lower and osteopenia more prevalent in patients receiving tenofovir-based regimens, we could not demonstrate a statistically significant association between current or cumulative tenofovir use and osteopenia or osteoporosis. The lack of significance despite this consistent pattern may be due to the relative short median tenofovir exposure time (28 months), which is only approximately one-half the median PI exposure time. Thus, exposure time may have been too short to result in significant quantitative differences in BMD. This concern is nourished by the significant higher osteoblast and osteoclast activity in tenofovir recipients, which, together with a trend for increased PTH levels, might indicate developing osteomalacia. To clarify these concerns, long-term follow-up data are needed.
Tenofovir has been associated with renal tubular toxicity and subsequent renal phosphate wasting [27]. Renal phosphate wasting may lead to increased bone turnover and hence elevated serum ALP. Significant tenofovir-related increases in ALP were identified after the initiation of tenofovir-based antiretroviral treatment but not tenofovir-sparing regimens in both treatment-naive and treatment-experienced patients [28].
Eleven patients (7.2%) had PRTD, which did not correlate with low BMD in multivariable analysis. The discriminatory power of the study for PRTD was limited by the lack of information on proteinuria (rather than albuminuria) or specific markers for tubular proteinuria. However, excessive phosphaturia in the fasting state and in the absence of vitamin D or PTH disturbances—as documented in our patients—is considered highly specific for proximal renal tubulopathy. Furthermore, HIV-associated nephropathy and diabetes mellitus, 2 main causes of nontubular proteinuria, were not evident in our patients. We therefore believe that PRTD truly represents tubolopathy. When we alternatively defined PRTD solely on the basis of impaired fractional tubular resorption of phosphorus in a post hoc analysis, we identified 27 patients (17.6%); the fact that most of them had no hyperparathyroidism suggests that a mild form of tubulopathy was present in these patients. Again, there was no association between osteopenia and alterations in bone metabolism (data not shown).
In the current era, with a growing proportion of HIV-infected persons aged >50 years, bone health is becoming a more important comorbid factor. It is unclear whether the high rates of osteopenia in men <50 years old will translate into increased fracture rates after an additional 10-20 years of antiretroviral treatment. A very large cohort study recently reported fracture prevalence to be >60% greater in HIV-infected adults than in HIV-uninfected adults [29]. BMD screening may be even more relevant as effective therapies become available. For example, the use of intravenous zoledronate appears to be a well tolerated and effective therapy for HIV-associated bone loss [30, 31]. Although the National US Osteoporosis Foundation does not recommend BMD screening for all patients with HIV, it explicitly states that postmenopausal women and men >50 years of age should be considered for BMD testing if the risk factor profile suggests cause for concern [32].
To improve the ability to predict subsequent fragility fracture in our patients, we used the WHO FRAX equation [14]. There are different recommendations for defining the threshold at which antiresorptive treatment is recommended. British guidelines recommend using a threshold based on age—that is, 7.5% for a man 45 years of age; this recommendation could translate into treatment for up to 16% of our HIV-infected population, significantly more than the 4.3% identified by documented osteoporosis only [16]. The National Osteoporosis Foundation recommendations suggest treating only patients with a risk for major osteoporotic fracture above a threshold of 20% at 10 years, meaning that only one-half of the patients (2.2% vs 4.3%) with osteoporosis would be identified [32]. In a relatively young population such as ours, with well-identified nonclassic risk factors (chronic disease, antiretroviral therapy), use of an age-dependent threshold as in the British recommendations may be more appropriate.
The FRAX score computed without BMD seems unable to discriminate adequately between patients with and those without osteopenia, and its guidance on when to initiate antiresorptive therapy is highly dependent on the chosen threshold, with such therapy recommended for 2.2% of our population at a 20% ten-year risk of major osteoporotic fracture, and 16% at a 7.5% ten-year risk.
Considering that the FRAX score includes only classic, HIV-independent risk factors and that HIV positivity and treatment have been associated with lower BMD, the score provides a very conservative fracture risk estimate for HIV-positive populations. Similar to rheumatoid arthritis, HIV infection promotes a chronic inflammatory state that may turn out to be an independent risk factor for bone fracture to be included in a FRAX-like score.
Taken together, these observations argue for using a FRAX score computed without BMD only as a screening tool in all HIV-positive patients with no indications for DXA scanning. Given our findings and published data, BMD measurement may be appropriate for HIV-positive postmenopausal women and men >50 years of age, all HIV-positive patients with documented hypogonadism, and bPI and/or tenofovir recipients.
This study has several limitations resulting from its cross-sectional nature. In particular, antiretroviral treatment regimens had not been chosen randomly, and drug-independent effects on BMD or bone metabolism may therefore have been falsely attributed to bPI or tenofovir treatment. The effect of insufficient vitamin D or testosterone levels as well as low body mass index may be obscured by the low rate of pathologic values found in the study population. Moreover, the use of the FRAX tool has not been validated for young HIV-positive individuals or for Australians. Given the epidemiology in Australia, our study results apply only to HIV-positive male patients and therefore cannot be generalized to women.
In conclusion, we found a high prevalence of low BMD in HIV-infected adults receiving combination antiretroviral therapy, particularly in those receiving a bPI. The use of a tool such as the WHO FRAX tool warrants further validation studies in HIV-infected patients.
Patients.The 153 participants were mostly men with long-standing HIV infection and a high rate of lipodystrophy (Table 1). Viral replication was undetectable in 127 patients (83%). Sixty-seven participants (44%) were currently receiving tenofovir, 81 (53%) were receiving a bPI, and 40 (26%) were receiving both tenofovir and a bPI. Eighteen patients (12%) had low plasma 25-hydroxyvitamin D levels (<35 nmol/L).



NOTE. Data are no. (%) of patients or median (interquartile range). ALP, alkaline phosphatase; cGFR, calculated glomerular filtration rate; HIV, human immunodeficiency virus; IQR, interquartile range; NNRTI, nonnucleoside reverse- transcriptase inhibitor; PI, protease inhibitor; PTH, parathyroid hormone. a Calculated as weight in kilograms divided by the square of height in meters
Prevalence of low BMD.
Sixty-five (42%) patients had low BMD (osteopenia or osteoporosis), 6 (4%) had osteoporosis, and 9 (6%) had a Z score less than -2.
Prevalence of PRTD.
Eleven patients (7.2%) had PRTD, all with impaired fractional tubular resorption of phosphorus and albuminuria; 1 patient also had glucosuria. No patient with PRTD had a low plasma bicarbonate level. When we alternatively defined PRTD solely on the basis of impaired fractional tubular resorption of phosphorus in a post hoc analysis, we identified 27 patients (17.6%).
PRTD was associated with a longer duration of HIV infection and longer exposure to tenofovir (with 8 of the 11 patients with PRTD exposed to tenofovir for >2 years) or bPIs (Table 2). BMD was slightly lower and cGFR significantly decreased in patients with PRTD (p=.002). Seven of the 11 patients (64%) had evidence of altered bone metabolism with either increased osteoclast activity (hydroxyproline-creatinine ratio) (6 patients [55%]), increased osteoblast activity (osteocalcin and/or bALP) (5 patients [46%]), or both (4 patients [37%]). Of the 27 patients with increased fractional excretion of phosphorus, 5 (19%) had elevated plasma PTH, 16 (59%) had evidence of altered bone metabolism, and 14 (52%) had low BMD; 18 (67%) had been exposed to tenofovir for a median of 33.5 months.
Risk factors for low BMD.
Univariable analysis revealed that patients with higher body mass index (OR, 0.87; 95% confidence interval [CI], 0.77-0.98), higher testosterone levels (OR, 0.94; 95% CI, 0.89-0.99), or higher creatinine clearance (OR, 0.99; 95% CI, 0.97-1.00) were less likely to have low BMD; any use of bPI, however, was significantly associated with low BMD (OR, 2.83; 95% CI, 1.36-5.92) (Table 3). Current tenofovir, bPI, thymidine analogue, abacavir, or nonnucleoside reverse-transcriptase inhibitor therapy; current lipodystrophy; and use of concomitant medications were not significant. In multivariable analysis, the history of any use of bPI remained significant (OR, 3.10; 95% CI, 1.30-7.21). Higher testosterone levels were also a significant protective factor (OR, 0.93; 95% CI, 0.88-0.99). Of note, "classic" risk factors, such as prior fracture, use of steroids, and alcohol consumption, were not risk factors for low BMD after adjustment for combination antiretroviral therapy exposure.
BMD and bone metabolism by antiretroviral treatment exposure.
Low BMD was consistently more frequent in patients treated with tenofovir or bPI, but differences only reached statistical significance for the Z score in bPI recipients, the spine T and Z scores in bPI recipients, and the hip Z and T scores in tenofovir recipients (Table 4). Levels of ALP (data not shown), its bone isoenzyme (bALP), osteocalcin, and urinary hydroxyproline excretion were significantly higher in individuals receiving tenofovir (p<.002), suggesting increases in both osteoblast and osteoclast activity. Interestingly, among 48 (35%) of 138 patients with elevated bALP or osteocalcin levels, 32 (68%) of 47 also had a high urinary hydroxyproline levels. Among the 52 (36%) of 144 patients with elevated urine hydroxyproline levels, 32 (63%) of 51 had elevated levels of bALP or osteocalcin. There was also a trend toward higher PTH levels in tenofovir-treated patients (p=.07). Patients currently receiving bPI were more likely to have elevated plasma osteocalcin levels (p=.004), but ALP and bALP and the hydroxyproline-creatinine ratio did not change significantly. There was no significant relationship between bPI duration (p=.194, by test for trend) or tenofovir duration (p=.731, by test for trend) and the prevalence of low BMD. Remarkably, significantly fewer patients treated with TDF showed a pathological fractional excretion of phosphate.
Ten-year estimation of fracture risk.
The FRAX score computed without BMD provided similar fracture risks for patients with normal BMD and those with low BMD (Table 5). The inclusion of BMD data in the equation significantly increased the calculated risk of fractures in patients with osteopenia, whereas it significantly reduced fracture risks of patients with normal BMD. The mean 10-year risk of fracture of the whole study population estimated by the FRAX equation (computed with the BMD) was 1.2% for hip fracture and 5.4% for major osteoporotic fracture.
Twenty-two (15.8%) of 139 patients had a 10-year probability of a major osteoporotic fracture of >7.5% (the threshold at which bisphosphonate therapy is considered to be cost-effective [16]), and only 3 (2.2%) had a 10-year probability of major osteoporotic fracture of >20%. We compared the characteristics of patients with a FRAX score above the 7.5% threshold with those of the rest of the study population. We did not find any significant difference between these 2 groups with regard to risk factors, demographic characteristics, renal function, or duration of antiretroviral treatment by class (data not shown).


NOTE. Data are no. (%) of patients or median (interquartile ranges). Statistically significant P values are shown in boldface font. BALP, bone alkaline phosphatase; bPI, boosted protease inhibitor; cGFR, calculated glomerular filtration rate; HIV, human immunodeficiency virus; NNRTI, nonnucleoside reverse-transcriptase inhibitor; PTH, PI, protease inhibitor; PTH, parathyroid hormone.
aCalculated as weight in kilograms divided by the square of height in meters bPercentages were calculated using the number of patients with available data as the denominator.


NOTE. Statistically significant P values are shown in boldface font. BPI, boosted PI; cGFR, calculated glomerular filtration rate; CI, confidence interval; NNRTI, nonnucleoside reverse-transcriptase inhibitor; OR, odds radio; PRTD, proximal renal tubular dysfunction.
aAnalyses have been adjusted for age and PRTD.
Table 4. Patient Characteristics, According to Current Tenofovir (TDF) or Boosted Protease Inhibitor (bPI) Exposure


NOTE.Data are no. (%) of patients or median (interquartile ranges). Statistically significant P values are shown in boldface font. ALP, alkaline phosphatase; BMD, bone mineral density; cGFR, calculated glomerular filtration rate; HIV, human immunodeficiency virus; PRTD, proximal renal tubular disease; PTH, parathyroid hormone; SD, standard deviation.
aCalculated as weight in kilograms divided by the square of height in meters bPercentages were calculated using the number of patients with available data as the denominator.
Materials and Methods
Study design and participants.

We performed a cross-sectional analysis in a hospital outpatient-based cohort. All patients who were receiving antiretroviral treatment and attending the HIV outpatient clinic at St Vincent's Hospital (Sydney, Australia) for routine appointments between January and April 2007 were invited to participate, except for those with an active opportunistic condition. The protocol was approved by the St Vincent's Hospital Human Research Ethics Committee. All patients provided written informed consent.
The following were evaluated by means of a questionnaire administered by a study nurse or physician: patient characteristics (age, sex, and duration of HIV infection), body composition (height, weight, body mass index, fat mass percentage, and lipodystrophy), risk factors for low BMD (previous fracture, prior fracture in a first-degree relative, smoking status, corticosteroid use, alcohol consumption, and concomitant medications), and type and duration of antiretroviral treatment.
Blood samples were collected after a minimum 10-h overnight fast for determination of serum creatinine levels, liver transaminase levels, metabolic parameters (total alkaline phosphatase [ALP], lactate, glucose, lipids [total, high-density lipoprotein and low-density lipoprotein cholesterol, and triglycerides]), HIV-related parameters (CD4+ lymphocyte count and HIV load), and risk factors for bone disease (calcium, phosphate, bone-specific ALP [bALP], 25-hydroxyvitamin D, total testosterone, parathyroid hormone [PTH], and osteocalcin). Creatinine clearance was calculated using the Cockcroft Gault formula (calculated glomerular filtration rate [GFR]) and analyzed as a continuous variable. From a spot urine sample we measured albumin, creatinine, glucose, phosphate, and hydroxyproline.
25-Hydroxyvitamin D was quantified using a competitive protein-binding assay (DiaSorin), total testosterone by radioimmunoassay (RIA) (ImmunoChem double-antibody testosterone iodine 125 RIA kit), PTH by RIA (Siemens Medical Solutions Diagnostics), osteocalcin by an in-house RIA, and bALP by Tandem Ostase immunoenzymometric assay (Beckman Coulter). The respective lower limits of detection for these 5 assays were 15 nmol/L, 0.1 nmol/L, 1 pmol/L, 3 µg/L, and 0.1 µg/L.
WHO criteria were used to classify patients as having osteoporosis (hip or spine T score, -2.5 or less; ie, 2.5 SDs below the mean BMD value for young adults of the same sex and race) or osteopenia (hip or spine T score, -1 or less). Patients classified as osteopenic or osteoporotic were compared with the "normal BMD group" (hip and spine T score above -1 SD). The Z score compares the BMD with the mean BMD for individuals of the same age and sex; any Z score greater than -2 was considered to be within the normal range.
PRTD was defined by the presence of >2 of the following 4 pathologies [13]: (1) renal tubular phosphate loss, defined as a ratio of maximal reabsorption capacity (tubular phosphate) to GFR of <0.8, as determined with the normogram of Walton and Bijvoet [15], which corrects the fractional excretion of phosphate ([phosphateurine/phosphateserum]/[creatinineurine/creatinineserum]) for the respective serum phosphate level; (2) a ratio of urine albumin to urine creatinine of >2.5 mg/mmol; (3) a urine glucose level of >1 mmol/L with a fasting plasma glucose level of <7.1 mmol/L; and (4) a plasma bicarbonate level of <20 mmol/L.
When bALP, the hydroxyproline-creatinine ratio, PTH, osteocalcin, and 25-hydroxyvitamin D were used as categorical variables, the upper limits of normal were set at 20.9 µg/L, 15 µmol/mmol, 7 pmol/L, 18.0 µg/L, and 35 nmol/L, respectively. We defined a boosted PI (bPI) as any HIV PI given with a ritonavir dose of 100 or 200 mg daily.
Dual-energy x-ray absorptiometry (DXA) was performed on a GE Lunar Prodigy DXA machine (GE Healthcare; software version 7.51). The in vivo precision for the bone measurement using the DXA technique is 0.5%-1.5% at the lumbar spine.
The FRAX tool integrates clinical risk factors (age, sex, weight, height, previous fracture, parent hip fracture, current smoking, current glucocorticoid use, rheumatoid arthritis, secondary osteoporosis, and alcohol use (>3 units/day) to produce a score computed with or without BMD (T score) at the femoral neck. The FRAX algorithm outputs are the 10-year probabilities of hip fracture and of a major osteoporotic fracture.
Statistical analysis.
All statistical tests were 2-sided, with a threshold of 5%. Continuous variables are reported using medians and interquartile ranges, except when stated otherwise. Logistic regression was used to determine factors associated with low BMD. Demographic (age, duration of HIV infection in years, prior fractures, family history of fracture, smoking status, and alcohol consumption) anthropometric (weight, height, and body composition parameters), treatment-related (current use of lipid-lowering drugs, steroids, antihypertensive therapy, proton pump inhibitors, hormonal substitution or nonsteroidal anti-inflammatory drugs; current or past use of tenofovir, zidovudine, abacavir, bPI, or nonnucleoside reverse-transcriptase inhibitors), HIV-related (CD4+ lymphocyte count and HIV load) and pathophysiologically plausible biologic variables (cGFR, testosterone, PTH, and 25-hydroxyvitamin D levels) were included in a univariable analysis. All variables with p<.02 in the univariable analysis were entered in a multivariable logistic model. The model was adjusted for patient age. FRAX scores were compared between groups using a 2-sided, nonparametric Mann-Whitney U test. Statistical analysis was performed using SPSS software, version 15 (SPSS).
  iconpaperstack view older Articles   Back to Top