| Office Location 2001 Wisconsin Avenue, N.W. Washington, D.C.
Telephone: (202) 334-2934 |
June 12, 1996
Dr. Ricardo Martinez
Administrator
National Highway Traffic
Safety Administration
U.S. Department of Transportation
400 7th Street, S.W.
Room 5220
Washington, D.C. 20590
Dear Dr. Martinez:
At the request of the National Highway Traffic Safety Administration (NHTSA), the National Research Council, acting through the Transportation Research Board (TRB), convened a committee to review recent NHTSA staff estimates of the relationship of vehicle weight to highway fatality and injury risk. The review was requested because the statistical methods employed to estimate the effects are complex, and the results of the analyses have proved controversial.
The NHTSA staff's analyses of the relationship between vehicle weight and safety are linked to the current policy debate over the desirability of raising corporate average fuel economy (CAFE) standards for automobiles and light trucks. More stringent fuel economy standards would have environmental benefits by reducing motor vehicle fossil fuel use. Some analysts, however, have maintained that higher standards will have negative effects on safety by encouraging automobile manufacturers to downsize or downweight passenger vehicles--the most straightforward method of increasing fuel economy--leading to increased risk of death and injury for the occupants of smaller and lighter vehicles. The most recent and previous NHTSA analyses--as well as research by other safety experts--suggest that the risk of death and injury does increase for the occupants of smaller and lighter vehicles. However, because of the difficulty of separating the effect of changing size and weight from the many other factors that affect highway safety, there is considerable uncertainty about the estimates of changes in risk.
The primary charge of the committee was to examine the assumptions and methods used by NHTSA analysts; to summarize the strengths and weaknesses of the analyses; and to recommend, where appropriate, additional procedures or analyses that should be undertaken to improve the estimates or alternative methods to test the magnitude and direction of the estimates. A secondary task was to examine whether the higher performance levels of new vehicles, which are not controlled for in the analysis, could be confounding the results, and if so, to suggest possible approaches for estimating the potential magnitude of the effect. The committee was cognizant of the policy implications of the studies, and of the substantial physical evidence of the effects of vehicle weight or mass on fatality and injury risk. It viewed as its central task, however, an examination of the NHTSA staff analyses to determine whether a statistical relationship between vehicle weight and safety had been established and quantified.
A committee of eight experts (see attached committee roster) was composed and reviewed according to NRC procedures, and was judged to be free of potential conflicts of interest and appropriately balanced regarding point of view. The committee met in Washington, D.C., on May 2-3, 1996, to conduct its review. Before the meeting, each member received the research papers and shared their initial comments with each other through correspondence. A public working session was held in conjunction with the committee meeting on May 2 at which time NHTSA analysts presented an overview of their methodologies, outside experts discussed their own research on this topic, and other invited participants provided additional commentary (see agenda attached). In the second day of its meeting, the committee met in executive session to deliberate on its findings. This letter report, which was approved by the committee and reflects its unanimous views, received an independent review according to NRC procedures.
The committee reviewed six separate staff reports, which were provided by NHTSA. It focused most of its attention and comments on a paper by Charles Kahane on the Relationship Between Vehicle Size and Fatality Risk in MY 1985-93 Passenger Cars and Light Trucks, because this is the most ambitious, complete, and complex of the analyses, and it went the furthest toward addressing the complicating factors that determine the probability and survivability of a crash.
This letter report summarizes the committee's unanimous findings and conclusions. Two technical appendices amplify the text. Appendix A provides an overview of the methodologies used in the NHTSA staff analyses; Appendix B expands several of the technical points raised by the committee in its review.
STUDY CONTEXT
The effect of vehicle weight or mass on safety seems self-evident. As has been shown in a large number of studies, all else being equal, vehicle occupants in a crash are better off in a heavier than in a lighter vehicle. The validity of this relationship depends on several factors: (a) the effect on the individual versus the overall effect on society, (b) the effect of variables other than weight, and (c) the type of crash.
First, it is important to distinguish individual from societal outcomes. For an individual, being in a heavier car will usually afford that individual greater protection in a crash. But the heavier car will increase the injury risk for the occupants of smaller vehicles that are involved in the crash or for pedestrians or bicyclists. Thus, the net safety effects from society's perspective could be quite different than from the individual's. Throughout this letter report we focus on societal risk.
Second, the caveat "all else being equal" is rarely so in practice. Other variables besides vehicle weight, such as driver behavior, can and do have large impacts on safety, particularly if the propensity of being in a crash, as well as the crashworthiness of the vehicle during a crash, is addressed.
Finally, even if the effects of other variables like driver behavior could be held constant, the importance of vehicle weight relative to other vehicle attributes depends on the type of crash. For example, in single-vehicle rollover crashes, which account for a disproportionately large share of highway fatalities relative to their incidence, vehicle track width (specifically, narrow track width) and center-of-gravity height appear to have a stronger contributing role to crash risk and fatality outcomes than the weight of the vehicle per se. In multiple-vehicle frontal crashes, however, vehicle weight is the dominant factor affecting crash outcomes.
These factors introduce considerable complexity to the analysis. An earlier review of studies on the relationship between vehicle weight and safety found that ". . . there are no conclusive answers to the question of whether, and to what extent, safety may be compromised by improvements in fuel economy" (from decreased vehicle weight) (NRC 1992, 62). Although, the report continues, "there is likely to be a safety cost if downweighting is used to improve fuel economy (all else being equal), . . . the available information is insufficient to make a specific estimation of the impact" (NRC 1992, 63). The report concludes that the importance of the safety issue in the fuel economy debate warrants a "comprehensive study of the effects of vehicle weight and size on safety . . . examin(ing) the full range of crash severity and crash types". . ., recognizing that "such a study will not provide definitive answers regarding the future safety performance of the vehicle fleet" (NRC 1992, 63).
The Kahane analysis goes a long way toward meeting that request. In comparison to earlier studies by agency staff, as well as by outside experts, it
The Kahane analysis attempts to answer the question: If there were an equivalent reduction in vehicle weight (e.g., 100-lb.) (and corresponding changes in other size parameters) in all passenger vehicles, and all non-weight related factors (e.g., driver age and gender) stayed the same, what would be the effect on fatality risk, measured as fatalities per million vehicle years? The analysis examines historical crash data and assumes that past relationships between vehicle weight and size parameters, such as track width, wheelbase, center-of-gravity height, and structural strength, would be maintained. These assumptions are important to bear in mind in interpreting the results.
FINDINGS AND CONCLUSIONS
The following findings and conclusions refer to the Kahane analysis unless otherwise noted.
1. The NHTSA analysts' most recent estimates of vehicle weight-safety relationships address many of the deficiencies of earlier research. Large uncertainties in the estimates remain, however, that make it impossible to use this analysis to predict with a reasonable degree of precision the societal risk of vehicle downsizing or downweighting. These uncertainties are elaborated below.
2. An indication of the uncertainty associated with the estimates is needed. Two types of uncertainty are relevant: (a) uncertainty in the estimates within the model, and (b) uncertainty regarding the model itself. The first type of uncertainty can be measured by confidence intervals around the estimate of fatality increases. Kahane provides a point estimate of 322 fatalities annually per 100-lb weight reduction; a confidence interval is needed to indicate the degree of precision of this point estimate within the model. This confidence interval would reflect Kahane's two-stage estimation process, under which uncertainty in the first stage affects the results in the second stage.
The second type of uncertainty is more difficult to address. Limitations of the data and in the specification of the model introduce uncertainty in the final estimates. For example, driver age, gender, aggressiveness, safety-belt use, alcohol and drug use, the geometric and structural stiffness differences between light trucks and cars, and other factors are related to fatalities and injuries. The way that the model and the data include or do not include these factors affects the estimates from the model. The uncertainty associated with the model and the data is not addressed in the report; however, this uncertainty appears to be large and needs to be explicitly considered.
3. The precision of the estimates is particularly vulnerable to the difficulty of isolating the effect of vehicle weight from other variables that are correlated with weight (e.g., younger drivers tend to drive smaller cars). If these factors are not accurately accounted for in the analysis, their effect will be erroneously attributed to weight. Yet accounting for them is extremely difficult. Kahane's analysis attempted to account for age and gender, but did not account for safety-belt use, alcohol and drug use, aggressiveness, or other factors related to the drivers. Insofar as more aggressive drivers tend to drive smaller cars, for example, the effect of aggressiveness is incorrectly incorporated into the estimated effect of weight--such that reductions in weight appear to have a greater impact on fatalities than is in fact true. The opposite direction of bias could also occur: for example, insofar as drivers of smaller cars tend to use safety-belts more frequently than drivers of larger cars, failure to account for safety-belt use will underestimate the impact of weight reductions on fatalities.
Kahane's attempt to control for driver age and gender was hampered by the data. In particular, because of the limitations of data available, he was required to use the age and gender data from induced-exposure collisions as a proxy for driver age and gender for each category of vehicle-registration years. Furthermore, he adjusted the fatality rates for age and gender differences before relating fatality rates to weight using two different methods of adjustment. The estimated impact of weight is seven times greater when fatality rates are adjusted to age and gender using one of the adjustment methods than when using the other method. This difference points out how susceptible the estimated weight impact is to how other factors--in this case, age and gender--are handled. Factors that were not included in the analysis, like driver aggressiveness, safety-belt use, and alcohol and drug use, could affect the estimated weight effect by even larger magnitudes.
4. Conceptually it is possible to develop a model that would isolate the effect of vehicle weight on fatality and injury risk from driver characteristics. In practice, however, the task is daunting. Developing such a model would require detailed exposure data (e.g., data on miles driven or even vehicle registration years) by driver age, gender, and ideally by safety-belt use, level of alcohol and drug use, and other factors. These data are not available directly from existing data bases, and the effort and funding required to obtain them would be quite significant (i.e., millions of dollars). Moreover, it is unclear how other critical variables, such as driver aggressiveness, could be measured, although proxy measures might be developed.
5. The analysis did not distinguish the effect of weight from other design components of vehicles, such as track width. The report correctly states that the estimates are not the effect of weight per se, but rather are the estimated effect of changes in weight assuming accompanying changes in other vehicle size characteristics (e.g., track width, wheelbase, center-of-gravity height, and structural strength) that historically have been linked with changes in vehicle weight. This distinction is important to recognize when evaluating and using the estimates. In particular, the estimates do not represent the effect of changes in weight that could occur from radical changes in materials or in the design of vehicles. Nor can it include the effects of potential future improvements in safety technology. These future changes could not be predicted, but vehicle variables, such as track width and wheelbase, could have been included together with vehicle weight in the regression equations in an attempt to sort out the relative importance of these factors on safety.
6. Vehicle horsepower, which may be confounded with vehicle weight in estimating safety effects, is a good example of the difficulty of isolating the sole effect of weight. For some models, automobile manufacturers have applied fuel economy technologies to meet CAFE standards and provided consumers with vehicles with increased horsepower. Thus crashes could be occurring at higher speeds, which could explain some of the fatality risk that is currently being attributed to weight. The Kahane analysis does not control for this possible effect, nor is it easy to do so. The problem is not only separating out horsepower effects across the fleet, but also the more difficult task of separating out driver effects. Because of the analytic complexities, and especially the lack of appropriate data, the committee is unable to suggest methods for handling this issue.
7. The predicted effect of a reduction in vehicle weight on societal risk depends on how the weight reduction is actually distributed across the fleet. For example, a reduction in the weight of small cars is likely to be far more harmful than a reduction in the weight of larger cars or light trucks. The Kahane analysis provides a glimpse of this effect. He finds that a 100-lb reduction in the weight of light trucks only, would have a net positive effect on fatalities; the small reduction in protection for the light-truck occupants would be more than offset by the increase in protection for the passenger car occupants. Predicting the effect of a reduction in vehicle weight in response to raising CAFE standards is not easy because the distribution of the weight reduction, which strongly affects the outcome, is determined by the automobile manufacturers. Empirically-based simulations of plausible distributions of weight reduction, however, could be conducted.
8. NHTSA staff estimates of the effects of vehicle weight on injury risk (Partyka and Hertz) are less comprehensive than the Kahane analysis. The analyses investigating the effect on moderate to serious injury did not attempt to control for driver variables nor did any of the analyses examine crash risk, that is, the probability of being in a crash. Thus, it is not surprising that the effects of weight on injury likelihood are larger, on average, than the effects on fatality risk where greater effort was made to control for confounding variables. The reports could all benefit from a more integrated approach to the question: how does vehicle weight affect societal fatality and injury risk?
TECHNICAL AND PROCEDURAL IMPROVEMENTS
1. The committee offers several suggestions for strengthening the Kahane analysis if it is to be published. The suggestions fall into two categories: (a) technical issues and (b) presentation. Technical issues are elaborated in Appendix B and include such issues as representativeness of the induced-exposure approach, model validation, clarification of data uses, treatment of uncertainty, and sensitivity analyses. Regarding presentation, the committee believes that the analysis could be made more transparent to policy makers and researchers if the policy context were more directly addressed and the rationale for the analytic approaches used and the limitations of the data were more clearly defined.
2. The committee commends NHTSA for inviting a review of these studies by experts outside the agency. In some respects the analysis by Kahane has broken new ground. Assessing these complex risks requires quite complicated analytical procedures, and the risk estimates necessarily reflect the limitations of available data. The committee recognizes that the analytic and data issues are not simple to resolve, but it believes that thorough discussion of these issues with the broader statistics, economics, and safety data community is warranted. We recommend that the agency develop appropriate procedures for ensuring rigorous peer review of its statistical analyses on a regular basis. Many federal agencies have procedures for outside peer review of complex analyses that could be adopted or modified for use by NHTSA.
CONCLUDING REMARKS
The committee applauds the efforts of the NHTSA researchers to tackle the complex and challenging issue of the relation between vehicle weight and safety. The data sets maintained by NHTSA are a crucial part of any such analyses. Nonetheless, the committee finds itself unable to endorse the quantitative conclusions in the reports about projected highway fatalities and injuries because of the large uncertainties associated with the results--uncertainties related both to the estimates and to the choice of the analytical model used to make the estimates. Plausible arguments exist that the total predicted fatalities and injuries could be substantially less, or possibly greater, than those predicted in the report. Moreover, possible model misspecification increases the range of uncertainty around the estimates. Although confidence intervals could be estimated and sensitivity analyses conducted to provide a better handle on the robustness of the results, the complexity of the procedures used in the analysis, the ad hoc adjustments to overcome data limitations, and model-related uncertainties are likely to preclude a precise quantitative assessment of the range of uncertainty. The committee believes that it is important, however, for the NHTSA analysts to provide a sense of the range of uncertainty, using qualitative judgments where necessary, so that policy makers and researchers can properly interpret the results.
The committee thanks the agency for the opportunity to review these important analyses, and hopes that its comments are useful.
Sincerely,
D. Warner North
Chair, Committee to Review Federal Estimates of the Relationship
of Vehicle Weight to Fatality and Injury Risk
NRC. 1992. Automotive Fuel Economy: How Far Should We Go? National Academy Press. Washington, D.C.