Multicriteria decision analysis and core values for enhancing vaccine-related decision-making

See allHide authors and affiliations

Science Translational Medicine  29 Jun 2016:
Vol. 8, Issue 345, pp. 345ps14
DOI: 10.1126/scitranslmed.aaf0756


Vaccines have the potential to transform the health of all individuals and to reduce the health inequality between rich and poor countries. However, to achieve these goals, it is no longer sufficient to prioritize vaccine development using cost-effectiveness as the sole indicator. During a symposium entitled “Mission Grand Convergence—The Role of Vaccines,” held in Siena, Italy, in July 2015, key stakeholders agreed that the prioritization of vaccine development and deployment must use multicriteria decision-making based on the following core concepts: (i) mortality and severity of the disease, (ii) vaccine safety considerations, and (iii) economic evaluation that captures the full benefits of vaccination.


During most of the 3 million years of human evolution, the average life span at birth was less than 35 years, and until a century ago, the U.S. life expectancy was less than 50 years, as children and infants often died from diseases such as diphtheria, smallpox, tuberculosis, pneumonia, tetanus, pertussis, and typhoid fever (13). Today, the average life span in the United States and Europe is more than 80 years. During this time interval, the impact of infectious diseases as a cause of death in developed countries has decreased dramatically. Historically, infectious diseases accounted for more than 50% of deaths in 1900 but currently account for fewer than 5%. Unfortunately, equivalent gains have not been reported in lower-income countries. In the year 2000, the average life spans in Africa and Southeast Asia were 50 and 63 years, respectively (4). Even within regions of the African continent, there is significant variability; for example, life expectancy at birth in Chad is 49 years and in Ghana is 67 years (5).

Along with clean water, antibiotics, and good hygiene, vaccines have accounted for this radical improvement in public health over the past century. A review of the literature cites a decrease in deaths from diphtheria, neonatal tetanus, measles, and the eradication of smallpox and polio, supporting the fact that vaccines are largely responsible for this increase in life expectancy (3, 5, 6).

On the 20th anniversary of the “Investing in Health” report published by the World Bank in 1993, an independent commission of 25 economists and public health experts reviewed the case for investment in health and subsequently developed an evidence-based investment case and framework to achieve dramatic health gains by the year 2035. A report on this work, “Global Health 2035: A World Converging Within a Generation,” was published in the journal Lancet in December 2013 (7). This report argued that not only is it feasible to substantially close the health gap between high- and low-income countries by 2035, it is also a sound economic investment. Although measures such as clean water and access to health care will play a role in achieving this goal, vaccination remains an extremely cost-effective tool to control infectious diseases and improve health outcomes. In order for this potential to be achieved by 2035, decisions regarding vaccine development, production, and introduction to the public must be aligned among manufacturers, public health policy agencies, nongovernmental organizations (NGOs), and ministries of finance. Traditional methods that estimate cost-effectiveness of an intervention emphasize the payer perspective and ignore the impact of improved health on family and national productivity. Therefore, vaccine assessment tools that accurately assess the full impact of vaccines are required to ensure that appropriate decisions are made. Here, we review economic models that support the Grand Convergence, discuss the history of vaccine decision-making to date, and propose a new assessment framework that can be used by all public-health stakeholders in developed and low- and middle-income countries (LMICs) to assess vaccines and vaccination.

A central thesis of the Lancet commission report is that it is possible and economically desirable for the level of public health in low income countries (LICs) to converge with that of industrialized countries by the year 2035. This could be accomplished through changes in fiscal policy that incentivize investment in public health infrastructure and universal access to health care, improvements in the usage of skilled birth assistance, appropriate use of oral rehydration therapy, and control of infectious diseases by further efforts to prevent and treat pneumonia, malaria, tuberculosis, HIV, and tropical diseases such as salmonella and shigella. Although vaccines will play an important role in these efforts, they must be viewed as part of the overall armamentarium of tools available to improve public health. Evidence that such a convergence is possible is provided by a specific example—the case of infant mortality in China, where, in 1850, mortality under 5 years of age was more than 300 deaths per 1000 live births (Fig. 1) (8). However, by 2015, infant mortality had fallen dramatically in China, so that it was only slightly above that seen in Sweden. Communities in China that began reaching high vaccine coverage rates in the late 1990s showed a 90% drop by 2006 in the prevalence of chronic hepatitis B virus (HBV) infection and other vaccine-preventable diseases in children under 5 (9). This decline resulted from economic improvements in China that were brought about in part by improvements in public health, thus establishing a positive feedback loop (7).

Fig. 1. Under-age-5 mortality.

Comparison of under-age-5 mortality in Sweden and China between 1751 and 2008. Credit: Adapted from (8) with permission.


Achievement of the convergence in public health by 2035 in other countries will require increased use of currently available interventions as well as development of new technology to combat TB, HIV, malaria, and other diseases that are especially prevalent in LICs such as typhoid, salmonella, shigella, and dengue. The development of new vaccines and introduction of new and existing vaccines are key to the prevention and control of these diseases. To receive the maximum benefit in a timely manner from such a vaccine program, it is important to appropriately target vaccines to develop and those to introduce.


The initial phase in the development of a vaccine is to decide which diseases present the most pressing public health need and then prioritize those diseases based on the impact of the targeted intervention. Historically, the decision to develop and test a vaccine, such as smallpox in 1796 or rabies in 1885, was made by scientists, such as physician Edward Jenner or chemist Louis Pasteur, respectively, without any formal institutional decision-making process. Now, with the developmental costs of vaccines approaching U.S. $1 billion in some cases and a more rigid and time-consuming clinical trials process, a more strategic approach is required. In the past, recommending bodies such as the U.S. National Academy of Medicine (NAM) used expert opinions to develop priority lists of target diseases, and, in 1978, NAM published the vaccine priorities for developed and developing countries (10). Two decades later, in 1998, NAM published a new list of vaccine priorities using cost-effectiveness based on quality-adjusted life years (QALY) and disability-adjusted life years (DALY) saved (11). These indicators were used to evaluate the introduction of a vaccine (12).

However, these approaches, while useful in the short term, had limitations that became apparent in subsequent years, for example, the 10-year (or more) lag time between the development of the vaccine priority list and availability of the vaccines. Because disease burden and medical needs in a specific population can change during the decade it takes for a product to come to market, some vaccines have been developed that no longer align with public health priorities. Furthermore, the development of such priority lists has not followed a transparent and explicit process. Possible criteria for inclusion of a disease target on the NAM list could include the intrinsic morbidity and mortality of the disease, the disease incidence, and a manufacturer’s perspective on the feasibility and potential profitability of a vaccine that prevents the disease. From the purchasers’ perspective, a vaccine that saves the health care system money might be preferred over a vaccine that prevents a rare disease with high morbidity and mortality. Historically, varying criteria were used by decision-makers in manufacturing, finance, and public health, which led to a disconnect between manufacturers’ priorities and those of public health agencies and purchasers. This lack of consensus introduced inefficiencies into the process, leading to the development of vaccines for which the market no longer existed. This situation has led to a call by many people in the vaccine field for a consensus on which evaluation criteria should be used, and to have these selection criteria reflect the evidence-based benefits of a vaccine.


Recent decisions regarding vaccine introduction and use have moved beyond consideration of the public health impact to include a health economic evaluation. The latter has been problematic for new vaccines for which the broad impact is often not measurable and for which overly conservative assumptions can lead to rejection of a beneficial vaccine. In addition, most health economic evaluations of vaccines have suffered from an overly restrictive approach when considering vaccine benefit.

In the United Kingdom, the tension in decision-making is evident, as seen in the meningococcus B vaccine example in which cost-effectiveness was considered only from the perspective of the payer and not from the perspective of benefits to the family or to an individual’s productivity. Recently, a petition to include, in the health service’s recommendations, meningococcus B vaccination in 1- to 11-year-old children was signed by 820,000 people (the largest such petition in the United Kingdom); this public preference was ignored after a cost-effectiveness analysis (13). To address the need for a more realistic assessment, Bloom et al. (14), among others, have proposed a further encompassing approach to evaluating the benefit of vaccines. Traditional health economic evaluations of vaccines take into account the health care system cost savings and increased productivity in the recipient. Bloom et al. argue that a comprehensive approach should include outcome-related productivity gains due to lack of disability, behavior-related productivity gains, community externalities, and the utilitarian value of health gains (14).

Bloom et al. also state that narrow approaches to the evaluation of vaccines have caused decision-makers to undervalue them as health care interventions because neither of these analyses accounts for the full economic benefit (14). Vaccination not only protects individuals against a specific illness but also protects against the long-term effects of that illness on their physical, emotional, and cognitive development. Disease sequelae, such as stunting growth or induced disability, can reduce future economic productivity. This is especially true in developing countries, where inability to perform manual labor affects people’s ability to support themselves and their families and to contribute to the growth of their communities. Thus, diseases such as measles not only cause cognitive disabilities directly but can also result in long-term effects such as malnutrition. Bloom et al. also argue that the importance of these factors has led to a recognition that improvements in health lead to improvements in economic growth not only at the individual level but at the national level as well—and further, that improvements in health as a result of vaccine interventions lead to economic gains to the society as a whole as measured by increased productivity, smaller family size, and improved education. Improvements in the economic status of a society, in turn, make available more funding for health care and health-related initiatives through the development and introduction of new interventions that provide further economic benefit to society. It is, indeed, this feedback loop that argues for the importance of an investment in health and, specifically, in the Grand Convergence effort (7, 14). By adopting this holistic approach, as shown in Table 1, to assessing vaccination, scientists and policy-makers can make appropriate choices regarding the development and introduction of a vaccine.

Table 1. Narrow and comprehensive approaches to the economic evaluation of vaccines (14).
View this table:

In support of this assessment, in September 2015, Lancet published a viewpoint called “Economists’ declaration of universal health care” led by Lawrence Summers on behalf of 267 global economists (15). The authors’ view addresses the importance of a comprehensive economic assessment to appropriate decision-making and the economic support for investment in health. The group of economists conclude that success of the next development chapter hinges on the ability to actually deliver proven health solutions to the most resource-poor populations and emphasize that universal access to basic health services, including vaccination, is key to achieving the Grand Convergence target. Such evaluation should include the impact of chronic as well as acute disease. The impact of chronic conditions such as hepatitis B, hepatitis C, and papillomavirus infections often has not been included in vaccine evaluation. By including the full “broad” impact of vaccines, they can more fairly be compared to other possible public health interventions such as clean water or mosquito control.


As is well documented by Bloom et al., vaccine decisions based on a single criterion such as cost-effectiveness cannot take into account the many benefits that such an intervention brings to individuals and society. Recently, multicriteria decision-making has been successfully applied to complex problems such as climate change, military actions, and urban traffic. In these analyses, multiple attributes are systematically evaluated to increase the likelihood of making a correct decision over that made from a single-parameter analysis (15). A working group of NAM has proposed the use of a multicriteria decision-making process for vaccines and developed a software tool—Strategic Multi-Attribute Ranking Tool (SMART) for Vaccines—with which to prioritize vaccine interventions. NAM also proposed 28 vaccine attributes divided into eight categories to be used with the SMART tool (16, 17). This approach allows the user to be explicit with regards to selecting a disease target or in recommending a vaccine for use. From the public’s perspective, this approach renders the decision-making process transparent. Although the SMART vaccine tool appears to be helpful, NAM invites users to bring their own perspective to the decision process, which does not foster alignment between different stakeholders (18). The availability of a multicriteria ranking tool and the urgency to align stakeholders so as to improve vaccine decision-making prompted the convening of a conference of potential stakeholders to debate the topic. Participants gathered in Siena, Italy, including leaders in public policy and health economics; representatives from vaccine-related NGOs such as the Bill and Melinda Gates Foundation, PATH, and Global Alliance for Vaccine and Immunization (GAVI); academic and industry scientists; and vaccine manufacturers.

The two key questions discussed at the meeting were (i) whether multicriteria decision- making should replace the use of a single criterion such as cost-effectiveness and (ii) whether the diverse set of stakeholders could begin with the 28 criteria proposed by NAM and reach a consensus on a subset of these criteria, or core values, that would be deemed necessary to include universally in all vaccine decisions. The workshop participants agreed that multicriteria decision-making should be used in the future for vaccine evaluations. The path to the development of a set of core values is described below.


To facilitate the discussion of vaccine attributes, the full set of 28 NAM criteria was reduced to 21 criteria through a preconference survey of participants that used hypothetical vaccine case studies. At the conference, attendees were divided into working groups that included vaccine evaluation in LMICs, in developed countries, and for emerging infections as well as a manufacturers’ working group. Using the 21 SMART Vaccine criteria as a starting point, each group was asked to identify three to five core criteria that they felt should be used to evaluate vaccines for development or introduction and implementation. Group leaders then met to develop a consensus across the working groups.

These groups recognized that the evaluation criteria needed to be clearly defined in order to reduce overlap and that stakeholders might want to include other evaluation criteria depending on their frame of reference, such as pandemic potential for emerging infections or the existence of a clear regulatory and development path for manufacturers. The three core values defined by this consensus were the mortality and severity of the disease, vaccine safety considerations, and an economic evaluation that captured the full benefits of vaccination, and it was the consensus of the group that the consistent use of the three core values would improve efficiency in consideration of new vaccines for development and their introduction. However, these criteria were not meant to be exclusive. Decision-making processes in developing countries, for emerging infections, or for manufacturers might make use of additional criteria as discussed within the working groups at the conference. A schematic of how these additional criteria might relate to the core criteria is shown in Fig. 2.

Fig. 2. Core values and multicriteria decision-making for vaccine development and deployment.

Additional attributes to be considered for other stakeholders or conditions (developed countries, low- and middle-income countries, emerging infections, and manufacturers) are shown surrounding the core values.



We have a unique opportunity over the next 20 years to greatly improve public health in LMICs globally. Improvement in the public health status of these countries is not only morally attractive but also economically desirable because it will drive economic development both locally and globally. Vaccines can play an important role in this effort, but, in order to achieve this potential, we need to align the processes by which we make decisions. Multicriteria decision-making using a core value set of criteria is a comprehensive and fair process and offers the potential for alignment of appropriate decisions regarding vaccine development and introduction. We must now move forward to ensure that, in the future, vaccines are available to achieve their full potential to improve public health worldwide.

The overall goal of our proposal is to align public and private stakeholders, including regulatory agencies, recommending bodies, funders of basic and applied research, and industry, behind the same priorities and objectives so that the global society can derive the sustainable health benefits that vaccines can provide. To reach this goal, meeting participants identified the following key actions: (i) the list of vaccine attributes proposed by NAM should be revised—eliminating duplications, redundancies, and inconsistencies—to generate a list containing less than 10 universally agreed-on and well-defined attributes; (ii) the economic evaluation of vaccines should evolve to make sure it captures the full broad benefits of vaccination (in a provocative way, why should we not include the value of increased life expectancy in the vaccine value?); (iii) make a list of vaccine priorities for all countries, using a multicriteria decision-making system (SMART Vaccines or a modification of it); (iv) generate global epidemiology data to populate the databases that will be used to make decisions; and (v) find new ways to mitigate the risks that innovative vaccine manufacturers take—for example, by working on ways to compensate investments if commonly agreed-on priorities change over time.


  1. Funding: This work was in part supported by the European Union Seventh Framework Program project “ADITEC” (Grant HEALTH-F4-2011-280873).
View Abstract

Stay Connected to Science Translational Medicine

Navigate This Article