Download PDF Version 

Standardized Monitoring and Assessment of Relief and Transitions (SMART)

An Inter-Agency Initiative

 

Improving the Monitoring, Reporting and Evaluation of Humanitarian Assistance

__________________________________________________________________________ 

Summary

 

Technical Working Session: July 23 - 25, 2002

"Standardizing Survey Methodology"

Policy Session: July 26, 2002

"Promoting Policy & Program Priorities Based on Data"

 

Background

Standardized Monitoring and Assessment of Relief and Transitions (SMART) is an inter-agency initiative to improve the monitoring, reporting and evaluation of humanitarian assistance interventions.  The program will pilot an approach to routinely collect, analyze and disseminate information on the nutrition and mortality experience of populations served by humanitarian interventions.  The program advocates the interpretation of health and nutrition data in the context of food security and vulnerability analysis.  As a start, the program addresses the basic health and nutrition indicators that are commonly used in the acute phase of an emergency.  Other indicators for humanitarian assistance will be incrementally reviewed and added as part of the collective effort.

The program will provide implementing partners and the broader humanitarian community with a range of tools to support humanitarian program assessment, monitoring and evaluation.  A specific website[1] has been established to serve as an organized workspace and knowledge repository to be used by participating organizations.  This will be linked with or mirrored at NutritionNet, the website that is currently being used for technical discussion.  It will also be linked with ReliefWeb, and coordinated with the United Nations Office for the Coordination of Humanitarian Affairs (UN-OCHA).  The program will develop information management tools for field reporting, web-based forum for posting survey reports, and a listserve for field practitioners to have direct, immediate access to the pool of experts drawn from various organizations.

SMART is a global effort to coordinate on assessment, monitoring, and evaluation of the overall effort of the humanitarian community.  As a joint U.S. Government policy advocacy effort, the United States Agency for International Development (USAID) and the United States Department of State, Bureau of Population, Refugees and Migration (State/PRM) have consulted and promoted the use of mortality rate and nutritional status indicators to other donors and international organizations.  The Canadian International Development Agency (CIDA) is one of the other major donors collaborating on this initiative.

Workshop 

Mortality rate and nutritional status are considered the most vital, basic public health indicators of the severity of a humanitarian crisis.  These indicators are useful to assess the severity of a crisis, identify need, and prioritize resources.  They monitor the extent to which the relief system is meeting the needs of the population and thus the overall impact and performance of the relief system (and not of the performance of any single organization or intervention).  These indicators are appropriate in complex humanitarian emergencies, as the response is system-wide in various sectors from the international community.

USAID's Bureau for Policy and Program Coordination convened a workshop, July 23 - 26, 2002, hosted by the American Red Cross, Washington, D.C.[2]   The objective of the Technical Working Session, July 23 - 25, was to establish a generic, standardized methodology for assessing mortality rate and nutritional status of populations in crisis.  The Policy Session held on July 26 promoted a better understanding on how policy and program decisions are made by donors and international agencies.

This was a landmark event.  For the first time, U.S., Canadian, and European private voluntary organizations (PVOs) and non-governmental organizations (NGOs), international and United Nations organizations, universities, and donors came together to review and resolve problems with surveys conducted in emergency situations.  They sought consensus on a generic methodology to be adopted by all relief organizations.  All organizations providing emergency health and nutrition relief shared this concern and participated in the workshop.  This included representation from 45 institutions, including 20 NGO/international organizations, seven UN organizations and several other institutions (universities, donors and government).  A common methodology will facilitate data sharing for policy and program decisions.[3]   Technical assistance will ensure that data collected is valid for making decision. The methodology sought was one that was simple for field practitioners and endorsed for its technical soundness and validity by leading emergency nutrition experts, public health specialists, medical epidemiologists, and demographers who participated in the discussions.  

Conclusion

There was agreement that timely, reliable, and standardized data is essential for prioritizing humanitarian assistance for policy and program decisions.  The humanitarian community needs to act quickly when a crisis erupts to ensure that critical, real time data is available for making appropriate policy decisions.  Otherwise, the opportunity to influence policy is lost.

There was consensus that implementing partners and donors need to demonstrate that significant investments in humanitarian response to complex emergencies are effective through the use of technically sound, valid measures.  Everyone, particularly the beneficiaries have a vital interest in ensuring that all data are accurate, timely, appropriately interpreted and free from bias or manipulation.

The two measures, Crude Mortality Rate (CMR) and nutritional status of children under five, are considered the most basic, essential indicators for assessing the overall severity of population stress and for monitoring the overall effort of the humanitarian community.  The current indicator, Crude Mortality Rate (CMR), should not be changed until research findings validate that Under-5 Mortality Rate is a better alternative.  The standard nutritional status indices to be used are wasting (thinness or marasmus) and edema (kwashiokor).  Wasting is measured using weight-for-height.  Consensus was reached on a generic, standardized methodology to be used in all emergencies for assessing nutritional status.  In essence, current best-practice survey methods should not be changed until new ones have been tested and validated.

Trend analysis is recommended for determining whether a situation warrants intervention, rather than using absolute thresholds established by the international community.[4]  Nutritional status that steadily continues to deteriorate, even if below the 10 cut-off percent wasting prevalence, merits attention and appropriate interventions.  Appropriately collected surveillance data triangulated with frequent surveys, undertaken with a simple and easily repeatable methodology, are recommended to recalibrate surveillance data and monitor trends.

Mortality data collection is much more difficult than collecting data on nutritional status.  It is prone to error.  Mortality survey data needs to be triangulated with other data such as nutritional status, surveillance (incidence, program coverage), grave counting, religious authority record, mother to child ratio, and demographic profile.

Mortality data, in isolation, cannot be used to decide how and which programs should be implemented.  Since prevalence surveys provide only a snapshot in time, surveillance (incidence) data is useful to give current and ongoing information on the evolution of the crisis and the nature of the problems encountered.  Where there is no system in place for gathering and collating surveillance data then survey data has to be collected to guide programs.  Edema (kwashiokor) develops over a very short period of time, and therefore, is often missed by surveys.  It also occurs in "pockets" without relationship to wasting (marasmus) prevalence.  Incidence data is important to inform us how to plan curative services.  Edema (functional nutrient deficiency) is due to a poor quality diet that lacks antioxidants essential for humans (micronutrients, minerals).  Wasting (growth-nutrient deficiency) is usually due to both a lack of food availability and the right mixture of food types (lack of diet diversity).  

Nutritional survey data cannot be interpreted in isolation. The food security context needs to be understood to interpret nutritional survey data.  Although there is not yet an agreed method or best practice, the Household Economy Approach (Livelihood method) has worked to predict quantitatively how an event, such as crop failure or price change, is likely to affect people's ability to get food.  It gives an estimate of who will be affected, how severely they will be affected, and when they will be affected. Other methods do not give this quality of information.

A study of 228 nutritional surveys conducted in 36 countries, in circumstances of poverty, conflict, drought or famine, showed that there was no change in the spread of wasting within the population as it becomes more malnourished.  All individuals within a defined population are equally affected by famine.  This may be due to social cohesiveness in the sampled populations.  Traditional definitions of vulnerability and strategies that target selected groups from within the same geographical zone may be inappropriate and inefficient use of resources.  Who becomes malnourished may relate more closely to geographically vulnerable areas than other vulnerability indicators. This study also showed that nutritional status is not necessarily a "trailing indicator" in population terms.

Lessons learned from Sudan[5] in pilot testing mortality data collection demonstrated this is do-able for PVOs in the context of a long-term emergency response and transitional program.  The use of mortality rates, in conjunction with nutrition and other key indicators, improved program planning and implementation.  It was useful in keeping Sudan on donor agendas by being able to provide accurate information on the severity of the emergency.  Challenges and lessons learned helped shape recommendations and conclusions reached by the workshop.

The workshop called for the investment of all concerned to build capacity to improve the health and nutritional assessment of populations in crisis.  Although this is of the highest priority for policy and program decision-making with implications for how we respond to emergencies, donor funding has been limited or non-existing.  A review by the Centers for Disease Control and Prevention (CDC) of data used in the Horn of Africa over the last decade (Somalia, Ethiopia) showed that the majority of surveys had methodological problems and established best practices had not been followed. This demonstrates a major problem with training and donor coordination, the lack of appropriate analytical tools and standardized reporting requirements, and problems in disseminating current best-practice methodologies.

The workshop was successful in meeting its stated objective to agree on a standardized survey methodology.  It was also successful in establishing a broad base consensus on the use of mortality and nutritional status indicators, and the importance of ensuring data is timely and reliable for policy and program decision making.  Most importantly, the workshop established a positive environment of mutual support with relief organizations, academia, donors and other institutions willing to work together so that technically sound standards are maintained in determining the health and nutritional status of populations in crisis.  This will help save lives, and it will tell us if our consolidated efforts have had impact.   

Next Steps

An inter-agency technical support system to build capacity at all levels (including donors, implementing partners, local partners) will be needed to ensure the standardized methodology and other recommendations are implemented.  This includes the development of a "SMART Manual for Dummies" and other tools accessible to all organizations.  As requested by workshop participants, it will also include an advisory analytical service to help analyze survey data, and an independent technical advisory group to review and accredit surveys to ensure validity for policy and program decision making.  Donors were requested to coordinate so that all donor-funded programs are required to use the same generic, standardized survey methodology and reporting format, and that the data is archived in the public domain.  Data will be shared and accessible to the humanitarian community to be used only for the good of the beneficiary.

As a joint effort, organizations will apply the standardized methodology and other recommendations in pilot countries, with an immediate request from U.S. PVOs and the World Food Program (WFP) to apply them to the Southern African drought situation.  Operational research and validated studies will help guide future recommendations and modifications on the current standardized methodology and the potential use of alternative indicators to monitor the impact of the overall performance of relief assistance.  The environment of mutual support demonstrated at the workshop will continue and technical assistance provided to ensure standards are met by all participating organizations.

 

ANNEX: DETAILS OF CONCLUSIONS

 

CRUDE MORTALITY RATE (CMR)

Crude Mortality Rate (CMR) is the most significant public health indicator for all populations, particularly for societies in crisis.  It is vital that humanitarian organizations have some idea of this measure in all populations, particularly for societies and populations in crisis. 

CMR is useful for:

·         Assessing the overall severity of population under stress.

·         Informing and prioritizing resource allocation, and advocating the urgency for intervention.

·         Documenting the crisis and assessing the overall effort of the humanitarian community using trend analysis, rather than fixed thresholds.  Typically, several organizations provide the package of interventions essential to meet critical needs in emergencies.  With the inter-dependency of the humanitarian community, this measure is not appropriate for evaluating individual interventions or performance of individual implementing partners.

·         Calibrating surveillance data and triangulating information.

CMR Methodology:

·         CMR should be collected by surveillance methods wherever possible and a survey should be used to triangulate the data and where there is no reliable surveillance data.

·         When a survey is necessary, this should be combined with the anthropometric survey for nutritional status for efficiency.  This should be undertaken during the peak season when populations are likely to suffer from food shortages.

·         There is no evidence basis for recommending a particular method at the moment. Organizations should use the methods they currently use; this is usually the current household method.  Other methods should be chosen when they are thought to have advantages, for example, past household census or a combination.  Prior birth history method seems to be a promising method and needs further review.

·         A three-month recall period is recommended (the most common methodology is a retrospective recalling of deaths in the household within a period of time).  However, this can be changed where there is a clear advantage from using a different recall period.

·         The scale of surveys is usually determined by each agency's needs, but consideration should be made of the broader needs of the beneficiary and the broader humanitarian community.  Coordinated multi-agency surveys are recommended in the future and data coordination is essential at country level.  A survey too large in scale averages out vulnerable pockets; and one that is too small a scale misses pockets and does not produce data that is generalizable.  The recommendation is to connect areas based on characteristics likely to be important determinants of nutritional status, for example, staple food areas, urban/rural areas, tribal/ethnic areas, internally displaced person (IDP) and refugee/resident areas, and not necessarily administrative districts.

NUTRITIONAL STATUS OF CHILDREN UNDER FIVE

Nutritional status of children under five is an essential indicator to assess the overall severity of nutritional status in a population.  The measures to be used are wasting and edema, using Z scores (standard deviations from the reference median) for overall reporting on the severity of the crisis.  Percentage of the reference median should also be reported as it is used as the entry criteria for feeding programs.

Mid-upper arm circumference (MUAC) is useful for rapid screening of individuals referred for further assessment, but population based prevalence surveys should always use weight-for-height.  Rapid assessments appropriately often use MUAC.

Physical growth in childhood is a proxy indicator of the nutritional well being of a population.  Finding a high prevalence of malnutrition in this group indicates that the whole population needs assistance and not just the age group that has been surveyed.  However, in some situations, other groups such as the elderly may be more vulnerable and should then be included in the survey.

Anthropometry[6] (survey) is useful for:

·         Assessing the overall severity of nutritional stress in a population.

·         Informing and prioritizing resource allocation, and the urgency for intervention.

·         Advocacy and documenting the overall effort of the humanitarian community using trend analysis.

·         Designing feeding programs.

·         When surveillance/program data show there is a problem.

·         When program "coverage" is likely to have changed.

Anthropometric Methodology:

·         Simple or systematic random samples should be used where feasible; in most chaotic emergency situations, cluster sampling should be used.

·         The clusters should be selected randomly with a chance of selection proportional to the population size of the natural groups under consideration, for example, villages.

·         30 x 30 cluster is recommended unless there are data and the skill available to make assumptions necessary to calculate a different sample size.  The total sample size should be split among at least 30 clusters.

·         Include all children in the household in the sample.  Houses with no children MUST be included for CMR/demographic and other data collection, even if they cannot provide any subjects for anthropometric measurements.

·         Interpret nutritional status data in the context of food security and agro-economic data.  Food security data will help to decide where and what programs should be established.

·         Secure areas are most likely to be used for surveys.  Reports should indicate that nutritional status is likely to be worse in insecure areas that are not accessible.

Additional Data to be Collected with Surveys:

·         The recommendation is start with the minimum set of essential data. These are measles, BCG (or vitamin A) coverage.  After programs are established coverage data should also be included.  Data on micronutrient deficiencies is optional, depending on the capacity and the likelihood of an existing problem.  Access to food and food security information is essential for data analysis and interpretation, but should be collected separately via focus groups or other methods. 

·         Any other additional data should be justified for relevance and usefulness.  A checklist of options includes cause of death, morbidity, water and sanitation, infant feeding, access to health services, and child protection.

 

OTHER CONCLUSIONS

Data Collection:

·         Every survey should have clearly stated objectives.  This will help to determine methods and sample size.

·         Integrate contextual data analysis to interpret and understand the causes of malnutrition. The Household Economy Approach (Livelihood method) has been successfully used in several emergencies to predict nutritional status decline.  It has been used to intervene early with appropriate interventions that prevented malnutrition.

Analysis:

·         An advisory analytical service is needed to help implementing partners analyze data. Each survey should be reviewed and validated by an independent, inter-agency advisory group to ensure there is technical soundness in the methodology, analysis and interpretation.  There is a need to create an environment of support system so there is no fear of criticism.  Those who fail to meet standards should be provided assistance.  

·         New windows-based, user-friendly software is needed for anthropometric surveys that facilitates edema analysis.  It should alert and allow for the correction of aberrant results.

Reporting:

·         A standardized reporting format should always be used, which includes all the information needed to evaluate the quality of the survey and demonstrate that the appropriate methodology has been used.  This will ensure that all surveys provide key essential information applicable to each emergency.

·         Reports should use the definitions "wasting", "stunting", "oedamatous malnutrition".  Reports should present data on wasting and edema separately, as well as aggregated as Global Acute Malnutrition (GAM) and total severe malnutrition.  

·         Results should be provided in both Z score and percentage of median as they have different purposes.  Percent of median is useful for admission for therapeutic and supplementary feeding as this has a higher correlation with mortality risk.

·         The report and raw data should be left in the country with the statistical office or another appropriate office so that there is historical data and the ability to analyze trends over time. 

·         A central repository, such as the Refugee Nutrition Information System (RNIS), United Nations Standing Committee on Nutrition (SCN), and/or the Centre for Research on the Epidemiology of Disasters (CRED), and/or Tulane University, should also archive the raw data and reports. These data should be accessible only to those who use it to improve the provision of humanitarian assistance.

·         Donors should include in their contracts and grant agreements that donor-funded survey results and raw data should be public domain.

Operational Research:

Current survey methods should not be changed until new methods have been tested and validated. Further research is needed on the following:

·         The Under-5 Mortality Rate as a proxy indicator for Crude Mortality Rate (CMR).

·         The use of prior birth history method to determine Under-5 Mortality Rate 

·         Supported by the hypothesis that there are relatively homogenous geographical "pockets" within which all individuals suffer to a similar degree, test the methodology of calculating from the mean and standard deviation to give a more rapid, efficient and precise estimate of the extent of malnutrition than counting individuals.  Evaluate the utility of using samples to map the geo-spatial distribution of the prevalence of malnutrition and to examine whether these data correlate with other measures of vulnerability such as those used in the Household Economy Approach for food security.

·         Evaluate and progressively adapt the Afghanistan (Badghis) survey method as a general model.

·         The Household Economy Approach "lite" methodology needs to be fully developed, tested and validated.  The current method requires three weeks of intensive training followed by supervised field experience before the methodology can be used accurately. This is too cumbersome for general use.

NEXT STEPS

Technical Support

Reliable data are needed for making policy, funding and program decisions.  Funding to build capacity to collect health and nutrition data in emergencies has been limited or non-existing.  Relief organizations are interested in improving their capacity and those of local partners.  For this, a coordinated technical assistance support system is needed with all organizations having access to technical expertise, manuals, and other tools.

·         Develop the "SMART Manual for Dummies" on methodology on Crude Mortality Rate (CMR) and Nutritional Status, including guidance on how to interpret data using food security analysis.

·         Undertake training at all levels - PVO/NGO headquarters, local capacity, donors, etc. This should use multi-media training tools distributed through the website and CD-ROM.  The training should include CMR/nutrition surveys and food security data collection and analysis.

·         Develop a global technical support system, and expand the inventory of organizations and individuals with expertise who can provide assistance. This should include:

(a)     An independent, advisory analytical service to provide assistance in analysis of data.

(b)     The Technical Advisory Group (TAG) should review, provide feedback and then officially give their assessment of the technical validity of surveys.

(c)     Listserve with rapid responses to field practitioners.

 

 

Information Management

The SMART website provides a central workspace for sharing and disseminating information to the humanitarian community.  The website will be linked to the NutritionNet site and both will make available the SMART manual on methodology and related guidelines, software and other tools to assist field practitioners.  It will serve as a forum for discussion groups on various issues that relate to the health and nutrition status of populations in crisis.

·         Continue to expand and refine the SMART website.

·         Develop the listserve capability with the Technical Advisory Group and other experts willing to provide technical advice.

·         Develop a specific user-friendly windows-based software for analysis of nutrition and CMR surveys and presenting all the data that should be reported in the appropriate format. The format should also allow the examination of the internal structure of the raw data to assess the quality of the data.

·         Disseminate multi-media training tools.

·         Develop a standard reporting format for all surveys.

 

Country Pilot Studies

 

The purpose of the country studies is to apply the SMART methodology as well as accompanying recommendations and best practices highlighted during the workshop.  It will demonstrate to what extent the inter-agency, collaborative approach to planning, designing and implementing surveys and interventions is realistic.  Depending on each country, the pilot study will provide an opportunity to integrate operational research issues that need to be tested and validated.  The pilot studies will help to build local capacity, which in most cases will be through the Ministry of Health and UNICEF.  With limited time available at the workshop, except for Angola, the other proposed country studies will need further discussion.

·         Angola:  Angola is prepared to be the first pilot site. There is consensus among donors, and other partners.  A plan of action and resource needs are being prepared by UNICEF with the Ministry of Health, to be reviewed by participating organizations. 

·         Southern African Drought:  Further discussion is needed with the U.S. PVO consortium (CARE, World Vision, CRS) and WFP, and other relevant organizations. The consortium is interested in application of the SMART methodology and other recommendations.  A joint proposal for six countries is being prepared for submission to USAID.

·         Other countries:  Further discussions are needed to decide how best to move forward in the Democratic Republic of Congo (DRC) and Afghanistan.

Operational Research

Alternative methodologies presented at the workshop merit further testing and validation.  Issues that merit further research include:  the use of prior birth history to determine Under-5 Mortality Rate, the use of Under-5 Mortality Rate as an alternative indicator of the severity of the situation (replacing Crude Morality Rate), evaluating and progressively adapting the Afghanistan (Badghis) survey method as a general model, developing a simplified model of the Household Economy Approach methodology, and testing the simplified method of calculating wasting from the mean weight-for-height.

The selection of indicators, methodologies and program interventions promoted by the SMART initiative will be guided by research findings that are practical for field application. Operational research and detailed audit of programs is crucial for the humanitarian community to continually evolve and refine best practices. 

·         Develop an operational research agenda with participating organizations, focusing those areas highlighted at the workshop. To the extent possible, integrate these studies in the context of the country pilot studies.

Donor and Other Coordination

To put recommendations into practice, an investment by ALL organizations is needed.  Each organization should incorporate recommended best practices into its policy and operational guidelines, proposal requests, training activities, and reporting.  In particular, donor coordination is essential.

·         Donors who sponsored the workshop will develop a plan of action for advocacy and outreach to other donors.

·         Coordinate on funding requirements for building capacity and institutionalizing a global technical assistance support system.  Donor investment is also needed for operational research to test and validate alternative methodologies and the theoretical basis for modifying current recommended methodologies or indicators.

·         Develop standardized grant requirements that include recommendations from the workshop.  Data collection funded by donors should be public domain for the humanitarian community, using standardized methods for collecting and analysis on mortality and nutritional status, and interpreting data in the context of food security.

·         Develop common reporting on humanitarian assistance that are jointly funded and the use of a common database for monitoring the overall effort of the humanitarian community.