|
LAUNCHING SMART 23-24 JUNE 2005 UNICEF HOUSE, NY
“Unofficial” Meeting Notes Bill Weiss
SMART (Standardized Monitoring & Assessment of Relief & Transitions) is a multi-partner effort to standardize assessment during emergencies. The current emphasis has been on standard methods to calculate crude and under-five death rates, nutritional status, and food security. This meeting launched a new software tool and provided updates about other SMART activities. A summary of key points followed by more detailed notes of the sessions is provided below.
SMART OVERVIEW – ANN RALTE (USAID) NUTRITION – MIKE GOLDEN (ABERDEEN U) MORTALITY SURVEY - MUIREANN BRENNAN (CDC) FOOD SECURITY – JOHN SEAMON (FORMERLY SC UK) NUTRISURVEY SOFTWARE – J ERHARDT (CONSULTANT) SOMALIA PILOT TEST – Noreen Prendeville (FAO) OTHER COMMENTS—June 23 sessions: PANEL Q&A ON METHODOLOGY—June 23 JUNE 24 AM – RECAP OF ISSUES FROM DAY 1 (FITSUM ASSEFA, UNICEF) CONTINUATION OF Q&A PANEL FROM JUNE 23 MIXED METHOD APPROACH FOR MEASURING VULNERABILITY – TIM FRANKENBURGER (TANGO) PANEL ON DONOR/UN ACTIVITIES AND POTENTIAL LINKAGES PANEL ON CAPACITY BUILDING – BILL BERTRAND (TULANE) PANEL ON INFORMATION & COMMUNICATION TECHNOLOGY – NATHAN MORROW MODERATOR (WV) DATABASE (CE-DAT) PRESENTATION – VICENTE TERAN (CRED) WRAP-UP & RECOMMENDATIONS – BILL BERTRAND MODERATOR (TULANE) KEY POINTS:Several things were introduced at the meeting that should be formally recognized and/or integrated into courses and field activities:
1. SMART manual that standardizes how crude mortality, under-five nutritional status and food security are measured, analyzed and reported in complex human emergencies. Think of this as way of helping make sure your mortality and nutrition surveys are ‘SMART-compliant’. Use the manual as is, or adjust your existing protocols to become ‘SMART-compliant’.
2. Windows-based software for data entry, analysis and reporting, with built-in data quality assurance functions. The food security module is “work in progress”. Version 1 is available. Other versions will follow based on lessons learned.
3. Complex Emergency Database (CE-DAT). This database focuses on mortality, morbidity and nutrition information for conflict situations. It compiles and ranks available data related to health status of conflict-affected populations. CE-DAT also places special emphasis on long-term health of IDPs. CE-DAT will serve as SMART’s primary data source for trend analysis, monitoring, reporting and briefings. CE-DAT complements EM-DAT (OFDA’s international disaster database). The website is www.cred.be/cedat.
4. Training and Technical Support. SMART is planning a mechanism for this including a comprehensive training system including short-term certification courses, long-distance courses, and training that is integrated into graduate MSH, MPH and doctoral programs (Tulane leading this effort). Other planned technical support include the SMART website (smartindicators.org), virtual library, listserve and rapid response system where an expert technical group can guide and review surveys and respond to technical enquiries. SMART OVERVIEW – ANN RALTE (USAID)· Standardized Monitoring & Assessment of Relief & Transitions (SMART) · Mention of 2002 Workshop that started SMART off (smartindicators.org) · Purpose: o Standardize methods for assessing key indicators o Establish mechanism for sharing information, tools · Approach: network of experts, practioners · Sequence: methodsà training à database à research (all informed by expert panel) · Components: o Methodologies o Software Version 1 o Training & Technical Support Component by Tulane: Graduate degree, short certification course, distance ed o Database (CE-DAT) provides trend analysis in priority country o Research Agenda to upgrade methods (need people to take the lead in this) o Expert Panel · How to support SMART: o Multi-partner surveys o Send data to CEDAT, NICS o Participate in research agenda NUTRITION – MIKE GOLDEN (ABERDEEN U)· What’s new? o Manual’s emphasis on quality of measurements rather than just the statistics o Integrate manual with software better o Manual is a living document o Change precision with expected prevalence o New method for estimating death rates · What does it do? o Integrates anthropometric, mortality and food-security data together, o avoids epi & statistical jargon, o makes survey as easy as possible, o identifies errors during data entry, o examines internal structure of nutrition data produced to assess quality of the data as well as presenting the data in a standard format o produces a report in standard format that allows one to also judge quality of the data. · Contentious issues? o Random error o Cut off points for flags o No substitution of children o Handling oedema o Nomenclature for death rates o Advice for catastrophic events o Others…. · Random measurement error importance o Random error is not neutral, especially at the tails of the distribution o Small measurement errors can greatly affect precision, especially at the tails of the distribution (regression toward the mean) o Prevalence will be exaggerated when working in populations at the lower end of the distribution (from normal) o Wide or narrow standard deviations (SD---narrow from overcleaning or bias---can increase prevalence dramatically (5 fold increases in prevalence) o SMART uses a calculated prevalence (vs. counting the prevalence) when data looks bad (when not consistent with best practice measurements) · Cut-off points for flags o Desire to remove erroneous measures but keep correct measures o More than 3Z above or below the sample mean is more likely to be an error than a true measurement (only remove 1 in 1000 subject incorrectly) o Use 3Z to clean the data o Reports the numbers removed from the data set due to flags. · Calculating the prevalence when data are suspect o Reports skewness and Kurtosis of the sample & the poisson distribution of the malnourished cases in clusters o Counts and calculates prevalence and recommends the calculated values when the internal structure of the data is suspect · Absent children o Report the number (required) o SMART does not recommend substitution · Oedema o Correction for oedema may be applied, what percent of body weight is due to oedema o Re-calculate weight for height correcting for oedema o No relationship between % oedema and % wasting o Design effect for oedema is different than design effect for wasting, no relationship between clusters, different distributions · Report both z-score and % median o % median used for adolescents and used for admission of children to feeding programs o Note: z score provides a higher prevalence of wasting than % median! o Note: differences are even larger with severe malnutrition o Great resource implications between use of the two measures · Choosing a design effect o Recent study of surveys: Mean 1.6 with a range of 0.8 and 2.4 o Need to continue to study this and use best in software in the future · Corrections for clothing weight in cold climates (e.g., N Korea) o Correction must be made o Important even if increase in weight is less than a scale increment · Who is affected by wasting? o As situation worsens, older children have a high prevalence MORTALITY SURVEY - MUIREANN BRENNAN (CDC)· Sampling – what design factor? o More tentative than the nutrition side (less of an evidence base) o Differs by site and type of mortality o Manual has tables to help choose the best design factor and how this fits with sample used for nutrition survey · Recall period – how long? o Usually cannot spread out mortality period over a long period such as one or two years in an emergency o 3 month time period will probably be most typical, look at data in 3 month increments, and see how this affects sample size & precision o denominator for mortality is person days · Questionnaire – past or current household census method? o In and out migration is counted using midpoint in time provided o Past or current?? · Cause of death – verbal autopsy? o CDC experience has not been validated… this needs to be done o Less experience with deaths in people over five years · Terminology – mortality rates, death rates? o UNICEF indicator is not a rate but a ratio (live births is denominator) o Use the term “crude death rate” and “age-specific death rate” · Interpretation o In context of quality of data and the nutrition data o Needs more work in the mortality section of the manual · Challenges o Poor evidence base o Non-standardized methods of collection o Terminology disagreements o Large controversy over this… need to prepare partners for this… enable partners to defend themselves… need to work harder on this in the manual… how to we report this data to policy makers… · Current Decision: o 3 months (up to one year) recall period o Use current household census techniques with migration adjustment o No verbal autopsy at this time o Use crude and under-five death rates for terminology o Requesting comments and analysis from the field, pending more evidence base, FOOD SECURITY – JOHN SEAMON (FORMERLY SC UK)· Permit interpretation of nutritional anthropometry and mortality surveys · Designed for small scale (e.g., district level) · Provides: o Recent history of people’s food security o The way in which people’s food security will change in the future (a famine prediction method based on Sen’s 1981 theory of exchange entitlements) · Interpretation issues: o Within year variation and between year variation in nutritional status: Is what we are seeing normal or abnormal? o Where are things going? · Method: o Uses data describing household economy in a reference year – ideally not a year of plenty nor want o Simulate the impact of a change or shock on household food access and how this will change over time · Data requirements: o Household income obtained from each income source o Asset household o Seasonal pattern o From: § “typical household in each wealth group” § for each population of interest § for a defined reference year § triangulate via multiple sources (key informants such as farmers & extension workers) · Analysis: o Seasonal spreadsheet: People’s ability to access food by month is charted over time and predicted ahead · Manual is being written now for straightforward situations (not camps, conflict etc.)… Needs field testing… can the target audience do it and use it? · Macro issues with sharing the spreadsheet · Interpretation needs to be matured…
NUTRISURVEY SOFTWARE – J ERHARDT (CONSULTANT)· Plausibility report for nutrition § indicating records with a problem (+/- 3 Z in height or age) § Stem and leaf on age § Count and calculation of prevalence § Digit preferences report § Poisson distribution of clusters · Easy transport to Excel · Standard Results for HA, WA, WH o By age, gender, cluster o Auto .RTF report (with some placeholders & with some generated text, tables and charts) · Planning section for sample sizes, and cluster selection/assignment for nutrition and mortality surveys, random number generator · Training section for enumerators: each enumerator measures 10 children for weight and height (evaluate enumerators on precision and accuracy) · Mortality questionnaire has total and under 5 numbers · Crude death rate with CI, with design effect, under 5 death rate with CI · Import function for Epi Info and Dbase files · Generates standard forms for paper data entry · Double data entry checking function
SOMALIA PILOT TEST – Noreen Prendeville (FAO)· Food Security Analysis Unit/Somalia: primary source of information on food security and nutrition in Somalia · Piloted the software on two occasions, and the survey method · Only use multi-agency assessments using standard guidelines · Assessments involve many sectors · Findings: o Nutrisurvey is more user friendly, required little training, performed almost all functions considered essential in undertaking analysis of nutrition survey data. o Somalia-related components added to questionnaire o Manual: easy for technical staff; field staff had some difficulty understanding nutrition standardization test, mortality interview (finding how to ask the question in a culturally acceptable way took a long time), household sampling o Piloting: § Guidelines practical and useful § Additional time required for training § Need more clear definitions of roles for various members of survey team § Training guide required § Need more guidance on process for analysis and interpretation of the information and making recommendations § Desire mapping of results § Other nutrition info needed too… (diet, coping, selective feeding)
OTHER COMMENTS—June 23 sessions:· Golden: With calculated prevalence for nutritional status you can rely on the mean z-scores (or prevalence if SD was 1) and eventually use a smaller sample (about 100 vs. 900) o Concern for losing ability to serve other information if you have a smaller sample for the nutritional status… · Frankenburger: think of sampling for food security alongside thinking about sample size for nutritional status and mortality rates… don’t short change one for the other PANEL Q&A ON METHODOLOGY—June 23· A. Loretti presentation on issues: o Provide criteria for decisions vs. strict guidelines o Surveillance vs. surveys? When/how often surveys? o Many surveys without communities seeing any benefits o Compromises due to difficult situations: security, randomness o Peer review processes, ownership of data and access, proper use of data given complexities and limitations of methods and agendas o Where to we go from here? Peer review of methods, sustaining operational research · Muireann Brennan: o How to widen the process? Get wider feedback? o Improve the manual, implementing the manual? How do we help others to do it? · WFP person: o Data sharing, management, storage thinking… lots of data out there is not easily available… data attrition… centralized storage in public domain needed o Existing coordination mechanisms are not being used to get the message out about this · Bruce Cogill: o More household measures of food security needed o Many in the community are not involved in this to date, need to bring these in o Conflicting guidance on MUAC o Software could incorporate other things and help widen the dialogue · Bill Bertrand: o Priority setting mechanism is needed among group for issues to address § See A. Loretti’s presentation · Other comments: o Both surveys and surveillance are needed (surveys can at least calibrate surveillance data) § Care due to improper assumptions about limitations of surveillance systems § Distinguish functions of surveillance and note where surveys can help serve the functions § We can also use community level screening (active case finding) serving functions of surveillance or survey (e.g., during polio and vitamin A vaccination for example) o Multi-agency surveys really need to be the standard---it increases the chance that one of the organizations is providing services and the community will benefit from findings o Use in quick onset vs. slow onset emergencies? Appropriate for both? Quick onset (shorter time period) means much greater sample size for mortality survey.
JUNE 24 AM – RECAP OF ISSUES FROM DAY 1 (FITSUM ASSEFA, UNICEF)· Nutrition o Emphasis on training o Software assesses data quality o Implications of random error o Calculated/counted prevalence o Research design effect · Mortality o Poor evidence base/pending evidence o Need to widen consultation/circle of experts · Food Security o Not field tested o Needs expansion to special situations · Field Test in Somalia o Tested how to promoted as well as tested methods and software o Issues with making mortality questions culturally appropriate o Manual easy to follow for technical o Training guided needed o Need more guidance on analysis & interpretation o 2nd stage of cluster sampling still not standardized o ACF people said they had a similar experience · Issues for discussion o Loretti’s points (WHO points) o 2nd stage sampling questions (how to select households within clusters) o Linking causal factors (e.g., food security info) o Integration into ongoing survey/surveillance efforts o Analysis and interpretation guide---not yet in manual o Process of integrating comments from pilot into Version 2 (or 1b) o Expand the expert group o How to provide capacity for implementers o Data ownership issues o Implementation of unfinished work (FS, analysis, integration of experiences into changes) o Links with other initiatives (MDGs that use DHS or national surveys) CONTINUATION OF Q&A PANEL FROM JUNE 23· This tool is a way to triangulate mortality, nutrition, and food security information---with the idea that there is confidence in each of the data points of the triangle. If these things do not track this is a flag that something else is important. These are key outcomes with food security being also a potential causal/context factor. · This is only 1 tool in the SMART package (others yet to come): avoid mission creep with this one tool · Include guidance that serves the function of IRB/ethical review · Ownership and use of data is still an unresolved issue · Big issue is the process by which the methods get out to the field and how lessons are fed back into updates · U5 mortality and age-specific death rate are two different terms with different meanings… so SMART recommends using “under five death rate” and “crude death rate.” · Mortality survey: Census of who is in the HH at the moment, list those who joined in the recall period, and those who left during the recall period… it also provides migration rates… would be nice for academic institution to compare current vs. past census · Avoidance of EPI household sampling method (bottle) where feasible… other methods are mentioned… but jury is still out… this is a research agenda item · Avoid using 30x30 cluster sample size automatically… calculate the sample size for each survey based on expected prevalence… · Must have at least 26 clusters… just use 30… use more clusters and fewer per person if this works for you… · Calculate prevalence from the mean (vs counted prevalence) when data are suspect · Handling oedema… severely malnourished category their Weight-for-Height calculation is adjusted from data about % of body weight due to oedema fluid (varies by severity of oedema)… this is an option but not required… Default value for software is up for discussion… · For catastrophic event like Tsunami, calculate deaths per event (a case fatality ratio concept) vs. a death rate… Death rates are fine for before and after the event… do not include catastrophic events “during” a recall period. · Use 85 cm vs. 2 years for deciding to use length or height measures when doing Weight-for-Height measures in emergencies… others are still using 2 years for development settings. · In the future, there will be tables for adolescents with a % of median calculations (z scores do not work with this population).
MIXED METHOD APPROACH FOR MEASURING VULNERABILITY – TIM FRANKENBURGER (TANGO)· Rural Kassala and Red Sea State, Sudan · Getting nutrition, mortality and food security information at the same time · Objectives of study: o Indications of impending crisis? o Assess chronic structural vulnerability and poverty · Methods: o HH livelihood survey o Nutrition survey o Qualitative study (interpretation of quantitative) · Sampling: o 30 x 30 in each state for nutrition o 30 x 15 for HH livelihood o (20 days for data collection) o PPS with poor census data o Selected 40 clusters in each state for backup · Link data between the 2 surveys to look at causal factors o Which factors associated with nutritional status? o Follow up with qualitative studies · Lessons Learned o Have back up clusters ready in case communities moved o Dispersed communities means more time is needed to collect data o Difficult to avoid duplication of interview questions between instruments (demographics) o Communities cite other causes of death than the MOH o Combining methods mean they have a much more convincing story to go with the results… more believable… more likely to mobilize resources · Findings: o Literacy, high dependency rations, low income, coping strategies associated with malnutrition o Water associated with malnutrition (time, money spend, source)
PANEL ON DONOR/UN ACTIVITIES AND POTENTIAL LINKAGES· CIDA presents the humanitarian donorship initiative o Strengthening consolidated appeals process including NGOs o Includes how humanitarian needs are assessed o 23 principles and good practice for humanitarian donors (GHD Initiative) o harmonize reporting requirements to reduce management burden o Humanitarian Policy Group document (Overseas Development Institute, ODI) o GHD portal on Reliefweb o Baseline pilots in DRC & Burundi · State Department presentation: o PRM supports SMART and CE-DAT o In line with GHD Initiative (good humanitarian donors) o Link SMART with GHD o Measure progress as donors collectively is of interest… SMART can help with this because it provide key data in a standard way o Work on increasing the number of donors on board with this · DFID presentation: o Pooled funding at global and country is an option o More resources for disaster prevention o Strengthen the humanitarian coordinator (HC) o More attention for forgotten emergencies o Standard information collection o Link with MDG summit vs. G8 o Have benchmarks for donors/agencies (for system as a whole) · Impact: mortality, nutrition, food security · Output: access or coverage of services · Process: funding, speed of response, GHD indicators · Better allocation of resources, planning, and accountability · UN Millenium Project 2005 presentation o Task force 2 on Hunger o Build local and national early warning systems objective – could link to SMART along with FEWS and SHPERE o FAO, WFP members o SMART is mentioned on the task force documents · Q&A for the panel: o GHD influenced by those knowledgeable of SMART o GHD not yet linked with SMART
PANEL ON CAPACITY BUILDING AND PLANNING FOR COUNTRY IMPLEMENTATION – SUSIE VILLENEUVE (UNICEF NY), Bjorn Ljungvist (UNICEF/Ethiopia)· In Ethiopia, new health, agriculture and emergency initiatives are an opportunity for using SMART at a national level · Emergency nutrition coordination unit in MOH, another unit in Min of Ag—all potential users of SMART methods · Plan/desire to start using methods in Ethiopia in August in 2 regions… SMART implementation units… need help in terms of funding (training, vehicles, operating costs) · Think of adjusting existing systems to be SMART compliant vs. pushing the SMART “method” and pushing people to use the guidelines (criteria vs directive; standards of conformity). o Publish standards for surveys such as what needs to be reported so people can adequately judge the surveys
PANEL ON CAPACITY BUILDING – BILL BERTRAND (TULANE)· Field data collection is the source of most of the error typically · Data collectors need to see the use of the data · SMART has probably gone as far as it can without a more formal system of institutionalizing and refining… · Need to define the specific competencies (for field workers and supervisors) that are placed within training materials… what you need to know and do at all levels from field to policy/donors · Translation is always an issue with these steps · Training with mentorship is recommended model · Turn over planned for among those who are working on SMART · Just-in-time training backed up by website or digital libraries is a model · Use analogies from roll out of Epi Info (something he worked on) including human resources to respond to queries… those who act as backup · Tie into networks… PVO networks… · Train local staff to find their own resources to deal with issues… · Others…. · What about food security modules and training? o Difficulty keeping up the numbers of trained people o Pool has increased but also the demand has increased more o Shortage of people who are capable of doing this o Must focus training local folks through local colleges and training institutes and there is no guarantee that local institutions will want to get on board
PANEL ON INFORMATION & COMMUNICATION TECHNOLOGY – NATHAN MORROW MODERATOR (WV)· Dennis King (State Department Humanitarian Information Unit): o Collaboration tools development: virtual local area networks where all members can upload and download information o Common repository of documents, maps o CHIC eRoom (Countries of Humanitarian Concern) o Current overload of information: data text/mining tools to extract critical data from documents within tagged websites---captures the actual key text in the documents o Visualization as an information technology: for example, GIS that requires geo-referenced data o Need for more granular visualizations (national rates not sufficient)… better to have predictions based on sentinel sites adjusted for population density… showing within country variations.. o Attempt to get better denominator data for sub-national levels for countries with poor census data o Lessons: § Don’t over emphasize the technology---demonstrate the value with products and services § Emphasize the importance of standardized data and meta-data § Employ visualization to represent complex data and display patterns · John Becker (USAID): o 3 classes of technology § Information communications § Geospatial data § Decision support systems o Clarify relationships with classes of technology and SMART o IT stakeholders: country IT, NGO IT, donor IT policies o Information communications § Handheld data collectors § Data transmittal systems are changing rapidly § Web-based systems o Geospatial (uneven quality by country) § Remote sensing & imaging (Landsat available) § GIS (continuity in projections, continent wide system) § GPS (privacy, security issues) o Decision support systems § Where there are no vital registration systems § Monitoring & surveillance systems for health and food security § Econometric modeling (NEPAD/SAKSS) · Nathan Morrow (WV): o Problems with poor advice from consultants in costly assessments o Great need for standard guidance for consultants to protect NGOs o Better communication platforms than email needed o Better visualization tools and summarizing tools for findings
DATABASE (CE-DAT) PRESENTATION – VICENTE TERAN (CRED)· Objectives : o Key indicators for decision making o Trend based analysis · Indicators : o Crude death rates o Under five death or mortalit o Infant mortality o Maternal mortality o Global acute malnutrition o Oedema o Muac o Measles, polio, DPT coverage o Others… · Other info: o Age group o Status (IDP, refugee) o Geographic o Methodology used o Source information · Quality ranking is made (validation process) · Confidentiality is an option · 8 pilot complex emergencies so far · 630 surveys (2000 – 2005) · TAG established · CE-DAT website up · Research and analysis is being done · Trying to get data straight from the source · Tailor made analysis for field collaborators
FINAL ISSUES TO RAISE?· How to get this out to wider audience? · Food Security module to be field tested · Training guide is a priority · Finding funding for rolling out capacity building activities including secretariat and web site is a priority · Research agenda is another priority · Note that prevalence rates will drop just from better collection of data… not proof of a better situation… message for policy makers… perhaps change 10% thresholds down to something like 8% will be needed. · Analysis and interpretation guidance is still needed · Sampling guidance is needed, perhaps live at a distance from experts through communication technologies WRAP-UP & RECOMMENDATIONS – BILL BERTRAND MODERATOR (TULANE)· Make SMART system as reliable and consistent as possible · Include current knowledge for best practices · Continue process for improvement including back up support · Sustainability system (funds, human resources, institutions) · NEXT STEPS: o Pilot testing of whole package by ACF by August 2005 o Process input from this session o Put conference materials on web o Version 1.0.0 in Ethiopia by Aug/Sep 05 o Funding search from new donors and/or at upcoming key meetings such as ECOSOC in July, IRC meeting on capacity building (Gates), Cochran Foundation, G8 July meeting, via multi-agency strategy formalized and funds raised for that (unicef, WHO, WFP, etc.) o Is food security an issue? Sufficient consultation? Make it an optional module? Getting more buy-in from WFP? Agree on the need for contextual data, however… never designed to replace WFP’s large needs assessment… focus of SMART food security is on context for mortality and nutrition findings… different instruments and audiences o Wider buy-in from other agencies like UNHCR who arrange for surveys is critical and this process needs to be managed… Good results that triangulates will help persuade other agencies |