Eliminating Childhood Lead Poisoning in New York State by 2010
VI. Evaluation Plan:
The Department of Health will take a lead role in developing and implementing an evaluation of the lead elimination plan, beginning in the first year of plan implementation. Evaluation design will follow the six-step evaluation cycle of the CDC's Framework for Program Evaluation in Public Health illustrated in Figure 2.41
Figure 2. CDC Framework for Program Evaluation in Public Health
Consistent with the CDC framework, the evaluation plan will be based on a logic model for the elimination plan. This logic model outlines the broad components of the elimination plan, including inputs, strategies, and expected short-, intermediate-, and long-term outcomes. Additional component-specific logic models (e.g., for the Healthy Neighborhoods Program expansion) will be developed to guide focused evaluation pieces, as described below.
Consistent with a Theory of Change approach, evaluation will incorporate both process and outcome components to assess progress toward fulfillment of the logic model. Initial efforts will focus on developing indicators and data sources for measuring inputs (e.g. stakeholder representation, adequacy of resources) and strategies (e.g. timely implementation of new initiatives, participation of providers, etc.). As plan components are implemented, focus will shift to measurement of short and intermediate term outcomes (e.g. changes in family knowledge/perceived benefits related to screening, timeliness of case management activities in target communities, demonstrated reduction of lead hazards in target communities, etc.). Finally blood lead levels (e.g. incidence and prevalence of elevated blood lead levels statewide and within target communities) will be monitored at regular intervals to assess success in achieving desired health outcomes.
Data to support evaluation will be drawn from multiple sources. Surveillance data collected by the DOH Childhood Lead Poisoning Prevention Program will serve as an ongoing core information source, with anticipated data quality enhancements as outlined in the strategic work plan. The new Leadtrac system, when implemented, will be powerful tool for collecting and analyzing data from local health department lead programs. Additional data will be collected as needed from a variety of primary and secondary sources, including surveys, focus groups (e.g. for pre-testing of media materials), and program data, such as monitoring reports. DOH staff will work with various stakeholders, especially local health departments and grant recipients, to ensure coordination and consistency of data collection in support of evaluation.
Evaluation design will focus on measuring progress and accomplishments of specific plan components, as well as on overall coordination of elimination plan efforts and statewide outcomes attributable to the combined effect of multiple plan components. As outlined in the strategic work plan, specific components to be evaluated individually include:
- Statewide Public Outreach Campaign — to include focus testing of new materials (formative evaluation), and both process and short-term outcome measurement of parents' knowledge, attitudes, and behaviors related to lead poisoning prevention.
- Healthy Neighborhoods Program — to include process evaluation of reach (number of target communities served) and scope (incorporation of lead-specific outreach components), as well as measurement of selected relevant outcomes, including number of dwellings assessed, number of hazards identified, lead hazard reduction activities implemented, and changes in screening behaviors and results.
- Community Coalitions — to include initially quantitative and qualitative process evaluation of coalition formation, including engagement of local partners, activities carried out, and new financial and non-financial resources obtained through coalition activities.
Evaluation findings will be summarized and shared with stakeholders, including the Lead Advisory Council, on a regular basis. Findings will be utilized within the Department of Health to refine the elimination plan, and to improve program development and implementation.