Findings of CHA Evaluations and Reviews

As noted above, there are a very limited number of published evaluations of CHA processes, models, or products. We summarize the findings from examples of those we located. Common themes include the importance of good data and a focus on sub-populations, and the importance of resources to allow CHA results to become actionable plans for health improvement.

CDC's Planned Approach to Community Health (PATCH) Program. Developed in the mid 1980s, PATCH was "designed to strengthen state and local health departments' capacities to plan, implement, and evaluate community- based health promotion activities targeted toward priority health problems." Though much less recent than the other CHAs described in this paper, PATCH sets the context for the early days of CHAs and perhaps a yardstick by which to measure the overall progress of assessment initiatives. PATCH involved not only assessment, but also community development and mobilization and was used by many health departments and other organizations around the United States to address both broad and targeted health issues. The steps in the PATCH process are: Mobilizing the community, collecting and organizing data, choosing health priorities, developing a comprehensive intervention plan, and evaluating PATCH. These are similar to the steps seen in more contemporary CHAs.

Two evaluations of PATCH pointed to the benefits of the assessment process for the participating communities, including enhanced organizing and data use skills, increased awareness and interest in health, networking and ability of groups and organizations to work together, and an increased number of health promotion interventions activities.

Though no longer in use, lessons from PATCH described in one review (Kreuter, 1992) show how some challenging aspects of CHAs at that time are similar to those still described today. One drawback noted was the amount of time required to collect and analyze data. Indeed, Kreuter explained that "data collection is often carried out by persons who have little or no experience and only marginal interest in the process; further, resources spent on data collection cannot be used to implement the program. Communities need systems that can routinely and efficiently gather data relevant to their prevention status. Such systems would not only facilitate but also would help to establish standard databases, thus enabling collection of comparable small-area data across divergent populations."

Additionally, PATCH challenged health departments to utilize scarce resources and devote time and money to a process directed at problems that are not definable at the outset: "With a focus on transfer of community intervention technology through states to localities, community PATCH applications do not start with an a priori health problem; they begin with community members trying to understand what their particular health problems are. Economic support is problematic in the absence of a discernable problem up front." Finally, Kreuter notes that the availability of resources for problems identified through the PATCH process likely depended on government priorities; problems identified for which there was little government support may have gone unfunded.

New York State Public Health Agenda Committee, 1998. In 1998, NYS's 58 local health departments CHAs were reviewed by a team of four or five central and regional NYSDOH evaluators using a standardized tool. Local health departments had been provided the suggested guidance and format developed by the Public Health Agenda Committee earlier that year. The criteria for evaluation included: whether and how the suggested sections in the format and guidance were addressed, presentation techniques, uses of data, priority-setting methods and profiling of community resources. The evaluators noted the following as among the key strengths of the review CHAs:

• Use of current data
• Use of charts, graphs, and other presentation aids
• Use of sub-county data
• Presenting information that was concise, easy to understand and find
• Summarizing priorities and major recommendations after each section
• Involving and expanding roles of community partners

The Committee noted the following as key weaknesses:

• Use of outdated data
• Presenting charts with no reference or explanation
• Presenting data in narrative form only
• Presenting regional data without drawing relevance to the county
• Not integrating information across the sections
• Not identifying or explaining local priorities
• Not identifying or explaining local health resources
• Not acknowledging or describing relationships with community collaborators

Finally, the Committee noted a number of opportunities for improvement:

• Locating and applying timely and comprehensive data from various sources
• Collecting, analyzing, and presenting sub-county data
• Clearly communicating and integrating local priorities across the CHA
• Identifying community assets and resources
• Linking priorities to an action plan

Washington State Assessment in Action Initiative, 2002. An evaluation of local health department CHAs in Washington State involved a participatory, qualitative evaluation of "factors that contribute to success, and develop strategies to help LHJs learn from model approaches. The project was led by Washington State's CDC-funded Assessment in Action Initiative. Success was defined as resulting in policy and programmatic action. This group also developed a logic model for community health assessment, which identified the intended short-term outcome of CHAs as changes in attitudes, awareness, and knowledge/skills regarding the use of assessment data in decisionmaking; the longer-term outcome as changes in programs, policies, and resources; and the overall goal as improved community health status.

The evaluation involved site visits to six local health departments and telephone interviews with others, aimed at gaining understanding about their CHA processes. Interviewees noted that key catalysts for CHAs include: having a champion for the CHA and having additional funds for assessment, staff capacity, state department of health support, technology and data, and community partners. Obstacles noted included: lack of time and money, resistance to change, competing priorities, lack of time and money, lack of understanding about assessment and what it can do, and lack of clear vision from the state health department. Key CHA funding sources noted included: Local Capacity Development Funds, grants, contracts, county general funds, and local funds. Factors leading to CHA sustainability included: participant belief that "assessment is an investment that leads to increased resources or improves their ability to do more with fewer resources and when communities come to view local health departments as vital partners because of their assessment capacity." The evaluation report also lists what participants viewed as key implementation characteristics of CHAs:

• Accurately describe the community and its subpopulations
• Use of quantitative and qualitative data
• Compelling presentation of data (GIS maps, fact sheets, etc.)
• Mobilization of action based on assessment data

The evaluation found that small health departments face greater obstacles to completing CHAs and therefore require more support from state health departments. Staffing can be a challenge and "the range of skills that are helpful for assessment personnel to have are too broad to be manifested in a single person." Participants indicated that the following are among the key skills of assessment staff: Experience in assessment, knowledge of epidemiology, community connections, and marketing skills.

Finally, the evaluators state their finding that "There is no one right way to conduct community health assessment" and that the main keys to success are:

• Leadership and vision
• Community as a partner
• Dedicating staff and staffing
• Commitment to assessment
• Access to data, technology, and peer learning

Baton Rouge, LA Community Health Assessment. Pearson describes the conduct and results of a 1993 health care organization-led community assessment in Baton Rouge, Louisiana. This assessment engaged a range of community representatives, including: a cancer center (which led the process), representatives from the Parish (county) Medical Society, and researchers from Louisiana State University. Pearson notes the challenges associated with a process that should ideally pull together a wide range of community members and organizations: "There is a tradition of service fragmentation and institutional competition to be overcome, not to mention adhering to a process that is sometimes seen as complicated." The assessment team collected feedback from participants around their motivation for collaborating in the process. Reasons included: Acquisition of information that could be used for each organization's own strategic planning; information collection completed with less duplication of effort; and combined costs, which also lowered individual institutions' data collection cost." They also noted that the collaborative effort might help ease perceived competition among providers in the community. This seven month CHA process included a review of existing community data and interviews and focus groups with a total of 300 community representatives. The processes resulted in not only an assessment report, but also a formal entity to drive the implementation of the action steps in the report—Baton Rouge Health Forum.

Community Health Assessment in Kansas. Curtis (2002) reports on an evaluation of CHAs in Kansas that aimed to "Describe community characteristics associated with CHA completion, factors contributing to success, as well as barriers and limitations that prevented Kansas communities from initiating a CHA or completing the process." In Kansas, local health departments in 1995 received a Community Health Assessment Process (CHAP) workbook produced by the Kansas Department of Health Environment, Kansas Association of Local Health Departments, and the Kansas Hospital Association. They also received technical assistance around their assessment efforts. The CHA model comprised six components: coalition building, reviewing community data, collecting community data, understanding community data, developing the community plan, and implementing and evaluating the community plan.

The evaluation of the resulting CHAs was carried out through two LPHA surveys. This work built upon community capacity theory, which Curtis notes includes the following dimensions: participation and leadership, people skills, community network, community power, and others. The evaluation aimed to assess the extent to which community capacity had an impact on the initiation and completion of the CHA process. By 1998, 64 Kansas counties (61%) had initiated a CHA and 64% of those had completed the process. These communities reported several positive outcomes: improved communications among community groups, problem understanding, and improved skills in accessing and interpreting data. They also noted a number of catalysts for their success, including: ready-to-use data, guidance from the CHAP creators, coalition strength, and effective media communication. Barriers included time and other resources, challenges obtaining buy-in, and an "exhausting data collection process." Finally, 72% of responding communities (n=25) reported having "initiated an intervention process."

Kansas communities that had not initiated a CHA attributed this to lack of community interest, lack of time, and lack of money, among other barriers. Those that initiated, but did not complete, the process pointed to "less representation of community leaders in their coalitions" as a key barrier, along with lack of financial resources, loss of interest over time, and "getting people to complete their tasks." The author notes that "some of the community limitations identified by the respondents could have easily been addressed

Evaluations of CHAs' Impact on Community Health. We identified no reports detailing the results of evaluations of specific CHA processes or products with respect to their usefulness and contribution to the health of a specific community. Several broader reports did describe the evaluation methods associated with their CHAs; the most frequent methods used were surveys of community participants. One state health department surveyed local health departments that had embarked on CHAs more broadly "to understand implementation challenges and barriers, including community characteristics, cooperation among agencies, and history of problem solving success."