ORIGINAL RESEARCH INVESTIGACIÓN ORIGINAL
Measuring progress of collaborative action in a community health effort
Medición del progreso de las actividades de colaboración en una iniciativa de salud comunitaria
Vicki L. Collie-Akers; Stephen B. Fawcett; Jerry A. Schultz
Work Group for Community Health and Development, University of Kansas, Lawrence, Kansas, United States of America. Send correspondence to Vicki L. Collie-Akers, e-mail: email@example.com
OBJECTIVE: To measure the progress made by the collaborative actions of multisectorial partners in a community health effort using a systematic method to document and evaluate community/system changes over time.
METHODS: This was a community-based participatory research project engaging community partners of the Latino Health for All Coalition, which based on the Health for All model, addresses health inequity in a low-income neighborhood in Kansas City, Kansas, United States of America. Guided by three research questions regarding the extent to which the Coalition catalyzed change, intensity of change, and how to visually display change, data were collected on community/system changes implemented by the community partners from 2009-2012. These changes were characterized and rated according to intensity (event duration, population reach, and strategy) and by other categories, such as social determinant of health mechanism and sector.
RESULTS: During the 4-year study period, the Coalition implemented 64 community/system changes. These changes were aligned with the Coalition's primary goals of healthy nutrition, physical activity, and access to health screenings. Community/system efforts improved over time, becoming longer in duration and reaching more of the population.
CONCLUSIONS: Although evidence of its predictive validity awaits further research, this method for documenting and characterizing community/system changes enables community partners to see progress made by their health initiatives.
Key words: Health promotion; equity in health; evaluation; community-based participatory research; United States.
OBJETIVO:Medir el progreso alcanzado por las actividades de colaboración de los socios multisectoriales en una iniciativa de salud comunitaria mediante el empleo de un método sistemático para verificar y evaluar los cambios en la comunidad y los sistemas con el transcurso del tiempo.
MÉTODOS: Se trata de un proyecto comunitario de investigación participativa en el que colaboraron los socios comunitarios de la Coalición Salud para Todos los Latinos, que, con base en el modelo de Salud para Todos, aborda las desigualdades en materia de salud en un vecindario de bajos ingresos de Kansas City, en el estado de Kansas (Estados Unidos). Adoptando como guía tres preguntas de investigación referentes a en qué medida la Coalición catalizó los cambios, qué intensidad alcanzaron y cómo mostrarlos gráficamente, se recogieron datos sobre los cambios en la comunidad y los sistemas introducidos por los socios comunitarios del 2009 al 2012. Estos cambios se describieron y evaluaron según su intensidad (la duración del acontecimiento, el porcentaje de población expuesta y la estrategia) y según otras categorías, tales como el mecanismo implicado como determinante social de la salud y el sector afectado.
RESULTADOS: Durante el período de estudio de cuatro años, la Coalición había introducido 64 cambios en la comunidad y los sistemas. Estos cambios estaban alineados con las principales metas de la Coalición: nutrición sana, ejercicio físico y acceso a los tamizajes de salud. Las iniciativas de la comunidad y los sistemas mejoraron con el transcurso del tiempo, eran más duraderas y llegaban a una parte más importante de la población.
CONCLUSIONES:Aunque se requieren investigaciones adicionales para establecer datos probatorios de su validez predictiva, este método para verificar y caracterizar los cambios en la comunidad y los sistemas permite a los socios comunitarios observar el progreso alcanzado por sus iniciativas en pro de la de salud.
Palabras clave: Promoción de la salud; equidad en salud; evaluación; investigación participativa basada en la comunidad; Estados Unidos.
Local actions that promote health and address its social determinants have a common vision: people working together to improve conditions for health and wellbeing for all (1). Optimal efforts deal with both the intermediary determinants, such as health behaviors and environmental conditions, and the broader social determinants of health (SDH), such as income inequality and socioeconomic position (2, 3). Moreover, the SDH conceptual framework for action (4), developed by the World Health Organization (WHO), suggests three "mechanisms" by which health inequities are produced: (a) different- ial exposure to health promoting/ damaging factors, e.g., access to healthy foods, opportunities for physical activity, and living conditions; (b) differential vulnerability/capabilities, e.g., education, income, and socioeconomic position; and (c) differential consequences, e.g., quality of health services determined by socioeconomic position. Components pivotal to progress in SDH are: com-munity empowerment, collaborative action through multisectoral partnerships, and systematic methods for measuring progress and making adjustments.
Empowering communities to take action is fundamental to improving and ensuring conditions for health and health equity (5); it drives the process by which individuals act collectively to gain greater influence and control over SDH and the quality of life in the community (1, 6). Empowerment can be seen when individuals and groups communicate their needs and concerns, engage in decisionmaking, and bring about changes in communities and systems to address identified concerns. Despite the importance of community empowerment and social participation for health promotion, there are relatively few systematic methods for measuring whether this process leads to change.
Collaborative action for population health and health equity (7) requires comprehensive and coordinated approaches to address both the intermediary determinants (e.g., environmental changes that make healthy nutrition and physical activity more likely) and the broader social determinants (e.g., policies that address differential exposures, vulnerabilities, and opportunities). These must be addressed through multiple sectors (e.g., health, education, government) and at multiple ecological levels (i.e., individuals, organizations, communities, and broader systems). In a participatory research/evaluation context, it is critical to communicate progress made along the way to enable systematic reflection and adjustments by community partners. Evaluating the effects of collaborative action at the community level requires a systematic method for documenting changes in communities/systems and characterizing their contribution to health promotion goals and SDH.
Evaluating the progress of local health promotion efforts requires the ability to measure discrete community and environmental changes in multiple sectors and at various levels; and to characterize their attributes, such as duration and reach, that may be related to their long-term contribution to population health and its social determinants. In community-based participatory research, scientific and community partners must be able to review progress along the way; for instance, the amount and type (intensity) of community/system changes (e.g., policies, programs, environmental conditions) being created and implemented by coalition members over time.
However, collaborative action poses measurement challenges for documenting and characterizing what was done to achieve intended collective impact (8). This can be resolved by a shared measurement system that can be used to help integrate and align efforts and to hold accountable a diverse set of partners working toward a common purpose. A shared monitoring and evaluation system can help partners document, characterize, and systematically reflect on their activities regarding, for instance, the number and type of new or modified community programs, policies, and practices aimed at community-established goals.
The present study describes a participatory research and evaluation process for measuring the progress made through collaborative action by documenting the number and scope of community/system changes carried out by a community-led effort in a low-income neighborhood in the United States, the Latino Health for All Coalition. The study uses intensity scores-rating the duration, the breadth or reach, and strength of and strategy employed by interventions-to characterize progress made through the collaborative action of multiple partners from a variety of sectors.
MATERIALS AND METHODS
The Latino Health for All Coalition (the Coalition) was formed in 2008 as a collaborative partnership with a mission of reducing risk for cardiovascular disease and diabetes by promoting healthy eating, active living, and access to health services. The Coalition has several named behavioral and health objectives: (a) to increase fruit and vegetable consumption; (b) to increase engagement in regular physical activity; (c) to decrease levels of obesity; and (d) to increase blood pressure, cholesterol, and hemoglobin A1C checks. A Community Advisory Board (CAB) was established in 2010 to guide the Coalition. It consisted of at least 10 community members, either Latino or representing local Latino-serving organizations (its composition has varied since, but is always at least 80% Latino). The CAB meets every other month to make decisions, such as which mini-grants will be dispersed to implement the Coalition's priority strategies. The Coalition's research partners focus on providing technical support to implement the CAB's guidance and to measure progress made by the Coalition's efforts.
The priority area for the Latin Health for All Coalition is a low-income neighborhood within the Kansas City metropolitan area in the state of Kansas, located in the central plains of the United States. At the time of the study, the neighborhood had a population of nearly 23 370 (9), of which approximately 6 480 was Latino-many recent immigrants from Mexico.
Health for All model. The Coalitions follows the Health for All model, described partially in Figure 1 and elsewhere in greater detail (10). This model outlines a process whereby different sectors engage in: (a) collaborative planning and capacity building; (b) implementing community and systems changes; and (c) achieving the intended result of environmental changes and improved health. As a backbone organization, the Coalition and its university partners provide technical assistance and other support for full implementation of the model's components: (a) establishment of an organizational structure (e.g., action committees for each goal area); (b) development of an action plan consisting of community-determined strategies; (c) mobilization of the community and stimulation of collaborative action; (d) distribution of grant resources through mini-grants to support activities; and (e) a documentation and feedback system to monitor progress and make adjustments.
Community-based participatory research (CBPR) was the approach used to guide implementation of this research and action project. The study is ongoing, but the data presented here are from 2009-2012. Consistent with CBPR principles (11), community members and research partners at the University of Kansas shared responsibility for the overall study's implementation, including prioritizing activities, designing data collection and documentation activities, analyzing data, and disseminating lessons learned. The primary mechanism through which this occurred was the Coalition's CAB.
The study was reviewed and approved by the Human Subjects Committee at the University of Kansas (Kansas City, Kansas, United States).
The study measured progress primarily by examining community and system changes, defined as new and modified programs, policies, and practices, and environmental changes that were facilitated by coalition partners and related to the Coalition's mission and goals. In this participatory research project, scientific and community coalition partners explored the following questions:
1. To what extent did collaborative action bring about a community/system change related to the Coalition's mission (stated previously)?
2. What was intensity (defined below) of the community/system change?
3. How can the progress be visually displayed?
Each community/system change-an intermediate outcome-was documented using an online system (12). Research partners entered data from reports submitted by community partners. Reports included descriptive information about the activities and accomplishments facilitated by partners and the number of people reached or engaged in the activity. First, each discrete activity documented on behalf of the Coalition was scored for whether it was a community or a system change. Second, each change was characterized by important dimensions, such as duration, estimated reach, and type of change-strategy employed. Research staff used a codebook (13) that included definitions and scoring instructions to determine whether the documented activity was to be scored as an instance of community/system change or another type of activity. Another staff member independently coded a sample of activities to ensure data quality through measures of inter-observer agreement regarding assignment of coding. Throughout the study period, inter-observer agreement was 96%; this is consistent with high levels of reliability seen in other studies using this method for community measurement.
Coding the changes. All documented community and system changes were characterized by dimensions related to:
Intensity (i.e., duration, reach, and strategy; further explanation follows)
Type of SDH mechanism (i.e., differential exposure, vulnerability, or consequences)
Other attributes (e.g., sector and ecological level at which implemented)
Intensity. To characterize intensity, each community/system change was coded according to three specific dimensions:
Duration (i.e., a one-time event, occurring more than once, or ongoing)
Reach (i.e., what proportion-high, medium, or low-of the total priority population experienced the change)
Strategy (i.e., provided information and enhanced skills; enhanced services and support; modified access, barriers, and opportunities; changed consequences; and modified policies and broader conditions).
Each category was assigned a numerical value based on the relative strength for the attribute. Table 1 describes the categories and assigned values for weighting the potential impact of a particular community/system change. The values across all three domains were summed to create a single intensity score (Σ Duration value + Reach value + Strategy = Intensity value). Using this approach, the score could range from 0.3 (weakest and potentially of least influence on longer-term outcomes) to 3.0 (strongest and potentially of most influence). Using the formula, intensity scores were calculated for each documented community/system change, and these scores were summed for all documented community/system changes occurring in a given year of the collaborative effort.
Strategy. In addition, each community/system change was coded according to the strategy it employed to address the social determinant. Drawing from the WHO conceptual framework for SDH (4), the strategies were coded as either:
Modifying exposures to health damaging/health promoting factors;
Reducing vulnerabilities/enha ncing capabilities; or
Modifying the social, economic, and health consequences.
Using descriptions provided by the WHO conceptual framework, research staff coded all entries according to these SDH strategy definitions to determine the frequency with which each was used. Other attributes that were coded include setting or sector, geographic location, and socioecological level addressed.
The data showed that from 1 January 2009-31 December 2012, Coalition members and partners had implemented a total of 64 distinct community/system changes. Responses to the three participatory research questions follow:
1. Extent to which collaborative action brought about mission-related changes
Table 2 shows several examples of community/system changes, organized by primary goal, type of characterization, and intensity score (using the method described in the table). More than 30 organizations or individuals led the implementation of these 64 activities on behalf of the Coalition. The community/system changes were implemented in multiple settings, including health care organizations, faith-based organizations, social services providers, schools, neighborhoods, and other settings.
2. Intensity of community/system changes
Across all 64 community/system changes, the mean intensity score was 0.975, with the full range of scores being observed-from 0.3 to 3.0. Two community/system changes from Table 2 are used below to demonstrate how actual intensity scores were computed:
Community health fair that distributed health information to 65 residents. This was a one-day event (0.1) in which information was provided (0.1) and a low-percentage of the population was reached (0.1) for a total intensity score of 0.3.
Unused space was converted into a soccer field. This was an ongoing effort (1.0) using a strong intervention for environmental change (1.0) with a potentially high reach (1.0) for a total intensity score of 3.0.
The annual mean intensity scores increased as the Coalition matured, from a mean score of 1.06 in 2009 to 1.58 in 2012. This reflected a shift to community/system changes of somewhat longer duration, greater reach, and increased strength of change strategy.
3. Visual display of progress
Figure 2 is a graphic display of the resulting information on the unfolding of community/system change over time (number of changes and intensity). Each community/system change is displayed as a different shade. The stacked bars reflect the composite annual intensity scores for all of the community/system changes in place during that calendar year. Each row within the stack is an individual community/system change. The width of each row reflects the intensity score for that community/system change. Wider rows reflect higher intensity scores (e.g., an ongoing environmental or policy change), while narrower rows reflect lower intensity scores (e.g., a one-day training event). The sum of the intensity score for each year reflects the new and ongoing community/systems changes; ongoing changes are included every year, whereas one-time events are only included in the year in which they occur.
During its first 4 years of collaborative action, the Latino Health for All Coalition fostered 64 community/system changes in this low-income Latino community. This participatory research study illustrates use of a systematic method for documenting and characterizing the intensity of an intermediate outcome of collaborative action-the number and extent of changes brought about in the community over time. This measurement approach has high utility in a participatory research context: it shows progress along the way, thereby prompting systematic reflection and adjustments by members of the community-led coalition.
This method of community measurement has several advantages. First, systematic documentation of community/systems change-the intended intermediate outcome of collective action-can help hold groups accountable for progress in changing conditions that affect population health and health equity. Second, characterization of community changes and associated intensity scoring-using attributes of duration, reach, and strength of intervention commended in other studies (14)-can help differentiate environmental changes that may have different levels of influence on ultimate longer-term outcomes. Third, visual display of progress can facilitate community participation in systematic reflection on progress in collaborative action, making it easier for community partners to see what is happening, make sense of it, and make needed adjustments to the effort. Fourth, the cost of doing such documentation is modest. Community members contribute time by reporting activities; and research/evaluation staff spend time documenting and characterizing activities and calculating intensity scores-an estimated total of 2 hours per month. Finally, this measurement approach-a type of monitoring and evaluation system-has implications for documenting the results of multisectoral action across the Region of the Americas. This documentation approach can be useful in capturing efforts to ensure health in all policies, whether in different sectors of the same community, or in multiple ministries at broader levels.
Community empowerment is a central value in local action to promote health and health equity. At the 7th Global Conference on Health Promotion (15), held in Nairobi, Kenya, in 2009, health promotion colleagues from around the world identified recommendations to enhance community empowerment and control of health promotion initiatives (6), including several reflected in this case study. First, the Call to Action called for community health efforts to listen to and start with the voices and aspirations of the community in planning and action. During action planning, the members of the Latino Health for All Coalition determined the community/system changes to be sought and who would do what to bring them about. Second, participants sought to ensure meaningful and equitable participation and control in decisionmaking and agenda setting (15). All decisions regarding the allocation of resources for the intervention were determined by the Community Advisory Board, which included members of socially-excluded groups. Third, the group recommended gathering empirical evidence about how communities create conditions for improved health and health equity (15). This study of collaborative action helps to expand the evidence base regarding the dose (number and kind) of community/system changes brought about to promote health and address its determinants.
Since documentation relies on reporting by community partners and review of documents, some data on community/system changes may be missing or inaccurate. Although community and research partners reviewed data for completeness, it is possible that information was still missing. Second, the current formula used for developing individual intensity scores has received limited testing for validity. Although its three dimensions used in characterizing intensity-duration, reach, and strategy-represent dimensions reported in the literature, future research is needed to test its validity. It is possible that other dimensions may also be important to the ultimate impact of a change; or, that identified aspects may have differential effect. Preliminary data for the Latino Health for All Coalition suggest that increased intensity scores are associated with improved population health outcomes (i.e., residents reporting higher fruit and vegetable intake and engagement in physical activity). However, these findings may be limited to the special conditions of this particular collaborative partnership. More research is required to better understand the validity of this method for estimating progress and the ultimate impact of collaborative action.
This report illustrates the potential for systematically measuring progress made through collaborative action. Data regarding the unfolding community/system changes (and their intensity) enables a visualization of the progress of collaborative action. By characterizing each documented change according to specific mechanisms and strategies, clarity is gained around how collaborative action is addressing priority goals and social determinants. In this participatory evaluation study, local people, including those most affected by health disparities, were engaged in systematically reflecting on: (a) What is it that we are seeing (e.g., in patterns of change; distribution among mechanisms of social determinants, sectors, and goals)?; (b) What does it mean (e.g., what is associated with increased/decreased rates of change or distributions)?; and, (c) What are the implications for improvement (e.g., how can leadership be enhanced to bring about other needed changes) (16)? Future research is needed to help understand the applicability of this participatory research approach to examining the ultimate impact of collaborative action in other contexts and at the level of broader systems.
Collaborative efforts to improve population health and health equity rarely use a single component intervention; instead, a combination of programs, policies, and environmental changes are facilitated over time by multiple partners across multiple settings and levels. Systematic measurement of the number and intensity of community/system changes (an intermediate outcome) helps capture the dynamic and unfolding interventions resulting from social participation in collaborative action (17). Measurement approaches for examining the effects of collaborative action can help detect progress in addressing intermediary and broader social determinants. Such participatory research methods enhance the understanding of collaborative action as a development process, that is, the gradual unfolding of changing conditions to promote health and address its social determinants.
Acknowledgements. The authors would like to acknowledge the important contributions of the Latino Health for All Coalition's Community Advisory Board (Kansas City, Kansas, United States), the Coalition's membership, Ana Paula Cupertino, and Monica Mendez.
Financing. Research and collaborative action with the Coalition were supported by a grant (NIH51190) from the National Institute on Minority Health and Health Disparities, part of the National Institutes of Health (Bethesda, Maryland, United States), to the University of Kansas for the purpose of testing a previously explored approach to collaborative action, the Health for All model.
Conflicts of interest. None.
1. World Health Organization. Health promotion glossary. Geneva: WHO; 1998.
2. World Health Organization Commision on Social Determinants of Health. Closing the gap in a generation: Healthy equity through action on the social determinants of health. Geneva: WHO; 2008.
3. Marmot M, Allen J, Goldblatt P, Boyce T, McNeish D, Grady M, et al. Fair society, healthy lives: Strategic review of health inequalities in England post-2010. London: Marmot Review; 2010.
4. Solar O, Irwin A. A conceptual framework for action on social determinants of health. Geneva: World Health Organization; 2010.
5. Marmot M. Harveian Oration: Health in an unequal world. Lancet. 2006;368(9552):2081-94.
6. Fawcett S, Abeykoon P, Arora M, Dobe M, Galloway-Gilliam L, Liburd L, et al. Constructing an action agenda for community empowerment at the 7th Global Conference on Health Promotion in Nairobi. Global Health. 2010;17(4):52-6.
7. Fawcett S, Schultz J, Watson-Thompson J, Fox M, Bremby R. Building multisectoral partnerships for population health and health equity. Prev Chronic Dis. 2010;7(6):A118.
8. Kania J, Kramer M. Collective impact. Stanford Social Innovation Review. 2011;1(9):36-41.
9. United States Census Bureau. American Factfinder. Available from: http://factfinder2.census.gov Accessed on 12 February 2013.
10. Fawcett SB, Collie-Akers VL, Schultz JA, Cupertino AC. Community-Based Participatory Research within the Latino Health for All Coalition. Prev Interv Community. 2013;41(3):142-154.
11. Israel B, Schulz, A., Parker, E., Becker, A. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19:173-202.
12. Fawcett SB, Schultz JA. Using the Community Tool Box's Online Documentation System to support participatory evaluation of community health initiatives. In: Minkler M, Wallerstein N, eds. Community-based participatory research for health: from process to outcomes. San Francisco: Jossey-Bass; 2008.
13. Work Group for Community Health and Development, University of Kansas. Online Documentation and Support System Codebook. Lawrence: University of Kansas, 2009.
14. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322-7.
15. World Health Organization. The Nairobi Call to Action. Proceedings of the 7th Global Conference on Health Promotion. Geneva: WHO; 2009.
16. Fawcett SB, Boothroyd R, Schultz JA, Francisco VT, Carson V, Bremby R. Building capacity for participatory evaluation within community initiatives. J Prev Interv Community. 2003;26(2):21-36.
17. Fawcett SB, Paine-Andrews A, Francisco VT, Schultz J, Richter KP, Berkley-Patton J, et al. Evaluating community initiatives for health and development. WHO Reg Publ Eur Ser. 2001;(92):241-70.
Manuscript received on 1 April 2013.
Revised version accepted for publication on 26 September 2013.