MARC Network Evaluation Toolkit : simplebooklet.com

Do It Yourself Strategies for Network Reflection and Evaluation Tools to Strengthen Networks Addressing ACEs Trauma and Resilience MOBILIZING ACTION FOR RESILIENT COMMUNITIES O V E RV I E W Networks addressing Adverse Childhood Experiences ACEs trauma and resilience can benefit from self reflection and evaluation The ever changing nature and distributed work of networks requires continual feedback on who their members are how they are engaged with one another what work is ongoing and the impact of that work The Mobilizing Action for Resilient Communities MARC Initiative comprised 14 established ACEs trauma and resilience ATR networks dedicated to building stronger and more equitable communities From the MARC evaluation experiences several do it yourself tools for strengthening networks were developed Tools to help networks reflect on progress include logic models and evaluability assessment to help assess the extent to which the network has agreed upon goals and is positioned to achieve them network analysis to take a close look at the network s composition and how the organizations and individuals within it are collaborating and outcome harvesting to take stock of what the network has accomplished This toolkit offers strategies that network members can use to help evaluate their efforts For each tool we provide a definition and description guidance on how to use it and recommended participants We offer a few key tips from the MARC Initiative and resources to further guide the work Logic Models and Evaluability Assessment What A logic model provides a visual blueprint for a network see Exhibit 1 It depicts the resources or inputs that a network has the overall goals and objectives that drive the network the portfolio of strategies or activities being conducted and the desired short term and longer term outcomes Logic models can be useful in and of themselves in describing a network and detailing how the various activities and other components link to accomplishments and desired outcomes Evaluability assessment is a methodology that uses a logic model to assess whether the network s activities as implemented are on course for achieving the desired outcomes Each MARC community together with the evaluators developed or refined a logic model to reflect the network s work and engaged in an evaluability assessment Why Logic models portray the underlying logic of a network displaying and connecting a network s resources goals activities and outcomes Because the efforts of networks can be organic and not prescribed logic models can include feedback loops and other elements to display the iterative nonlinear nature of the work A logic model also offers a tool for coalescing members perspectives on the general focus and direction of the network the types of activities they believe fall within its scope and what they believe the network can accomplish and impact Evaluability assessment uses the logic model as an analytic tool to determine whether the work of the network aligns with the model and whether the outcomes are plausible to achieve given the work underway The assessment is typically conducted to determine if an outcome evaluation of the network can produce results that would be credible reliable and useful 1

Page 2

Tools to Strengthen Networks Addressing ACEs Trauma and Resilience Exhibit 1 Sample Network Logic Model Illinois 7 7 16 GOAL The Illinois ACEs Response Collaborative seeks to build capacity to foster a movement around ACEs and resilience in Cook County INPUTS OBJECTIVES ACTIVITIES OUTPUTS SHORT TERM OUTCOMES Illinois ACEs Response Collaborative 32 organizations Backbone organizations BOD staff and partners 4 Neighborhood networks within Cook County National Expert Advisors Expert ACEs trainers and evaluation team Baseline ACEs data through IL BRFSS Funding MARC 299 631 Children s Health Foundation Raise awareness and inform on the pervasiveness and impact of ACEs Provide practitioners across various sectors with skills that promote TI practice To institute systems change through TI policies and procedures on the state regional organizational and community levels Identify promising principles practices for addressing ACEs in various sectors across the life course Support neighborhood initiatives to implement TI services Use data to capture targeted areas of the Collaborative s impact to improve policies services and practices and drive systems change in IL Large scale community and service provider awareness with Paper Tiger screenings and ACEs Trauma 101 across health education and criminal justice sectors attendees type of practitioner community member location of training Meeting evaluations with awareness attitude or usefulness measures Education on TI practices Trauma 201 and implementation of NPP and CPP for DFCS staff Screening Brief Interv for pediatric residents in school based health TI Hospital Collaborative Restorative Justice of contacts with funders policy makers of foundations with changed priorities portfolios of Universities with curricula including ACES type creator name of strategic plans Informing foundations changing funding priorities portfolios Incorporating ACES training into curriculum Including TI vision in strategic planning documents type of network meetings of attendees of new Collaborative members Neighborhood Initiative Planning Environmental scan Ongoing assessment of membership alignment with network activities Implementation evaluation plans for pilots including agreed upon practices measures Products summarizing scan and guiding pilot work Implement TI initiatives in 4 neighborhoods in Cook Co Outputs and outcomes TBD Data Committee analyzing BRFSS data to create policy recommendations Program manager outputs training evaluations Members Environmental scan pilot evaluation Products summarizing analysis feedback and evaluation activities for strategic planning and service planning and policy development Increase community awareness and knowledge of ACEs trauma Increase service provider awareness and knowledge of ACEs and TI practice Increase the use of TI strategies by service providers Demonstrate impact in neighborhoods for TI initiatives such as increased number of community leaders focused on building capacity for TI practice evidence of improved community safety and evidence of TI principles applied in service provision across sectors LONG TERM OUTCOMES Reduce the frequency and severity of ACEs Reduce the frequency of vicarious trauma or retraumatization Increase access to TI services for children and adults Increase the number of TI policies Increase funding reimbursement for TI services CONTEXT Cook County comprises 40 of the population of Illinois representing a large geographic and jurisdictional area It includes Chicago and 127 suburbs When Logic models are best developed at the outset of a network to help it determine the goals members have for its work the types of activities it could engage in and the changes it hopes to contribute to or create They can however be developed at any stage of a network to provide a visual understanding of the underlying logic of the network Similarly though evaluability assessment may be best conducted once a network is underway in conducting activities the process can be useful at any time in a network s life course It may be especially helpful to engage in an evaluability assessment when a network is pivoting in its work or when changes in the membership or the context are occurring and affecting the network s work and what it can accomplish Who Developing logic models and conducting evaluability assessments are activities that benefit from engaging as many key members of the network as possible The process of logic model development for example could be facilitated by an individual member a small group or committee of members or an outside consultant with broad based support and input from the network 2

Page 3

Tools to Strengthen Networks Addressing ACEs Trauma and Resilience How Logic models can be developed in a variety of ways A small group can do the initial work and prepare it for review by the entire network or the initial work can be done by the entire network refined by a small group and then reviewed revised and validated by the entire network The initial work involves first developing lists of the network s resources goals and objectives key strategies and sets of activities and outputs and outcomes expected from specific activities as well as those expected from the cluster of activities These lists are often developed through a review of documents Once the lists are completed the network determines how these different elements link together Using an if then logic i e if we do this activity in this way then we expect this output which should lead to this outcome or outcomes can facilitate drawing connections among the different elements Drawing the models can be a dynamic process and may be best done initially by a small group which then checks in with the larger network Using the listing of items the group can build the logic model either using post it notes with each item written on a note to have flexibility in positioning the different elements or drawing on white boards visible to all The process can take several hours and may be best completed in stages depending on the complexity of the network differences in what members believe are the goals and objectives the different ways in which the activities can lead to outcomes and so forth In some situations a facilitator can be brought in to provide all members in the group an opportunity to participate A variety of software packages are available for developing models from using basic Microsoft Word or PowerPoint to more specialized tools https www betterevaluation org en evaluation options TOC_software Once the logic model is developed it can guide the development of data collection and analysis protocols for the evaluability assessment Data are collected from a broad array of network members and other key stakeholders to assess how they perceive the goals of the network the objectives and implementation of specific activities and the outcomes they expect to be accomplished from specific activities and by the network as a whole The analysis is guided by the questions outlined in the box below The main focus of the analysis is to determine if the network s activities and strategies as implemented can achieve the desired outcomes The assessment focuses on whether the work underway aligns sufficiently with expectations and if not how the activities need to be changed or enhanced or how outcomes need to be adjusted to be more aligned with the activities in place Although an evaluability assessment is often conducted by an outside organization network members can also conduct it It is likely best for a small committee to conduct the assessment and then have it reviewed by the larger network The evaluability assessment process and its results can help the network review and revise its work to be most effective Analyzing the Logic Model Are the goals clearly specified and agreed upon Are the network resources e g funding staffing partnerships sufficient to implement the activities as intended Are activities implemented in a way that aligns with the types of outcomes that are desired e g are they reaching enough people Are they reaching enough of the intended audience Are they involving the expected participants and organizations Are the outcomes specific enough to be measurable and meaningful Are the linkages between the activities and outcomes plausible given the level of implementation resources and context 3

Page 4

Tools to Strengthen Networks Addressing ACEs Trauma and Resilience Key tips Logic models can be developed at different levels of detail from groups of activities within the network to each organization s specific activities If the network is large and complex the model developers might consider trying to develop a high level picture of the network first and then develop more detailed models of strategies or sets of activities if desired Developing reviewing and revising the logic models in iterative sprints can be useful in providing manageable sets of activities and time for reflection rather than trying to produce the perfect model in one cycle of development Logic models and evaluability assessment also are best done when the views of all network members are considered This diversity of views may make it more difficult to reconcile differences and reach a consensus but will ultimately result in a more representative and accurate depiction of the network Additional resources Logic Models https www slideshare net InnoNet_Eval doityourself logic models https www innonet org media logic_model_workbook_0 pdf McLaughlin J A Jordan G B 2010 Using logic models In Wholey J S Hatry H P Newcomer K E Eds Handbook of practical program evaluation San Francisco Jossey Bass Evaluability Assessment https www betterevaluation org en themes evaluability_assessment DFID 2013 Planning Evaluability Assessments A Synthesis of the Literature with Recommendations Working Paper 40 Template for a MARC ACEs Trauma and Resilience Network Evaluability Assessment Report Westat Evaluation reports Mobilizing Action for Resilient Communities MARC Initiative 4

Page 5

Tools to Strengthen Networks Addressing ACEs Trauma and Resilience Social Network Analysis What Social network analysis SNA is a technique to examine the relationships between members participating in a network It can measure both the quality and types of relationships members have with one another Although the formal SNA method can be highly technical and require the expertise of a researcher trained in analyzing social network data a less technical approach can be used by networks themselves to understand the extent to which network members relate to one another Why SNA helps a network examine which members work with whom in a network and the extent to which they connect with one another especially across sectors This information can help a network understand where more work is needed in facilitating linkages among network members which members are at the hub of interactions among members and which members need to be integrated more into the network Exhibit 2 Example of a network map showing different levels of collaboration When SNA can be conducted at any point the network wants to have an understanding of the relationships among members It also can be done at multiple points in time to observe changes in the network over time either in response to specific efforts or to look at evolution in the network In MARC we conducted two SNAs with each community to show both the level of collaboration at two given time points as well as the change in collaboration over time Who All organizations that participate in a network should be involved in the analysis The analysis could be conducted by the backbone organization 1 or a small committee of network members How The formal SNA method involves conducting a survey of all network members and having members rate their or their organizations relationships with all other organizations members in the network The rating could be the extent of collaboration overall or could be about specific types of collaboration e g working together on activities sharing staff etc The data are then analyzed with specific software producing pictures of the networks see Exhibit 2 as well as metrics on the average number of unique connections among members and network density the number of connections among members relative to the total number possible in the network among others These type of measures tell us something about the connectivity among members Formal SNA requires the assistance of a professional researcher However networks can engage in other methods to understand the interconnectivity communication and relationships among members For example during a meeting or retreat for the network an individual network member or subcommittee can facilitate a process in which members examine the extent to which networking is occurring One way could be to have an interactive session in which each organization is represented on a wall with a different colored circle see Exhibit 3 A representative from each organization then draws a line from their circle to each other organization s circle with which they have worked together or exchanged information Those running the exercise would pose a specific collaboration question that can be answered For example it may be that the network is most interested in learning about any contacts that have occurred within the last week or contacts that involve joint work as opposed to solely information exchange Butterfoss F 2007 Coalitions and Partnerships in Community Health San Francisco CA Jossey Bass 1 5

Page 6

Tools to Strengthen Networks Addressing ACEs Trauma and Resilience Or it may be that the network is interested in examining how frequently members work with one another If there are concerns of highlighting individual members that might have little involvement with others or those that may be highly involved with many others members could be grouped by sector so the process is examining interactions between sectors Alternatively the question of collaboration could be focused on high levels of collaboration that would highlight key subsets of members with no risk of leaving out only one or two members Exhibit 3 Example of a social network analysis developed through a group process Virtual meetings might also provide an opportunity to conduct an interactive network analysis in an anonymous and real time manner On the Zoom virtual platform for example stamps e g a star a checkmark etc are available in the annotation tool and can be used by participants to endorse something on the screen As depicted below a meeting facilitator could show slides with several members on the screen at a time and ask participants to place a stamp on those with whom they collaborate or collaborate a great deal or some other version of a collaboration question Participants within each sector might be asked to use the same stamp such as the Photo credit Elevate Montana Helena Affiliate star for education the checkmark for mental health and so forth so that the visual display would show not only the extent of collaboration but the sectors with whom they collaborate See Exhibit 4 Putting up a few members at a time might also minimize the opportunity of a member standing out as having few or no collaborations Although the exercise itself will provide a sense of the nature of the interactions the data from the process can be analyzed and summarized more completely after the meeting Exhibit 4 Example of a version of social networks produced in Zoom meeting Which members do you collaborate with once a month or more Member Member Member Member Member Member 6

Page 7

Tools to Strengthen Networks Addressing ACEs Trauma and Resilience Key tips A network s approach to this exercise would ideally be guided by the type of information needed to understand the networking in place If the network is interested in the interactions between member representatives of different sectors the team can set up the data collection in that way masking the identity of specific members If the network is interested in only high level or more intensive levels of collaboration the team would want to frame the questions accordingly If the network is interested in understanding hubs of activity or from where information flows that interest would suggest a different set of questions For example rather than ask each member about its interaction with each other member the facilitator would inquire about the five members with which they collaborate most The information from this inquiry could then provide a picture of the interactions that are happening most often and offer some guidance in how to broaden the work Additional resources https assets publishing service gov uk government uploads system uploads attachment_data file 491572 socnet_howto pdf Outcome Harvesting What Outcome harvesting is an evaluation method that networks can use to collect evidence on changes outcomes they have helped create in a community and or in organizations Outcomes include any changes in the behavior relationships actions activities policies or practices of an individual group community organization or institution2 An outcome can be positive or negative intended or unintended as long as the link between the network activities and the outcome is verifiable and the outcome itself is verifiable Why Although networks can use logic models to summarize their expected and intended outcomes it is also true that a range of spontaneous outcomes can occur due to the multi sector nature of the networks the multiplicity of activities that can take place under the network and changes in the context that can shape network efforts Outcome harvesting is particularly well suited to assessing the work of networks as it permits examination of both planned and unplanned outcomes and outcomes generated through the ripple effects of members and through more opportunistic ways When Outcome harvesting is most appropriate to use with networks that are somewhat well established are well underway in conducting activities and can expect to see some accomplishments A network may want to consider using the process to take stock of a network s progress after a set of specific activities inform strategic planning or guide the network at a time when it expects to pivot or revise its course In MARC networks engaged in a collaborative outcome harvesting process near the completion of the MARC funding period 2 https www betterevaluation org en plan approach outcome_harvesting 7

Page 8

Tools to Strengthen Networks Addressing ACEs Trauma and Resilience Who The backbone organization of the network and other key network members can conduct the process A small group of the network can operate as the outcome harvesting team and identify other key informants who can speak to the occurrence of each outcome How The basic method involves collecting evidence of outcomes that have occurred and working backward to trace whether and how the network contributed to the change It involves gleaning outcomes from multiple sources documents websites interviews data and then examining the network s influence on each outcome The method was designed to be highly participatory and provides an opportunity for network members to learn about a network s accomplishments The process begins by developing an initial listing of outcomes working with the backbone organization and key network members In parallel the outcome harvesting team can identify additional outcomes through documents reports or websites If the list is long outcomes to trace can be prioritized to those that are most significant in their impact It is ideal is to have a clear statement for each outcome noting when the outcome occurred the concrete change that occurred in active voice and active verb and the actors involved in bringing about the change We provide an example of this below For each outcome the outcome harvesting team seeks out key informants and other sources to verify that the outcome occurred and obtain a description of how it was accomplished and the extent to which the network contributed to it Data from key sources can come from in person or telephone interviews and email exchanges The data collection includes examining the processes and sequence of events by which each outcome unfolded over time and tracing back and analyzing the way in which the network as well as other factors played a role in the occurrence of the outcome Below is an example of an outcome and the process leading up to it The set of outcomes encompasses the Albany Police Department s ACEs informed program and policies in dealing with the public and internal staff and requires all officers and staff to be trained in ACEs The outcomes are traced back to the activity in orange showing that the Albany Police Department was brought in as a member and partner of the HEARTS network and trained on ACEs and trauma Other network activities include the collaboration between the police department and Osborne Association that together led to additional trauma informed policy changes including Handle with Care which was adopted based on a model developed in West Virginia Exhibit 5 Example of process tracing with a harvested outcome The Osborne Association collaborates with the Albany Police Department on reducing risks to children during parent arrests The Albany Police Department adds questions about presence of dependent children to arrest protocols Albany Police Department becomes a member of the HEARTS network and requests training from HEARTS member LaSalle School Department staff attend training in West Virginia on Handle with Care program Trauma informed training offered throughout department Handle with Care program adopted by department Trauma informed training embedded in Academy classes New focus on crisis intervention training in Academy and active force 8

Page 9

Tools to Strengthen Networks Addressing ACEs Trauma and Resilience The questions below lay out the information needed to define and analyze each outcome These attributes include THE OUTCOME WHAT happened What or who changed What changed in behaviors relationships activities actions policies or practices Is the change qualitative and or quantitative What evidence substantiates the change What are the sources for the information THE ACTORS WHO individual or group was involved in making the change WHO made this happen TIMING and LOCATION WHEN and WHERE did the change take place SIGNIFICANCE WHY is the change relevant significant important Does it address a problem or need in the community What is the reach and scope of the outcome CONTRIBUTION HOW did the network contribute to the change How did network activities outputs contribute to the change e g was it a direct cause a large contribution as a catalyst or tipping point How did others unrelated to your network contribute to the change Key tips Each outcome should have multiple sources ideally self reported outcomes corroborated with some documentary or data based evidence Some outcomes may overlap or interrelate the interconnectivity of outcomes can provide a full picture of what changes have taken place It is possible for example that one outcome has led to another within the list or that there was yet another result of an outcome that was overlooked It may also be that a few of the network activities lead to multiple outcomes and some of the outcomes can have multiple activities contributing to it Keep the focus on behavior changes Changes in awareness are important but are not considered outcomes until they lead to some verifiable change in action Engaging a broad array of network members in the harvesting helps to identify the outcomes that occur through the ripple effects of members efforts 9

MARC Network Evaluation Toolkit

Page 1