Return to flip book view

The Best They Can Be 2013 2014 Annual Quality Improvement Report for Out of School Time Programs in Palm Beach County Lisa M Lindeman Ph D Director of Research Evaluation Prime Time Palm Beach County Inc Prime Time Palm Beach County Inc

CONTENTS Summary 1 Quality Matters 2 How Prime Time Measures Quality 3 The Palm Beach County Program Quality Assessment Tool 3 Form A Quality of the Environment and Interactions with Youth 3 Form B Quality of the Organizational Structure and Family Involvement 4 Scoring and Levels 4 Recent Changes to the Assessment Tool 4 What Changed 5 The Impact on Quality Scores 5 Program Quality in 2013 2014 5 New Tool Old Target Outcomes 5 Participation Linked to Quality 6 How Strong is the Link 6 What Does This Mean For Youth in Palm Beach County 6 What Makes More Established Programs Stand Out 6 Performance on Specific Elements of Quality 7 I Safe Environment 7 II Supportive Environment 9 III Interaction 11 IV Engagement 12 V Youth Centered Policies and Practices 14

VI High Expectations for Youth and Staff 15 VII Organizational Management 16 VIII Family 17 Quality Improvement 17 Programs Encouraged Youth By Asking More Open Ended Questions 18 Programs Fostered More Cooperative Group Interactions 19 Programs Gave Youth More Choices 19 Programs Helped More Youth Reflect 20 Programs Created More Structured Opportunities for Youth to Facilitate Groups 20 Youth Need More Opportunities to Make Plans 21 Serving High Need Areas 21 Conclusion 23 Annotated Bibliography 24 Appendix Domain and Scale Scores 30

Prime Time wrote the book on quality Out of school time practitioner The mission of Prime Time Palm Beach County is to foster high quality in out of school time programs which provide opportunities for children and youth to succeed To fulfill this mission Prime Time launched the Palm Beach County Quality Improvement System QIS nearly a decade ago with support from the David P Weikart Center for Youth Program Quality a division of the Forum for Youth Investment As part of this system Prime Time s Quality Improvement team consisting of highly skilled quality advisors began working closely with out of school time OST programs offering personalized coaching guidance and support Complementing this effort Prime Time s Professional Development team offers training workshops to OST practitioners including the Youth Work Method series developed by Weikart Each series e g Planning and Reflection is tied to elements of quality Finally Prime Time s Community Engagement and Supports team contracts with external organizations such as the Palm Beach Zoo the South Florida Science Center and Aquarium Youth Speak Out and Junior Achievement to provide learning and enrichment activities to youth This model has empowered more than 100 programs in Palm Beach County to signficantly improve the quality of their environment organizational structure skills and activities leading to positive outcomes for youth SUMMARY As a result of Prime Time s work with OST programs in Palm Beach County programs significantly improve in quality from year to year Programs that have participated in the QIS for many years demonstrate higher overall quality than programs newer to the system In the 20132014 quality improvement cycle overall program quality was highest among those that had been in the QIS the longest while new programs just beginning their work in the QIS scored lowest on quality 1

Notably programs showed dramatic improvements on areas of quality that they chose as a focus in 2012 and 2013 Specifically programs Encouraged youth by asking more open ended questions Fostered more cooperative group interactions Offered youth more choices Gave youth more opportunities to reflect Enabled more youth to experience structured opportunities to lead and facilitate activities Note The results presented in this report are based on a quality assessment tool that experienced significant revisions before assessors observed programs during the 2013 2014 quality improvement cycle Programs achieving the same level of quality in 2012 2013 and 2013 2014 nevertheless received a lower score on the tool in 2013 2014 due to these revisions QUALITY MATTERS Prime Time s work is rooted in a wide body of research demonstrating that quality OST programs lead to substantial academic social and emotional benefits for youth These benefits include higher grades improved standardized test scores on time grade promotion reduced dropout rates and increased school day attendance in programs throughout the country Naftzger et al 2014 Vandell Reisner Pierce 2007 Huang et al 2000 2005 2007 London Gurantz Norman 2011 Metz Goldsmith Arbreton 2008 Durlak Weissberg Pachan 2010 Durlak Weissberg 2007 Programs that attend to the social and emotional needs of youth lead to skill development wellbeing prosocial behavior as well as improved academic performance according to a metaanalysis of 73 programs by Durlak and Weissberg 2007 also see Durlak Weissberg Pachan 2010 Importantly program quality is often critical for seeing benefits In many cases no benefits are seen unless a program is of high quality Vandell Reisner Pierce 2007 In Palm Beach County high quality OST programs lead more youth to move to the next grade level on time compared to low quality programs according to a rigorous study conducted by the American Institutes for Research Naftzger et al 2014 Further research on youth outcomes in Palm Beach County is currently in progress For a more detailed summary of research demonstrating the benefits of high quality OST programs please see the annotated bibliography 2

Palm Beach County Quality Standards STANDARD ONE Solid Organizational Framework The OST program is structured and organized to ensure the health and safety of children and youth in the program The administration utilizes sound business practices and promotes the development training and retention of qualified staff STANDARD TWO Supportive Ongoing Relationships The OST program staff involves youth as partners in the program and encourages children and youth to work together STANDARD THREE Positive and Inclusive Environment The OST program promotes psychological and emotional safety The afterschool program staff creates a welcoming environment that fosters a sense of belonging for children and youth families and staff STANDARD FOUR Challenging Learning Experiences The OST program provides positive learning experiences for children and youth which build upon youth interest and supports active engagement in enrichment activities STANDARD FIVE Family Outreach and Involvement The OST program promotes positive communication with families and supports parental involvement in the educational experiences of children and youth HOW PRIME TIME MEASURES QUALITY Raising quality begins with clear assessments of existing strengths and areas in need of improvement Prime Time employs an in depth nationally vetted method for measuring quality Each year external assessors observe every program in the QIS three times using the Palm Beach County Program Quality Assessment PBC PQA an adaptation of the Youth Program Quality Assessment YPQA which was developed and tested in 2005 by the High Scope Educational Research Foundation Smith Hohmann 2005 and instituted by the David P Weikart Center for Youth Program Quality Prime Time has one of the largest and most robust sets of OST program quality data in the nation External assessors have observed programs in Palm Beach County using this reliable validated tool for seven years Assessors are trained in the use of the tool and must achieve a high standard of inter rater reliability before conducting observations The Palm Beach County Program Quality Assessment Tool The PBC PQA consists of 106 items that form 30 scales in eight domains of quality These domains correspond to elements of the Palm Beach County Quality Standards see sidebar Form A Quality of the Environment and Interactions with Youth The first portion of the PBC PQA Form A which consists of 69 items in 20 scales examines the program environment and how staff interact with and engage youth Form A measures four major domains of quality I Safe Environment 3

II III IV Supportive Environment Interaction Engagement Form B Quality of the Organizational Structure and Family Involvement The second portion of the PBC PQA involves an interview in which assessors evaluate the policies supports culture management and communication practices of the program Form B consists of 37 items in ten scales Results of the interview are recorded using Form B V VI VII VIII Youth Centered Policies and Practices High Expectations for Youth and Staff Organizational Management Family Scoring and Levels Scores on each item of the PBC PQA range from 1 to 5 On the first domain of quality Safe Environment programs are given a 1 or 5 for each item On the other domains programs are given a 1 3 or 5 for each item In general these scores indicate the following 1 The program did not demonstrate this area of quality 3 The program demonstrated this area of quality sometimes or with some youth or the program was neutral or mixed on this measure 5 The program demonstrated this area of quality always or with all youth The Quality Improvement department considers scores of 3 0 or above as indicators of acceptable quality Programs that achieve an overall average score of 4 1 or above on Form A have met a high standard of quality and after two consecutive years with a 4 1 or above may move to maintenance level Programs begin their journey in the QIS at entry level After one year in the system a program moves from entry level to intermediate level shifting the coaching and supports from all staff to program directors in order to allow them to develop maintenancelevel skills Recent Changes to the Assessment Tool Before reviewing results of the latest program quality assessments it is important to note that the Weikart Center and Prime Time Quality Improvement team made significant revisions to the PBC PQA before assessors observed programs during the 2013 2014 quality improvement cycle Many items on the new version were more difficult than before The new version raised the bar for quality making it possible to discern increasingly finer levels of excellence among high4

performing programs The new version also reflected changes made to the Youth PQA tool developed by the Weikart Center CYPQ to improve inter rater reliability Finally many items on the tool were adjusted to make them more readable What Changed Revisions only affected items in three domains of quality on Form A II Supportive Environment III Interaction and IV Engagement 1 Form B remained unchanged No existing items were removed and no new items were added The Impact on Quality Scores Because many items on the assessment tool became more difficult programs that achieved the same level of quality in the 2013 2014 quality improvement cycle as they did in the previous cycle received lower scores on the assessment For this reason this change must be considered when comparing scores on the revised tool to scores on previous versions of the tool PROGRAM QUALITY IN 2013 2014 In the 2013 2014 quality improvement cycle 121 OST programs were observed and intereviewed using the revised PBC PQA All programs without exception achieved an overall score above 3 0 on Form A The tables in the appendix provide the means and standard deviations for all scales and domains of the PBC PQA Among 111 programs participating in the QIS for at least one year the average Form A score was 3 93 SD 40 and the average Form B score was 4 27 New Tool Old Target Outcomes In previous years Prime Time met a target outcome of 90 of programs attaining a score of 3 6 or above on Form A of the PBC PQA However a score of 3 6 on the previous version of the PBC PQA is not equivalent to a 3 6 on the new version of the tool The revised version caused scores to shift Similarly while most programs improved or maintained their PBC PQA scores in previous years changes to the tool resulted in a smaller percentage 45 improving or maintaining a score at or above 4 1 With respect to the target outcome 90 of programs in the QIS for at least one year had an overall score of 3 43 or above on the new version of the tool and 76 had an overall score of 3 6 or above Again a lower score on the recent PBC PQA is not necessarily an indication that quality went down but rather reflects changes in the tool 1 A list of specific changes is available upon request 5

Participation Linked to Quality Programs that have participated in the QIS for many years are better equipped to serve the needs of youth than programs new to the system 2 Specifically programs demonstrate significantly higher quality on the PBC PQA after three to four years in the QIS 3 The average overall score on Form A was 3 75 SD 40 for 19 newer programs in the QIS between zero and two years 3 84 SD 37 for 36 more experienced programs three to four years and 4 01 SD 40 for 66 established programs in the system for five or more years How Strong is the Link The link between participation in the QIS and program quality based on a comparison between programs is respectable One way of examining the strength of this link is by looking at the amount of difference in program quality between established programs and newer programs The difference in quality between newer and more established programs is statistically speaking considered medium a medium effect size or Cohen s d 64 4 What Does This Mean For Youth in Palm Beach County Elaine Mancini Assessment Manager at Prime Time describes the difference between newer and more established programs in terms of trends in item scores In the beginning we see a lot of ones and threes Later we see more threes and fives Recall that a score of one on the PBCPQA indicates that an element of quality is not present and a score of three indicates that it is present sometimes or in some form A score of five means that the element of quality is fully present for all youth An increase in ratings of three and five for a particular element of quality means that more youth across the county are experiencing it or that the same youth are experiencing it more often For example in fostering more cooperative group interactions participation in the QIS for several years could mean that hundreds of youth benefit from group interactions several times per week instead of once every month or so What Makes More Established Programs Stand Out Differences are most apparent for domains II III and IV according to a repeated measures multivariate analysis with domain averages as the outcome variables and year in the QIS new 2 According to a univariate analysis of variance with overall Form A scores as the dependent variable and years in QIS 0 to 2 3 to 4 or 5 to 6 as the independent variable F 2 118 4 25 p 017 2p 067 Statistical output and supplemental materials are available upon request 3 According to planned comparisons p 004 4 One way to describe this difference is in terms of the probability that an established program will demonstrate higher quality than a newer program known as the common language effect size McGraw and Wong 1992 or the probability of superiority Grissom and Kim 2005 The probability that a program in the QIS for five or six years will demonstrate higher quality than a program in the QIS for zero to two years is 64 6

more experienced or established and quality improvement cycle 2012 2013 or 2013 2014 as predictor variables F 8 220 2 76 p 006 2p 09 see Figure 1 Youth engagement is particularly impacted by experience in the QIS Figure 1 Average PBC PQA scores for each domain on Form A by number of years programs have participated in the QIS as of the 2013 2014 quality improvement cycle Baseline to 2 years 3 to 4 years 5 to 6 years 5 4 3 2 1 I Safe Environment III Interaction II Supportive Environment IV Engagement Performance on Specific Elements of Quality All programs achieved the highest quality as measured by the PBC PQA on the safety of their environment The most established programs achieve the highest quality in their interactions with youth domain III and how well they engage youth domain IV The following sections summarize quality assessment scores for the 2013 2014 quality improvement cycle and highlight supporting research for each scale Average scores for each domain and scale are listed in the appendix I Safe Environment In assessing the safety of their environment all programs participating in the QIS demonstrate exceedingly high quality The first domain of quality measured by the PBC PQA is the physical and psychological safety of the environment For items within this domain programs are given a score of 1 or 5 which corresponds to No or Yes indicating whether the program demonstrated what is described in the item No items in this domain were revised for the new version of the tool The average score for all programs is 4 98 for the domain of safe environment No programs regardless of the length of time they have participated in the QIS received an average score less 7

than 4 5 on any of the five scales in this domain Programs score this high on this domain every year The vast majority of items in the domain of safety are closely related to Florida state licensing requirements Programs must be licensed or license exempt to join the QIS Nevertheless this domain on the PBC PQA serves a valuable purpose Programs are visited by licensing officials only once per year In contrast Prime Time quality advisors perform quarterly progress checks in addition to annual assessments each consisting of three observations A Psychological and emotional safety is promoted The first scale in the domain of safety examines psychological and emotional safety Staff at all programs without exception demonstrated respect for and inclusion of others regardless of religion ethnicity class gender ability appearance or sexual orientation and at no program was there any evidence of bias B The physical environment is safe and free of health hazards Programs in the QIS are free of health and safety hazards as measured by the second scale in this domain Their spaces are kept clean and sanitary ventilation and lighting are adequate and the temperature is comfortable C Emergency and safety procedures are in place to protect youth Safety procedures are well established in QIS programs This includes the accessibility of first aid kits fire extinguishers and written procedures as well as supervision of youth D Program space and furniture materials accommodate activities All programs in the QIS have ample space for OST activities E Healthy food and drink are provided In 2014 results of a Prime Time youth survey highlighted the importance of healthy food and drink for program quality and youth outcomes Youth participating in expanded learning opportunities were asked whether the activities involved new learning problem solving collaboration challenge a sense of belonging a positive social atmosphere and more Youth were also asked to indicate whether they felt hungry tired or upset just prior to taking the survey Youth who were hungry rated the activities as substantially less engaging in every way suggesting that healthy food and drink can translate into greater youth engagement Without exception all QIS programs ensured that at least one or two of the food choices are healthy e g there are vegetables fresh fruit and or real juice All but one program ensured that food and drinks are plentiful and available at appropriate times for all youth 8

II Supportive Environment All programs in the QIS provide a welcoming atmosphere for youth scale F ensure that their session flow is planned presented and paced for youth scale G and effectively maintain clear limits scale H No programs earned scores less than 3 0 on these scales In contrast all programs continue to improve on their capacity to encourage youth build youth skills and reframe conflict scales I J K and L Figure 2 Average PBC PQA scores for each scale in domain II by number of years programs have participated in the QIS Baseline to 2 years 3 to 4 years 5 or more years 5 3 1 F Welcoming atmosphere G Optimum session flow H Clear limits I Engaging activities J K Support for Encouragement skill building L Conflict reframing F Staff provides a welcoming atmosphere Among all programs in the QIS staff provide a welcoming atmosphere for all youth The lowest score on this scale was 4 0 Prime Time quality advisors support programs in maintaining this positive atmosphere G Session flow is planned presented and paced for youth Session flow includes starting and ending sessions within ten minutes of scheduled times having ample materials and supplies ready clearly explaining activities and providing instructions to youth and allowing an appropriate amount of time for activities The lowest score on this scale was 3 53 H Staff effectively maintains clear limits Programs are encouraged to create a balance between choice and structure although these goals are not mutually exclusive Baumrind 1996 Setting clear limits requires classroom 9

management skills which have a huge impact on quality Average scores on this scale are high for all programs in the QIS with a minimum score of 3 0 As a result youth likely experience positive outcomes associated with clear limits Numerous researchers have discovered that when youth experience consistency clear rules reasonable boundaries clear expectations and predictability they benefit tremendously Eccles Gootman 2002 Caron Weiss Harris Catron 2006 Day Peterson Badali Shea 2002 Jackson Henricksen Foshee 1998 Lamborn Mounts Steinberg Dornbusch 1991 Smetana 1995 Steinberg 2001 Steinberg Elmen 1986 Steinberg Elmen Mounts 1989 Barber et al 2005 Meteyer Perry Jenkens 2009 I Activities support active engagement Established programs support more active engagement than programs newer to the QIS When youth are actively engaged they are creating with materials working with ideas or improving skills with guided practice Their engagement leads to tangible products or performances reflecting their contributions They have clear opportunities to share what they ve created developed or learned and a balance evolves between the physical e g sculpting visiting the museum writing a story and mental or emotional e g learning reflection discussion elements of the activity J Staff support youth in building new skills Differences in staff support of skill building were not apparent between established and less experienced programs However nearly 70 of all programs scored above 3 0 on this scale Thirty seven programs scored below 3 0 on this scale with one program receiving an average of 1 0 indicating no support for youth in building new skills Programs can support youth by encouraging them to try new skills providing useful feedback communicating a specific learning or skill building focus or breaking difficult tasks into smaller simpler steps K Staff support youth with encouragement Encouraging youth begins with active involvement between staff and youth Staff encourage youth by acknowledging their contributions and asking frequent open ended questions In offering encouragement only five programs received an average score below 3 0 L Staff use youth centered approaches to reframe conflict Because conflicts do not always occur during observations by assessors only 33 programs received a score for this scale Reframing conflict also draws upon classroom management skills Using youth centered approaches means that staff ask about or acknowledge the feelings of all youth involved help youth respond calmly ask those involved what happened and encourage youth to develop solutions 10

III Interaction High quality interactions between staff and youth as well as youth and their peers enable youth to experience the benefits of healthy relationships and engaging social activities In the domain of interaction programs have some room for improvement However more established programs demonstrate higher quality on this domain overall compared to newer programs Figure 3 Average PBC PQA scores for each scale in domain III by number of years programs have participated in the QIS Baseline to 2 years 3 to 4 years 5 or more years O Youth can facilitate groups and P Youth can partner with adults 5 3 1 M Sense of belonging N Participation in cooperative groups Q Positive peer relationships M Youth have opportunities to develop a sense of belonging Programs facilitate a sense of belonging by providing youth with structured opportunities to get to know their peers preventing exclusion and enabling youth to receive public acknowledgment or attention for their achievements or contributions Assessors also gauge whether youth strongly identify with the program offering These elements of quality correspond to the four items in scale M All programs successfully prevent exclusion or take actions to include youth who are in danger of being left out The average score for this item was 4 94 The vast majority of programs also scored high on youth identification M 4 48 SD 99 In contrast not all programs provide opportunities for youth to receive attention from staff and their peers for their contributions e g group presentations celebrations exhibitions performances However more established programs M 3 34 SD 1 16 provided more opportunities than newer programs M 2 86 SD 1 29 suggesting that programs improve in this area as a result of participation in the QIS 11

N Youth have opportunities to participate in cooperative groups Participation in cooperative groups involves more than group activities When youth have special roles and a purpose toward which all group members strive cooperation is enhanced In general programs receive adequate scores on two out of three items on this scale Staff provide opportunities for youth to work together in teams or groups M 3 14 SD 1 82 and cooperative groups have a purpose M 3 07 SD 1 81 However fewer programs provide youth with interdependent roles during group activities M 2 60 SD 1 72 More established programs score higher on this item than newer programs O Youth have opportunities to act as group facilitators and mentors Opportunities to lead and mentor allow youth to develop critical social skills While most programs particularly those that have participated in the QIS for more than two years succeed in providing a variety of opportunities for youth to develop group interaction skills M 4 05 SD 1 35 they provide mentorship and leadership opportunities less often This represents an area for improvement for all programs in the system P Youth have opportunities to partner with adults Effective relationships between youth and adults are connected to positive feelings in youth and reduced discipline problems in school Marzano Marzano 2003 This scale comprises two items The first item measures the extent to which staff share control of most activities with youth providing guidance and facilitating while retaining overall responsibility The second item measures whether staff provide explanations for their expectations directions or guidelines In general programs receive satisfactory scores for both items Q Youth have opportunities to develop positive peer relationships Programs in the QIS foster positive relationships among youth The final scale in the domain of interaction explores how youth in each program treat one another Positive peer relationships are inferred based on respectful language a warm tone of voice eye contact and friendly gestures The vast majority of programs 73 received a perfect score on this scale and only seven scored below 4 0 IV Engagement Youth engagement is a critical program goal Many youth benefits do not occur unless youth are engaged Engagement or participation is a better predictor of youth benefits than attendance alone Roth Malone Brooks Gunn 2010 Cross et al 2010 Shernoff Vandell 2010 As programs participate in the QIS they engage youth more frequently and effectively with each passing year Programs struggle most with this domain of quality but scores for this domain rise dramatically as a result of targeted improvement efforts 12

Figure 4 Average PBC PQA scores for each scale in domain IV by number of years programs have participated in the QIS Baseline to 2 years 3 to 4 years 5 or more years 5 3 1 R Opportunities to make plans S Opportunities to make choices based on interests T Opportunities to reflect R Youth have opportunities to make plans More established programs scored higher on this scale than newer programs Nevertheless most programs do not provide youth sufficient opportunities to make plans Often practitioners must follow lesson plans and activity plans from which they cannot deviate Scores on this scale are lower than any other scale and among the 42 programs that chose to focus on this aspect of quality in the 2012 2013 quality improvement cycle only nine improved Future quality improvement efforts will focus on guiding and training practitioners to create opportunities for youth to make plans S Youth have opportunities to make choices based on their interests Programs in the QIS provide youth many opportunities to make choices about how they complete an activity i e the process involved their roles use of tools or materials or the order of activities according to average scores on the first item in scale S M 3 58 SD 1 10 Often youth are able to choose among options they themselves determine Programs also provide youth opportunities to make choices about content e g deciding on the topic of an activity but the choices are more often restricted to those presented by the practitioner M 2 91 SD 1 24 T Youth have opportunities to reflect Reflection allows youth to understand the rationale and take home message of activities they experience in their program Time for reflection can be pivotal for achieving youth benefits A recent meta analysis of 49 studies from around the world found that structured time for reflection 13

enabled community service programs to achieve substantial benefits for youth whereas programs lacking time for reflection achieved almost no noticeable effects Society for Research in Child Development 2014 V Youth Centered Policies and Practices External assessors interview programs to determine the extent to which the needs of youth shape their policies and practices Scale A gauges whether program offerings tap youth interests to build multiple skills Scale B examines whether youth have an influence on organizational decisions Only ten programs received a score on this scale because this only applies to middleschool youth More established programs those participating in the QIS for five or more years score higher on this domain than newer programs according to a repeated measures multivariate analysis F 2 97 3 39 p 038 see Figure 5 and Table 3 in the appendix Figure 5 Average PBC PQA scores for each scale in domain V by number of years programs have participated in the QIS 5 Baseline to 2 years 3 to 4 years 5 to 6 years 3 1 A Youth interests influence programming B Youth influence organizational decisions 14

VI High Expectations for Youth and Staff Assessors also explore each program s expectations for youth and staff Assessors determine whether organizations promote staff development scale C supportive social norms scale D and academic enrichment scale E More established programs score higher on this domain than newer programs F 2 97 4 39 p 015 see Figure 6 Figure 6 Average PBC PQA scores for each scale in domain VI by number of years programs have participated in the QIS Baseline to 2 years 3 to 4 years 5 to 6 years 5 3 1 C Staff development D Supportive social norms 15 E Academic enrichment

VII Organizational Management Programs are assessed on their business practices scale F the effectiveness of their organizational logistics scale G and whether staff education and field specific training meet county standards scale H Importantly programs can receive a score of zero on items in scale H If a program receives a zero on scale H staff have less than one year of experience working with youth less than ten hours of field specific training no credentials no Youth Development College Credit Certificates YDCCC and less than 30 college credits in a related field In figure 7 zero is not included in the axis on the left While county standards factor in scores on scale H for programs in Palm Beach County are not indicative of how well programs meet county standards Rather higher scores indicate that programs are going above and beyond basic requirements set by a variety of state and local entities Figure 7 Average PBC PQA scores for each scale in domain VII by number of years programs have participated in the QIS Baseline to 2 years 3 to 4 years 5 to 6 years 5 3 1 F Sound business practices G Effective logistics H Staff education meets standards No significant differences in scores on this domain distinguish new and more established programs p 09 However scores on scale H staff education and training are higher for more established programs see Figure 7 16

VIII Family The last domain on the PBC PQA examines family involvement As part of their interview programs are assessed on whether they support positive communication with family scale I and family involvement scale J More established programs show higher levels of positive communication and family involvement F 2 97 10 84 p 0001 Figure 8 Average PBC PQA scores for each scale in domain VIII by number of years programs have participated in the QIS Baseline to 2 years 3 to 4 years 5 to 6 years 5 3 1 I Positive communication with family J Family involvement QUALITY IMPROVEMENT The overarching goal of the QIS is to drive meaningful progress in quality from one year to the next Substantial changes to the PBC PQA tool make it difficult to compare the latest quality assessment scores with scores from the previous year Despite increases in actual quality scores on the new version of the tool were lower than scores on the previous version according to a multivariate analysis with year included as a within subjects factor F 8 90 5 86 p 0001 This shift lower scores on the revised version of the PBC PQA was only evident in domains of Form A that were revised according to within subjects contrasts Again this is due to differences in how quality was measured by the two versions of the tool Despite this shift in the tool and resulting scores improvements were seen in some areas that programs chose to focus on throughout the year Each year programs select several areas of 17

quality as focal points for improvement Between the 2011 2012 and 2012 2013 quality improvement cycles programs made significant progress on their improvement plans The most common areas chosen for improvement in 2013 2014 were encouraging youth in particular asking open ended questions II K item 3 giving youth opportunities to make plans IV R giving youth opportunities to participate in cooperative groups III N giving youth opportunities to reflect IV T giving youth opportunities to make choices based on their interests IV S giving youth opportunities to act as group facilitators and mentors in particular providing structured opportunities for youth to lead a group III O item 3 Programs that chose to focus on these areas as part of their improvement plans significantly improved except on IV R giving youth opportunities to make plans while programs that did not focus on these areas experienced a decrease in scores according to repeated measures analyses of variance see below Programs Encouraged Youth By Asking More Open Ended Questions Thirty six programs made it their goal to ask more open ended questions of youth II K item 3 As a result of coaching training and other QIS supports these programs significantly improved on this item from an average score of 2 10 to 2 72 F 1 109 7 26 p 008 Of note in the revised version of the PBC PQA tool only grammatical changes were made to this item Figure 9 Changes in average PBC PQA scores for item 3 scale K Domain II for programs that did and did not choose to focus on this item as part of their improvement plan 5 Year 6 3 Year 7 1 Not On Plan On Plan 18

Programs Fostered More Cooperative Group Interactions Thirty three programs focused on giving youth more opportunities to participate in cooperative groups III N These programs improved significantly from an average score of 2 34 to 2 92 F 1 109 5 01 p 027 despite changes to the tool Figure 10 Changes in average PBC PQA scores for scale N Domain III for programs that did and did not choose to focus on this item as part of their improvement plan 5 Year 6 3 Year 7 1 Not On Plan On Plan Programs Gave Youth More Choices Fifteen programs focused on giving youth opportunities to make choices based on their interests IV S For these programs despite the increased difficulty of achieving the same score on items in this domain scores dramatically improved from 1 94 to 3 84 F 1 109 20 67 p 001 Figure 11 Changes in average PBC PQA scores for scale S Domain IV for programs that did and did not choose to focus on this item as part of their improvement plan 5 Year 6 3 Year 7 1 Not On Plan On Plan 19

Programs Helped More Youth Reflect Seventeen programs focused on giving youth more opportunities to reflect on what they are doing or have done to share and present what they have done and to give feedback on activities IV T Average scores increased from 2 97 to 3 83 F 1 109 8 73 p 004 Figure 12 Changes in average PBC PQA scores for scale T Domain IV for programs that did and did not choose to focus on this scale as part of their improvement plan 5 Year 6 3 Year 7 1 Not On Plan On Plan Programs Created More Structured Opportunities for Youth to Facilitate Groups Twelve programs focused on providing youth more structured opportunities to lead or facilitate activities among a group of their peers III O item 3 These programs made significant improvements while other programs did not F 1 113 7 70 p 006 Figure 13 Changes in average PBC PQA scores for item 3 in scale O Domain III for programs that did and did not choose to focus on this item as part of their improvement plan 5 Year 6 3 Year 7 1 Not On Plan On Plan 20

Youth Need More Opportunities to Make Plans Programs are encouraged to continue striving to create opportunities for youth to make plans Forty two programs focused on giving youth more opportunities to make plans IV R which includes supporting youth in making plans and encouraging youth to represent their plans in tangible ways e g writing diagrams etc Average scores on this scale fell However scores fell a little less dramatically for those programs that chose this as part of their improvement plan and items in this domain became more difficult in the revised version of the PBC PQA tool Figure 14 Changes in average PBC PQA scores for scale R Domain III for programs that did and did not choose to focus on this scale as part of their improvement plan 5 Year 6 3 Year 7 1 Not On Plan On Plan SERVING HIGH NEED AREAS Most OST programs in the QIS are located in high need areas where high need is defined by the number of youth receiving free or reduced priced lunch FRL Across Palm Beach County nearly 9 000 out of 13 500 elementary school youth or 65 receive FRL 5 Of those programs in the QIS more than half are located in zip code regions where more than 80 of youth receive FRL see Figure 14 5 According to the American Community Survey reported by Palm Beach County Counts more than 77 000 youth between the ages of 5 and 9 live in Palm Beach County However data from the Early Learning Coalition on the number of youth receiving free or reduced priced lunch include only 13 500 enrolled elementary school students 21

Figure 2 Number of QIS programs in regions by percent of youth receiving free or reduced price lunch More QIS programs exist in areas where more youth receive free and reduced priced lunch 90 p 0001 adjusted R2 48 The total number of youth in a zip code region does not however predict the number of QIS programs in that region 0 27 p 37 Program quality did not differ substantially between lower need areas and higher need areas as defined by the percent of youth receiving free or reduced price lunch However quality assessment scores on Form B of the PBC PQA did differ somewhat based on need R2 15 p 002 A multiple regression analysis revealed that programs in higher need areas scored slightly lower on domains V Youth Centered Policies and Practices and VI High Expectations for Youth and Staff but slightly higher on domain VII Organizational Management 22

CONCLUSION Prime Time launched the Palm Beach County Quality Improvement System in 2007 Since then OST programs across the county particularly those in high need areas have joined the system and benefited from expert coaching guidance and training from quality advisors and professional development specialists at Prime Time Program quality assessments provide strong evidence that the Quality Improvement System raises the quality of OST programs over time More established programs those that have participated in the system for five or more years demonstrate higher quality in multiple domains compared to newer programs Further programs have shown dramatic improvement on areas of quality they chose to focus on Future research on Prime Time s impact will explore change over time for each program based on two years of assessments using the same version of the assessment tool as well as the social and emotional benefits of improved quality for youth 23

ANNOTATED BIBLIOGRAPHY Afterschool Alliance 2005 Afterschool Programs A Wise Public Investment Afterschool Alert Issue Brief No 22 Afterschool Alliance Washington D C In this brief review the Afterschool Alliance emphasizes the critical support that OST programs provide to youth and explain why programs are a worthwhile investment High quality OST programs reduce costs associated with remedial services grade level recurrence dropout substance abuse crime teen pregnancy and missed higher income opportunities in adulthood Barber B Stolz H Olsen J et al 2005 Parental support psychological control and behavioral control Assessing relevance across time culture and method Monographs of the Society for Research in Child Development 70 4 i 147 Baumrind D 1996 The discipline controversy revisited National Council on Family Relations 45 4 405 414 Caron A Weiss B Harris V Catron T 2006 Caregiver behaviors and child psychopathology Specificity task dependency and interactive relations Journal of Clinical Child and Adolescent Psychology 35 34 45 Day D M Peterson Badali M Shea B 2002 May Parenting style as a context for the development of adolescents thinking about rights Poster presented at the 9th Biennial Meeting of the Society for Research on Adolescence New Orleans L A ERIC Document Reproduction Service No ED 464 746 Durlak J A Weissberg R P Pachan M K 2010 A meta analysis of after school programs that seek to promote personal and social skills in children and adolescents American Journal of Community Psychology 16 294 309 OST programs with a focus on social and emotional learning lead to increased prosocial behavior improved academic performance reduced behavioral problems and enhanced self awareness according to their meta analysis of 68 studies The greatest impact is associated with programs whose activities are well coordinated actively engaging focused and explicit in their focus on social skills Durlak J A Weissberg R P 2007 The impact of after school programs that promote personal and social skills Chicago Collaborative for Academic Social and Emotional Learning 24

A meta analysis of 73 OST programs found that social emotional learning leads to positive outcomes for youth Youth experienced a variety of personal social and academic benefits These included an increase in self confidence self esteem and school pride Additionally problem behaviors such as drug use and aggression were reduced Eccles J Gootman J A 2002 Community programs to promote youth development Committee on Community Level Programs for Youth Board on Children Youth and Families Commission on Behavioral and Social Sciences Education National Research Council and Institute of Medicine Washington DC National Academy Press Grissom R J Kim J J 2005 Effect Sizes for Research A Broad Practical Approach Mahwah NJ Lawrence Erlbaum Associates Hirsch B J Mekinda M A Stawicki J 2010 More than attendance The importance of after school program quality American Journal Of Community Psychology 45 3 4 447452 Hirsh and colleagues review several studies demonstrating that positive youth outcomes of OST program attendance depend on program quality and levels of participation Huang D Gribbons B Kim K S Lee C Baker E L 2000 A decade of results The impact of the LA s BEST after school enrichment initiative on subsequent student achievement and performance Los Angeles CA UCLA Center for the Study of Evaluation Graduate School of Education Information Studies University of California Huang D Kim K S Marshall A Perez P 2005 Keeping kids in school An LA s BEST example A study examining the long term impact of LA s BEST on students dropout rates Los Angeles CA National Center for Research on Evaluation Standards and Student Testing Huang D Coordt A La Torre D Leon S Miyoshi J Perez P Peterson C 2007 The afterschool hours Examining the relationship between afterschool staff based social capital and student engagement in LA s BEST Los Angeles UCLA CRESST Huang and colleagues preceding three papers explored the impact of LA s BEST Better Educated Students for Tomorrow an afterschool program that incorporates educational enrichment activities and support into their regular program One hundred elementary school programs serving 14 000 second through fifth graders in the Los Angeles area participated in their studies Participation in the program led to increased school attendance improved performance on standardized tests of reading and math higher 25

aspirations for graduation and postsecondary education and reduced high school dropout rates Huang D Dietel R 2011 Making afterschool programs better Policy Brief No 11 CRESST National Center for Research on Evaluation Standards Student Testing UCLA Graduate School of Education Information Studies Jackson C Henriksen L Foshee V A 1998 The authoritative parenting index Predicting health risk behaviors among children and adolescents Health Education and Behavior 25 3 319 doi 10 1177 109019819802500307 Kataoka S Vandell D 2013 Quality of Afterschool Activities and Relative Change in Adolescent Functioning Over Two Years Applied Developmental Science 17 3 123134 A two year longitudinal study examined feedback from 186 middle school youth regarding their participation in high quality OST program experiences to address 1 whether youth who described overall aspects of positive OST experiences would show greater gains in youth functioning in year two as reported by teachers and 2 whether youth feedback regarding specific aspects of quality OST programs were linked to teachers reports of youth functioning in year two Through use of the After School Environment Scale development of autonomy emotional support from staff and positive relationship building with peers were examined Elements of youth functioning were assessed such as social skills relationships with peers behavior and persistence Positive feedback from youth was linked to their teachers reports Decreases in aggressive behavior were associated with specific elements of program quality such as greater emotional support for youth Lamborn S Mounts N Steinberg L et al 1991 Patterns of competence and adjustment among adolescents from authoritative authoritarian indulgent and neglectful families Child Development 62 5 1049 1065 London R Gurantz O Norman J R 2011 The effect of afterschool program participation on English language acquisition Afterschool Matters 13 22 29 London and colleagues examined Hispanic youth over a four year period elementary tomiddle school to identify connections between OST program participation and English learning Assessments used to measure language growth include the Annual Measureable Achievement Objectives AMAO 1 assessment Youth in OST programs scored higher on English language milestone assessments than non participants and youth with greater attendance in OST programs were more likely to reach AMAO 1 status 26

Little P M D Wimer C Weiss H B 2008 Afterschool programs in the 21st Century Their potential and what it takes to achieve it Harvard Family Research Project Issues and Opportunities in Out of School Time Evaluation 10 1 12 For youth to be successful in a global economy OST programs can contribute to the required knowledge and skills through sustained participation in well structured and well implemented after school programs and activities A decade of research and evaluation has shown positive benefits in academic performance social emotional development crime drug sex prevention and promotion of health wellness Examples include significant gains in standardized test scores positive effects on reading and math achievement improvements in self worth self efficacy decreases in delinquency and health risk behaviors greater knowledge of nutrition and healthy behaviors These positive outcomes are more likely when youth continuously attend high quality programs with high quality staff and inter organization partnerships Marzano R J Marzano J S Pickering D J 2003 Classroom management that works Research based strategies for every teacher Alexandria VA Association for Supervision Curriculum Development McGraw K O Wong S P 1992 A common language effect size statistic Psychological Bulletin 111 361 365 Meteyer K Perry Jenkins M 2009 Dyadic parenting and children s externalizing symptoms Family Relations 58 3 289 302 Metz R A Goldsmith J Arbreton A J A 2008 Putting it all together Guiding principles for quality after school programs serving preteens Lucile Packard Foundation for Children s Health In this literature review the authors find that positive outcomes for preteens such as emotional well being prosocial behavior and improved educational achievement result when OST programs give youth opportunities to create clear goals build skills participate for longer durations in a variety of activities and form positive adult youth relationships Programs also benefit youth when they include families by using effective communication techniques and creating a warm atmosphere building a diverse staff teaching youth to value unique cultures and focusing on continuous program development Naftzger N Hallberg K Yang T 2014 Exploring the relationship between afterschool program quality and youth outcomes Findings from the Palm Beach County Quality Improvement Study Washington D C American Institutes of Research 27

Roth J L Malone L M Brooks Gunn J 2010 Does the amount of participation in afterschool programs relate to developmental outcomes A review of the literature American Journal of Community Psychology 45 3 4 310 324 Shernoff D J Vandell D L 2010 Engagement in after school programs as a predictor of social competence and academic performance American Journal Community Psychology 45 325 337 Smetana J 1995 Parenting styles and conceptions of parental authority during adolescence Child Development 66 2 299 316 Smith C Hohmann C 2005 Full findings from the Youth PQA validation study Ypsilanti MI HighScope Educational Research Foundation Society for Research in Child Development 2014 July 24 Community service programs that include reflection found to be more beneficial to youth ScienceDaily Retrieved March 10 2015 from www sciencedaily com releases 2014 07 140724094212 htm Steinberg L 2001 We know some things Parent adolescent relationships in retrospect and prospect Journal of Research on Adolescence 11 1 1 19 Steinberg L Elmen J 1986 Authoritative parenting promotes adolescent school achievement and attendance p 15 Wisconsin Corp Author National Center on Effective Secondary Schools Madison WI Steinberg L Elmen J Mounts N 1989 Authoritative parenting psychosocial maturity and academic success among adolescents Child Development 60 6 1424 1436 Tooley M Bronfreund L 2014 Skills for Success Supporting and Assessing Key Habits Mindsets and Skills in PreK 12 Washington D C New America Education Policy University of California at Irvine Department of Education 2001 Evaluation of California s After School Learning and Safe Neighborhoods Partnerships Program 1999 2000 preliminary report Irvine CA University of California researchers found that participants in afterschool programs that included academic and enrichment activities scored higher on standardized reading and math tests than non participating youth statewide Participating youth also missed fewer days of school and fewer were held back in school saving the state approximately 11 million in 2000 28

Vandell D Reisner E Pierce K 2007 Outcomes linked to high quality afterschool programs Longitudinal finding from the study of promising afterschool programs Irvine CA University of California Washington DC Policy Studies Associates Thirty five elementary and middle school OST programs serving nearly 3000 youth in 14 cities in eight states participated in this 2 year longitudinal study of youth outcomes Participation in high quality programs led to improvements in academic performance social skills and behavior Viadero D 2007 High quality after school programs tied to test score gains Education Week 27 13 1 Based on an eight state study of 35 OST programs youth from disadvantaged backgrounds who frequently attended high quality programs over a two year period made greater academic gains over peers that did not participate in supervised activities This trumps a previous study of federally funded 21st Century Community Learning Centers that found no significant learning growth with results based on comparison groups of youth who did not regularly attend OST programs The new study looked at three groups of youth 1 program only group who only attended OST programs and participated in nothing else 2 program plus group who participated in OST programs and other extracurricular activities and 3 low supervision group who did not attend OST programs on a regular basis Not only did the researchers find growth in academics but social and behavioral outcomes were greater 29

APPENDIX DOMAIN AND SCALE SCORES Table 1 Scores for each domain and scale on Form A N 121 Domain and Scale Mean Standard Deviation Min Max I Safe Environment 4 98 0 07 4 56 5 00 A Psychological and emotional safety is promoted 5 00 0 00 5 00 5 00 B The physical environment is safe and free of health hazards 4 96 0 15 4 00 5 00 C Policies and procedures protect children and youth 4 95 0 19 3 67 5 00 D Program space and furniture accommodate the activities offered 4 99 0 12 3 67 5 00 E Healthy foods and drinks are provided 4 99 0 12 3 67 5 00 II Supportive Environment 4 24 0 37 3 06 4 94 F Staff provides a welcoming atmosphere 4 82 0 26 4 00 5 00 G Session flow is planned presented and paced for youth 4 75 0 33 3 53 5 00 H Staff effectively maintains clear limits 4 82 0 39 3 00 5 00 I Activities support active engagement 3 98 0 64 2 17 5 00 J Staff support youth in building new skills 3 46 1 03 1 00 5 00 K Staff support youth with encouragement 3 64 0 56 2 56 5 00 L Staff use youth centered approaches to reframe conflict 3 42 1 09 1 00 5 00 III Interaction 3 64 0 57 2 46 4 74 M Youth have opportunities to develop a sense of belonging 4 01 0 51 2 50 5 00 N Youth have opportunities to participate in cooperative groups 2 94 1 14 1 00 5 00 O Youth have opportunities to act as group facilitators and mentors 2 94 0 84 1 44 4 78 P Youth have opportunities to partner with adults 3 49 1 07 1 00 5 00 30

Q Youth have opportunities to develop positive peer relationships 4 79 0 41 3 00 5 00 IV Engagement 2 81 0 82 1 33 4 93 R Youth have opportunities to make plans 1 88 1 10 1 00 4 78 S Youth have opportunities to make choices based on their interests 3 24 0 95 1 33 5 00 T Youth have opportunities to reflect 3 30 1 12 1 00 5 00 Overall 3 91 0 40 3 01 4 82 Table 2 Scores for each domain and scale on Form B N 110 Domain and Scale Mean Standard Deviation Min Max V Youth Centered Policies and Practices 4 42 0 62 2 50 5 00 A Program offerings tap youth interests to build multiple skills 4 46 0 59 2 50 5 00 B Youth have influence on structure and policy in the organization 3 76 1 36 1 40 5 00 VI High Expectations for Youth and Staff 4 70 0 45 2 55 5 00 C Organization promotes staff development 4 81 0 52 1 00 5 00 D Organization promotes supportive social norms 4 70 0 61 3 00 5 00 E Organization supports academic enrichment 4 62 0 60 2 33 5 00 VII Organizational Management 3 50 0 59 2 25 5 00 F The administration utilizes sound business practices 4 14 1 55 1 00 5 00 G Organizational logistics are effective 4 31 0 57 1 00 5 00 H Staff education and field specific training meet county standards 2 61 0 86 0 50 5 00 VIII Family 4 40 0 57 2 33 5 00 I Organization supports positive communication with family 4 57 0 41 3 00 5 00 J Organization supports family involvement 4 24 0 91 1 00 5 00 Overall 4 25 0 40 2 47 4 97 31

Prime Time Palm Beach County Inc receives significant funding from the Children s Services Council of Palm Beach County Prime Time Palm Beach County Inc 2300 High Ridge Road Suite 330 Boynton Beach FL 33426 561 732 8066 ph 561 732 8094 fax www primetimepbc org