As a result of Prime Time’s work with OST programs in Palm Beach County, programs significantly improve in quality from year to year. Programs that have participated in the QIS for many years demonstrate higher overall quality than programs newer to the system. In the 2013-2014 quality improvement cycle, overall program quality was highest among those that had been in the QIS the longest, while new programs just beginning their work in the QIS scored lowest on quality. Notably, programs showed dramatic improvements on areas of quality that they chose as a focus in 2012 and 2013. Specifically, programs: • Encouraged youth by asking more open-ended questions • Fostered more cooperative group interactions • Offered youth more choices • Gave youth more opportunities to reflect • Enabled more youth to experience structured opportunities to lead and facilitate activities Note: The results presented in this report are based on a quality assessment tool that experienced significant revisions before assessors observed programs during the 2013-2014 quality improvement cycle. Programs achieving the same level of quality in 2012-2013 and 2013-2014 nevertheless received a lower score on the tool in 2013-2014 due to these revisions.

The Best They Can Be 2013-2014 Annual Quality Improvement Report for Out-of-School Time Programs in Palm Beach County Lisa M. Lindeman, Ph.D. Director of Research & Evaluation Prime Time Palm Beach County, Inc. Prime Time Palm Beach County, Inc.
The Best They Can Be 2013-2014 Annual Quality Improvement Report for Out-of-School Time Programs in Palm Beach County Lisa...
CONTENTS Summary ....................................................................................................... 1 Quality Matters ........................................................................................... 2 How Prime Time Measures Quality ...................................................... 3 The Palm Beach County Program Quality Assessment Tool ...............................3 Form A: Quality of the Environment and Interactions with Youth ................................... 3 Form B: Quality of the Organizational Structure and Family Involvement .................... 4 Scoring and Levels........................................................................................................................ 4 Recent Changes to the Assessment Tool ................................................................. 4 What Changed ............................................................................................................................. 5 The Impact on Quality Scores................................................................................................... 5 Program Quality in 2013-2014 ................................................................ 5 New Tool, Old Target Outcomes.............................................................................. 5 Participation Linked to Quality ................................................................................... 6 How Strong is the Link?.............................................................................................................. 6 What Does This Mean For Youth in Palm Beach County? ............................................... 6 What Makes More Established Programs Stand Out? ...................................................... 6 Performance on Specific Elements of Quality .......................................................... 7 I. Safe Environment ..................................................................................................................... 7 II. Supportive Environment......................................................................................................... 9 III. Interaction ............................................................................................................................. 11 IV. Engagement.......................................................................................................................... 12 V. Youth Centered Policies and Practices ............................................................................ 14
CONTENTS Summary ....................................................................................................... 1...
VI. High Expectations for Youth and Staff.......................................................................... 15 VII. Organizational Management .......................................................................................... 16 VIII. Family................................................................................................................................... 17 Quality Improvement .............................................................................. 17 Programs Encouraged Youth By Asking More Open-Ended Questions ....................... 18 Programs Fostered More Cooperative Group Interactions ............................................. 19 Programs Gave Youth More Choices ................................................................................... 19 Programs Helped More Youth Reflect ................................................................................. 20 Programs Created More Structured Opportunities for Youth to Facilitate Groups .. 20 Youth Need More Opportunities to Make Plans .............................................................. 21 Serving High-Need Areas ....................................................................... 21 Conclusion ................................................................................................. 23 Annotated Bibliography........................................................................... 24 Appendix: Domain and Scale Scores ................................................... 30
VI. High Expectations for Youth and Staff.......................................................................... 15 VII...
“Prime Time wrote the book on quality.” ─ Out-of-school time practitioner The mission of Prime Time Palm Beach County is “to foster high quality in out-of-school time programs, which provide opportunities for children and youth to succeed.” To fulfill this mission, Prime Time launched the Palm Beach County Quality Improvement System (QIS) nearly a decade ago with support from the David P. Weikart Center for Youth Program Quality, a division of the Forum for Youth Investment. As part of this system, Prime Time's Quality Improvement team, consisting of highly skilled quality advisors, began working closely with out-of-school time (OST) programs, offering personalized coaching, guidance, and support. Complementing this effort, Prime Time's Professional Development team offers training workshops to OST practitioners, including the Youth Work Method series developed by Weikart. Each series (e.g., “Planning and Reflection”) is tied to elements of quality. Finally, Prime Time's Community Engagement and Supports team contracts with external organizations, such as the Palm Beach Zoo, the South Florida Science Center and Aquarium, Youth Speak Out, and Junior Achievement, to provide learning and enrichment activities to youth. This model has empowered more than 100 programs in Palm Beach County to signficantly improve the quality of their environment, organizational structure, skills, and activities, leading to positive outcomes for youth. SUMMARY As a result of Prime Time’s work with OST programs in Palm Beach County, programs significantly improve in quality from year to year. Programs that have participated in the QIS for many years demonstrate higher overall quality than programs newer to the system. In the 20132014 quality improvement cycle, overall program quality was highest among those that had been in the QIS the longest, while new programs just beginning their work in the QIS scored lowest on quality. 1
   Prime Time wrote the book on quality.        Out-of-school time practitioner The mission of Prime Time Palm Beach Count...
Notably, programs showed dramatic improvements on areas of quality that they chose as a focus in 2012 and 2013. Specifically, programs: • Encouraged youth by asking more open-ended questions • Fostered more cooperative group interactions • Offered youth more choices • Gave youth more opportunities to reflect • Enabled more youth to experience structured opportunities to lead and facilitate activities Note: The results presented in this report are based on a quality assessment tool that experienced significant revisions before assessors observed programs during the 2013-2014 quality improvement cycle. Programs achieving the same level of quality in 2012-2013 and 2013-2014 nevertheless received a lower score on the tool in 2013-2014 due to these revisions. QUALITY MATTERS Prime Time’s work is rooted in a wide body of research demonstrating that quality OST programs lead to substantial academic, social, and emotional benefits for youth. These benefits include higher grades, improved standardized test scores, on-time grade promotion, reduced dropout rates, and increased school day attendance in programs throughout the country (Naftzger et al., 2014; Vandell, Reisner, & Pierce, 2007; Huang et al., 2000, 2005, 2007; London, Gurantz, & Norman, 2011; Metz, Goldsmith, & Arbreton, 2008; Durlak, Weissberg, & Pachan, 2010; Durlak & Weissberg, 2007). Programs that attend to the social and emotional needs of youth lead to skill development, wellbeing, prosocial behavior, as well as improved academic performance, according to a metaanalysis of 73 programs by Durlak and Weissberg (2007; also see Durlak, Weissberg, & Pachan, 2010). Importantly, program quality is often critical for seeing benefits. In many cases, no benefits are seen unless a program is of high quality (Vandell, Reisner, & Pierce, 2007). In Palm Beach County, high quality OST programs lead more youth to move to the next grade level on time compared to low quality programs, according to a rigorous study conducted by the American Institutes for Research (Naftzger et al., 2014). Further research on youth outcomes in Palm Beach County is currently in progress. For a more detailed summary of research demonstrating the benefits of high quality OST programs, please see the annotated bibliography 2
Notably, programs showed dramatic improvements on areas of quality that they chose as a focus in 2012 and 2013. Specifical...
Palm Beach County Quality Standards STANDARD ONE: Solid Organizational Framework The OST program is structured and organized to ensure the health and safety of children and youth in the program. The administration utilizes sound business practices and promotes the development, training, and retention of qualified staff. STANDARD TWO: Supportive Ongoing Relationships The OST program staff involves youth as partners in the program and encourages children and youth to work together. STANDARD THREE: Positive and Inclusive Environment The OST program promotes psychological and emotional safety. The afterschool program staff creates a welcoming environment that fosters a sense of belonging for children and youth, families and staff. STANDARD FOUR: Challenging Learning Experiences The OST program provides positive learning experiences for children and youth which build upon youth interest and supports active engagement in enrichment activities. STANDARD FIVE: Family Outreach and Involvement The OST program promotes positive communication with families and supports parental involvement in the educational experiences of children and youth. HOW PRIME TIME MEASURES QUALITY Raising quality begins with clear assessments of existing strengths and areas in need of improvement. Prime Time employs an in-depth, nationally vetted method for measuring quality. Each year, external assessors observe every program in the QIS three times using the Palm Beach County Program Quality Assessment (PBC-PQA), an adaptation of the Youth Program Quality Assessment (YPQA), which was developed and tested in 2005 by the High/Scope Educational Research Foundation (Smith & Hohmann, 2005) and instituted by the David P. Weikart Center for Youth Program Quality. Prime Time has one of the largest and most robust sets of OST program quality data in the nation. External assessors have observed programs in Palm Beach County using this reliable, validated tool for seven years. Assessors are trained in the use of the tool and must achieve a high standard of inter-rater reliability before conducting observations. The Palm Beach County Program Quality Assessment Tool The PBC-PQA consists of 106 items that form 30 scales in eight domains of quality. These domains correspond to elements of the Palm Beach County Quality Standards (see sidebar). Form A: Quality of the Environment and Interactions with Youth The first portion of the PBC-PQA (Form A), which consists of 69 items in 20 scales, examines the program environment and how staff interact with and engage youth. Form A measures four major domains of quality: I. Safe Environment 3
Palm Beach County Quality Standards STANDARD ONE  Solid Organizational Framework The OST program is structured and organiz...
II. III. IV. Supportive Environment Interaction Engagement Form B: Quality of the Organizational Structure and Family Involvement The second portion of the PBC-PQA involves an interview in which assessors evaluate the policies, supports, culture, management, and communication practices of the program. Form B consists of 37 items in ten scales. Results of the interview are recorded using Form B. V. VI. VII. VIII. Youth Centered Policies and Practices High Expectations for Youth and Staff Organizational Management Family Scoring and Levels Scores on each item of the PBC-PQA range from 1 to 5. On the first domain of quality, “Safe Environment,” programs are given a 1 or 5 for each item. On the other domains, programs are given a 1, 3, or 5 for each item. In general, these scores indicate the following: 1 The program did not demonstrate this area of quality. 3 The program demonstrated this area of quality sometimes or with some youth, or the program was neutral or mixed on this measure. 5 The program demonstrated this area of quality always or with all youth. The Quality Improvement department considers scores of 3.0 or above as indicators of acceptable quality. Programs that achieve an overall average score of 4.1 or above on Form A have met a high standard of quality and after two consecutive years with a 4.1 or above, may move to “maintenance” level. Programs begin their journey in the QIS at entry level. After one year in the system, a program moves from entry level to intermediate level, shifting the coaching and supports from all staff to program directors in order to allow them to develop maintenancelevel skills. Recent Changes to the Assessment Tool Before reviewing results of the latest program quality assessments, it is important to note that the Weikart Center and Prime Time Quality Improvement team made significant revisions to the PBC-PQA before assessors observed programs during the 2013-2014 quality improvement cycle. Many items on the new version were more difficult than before. The new version raised the bar for quality, making it possible to discern increasingly finer levels of excellence among high4
II. III. IV.  Supportive Environment Interaction Engagement  Form B  Quality of the Organizational Structure and Family In...
performing programs. The new version also reflected changes made to the Youth PQA tool developed by the Weikart Center (CYPQ) to improve inter-rater reliability. Finally, many items on the tool were adjusted to make them more readable. What Changed Revisions only affected items in three domains of quality on Form A: II. Supportive Environment, III. Interaction, and IV. Engagement. 1 Form B remained unchanged. No existing items were removed, and no new items were added. The Impact on Quality Scores Because many items on the assessment tool became more difficult, programs that achieved the same level of quality in the 2013-2014 quality improvement cycle as they did in the previous cycle received lower scores on the assessment. For this reason, this change must be considered when comparing scores on the revised tool to scores on previous versions of the tool. PROGRAM QUALITY IN 2013-2014 In the 2013-2014 quality improvement cycle, 121 OST programs were observed and intereviewed using the revised PBC-PQA. All programs without exception achieved an overall score above 3.0 on Form A. The tables in the appendix provide the means and standard deviations for all scales and domains of the PBC-PQA. Among 111 programs participating in the QIS for at least one year, the average Form A score was 3.93 (SD = .40), and the average Form B score was 4.27. New Tool, Old Target Outcomes In previous years, Prime Time met a target outcome of 90% of programs attaining a score of 3.6 or above on Form A of the PBC-PQA. However, a score of 3.6 on the previous version of the PBC-PQA is not equivalent to a 3.6 on the new version of the tool. The revised version caused scores to shift. Similarly, while most programs improved or maintained their PBC-PQA scores in previous years, changes to the tool resulted in a smaller percentage (45%) improving or maintaining a score at or above 4.1. With respect to the target outcome, 90% of programs in the QIS for at least one year had an overall score of 3.43 or above on the new version of the tool, and 76% had an overall score of 3.6 or above. Again, a lower score on the recent PBC-PQA is not necessarily an indication that quality went down but rather reflects changes in the tool. 1 A list of specific changes is available upon request. 5
performing programs. The new version also reflected changes made to the Youth PQA tool developed by the Weikart Center  CY...
Participation Linked to Quality Programs that have participated in the QIS for many years are better equipped to serve the needs of youth than programs new to the system. 2 Specifically, programs demonstrate significantly higher quality on the PBC-PQA after three to four years in the QIS. 3 The average overall score on Form A was 3.75 (SD = .40) for 19 newer programs (in the QIS between zero and two years), 3.84 (SD = .37) for 36 more experienced programs (three to four years), and 4.01 (SD = .40) for 66 established programs (in the system for five or more years). How Strong is the Link? The link between participation in the QIS and program quality, based on a comparison between programs, is respectable. One way of examining the strength of this link is by looking at the amount of difference in program quality between established programs and newer programs. The difference in quality between newer and more established programs is, statistically speaking, considered medium (a medium effect size, or Cohen’s d = .64). 4 What Does This Mean For Youth in Palm Beach County? Elaine Mancini, Assessment Manager at Prime Time, describes the difference between newer and more established programs in terms of trends in item scores: “In the beginning, we see a lot of ones and threes. Later, we see more threes and fives.” Recall that a score of one on the PBCPQA indicates that an element of quality is not present, and a score of three indicates that it is present sometimes or in some form. A score of five means that the element of quality is fully present for all youth. An increase in ratings of three and five for a particular element of quality means that more youth across the county are experiencing it, or that the same youth are experiencing it more often. For example, in fostering more cooperative group interactions, participation in the QIS for several years could mean that hundreds of youth benefit from group interactions several times per week instead of once every month or so. What Makes More Established Programs Stand Out? Differences are most apparent for domains II, III, and IV, according to a repeated-measures multivariate analysis with domain averages as the outcome variables and year in the QIS (new, 2 According to a univariate analysis of variance with overall Form A scores as the dependent variable and years in QIS (0 to 2, 3 to 4, or 5 to 6) as the independent variable, F(2,118) = 4.25, p = .017, η2p = .067. Statistical output and supplemental materials are available upon request. 3 According to planned comparisons (p = .004). 4 One way to describe this difference is in terms of the probability that an established program will demonstrate higher quality than a newer program (known as the “common language effect size,” McGraw and Wong, 1992, or the “probability of superiority,” Grissom and Kim, 2005). The probability that a program in the QIS for five or six years will demonstrate higher quality than a program in the QIS for zero to two years is 64%. 6
Participation Linked to Quality Programs that have participated in the QIS for many years are better equipped to serve the...
more experienced, or established) and quality improvement cycle (2012-2013 or 2013-2014) as predictor variables, F(8,220) = 2.76, p = .006, η2p = .09 (see Figure 1). Youth engagement is particularly impacted by experience in the QIS. Figure 1. Average PBC-PQA scores for each domain on Form A by number of years programs have participated in the QIS (as of the 2013-2014 quality improvement cycle). Baseline to 2 years 3 to 4 years 5 to 6 years 5 4 3 2 1 I. Safe Environment III. Interaction II. Supportive Environment IV. Engagement Performance on Specific Elements of Quality All programs achieved the highest quality, as measured by the PBC-PQA, on the safety of their environment. The most established programs achieve the highest quality in their interactions with youth (domain III) and how well they engage youth (domain IV). The following sections summarize quality assessment scores for the 2013-2014 quality improvement cycle and highlight supporting research for each scale. Average scores for each domain and scale are listed in the appendix. I. Safe Environment In assessing the safety of their environment, all programs participating in the QIS demonstrate exceedingly high quality. The first domain of quality measured by the PBC-PQA is the physical and psychological safety of the environment. For items within this domain, programs are given a score of 1 or 5 which corresponds to “No” or “Yes,” indicating whether the program demonstrated what is described in the item. No items in this domain were revised for the new version of the tool. The average score for all programs is 4.98 for the domain of “safe environment.” No programs, regardless of the length of time they have participated in the QIS, received an average score less 7
more experienced, or established  and quality improvement cycle  2012-2013 or 2013-2014  as predictor variables, F 8,220  ...
than 4.5 on any of the five scales in this domain. Programs score this high on this domain every year. The vast majority of items in the domain of safety are closely related to Florida state licensing requirements. Programs must be licensed or license-exempt to join the QIS. Nevertheless, this domain on the PBC-PQA serves a valuable purpose. Programs are visited by licensing officials only once per year. In contrast, Prime Time quality advisors perform quarterly progress checks in addition to annual assessments, each consisting of three observations. A. Psychological and emotional safety is promoted. The first scale in the domain of safety examines psychological and emotional safety. Staff at all programs, without exception, demonstrated “respect for and inclusion of others regardless of religion, ethnicity, class, gender, ability, appearance or sexual orientation,” and at no program was there any evidence of bias. B. The physical environment is safe and free of health hazards. Programs in the QIS are free of health and safety hazards, as measured by the second scale in this domain. Their spaces are kept clean and sanitary, ventilation and lighting are adequate, and the temperature is comfortable. C. Emergency and safety procedures are in place to protect youth. Safety procedures are well-established in QIS programs. This includes the accessibility of first aid kits, fire extinguishers, and written procedures as well as supervision of youth. D. Program space and furniture/materials accommodate activities. All programs in the QIS have ample space for OST activities. E. Healthy food and drink are provided. In 2014, results of a Prime Time youth survey highlighted the importance of healthy food and drink for program quality and youth outcomes. Youth participating in expanded learning opportunities were asked whether the activities involved new learning, problem solving, collaboration, challenge, a sense of belonging, a positive social atmosphere, and more. Youth were also asked to indicate whether they felt hungry, tired, or upset just prior to taking the survey. Youth who were hungry rated the activities as substantially less engaging in every way, suggesting that healthy food and drink can translate into greater youth engagement. Without exception, all QIS programs ensured that “at least one or two of the food choices are healthy (e.g., there are vegetables, fresh fruit, and/or real juice).” All but one program ensured that “food and drinks are plentiful and available at appropriate times for all youth.” 8
than 4.5 on any of the five scales in this domain. Programs score this high on this domain every year. The vast majority o...
II. Supportive Environment All programs in the QIS provide a welcoming atmosphere for youth (scale F), ensure that their session flow is planned, presented, and paced for youth (scale G), and effectively maintain clear limits (scale H). No programs earned scores less than 3.0 on these scales. In contrast, all programs continue to improve on their capacity to encourage youth, build youth skills and reframe conflict (scales I, J, K, and L). Figure 2. Average PBC-PQA scores for each scale in domain II by number of years programs have participated in the QIS. Baseline to 2 years 3 to 4 years 5 or more years 5 3 1 F. Welcoming atmosphere G. Optimum session flow H. Clear limits I. Engaging activities J. K. Support for Encouragement skill building L. Conflict reframing F. Staff provides a welcoming atmosphere. Among all programs in the QIS, staff provide a welcoming atmosphere for all youth. The lowest score on this scale was 4.0. Prime Time quality advisors support programs in maintaining this positive atmosphere. G. Session flow is planned, presented, and paced for youth. Session flow includes starting and ending sessions within ten minutes of scheduled times, having ample materials and supplies ready, clearly explaining activities and providing instructions to youth, and allowing an appropriate amount of time for activities. The lowest score on this scale was 3.53. H. Staff effectively maintains clear limits. Programs are encouraged to create a balance between choice and structure, although these goals are not mutually exclusive (Baumrind, 1996). Setting clear limits requires classroom 9
II. Supportive Environment All programs in the QIS provide a welcoming atmosphere for youth  scale F , ensure that their s...
management skills, which have a huge impact on quality. Average scores on this scale are high for all programs in the QIS with a minimum score of 3.0. As a result, youth likely experience positive outcomes associated with clear limits. Numerous researchers have discovered that when youth experience consistency, clear rules, reasonable boundaries, clear expectations, and predictability, they benefit tremendously (Eccles & Gootman, 2002; Caron, Weiss, Harris, & Catron, 2006; Day, Peterson-Badali, & Shea, 2002; Jackson, Henricksen, & Foshee, 1998; Lamborn, Mounts, Steinberg, & Dornbusch, 1991; Smetana, 1995; Steinberg, 2001; Steinberg & Elmen, 1986; Steinberg, Elmen, & Mounts, 1989; Barber et al., 2005; Meteyer & Perry-Jenkens, 2009). I. Activities support active engagement. Established programs support more active engagement than programs newer to the QIS. When youth are actively engaged, they are creating with materials, working with ideas, or improving skills with guided practice. Their engagement leads to tangible products or performances reflecting their contributions. They have clear opportunities to share what they’ve created, developed, or learned, and a balance evolves between the physical (e.g., sculpting, visiting the museum, writing a story) and mental or emotional (e.g., learning, reflection, discussion) elements of the activity. J. Staff support youth in building new skills. Differences in staff support of skill building were not apparent between established and less experienced programs. However, nearly 70% of all programs scored above 3.0 on this scale. (Thirty-seven programs scored below 3.0 on this scale with one program receiving an average of 1.0, indicating no support for youth in building new skills.) Programs can support youth by encouraging them to try new skills, providing useful feedback, communicating a specific learning or skill building focus, or breaking difficult tasks into smaller, simpler steps. K. Staff support youth with encouragement. Encouraging youth begins with active involvement between staff and youth. Staff encourage youth by acknowledging their contributions and asking frequent, open-ended questions. In offering encouragement, only five programs received an average score below 3.0. L. Staff use youth-centered approaches to reframe conflict. Because conflicts do not always occur during observations by assessors, only 33 programs received a score for this scale. Reframing conflict also draws upon classroom management skills. Using youth-centered approaches means that staff ask about or acknowledge the feelings of all youth involved, help youth respond calmly, ask those involved what happened, and encourage youth to develop solutions. 10
management skills, which have a huge impact on quality. Average scores on this scale are high for all programs in the QIS ...
III. Interaction High-quality interactions between staff and youth as well as youth and their peers enable youth to experience the benefits of healthy relationships and engaging social activities. In the domain of interaction, programs have some room for improvement. However, more established programs demonstrate higher quality on this domain overall compared to newer programs. Figure 3. Average PBC-PQA scores for each scale in domain III by number of years programs have participated in the QIS. Baseline to 2 years 3 to 4 years 5 or more years O. Youth can facilitate groups and… P. Youth can partner with adults 5 3 1 M. Sense of belonging N. Participation in cooperative groups Q. Positive peer relationships M. Youth have opportunities to develop a sense of belonging. Programs facilitate a sense of belonging by providing youth with structured opportunities to get to know their peers, preventing exclusion, and enabling youth to receive public acknowledgment or attention for their achievements or contributions. Assessors also gauge whether youth “strongly identify with the program offering.” These elements of quality correspond to the four items in scale M. All programs successfully prevent exclusion or take actions to include youth who are in danger of being left out. (The average score for this item was 4.94.) The vast majority of programs also scored high on youth identification (M = 4.48, SD = .99). In contrast, not all programs provide opportunities for youth to receive attention from staff and their peers for their contributions (e.g., group presentations, celebrations, exhibitions, performances). However, more established programs (M = 3.34, SD = 1.16) provided more opportunities than newer programs (M = 2.86, SD = 1.29), suggesting that programs improve in this area as a result of participation in the QIS. 11
III. Interaction High-quality interactions between staff and youth as well as youth and their peers enable youth to experi...
N. Youth have opportunities to participate in cooperative groups. Participation in cooperative groups involves more than group activities. When youth have special roles and a purpose toward which all group members strive, cooperation is enhanced. In general, programs receive adequate scores on two out of three items on this scale. Staff provide opportunities for youth to work together in teams or groups (M = 3.14, SD = 1.82), and cooperative groups have a purpose (M = 3.07, SD = 1.81). However, fewer programs provide youth with interdependent roles during group activities (M = 2.60, SD = 1.72). More established programs score higher on this item than newer programs. O. Youth have opportunities to act as group facilitators and mentors. Opportunities to lead and mentor allow youth to develop critical social skills. While most programs, particularly those that have participated in the QIS for more than two years, succeed in providing a variety of opportunities for youth to develop group-interaction skills (M = 4.05, SD = 1.35), they provide mentorship and leadership opportunities less often. This represents an area for improvement for all programs in the system. P. Youth have opportunities to partner with adults. Effective relationships between youth and adults are connected to positive feelings in youth and reduced discipline problems in school (Marzano & Marzano, 2003). This scale comprises two items. The first item measures the extent to which staff share “control of most activities with youth, providing guidance and facilitating while retaining overall responsibility.” The second item measures whether staff provide explanations for their expectations, directions, or guidelines. In general, programs receive satisfactory scores for both items. Q. Youth have opportunities to develop positive peer relationships. Programs in the QIS foster positive relationships among youth. The final scale in the domain of interaction explores how youth in each program treat one another. Positive peer relationships are inferred based on respectful language, a warm tone of voice, eye contact, and friendly gestures. The vast majority of programs (73%) received a perfect score on this scale, and only seven scored below 4.0. IV. Engagement Youth engagement is a critical program goal. Many youth benefits do not occur unless youth are engaged. Engagement, or participation, is a better predictor of youth benefits than attendance alone (Roth, Malone, Brooks-Gunn, 2010; Cross et al., 2010; Shernoff & Vandell, 2010). As programs participate in the QIS, they engage youth more frequently and effectively with each passing year. Programs struggle most with this domain of quality, but scores for this domain rise dramatically as a result of targeted improvement efforts. 12
N. Youth have opportunities to participate in cooperative groups. Participation in cooperative groups involves more than g...
Figure 4. Average PBC-PQA scores for each scale in domain IV by number of years programs have participated in the QIS. Baseline to 2 years 3 to 4 years 5 or more years 5 3 1 R. Opportunities to make plans S. Opportunities to make choices based on interests T. Opportunities to reflect R. Youth have opportunities to make plans. More established programs scored higher on this scale than newer programs. Nevertheless, most programs do not provide youth sufficient opportunities to make plans. Often, practitioners must follow lesson plans and activity plans from which they cannot deviate. Scores on this scale are lower than any other scale, and among the 42 programs that chose to focus on this aspect of quality in the 2012-2013 quality improvement cycle, only nine improved. Future quality improvement efforts will focus on guiding and training practitioners to create opportunities for youth to make plans. S. Youth have opportunities to make choices based on their interests. Programs in the QIS provide youth many opportunities to make choices about how they complete an activity (i.e., the process involved, their roles, use of tools or materials, or the order of activities), according to average scores on the first item in scale S (M = 3.58, SD = 1.10). Often, youth are able to choose among options they themselves determine. Programs also provide youth opportunities to make choices about content (e.g., deciding on the topic of an activity), but the choices are more often restricted to those presented by the practitioner (M = 2.91, SD = 1.24). T. Youth have opportunities to reflect. Reflection allows youth to understand the rationale and take-home message of activities they experience in their program. Time for reflection can be pivotal for achieving youth benefits. A recent meta-analysis of 49 studies from around the world found that structured time for reflection 13
Figure 4. Average PBC-PQA scores for each scale in domain IV by number of years programs have participated in the QIS. Bas...
enabled community service programs to achieve substantial benefits for youth, whereas programs lacking time for reflection achieved almost no noticeable effects (Society for Research in Child Development, 2014). V. Youth Centered Policies and Practices External assessors interview programs to determine the extent to which the needs of youth shape their policies and practices. Scale A gauges whether program offerings tap youth interests to build multiple skills. Scale B examines whether youth have an influence on organizational decisions. (Only ten programs received a score on this scale, because this only applies to middleschool youth.) More established programs (those participating in the QIS for five or more years) score higher on this domain than newer programs, according to a repeated-measures multivariate analysis, F(2,97) = 3.39, p = .038 (see Figure 5 and Table 3 in the appendix). Figure 5. Average PBC-PQA scores for each scale in domain V by number of years programs have participated in the QIS. 5 Baseline to 2 years 3 to 4 years 5 to 6 years 3 1 A. Youth interests influence programming B. Youth influence organizational decisions 14
enabled community service programs to achieve substantial benefits for youth, whereas programs lacking time for reflection...
VI. High Expectations for Youth and Staff Assessors also explore each program’s expectations for youth and staff. Assessors determine whether organizations promote staff development (scale C), supportive social norms (scale D), and academic enrichment (scale E). More established programs score higher on this domain than newer programs, F(2,97) = 4.39, p = .015 (see Figure 6). Figure 6. Average PBC-PQA scores for each scale in domain VI by number of years programs have participated in the QIS Baseline to 2 years 3 to 4 years 5 to 6 years 5 3 1 C. Staff development D. Supportive social norms 15 E. Academic enrichment
VI. High Expectations for Youth and Staff Assessors also explore each program   s expectations for youth and staff. Assess...
VII. Organizational Management Programs are assessed on their business practices (scale F), the effectiveness of their organizational logistics (scale G), and whether staff education and field-specific training meet county standards (scale H). Importantly, programs can receive a score of zero on items in scale H. If a program receives a zero on scale H, staff have less than one year of experience working with youth, less than ten hours of field specific training, no credentials, no Youth Development College Credit Certificates (YDCCC) and less than 30 college credits in a related field. In figure 7, zero is not included in the axis on the left. While county standards factor in, scores on scale H for programs in Palm Beach County are not indicative of how well programs meet county standards. Rather, higher scores indicate that programs are going above and beyond basic requirements set by a variety of state and local entities. Figure 7. Average PBC-PQA scores for each scale in domain VII by number of years programs have participated in the QIS. Baseline to 2 years 3 to 4 years 5 to 6 years 5 3 1 F. Sound business practices G. Effective logistics H. Staff education meets standards No significant differences in scores on this domain distinguish new and more established programs (p = .09). However, scores on scale H (staff education and training) are higher for more established programs (see Figure 7). 16
VII. Organizational Management Programs are assessed on their business practices  scale F , the effectiveness of their org...
VIII. Family The last domain on the PBC-PQA examines family involvement. As part of their interview, programs are assessed on whether they support positive communication with family (scale I) and family involvement (scale J). More established programs show higher levels of positive communication and family involvement, F(2,97) = 10.84, p < .0001. Figure 8. Average PBC-PQA scores for each scale in domain VIII by number of years programs have participated in the QIS Baseline to 2 years 3 to 4 years 5 to 6 years 5 3 1 I. Positive communication with family J. Family involvement QUALITY IMPROVEMENT The overarching goal of the QIS is to drive meaningful progress in quality from one year to the next. Substantial changes to the PBC-PQA tool make it difficult to compare the latest quality assessment scores with scores from the previous year. Despite increases in actual quality, scores on the new version of the tool were lower than scores on the previous version, according to a multivariate analysis with year included as a within-subjects factor, F(8,90) = 5.86, p < .0001. This shift (lower scores on the revised version of the PBC-PQA) was only evident in domains of Form A that were revised (according to within-subjects contrasts). Again, this is due to differences in how quality was measured by the two versions of the tool. Despite this shift in the tool and resulting scores, improvements were seen in some areas that programs chose to focus on throughout the year. Each year, programs select several areas of 17
VIII. Family The last domain on the PBC-PQA examines family involvement. As part of their interview, programs are assessed...
quality as focal points for improvement. Between the 2011-2012 and 2012-2013 quality improvement cycles, programs made significant progress on their improvement plans. The most common areas chosen for improvement in 2013-2014 were: • • • • • • encouraging youth (in particular, asking open-ended questions, II-K, item 3) giving youth opportunities to make plans (IV-R) giving youth opportunities to participate in cooperative groups (III-N) giving youth opportunities to reflect (IV-T) giving youth opportunities to make choices based on their interests (IV-S) giving youth opportunities to act as group facilitators and mentors (in particular, providing structured opportunities for youth to lead a group, III-O, item 3). Programs that chose to focus on these areas as part of their improvement plans significantly improved (except on IV-R, giving youth opportunities to make plans), while programs that did not focus on these areas experienced a decrease in scores (according to repeated-measures analyses of variance; see below). Programs Encouraged Youth By Asking More Open-Ended Questions Thirty-six programs made it their goal to ask more open-ended questions of youth (II-K, item 3). As a result of coaching, training, and other QIS supports, these programs significantly improved on this item from an average score of 2.10 to 2.72, F(1,109) = 7.26, p = .008. Of note, in the revised version of the PBC-PQA tool, only grammatical changes were made to this item. Figure 9. Changes in average PBC-PQA scores for item 3, scale K (Domain II) for programs that did and did not choose to focus on this item as part of their improvement plan. 5 Year 6 3 Year 7 1 Not On Plan On Plan 18
quality as focal points for improvement. Between the 2011-2012 and 2012-2013 quality improvement cycles, programs made sig...
Programs Fostered More Cooperative Group Interactions Thirty-three programs focused on giving youth more opportunities to participate in cooperative groups (III-N). These programs improved significantly from an average score of 2.34 to 2.92, F(1,109) = 5.01, p = .027, despite changes to the tool. Figure 10. Changes in average PBC-PQA scores for scale N (Domain III) for programs that did and did not choose to focus on this item as part of their improvement plan. 5 Year 6 3 Year 7 1 Not On Plan On Plan Programs Gave Youth More Choices Fifteen programs focused on giving youth opportunities to make choices based on their interests (IV-S). For these programs, despite the increased difficulty of achieving the same score on items in this domain, scores dramatically improved from 1.94 to 3.84, F(1,109) = 20.67, p < .001. Figure 11. Changes in average PBC-PQA scores for scale S (Domain IV) for programs that did and did not choose to focus on this item as part of their improvement plan. 5 Year 6 3 Year 7 1 Not On Plan On Plan 19
Programs Fostered More Cooperative Group Interactions Thirty-three programs focused on giving youth more opportunities to ...
Programs Helped More Youth Reflect Seventeen programs focused on giving youth more opportunities to reflect on what they are doing or have done, to share and present what they have done, and to give feedback on activities (IV-T). Average scores increased from 2.97 to 3.83, F(1,109) = 8.73, p = .004. Figure 12. Changes in average PBC-PQA scores for scale T (Domain IV) for programs that did and did not choose to focus on this scale as part of their improvement plan. 5 Year 6 3 Year 7 1 Not On Plan On Plan Programs Created More Structured Opportunities for Youth to Facilitate Groups Twelve programs focused on providing youth more structured opportunities to lead or facilitate activities among a group of their peers (III-O, item 3). These programs made significant improvements while other programs did not, F(1,113) = 7.70, p = .006. Figure 13. Changes in average PBC-PQA scores for item 3 in scale O (Domain III) for programs that did and did not choose to focus on this item as part of their improvement plan. 5 Year 6 3 Year 7 1 Not On Plan On Plan 20
Programs Helped More Youth Reflect Seventeen programs focused on giving youth more opportunities to reflect on what they a...
Youth Need More Opportunities to Make Plans Programs are encouraged to continue striving to create opportunities for youth to make plans. Forty-two programs focused on giving youth more opportunities to make plans (IV-R), which includes supporting youth in making plans and encouraging youth to represent their plans in tangible ways (e.g., writing, diagrams, etc.). Average scores on this scale fell. However, scores fell a little less dramatically for those programs that chose this as part of their improvement plan, and items in this domain became more difficult in the revised version of the PBC-PQA tool. Figure 14. Changes in average PBC-PQA scores for scale R (Domain III) for programs that did and did not choose to focus on this scale as part of their improvement plan. 5 Year 6 3 Year 7 1 Not On Plan On Plan SERVING HIGH-NEED AREAS Most OST programs in the QIS are located in high-need areas, where high need is defined by the number of youth receiving free or reduced-priced lunch (FRL). Across Palm Beach County, nearly 9,000 out of 13,500 elementary school youth (or 65%) receive FRL. 5 Of those programs in the QIS, more than half are located in zip code regions where more than 80% of youth receive FRL (see Figure 14). 5 According to the American Community Survey reported by Palm Beach County Counts, more than 77,000 youth between the ages of 5 and 9 live in Palm Beach County. However, data from the Early Learning Coalition on the number of youth receiving free or reduced-priced lunch include only 13,500 enrolled elementary school students. 21
Youth Need More Opportunities to Make Plans Programs are encouraged to continue striving to create opportunities for youth...
Figure 2. Number of QIS programs in regions by percent of youth receiving free or reduced-price lunch. More QIS programs exist in areas where more youth receive free and reduced-priced lunch (β = .90, p < .0001, adjusted R2 = .48). The total number of youth in a zip code region does not, however, predict the number of QIS programs in that region (β = -0.27, p = .37). Program quality did not differ substantially between lower need areas and higher need areas, as defined by the percent of youth receiving free or reduced-price lunch. However, quality assessment scores on Form B of the PBC-PQA did differ somewhat based on need, R2 = .15, p = .002. A multiple regression analysis revealed that programs in higher need areas scored slightly lower on domains V (Youth Centered Policies and Practices) and VI (High Expectations for Youth and Staff) but slightly higher on domain VII (Organizational Management). 22
Figure 2. Number of QIS programs in regions by percent of youth receiving free or reduced-price lunch.  More QIS programs ...
CONCLUSION Prime Time launched the Palm Beach County Quality Improvement System in 2007. Since then, OST programs across the county, particularly those in high need areas, have joined the system and benefited from expert coaching, guidance, and training from quality advisors and professional development specialists at Prime Time. Program quality assessments provide strong evidence that the Quality Improvement System raises the quality of OST programs over time. More established programs, those that have participated in the system for five or more years, demonstrate higher quality in multiple domains compared to newer programs. Further, programs have shown dramatic improvement on areas of quality they chose to focus on. Future research on Prime Time's impact will explore change over time for each program based on two years of assessments using the same version of the assessment tool as well as the social and emotional benefits of improved quality for youth. 23
CONCLUSION Prime Time launched the Palm Beach County Quality Improvement System in 2007. Since then, OST programs across t...
ANNOTATED BIBLIOGRAPHY Afterschool Alliance (2005). Afterschool Programs: A Wise Public Investment. Afterschool Alert. Issue Brief No. 22, Afterschool Alliance. Washington, D.C. In this brief review, the Afterschool Alliance emphasizes the critical support that OST programs provide to youth and explain why programs are a worthwhile investment. High quality OST programs reduce costs associated with remedial services, grade level recurrence, dropout, substance abuse, crime, teen pregnancy and missed higher income opportunities in adulthood. Barber, B., Stolz, H., Olsen, J., et al. (2005). Parental support, psychological control, and behavioral control: Assessing relevance across time, culture, and method. Monographs of the Society for Research in Child Development, 70(4), i-147. Baumrind, D. (1996). The discipline controversy revisited. National Council on Family Relations, 45(4), 405-414. Caron, A., Weiss, B., Harris, V. & Catron, T. (2006). Caregiver behaviors and child psychopathology: Specificity, task dependency, and interactive relations. Journal of Clinical Child and Adolescent Psychology, 35, 34-45. Day, D. M., Peterson-Badali, M., & Shea, B. (2002, May). Parenting style as a context for the development of adolescents’ thinking about rights. Poster presented at the 9th Biennial Meeting of the Society for Research on Adolescence, New Orleans, L A. (ERIC Document Reproduction Service No. ED 464 746). Durlak, J.A., Weissberg, R.P., & Pachan, M.K. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 16, 294-309. OST programs with a focus on social and emotional learning lead to increased prosocial behavior, improved academic performance, reduced behavioral problems, and enhanced self-awareness, according to their meta-analysis of 68 studies. The greatest impact is associated with programs whose activities are well-coordinated, actively engaging, focused, and explicit in their focus on social skills. Durlak, J.A. & Weissberg, R.P. (2007). The impact of after-school programs that promote personal and social skills. Chicago: Collaborative for Academic, Social, and Emotional Learning. 24
ANNOTATED BIBLIOGRAPHY Afterschool Alliance  2005 . Afterschool Programs  A Wise Public Investment. Afterschool Alert. Iss...
A meta-analysis of 73 OST programs found that social-emotional learning leads to positive outcomes for youth. Youth experienced a variety of personal, social and academic benefits. These included an increase in self-confidence, self-esteem and school pride. Additionally, problem behaviors such as drug use and aggression were reduced. Eccles, J. & Gootman, J. A. (2002). Community programs to promote youth development. Committee on Community-Level Programs for Youth. Board on Children, Youth, and Families, Commission on Behavioral and Social Sciences Education, National Research Council and Institute of Medicine. Washington, DC: National Academy Press. Grissom R. J., Kim J. J. (2005). Effect Sizes for Research: A Broad Practical Approach. Mahwah, NJ: Lawrence Erlbaum Associates. Hirsch, B. J., Mekinda, M. A., & Stawicki, J. (2010). More than attendance: The importance of after-school program quality. American Journal Of Community Psychology, 45(3/4), 447452. Hirsh and colleagues review several studies demonstrating that positive youth outcomes of OST program attendance depend on program quality and levels of participation. Huang, D., Gribbons, B., Kim, K. S., Lee, C., & Baker, E. L. (2000). A decade of results: The impact of the LA's BEST after school enrichment initiative on subsequent student achievement and performance. Los Angeles, CA: UCLA Center for the Study of Evaluation, Graduate School of Education & Information Studies, University of California. Huang, D., Kim, K. S., Marshall, A., & Perez, P. (2005). Keeping kids in school: An LA's BEST example—A study examining the long-term impact of LA's BEST on students' dropout rates. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing. Huang, D., Coordt, A., La Torre, D., Leon, S., Miyoshi, J., Perez, P., & Peterson, C. (2007). The afterschool hours: Examining the relationship between afterschool staff-based social capital and student engagement in LA's BEST. Los Angeles: UCLA/CRESST. Huang and colleagues [preceding three papers] explored the impact of LA's BEST (Better Educated Students for Tomorrow), an afterschool program that incorporates educational enrichment activities and support into their regular program. One hundred elementary school programs serving 14,000 second through fifth graders in the Los Angeles area participated in their studies. Participation in the program led to increased school attendance, improved performance on standardized tests of reading and math, higher 25
A meta-analysis of 73 OST programs found that social-emotional learning leads to positive outcomes for youth. Youth experi...
aspirations for graduation and postsecondary education, and reduced high school dropout rates. Huang, D. & Dietel, R. (2011). Making afterschool programs better. Policy Brief No. 11, CRESST National Center for Research on Evaluation, Standards, & Student Testing. UCLA: Graduate School of Education & Information Studies. Jackson, C., Henriksen, L., & Foshee, V.A. (1998). The authoritative parenting index: Predicting health risk behaviors among children and adolescents. Health Education and Behavior, 25(3), 319, doi: 10.1177/109019819802500307. Kataoka, S., & Vandell, D. (2013). Quality of Afterschool Activities and Relative Change in Adolescent Functioning Over Two Years. Applied Developmental Science, 17(3), 123134. A two-year longitudinal study examined feedback from 186 middle school youth regarding their participation in high-quality OST program experiences to address (1) whether youth who described overall aspects of positive OST experiences would show greater gains in youth functioning in year two as reported by teachers and (2) whether youth feedback regarding specific aspects of quality OST programs were linked to teachers’ reports of youth functioning in year two. Through use of the After-School Environment Scale, development of autonomy, emotional support from staff and positive relationship-building with peers were examined. Elements of youth functioning were assessed, such as social skills, relationships with peers, behavior, and persistence. Positive feedback from youth was linked to their teachers’ reports. Decreases in aggressive behavior were associated with specific elements of program quality, such as greater emotional support for youth. Lamborn, S., Mounts, N., Steinberg, L., et al. (1991). Patterns of competence and adjustment among adolescents from authoritative, authoritarian, indulgent, and neglectful families. Child Development, 62(5), 1049-1065. London, R., Gurantz, O., & Norman, J. R. (2011). The effect of afterschool program participation on English language acquisition. Afterschool Matters, (13), 22-29. London and colleagues examined Hispanic youth over a four-year period (elementary-tomiddle school) to identify connections between OST program participation and English learning. Assessments used to measure language growth include the Annual Measureable Achievement Objectives (AMAO) 1 assessment. Youth in OST programs scored higher on English language milestone assessments than non-participants, and youth with greater attendance in OST programs were more likely to reach AMAO 1 status. 26
aspirations for graduation and postsecondary education, and reduced high school dropout rates. Huang, D.   Dietel, R.  201...
Little, P. M. D., Wimer, C., & Weiss, H. B. (2008). Afterschool programs in the 21st Century: Their potential and what it takes to achieve it. Harvard Family Research Project, Issues and Opportunities in Out-of-School Time Evaluation, 10, 1-12. For youth to be successful in a global economy, OST programs can contribute to the required knowledge and skills through “sustained participation in well-structured and well-implemented after school programs and activities.” A decade of research and evaluation has shown positive benefits in academic performance, social/emotional development, crime/drug/sex prevention and promotion of health & wellness. Examples include significant gains in standardized test scores, positive effects on reading and math achievement, improvements in self-worth/self-efficacy, decreases in delinquency and health risk behaviors, greater knowledge of nutrition and healthy behaviors. These positive outcomes are more likely when youth continuously attend high quality programs with high quality staff and inter-organization partnerships. Marzano, R. J., Marzano, J. S., & Pickering, D. J. (2003). Classroom management that works: Research-based strategies for every teacher. Alexandria, VA: Association for Supervision & Curriculum Development. McGraw K. O., & Wong S. P. (1992). A common language effect size statistic. Psychological Bulletin, 111, 361–365. Meteyer, K. & Perry-Jenkins, M. (2009). Dyadic parenting and children’s externalizing symptoms. Family Relations, 58(3), 289-302. Metz, R.A., Goldsmith, J., & Arbreton, A. J.A. (2008). Putting it all together: Guiding principles for quality after-school programs serving preteens. Lucile Packard Foundation for Children's Health. In this literature review, the authors find that positive outcomes for preteens, such as emotional well-being, prosocial behavior, and improved educational achievement, result when OST programs give youth opportunities to create clear goals, build skills, participate for longer durations in a variety of activities, and form positive adult-youth relationships. Programs also benefit youth when they include families by using effective communication techniques and creating a warm atmosphere, building a diverse staff, teaching youth to value unique cultures, and focusing on continuous program development. Naftzger, N., Hallberg, K., & Yang, T. (2014). Exploring the relationship between afterschool program quality and youth outcomes: Findings from the Palm Beach County Quality Improvement Study. Washington, D.C.: American Institutes of Research. 27
Little, P. M. D., Wimer, C.,   Weiss, H. B.  2008 . Afterschool programs in the 21st Century  Their potential and what it ...
Roth, J. L., Malone, L. M., & Brooks-Gunn, J. (2010). Does the amount of participation in afterschool programs relate to developmental outcomes?: A review of the literature. American Journal of Community Psychology, 45(3-4), 310-324. Shernoff, D. J., & Vandell, D. L. (2010). Engagement in after-school programs as a predictor of social competence and academic performance. American Journal Community Psychology, 45, 325-337. Smetana, J. (1995). Parenting styles and conceptions of parental authority during adolescence. Child Development, 66(2), 299-316. Smith, C., & Hohmann, C. (2005). Full findings from the Youth PQA validation study. Ypsilanti, MI: HighScope Educational Research Foundation. Society for Research in Child Development. (2014, July 24). Community service programs that include reflection found to be more beneficial to youth. ScienceDaily. Retrieved March 10, 2015 from www.sciencedaily.com/releases/2014/07/140724094212.htm. Steinberg, L. (2001). We know some things: Parent-adolescent relationships in retrospect and prospect. Journal of Research on Adolescence, 11(1), 1-19. Steinberg, L. & Elmen, J. (1986). Authoritative parenting promotes adolescent school achievement and attendance (p. 15). Wisconsin: Corp Author National Center on Effective Secondary Schools Madison WI. Steinberg, L., Elmen, J., & Mounts, N. (1989). Authoritative parenting, psychosocial maturity, and academic success among adolescents. Child Development, 60(6), 1424-1436. Tooley, M. & Bronfreund, L. (2014). Skills for Success: Supporting and Assessing Key Habits, Mindsets, and Skills in PreK-12. Washington, D.C.: New America, Education Policy. University of California at Irvine, Department of Education. (2001). Evaluation of California’s After School Learning and Safe Neighborhoods Partnerships Program: 1999-2000 preliminary report. Irvine, CA. University of California researchers found that participants in afterschool programs that included academic and enrichment activities scored higher on standardized reading and math tests than non-participating youth statewide. Participating youth also missed fewer days of school, and fewer were held back in school, saving the state approximately $11 million in 2000. 28
Roth, J. L., Malone, L. M.,   Brooks-Gunn, J.  2010 . Does the amount of participation in afterschool programs relate to d...
Vandell, D., Reisner, E., & Pierce, K. (2007). Outcomes linked to high-quality afterschool programs: Longitudinal finding from the study of promising afterschool programs. Irvine, CA: University of California & Washington, DC: Policy Studies Associates. Thirty-five elementary and middle-school OST programs serving nearly 3000 youth in 14 cities in eight states participated in this 2-year longitudinal study of youth outcomes. Participation in high-quality programs led to improvements in academic performance, social skills, and behavior. Viadero, D. (2007). High-quality after-school programs tied to test-score gains. Education Week, 27(13), 1. Based on an eight-state study of 35 OST programs, youth from disadvantaged backgrounds who frequently attended high-quality programs over a two-year period made greater academic gains over peers that did not participate in supervised activities. This trumps a previous study of federally-funded 21st Century Community Learning Centers that found no significant learning growth with results based on comparison groups of youth who did not regularly attend OST programs. The new study looked at three groups of youth: 1) “program-only group” who only attended OST programs and participated in nothing else; 2) “program plus group” who participated in OST programs and other extracurricular activities; and 3) “low supervision group” who did not attend OST programs on a regular basis. Not only did the researchers find growth in academics, but social and behavioral outcomes were greater. 29
Vandell, D., Reisner, E.,   Pierce, K.  2007 . Outcomes linked to high-quality afterschool programs  Longitudinal finding ...
APPENDIX: DOMAIN AND SCALE SCORES Table 1. Scores for each domain and scale on Form A. (N = 121) Domain and Scale Mean Standard Deviation Min Max I. Safe Environment 4.98 0.07 4.56 5.00 A. Psychological and emotional safety is promoted. 5.00 0.00 5.00 5.00 B. The physical environment is safe and free of health hazards. 4.96 0.15 4.00 5.00 C. Policies and procedures protect children and youth. 4.95 0.19 3.67 5.00 D. Program space and furniture accommodate the activities offered. 4.99 0.12 3.67 5.00 E. Healthy foods and drinks are provided. 4.99 0.12 3.67 5.00 II. Supportive Environment 4.24 0.37 3.06 4.94 F. Staff provides a welcoming atmosphere. 4.82 0.26 4.00 5.00 G. Session flow is planned, presented, and paced for youth. 4.75 0.33 3.53 5.00 H. Staff effectively maintains clear limits. 4.82 0.39 3.00 5.00 I. Activities support active engagement. 3.98 0.64 2.17 5.00 J. Staff support youth in building new skills. 3.46 1.03 1.00 5.00 K. Staff support youth with encouragement. 3.64 0.56 2.56 5.00 L. Staff use youth-centered approaches to reframe conflict. 3.42 1.09 1.00 5.00 III. Interaction 3.64 0.57 2.46 4.74 M. Youth have opportunities to develop a sense of belonging. 4.01 0.51 2.50 5.00 N. Youth have opportunities to participate in cooperative groups. 2.94 1.14 1.00 5.00 O. Youth have opportunities to act as group facilitators and mentors. 2.94 0.84 1.44 4.78 P. Youth have opportunities to partner with adults. 3.49 1.07 1.00 5.00 30
APPENDIX  DOMAIN AND SCALE SCORES Table 1. Scores for each domain and scale on Form A.  N   121  Domain and Scale  Mean  S...
Q. Youth have opportunities to develop positive peer relationships. 4.79 0.41 3.00 5.00 IV. Engagement 2.81 0.82 1.33 4.93 R. Youth have opportunities to make plans. 1.88 1.10 1.00 4.78 S. Youth have opportunities to make choices based on their interests. 3.24 0.95 1.33 5.00 T. Youth have opportunities to reflect. 3.30 1.12 1.00 5.00 Overall 3.91 0.40 3.01 4.82 Table 2. Scores for each domain and scale on Form B. (N=110) Domain and Scale Mean Standard Deviation Min Max V. Youth Centered Policies and Practices 4.42 0.62 2.50 5.00 A. Program offerings tap youth interests to build multiple skills. 4.46 0.59 2.50 5.00 B. Youth have influence on structure and policy in the organization. 3.76 1.36 1.40 5.00 VI. High Expectations for Youth and Staff 4.70 0.45 2.55 5.00 C. Organization promotes staff development. 4.81 0.52 1.00 5.00 D. Organization promotes supportive social norms. 4.70 0.61 3.00 5.00 E. Organization supports academic enrichment. 4.62 0.60 2.33 5.00 VII. Organizational Management 3.50 0.59 2.25 5.00 F. The administration utilizes sound business practices. 4.14 1.55 1.00 5.00 G. Organizational logistics are effective. 4.31 0.57 1.00 5.00 H. Staff education and field specific training meet county standards. 2.61 0.86 0.50 5.00 VIII. Family 4.40 0.57 2.33 5.00 I. Organization supports positive communication with family. 4.57 0.41 3.00 5.00 J. Organization supports family involvement. 4.24 0.91 1.00 5.00 Overall 4.25 0.40 2.47 4.97 31
Q. Youth have opportunities to develop positive peer relationships.  4.79  0.41  3.00  5.00  IV. Engagement  2.81  0.82  1...
Prime Time Palm Beach County, Inc. receives significant funding from the Children’s Services Council of Palm Beach County. Prime Time Palm Beach County, Inc. 2300 High Ridge Road, Suite 330 Boynton Beach, FL 33426 561-732-8066 ph 561-732-8094 fax www.primetimepbc.org
Prime Time Palm Beach County, Inc. receives significant funding from the Children   s Services Council of Palm Beach Count...