CONFERENCE EVALUATION TOOLKIT
REFERENCE GUIDE

TABLE OF CONTENTS

Introduction

Overview of OSTRC Conference Evaluation Toolkit
Common Questions

Conference Evaluation Toolkit Surveys

Survey Descriptions
Survey Description Summary

Appendix A:  Development of the OSTRC Conference Evaluation Toolkit

Appendix B:  Additional Resources of Planning and Implementing Conference Evaluations

Overview of the Evaluation Toolkit

The Out-of-School Time Resource Center’s (OSTRC’s) Conference Evaluation Toolkit is designed for organizations which develop, provide, and/or evaluate OST conferences. The Toolkit is the result of a two-year study in which surveys were pilot-tested at multiple OST conferences. By using this Toolkit and contributing to an OSTRC database, organizations can evaluate their own conferences while providing the field with information to enhance professional development overall.

back to top

Common Questions

What is the OSTRC Conference Evaluation Toolkit?

The Conference Evaluation Toolkit is a collection of research-based surveys developed to measure the impact of OST conferences on staff, programs, and students.

Why should organizations use the Toolkit?

Information gathered through the Toolkit enhances specific efforts, provides evidence of success to administrators and funders, and can influence OST professional development on a larger scale. Using the OSTRC’s evaluation toolkit also helps develop and refine the tools themselves.

Can the Toolkit be used to evaluate workshops other than in conference settings?

The OSTRC does not recommend using these surveys outside of conference settings. The surveys were pilot-tested and are only reliable in conference settings. Moreover, there are differences between stand-alone workshops and conference workshops that are not addressed by these surveys. The OSTRC is currently pilot-testing other surveys designed to evaluate non-conference workshops.

What surveys are in the Toolkit?

• Post Workshop Survey
  
  
• Follow-Up Workshop Survey
  
  
• Overall Conference Survey
  
  
• Presenter Self-Assessment

Can surveys be combined or eliminated?

In certain circumstances, conference surveys can be combined. For example, the Post Workshop Survey can be divided into a Pre-Workshop and Post-Workshop Survey. The Overall Conference Survey can be expanded to include workshop questions, thus eliminating the individual Post-Workshop Surveys.

What services can the OSTRC provide with the Toolkit?

• Survey Design: The OSTRC can identify and adapt the evaluation tools to meet the particular needs of an organization.
• Staff Training: The OSTRC can train an organization’s staff to facilitate all aspects of the evaluation process including planning the activities, implementing the surveys, and analyzing the data.
• On-Site Evaluation: The OSTRC will implement the evaluation process, communicate with presenters and participants, and distribute and collect the surveys. 
• Analysis: The Conference Evaluation surveys utilize a rich mix of close-ended (quantitative) and open-ended (qualitative) questions.  There are three types of analysis that can be performed from the data: 
  1. Basic Analysis (which determines “frequencies,” or how often participants respond to specific questions)
  2. Intermediate Analysis (which explores multiple connections and comparisons)
  3. Advanced Analysis (which addresses significance testing)

Is there a fee for the Toolkit or services?

There is no fee for using the Toolkit as-is or for having OSTRC staff design organization-specific surveys. There is a fee for staff training, on-site evaluation, and analysis. All Toolkit users must sign a Letter of Agreement and share (anonymous) data gathered through the surveys with the OSTRC database. This data contributes to tool refinement and ongoing professional development research.

back to top

Survey Descriptions

A.  Post Workshop Survey

The Post Workshop Survey is administered at the end of each workshop and gathers the following information: 

• Comparative (before and after the workshop) responses regarding participants’ level of knowledge, skills, and belief in the importance of the workshop topic in terms of benefiting program youth.
• Participants’ feedback regarding workshop components such as the relevance of the information, the quality of the presentation, and the effectiveness of the activities.
• Participants’ expectations of how successfully they will apply the new information, receive organizational support, share information with others, and impact program youth. These questions are asked again in the Follow-Up Survey to ascertain the intended versus actual application of new information.

B.  Follow-Up Workshop Survey

The Follow-Up Workshop Survey is a web-based survey which is e-mailed to participants one month after the conference. It provides insight into whether new information was used, whether any barriers were encountered, what support systems were utilized, and whether student impact was observed.     

C.  Overall Conference Survey

The Overall Conference Survey is distributed at the end of the conference and used in conjunction with individual workshop surveys. It measures satisfaction with the conference logistics, reactions to keynote speakers and exhibitors, and similar elements applicable to the entire conference.

D.  Presenter Self-Assessment Survey

The Presenter Self-Assessment survey is completed by the presenter(s) after each workshop. Presenters answer questions similar to those on the Post-Workshop Survey, which allows analysts to compare different orientations to and perceptions of the workshop. Presenters are also asked to comment on the conference organization, their support from the conference committee, any “surprises” they encountered in their workshops, and similar issues. 

back to top

Survey Description Summary

Below is a grid that summarizes the data that can be gathered through each of the Surveys:

back to top

In the summer of 2004, the Out-of-School Time Resource Center (OSTRC) attempted to locate research-based survey instruments used to evaluate OST professional development. After an extensive literature review and conversations with local and national colleagues, staff determined that these instruments did not exist. The OSTRC then implemented a mixed method pilot study to design and test multiple surveys for use in OST conferences. 
As part of the planning process, the OSTRC reviewed literature on effectively implementing professional development [1]. This included several models of professional development evaluation from researchers Thomas Guskey, Joellen Killion, Donald Kirkpatrick, and others [2]. These models each identify and define “levels” of evaluating professional development (participants’ satisfaction, learning, application, and results). However, they were designed for formal education and relate only peripherally to OST programs. In creating an OST model, the OSTRC added another level of evaluating professional development:  extension, or adapting knowledge to suit a particular program and/or sharing this knowledge with OST staff, programs, or youth [3].
Using the theoretical frameworks described in the literature, the OSTRC developed a series of survey instruments that utilize a pre/post-test design and measure indicators of knowledge, skills, and attitudes as well as intended vs. actual application.  The surveys incorporate both close-ended (quantitative) questions as well as open-ended (qualitative) questions.
Research Questions
Initially, the OSTRC identified several research questions:

1. In what ways is an OST conference setting a valuable professional development experience for its participants, and in what ways is it limited?
2. What steps are involved in learning and applying information from an OST conference workshop?
3. Which types of workshops contribute to a substantial change in participants’ content knowledge, skills, and/or belief in the importance of the topic in terms of benefiting youth? 
4. To what extent do participants apply new knowledge/skills, or experience a change in belief, one month after the conference?

Pilot study

The OSTRC performed two major pilot studies with the evaluation surveys. The first involved one local conference while the second involved one statewide and one regional conference [4]. Between the first and second pilot, the OSTRC also conducted five focus groups with 50 local OST staff, including those who worked directly with children as well as those in administrative positions [5]. The purpose of the focus groups was to explore how participants felt they benefited from professional development, specifically in terms of affecting positive change in participants’ knowledge, skills, and attitudes. 
The surveys were revised after each pilot test and after the focus groups. Some insights gathered in the focus groups modified or contributed to the survey questions, while other information informed analysis of the survey data. 

Between November 2004 and May 2005, a total of 1,426 OST staff participated and 4,733 surveys have been collected.  Specifically, the OSTRC has collected 1,863 Pre Workshop Surveys, 2,084 Post Workshop Surveys, 457 Follow-Up Surveys, 93 Presenter Self-Assessment Surveys, and 236 Overall Conference Evaluations.

CITATIONS

1. National Staff Development Council (1995).  National Staff Development Council’s Standards for Staff Development, http://www.nsdc.org/standards/about/index.cfm; Gardner, H. (1983).  Frames of Mind:  The Theory of Multiple Intelligences.  New York:  Basic Books, PA Pathways (2002).  PA Pathways Training Resource Manual: Guidelines for Training Organizations, http://www.papathways.org/PDFs/TRM_AdLearn.pdf.

2. Guskey, T. (2000).  Evaluating Professional Development.  Thousand Oaks, California:  Corwin Press, Inc., Killion, J. (2002).  Assessing Impact:  Evaluating Staff Development.  Oxford, Ohio:  National Staff Development Council, Kirkpatrick , D.L. (1994). Evaluating Training Programs: The Four Levels.  San Francisco, CA: Berrett-Koehler.

3. Peter, N. (2004).  Out-of-School Time (OST) Professional Development Workshops:  An Evaluation Framework,    http://www.sp2.upenn.edu/ostrc/pdf/OSTWorkshopEvaluation.pdf

4. “Regional” in this context refers to the Mid-Atlantic region of the United States .

5. For more information on the findings from the OSTRC focus groups, see: http://www.sp2.upenn.edu/ostrc/pdf/OSTRCFocusGroupSummary.pdf

back to top

Bouffard, S. (2004).  “Promoting Quality Out-of-School Time Programs through Professional Development”.  Harvard Family Research Project, Evaluation Exchange, Volume X, No. 1, Spring 2004.

Gardner, H. (1993).  Frames of Mind:  The Theory of Multiple Intelligences.  New York:  Basic Books.

Guskey, T. (2000).  Evaluating Professional Development.  Thousand Oaks, California:  Corwin Press, Inc.

Killion, J. (2002).  Assessing Impact:  Evaluating Staff Development.  Oxford, Ohio:  National Staff Development Council.

Kirkpatrick , D.L. (1994). Evaluating Training Programs: The Four Levels.  San Francisco, CA: Berrett-Koehler.

National Staff Development Council (1995).  National Staff Development Council’s Standards for Staff Development.  Retrieved September 20, 2004 from http://www.nsdc.org/standards/about/index.cfm.

Halpern, R. (1999). After-school programs for low-income children: Promise and challenges. The Future of Children, 9, 81-93.

Harvard Family Research Project, Evaluation Exchange.  Issue Topic:  Professional Development.  Volume XI, No. 4, Winter 2005/2006.

Lauver, S. (2004). Issue Topic: Evaluating Out-of-School Time Program Quality.  The Evaluation Exchange, 10 (1).

Shortt, J. (2002).  Out-of-School Time programs: At a critical juncture.  New Directions for Youth Development, vol. 2002, no. 94, 119-123.

PA Pathways (2002).  PA Pathways Training Resource Manual: Guidelines for Training Organizations, http://www.papathways.org/PDFs/TRM_AdLearn.pdf.

Harms, T., Jacobs, E. & White, D.  (1995). School Age Care Environmental Rating Scale.  New York:  Teachers College Press.

Heck, S., Loucks, S.F. et al (1981). Measuring Innovation Configurations.  Austin, Texas:  Southwest Educational Development Laboratory.

Horsley, D.L, & Loucks-Horsley, S. (1998).  CBAM brings order to the tornado of change.  Journal of Staff Development, Vol. 19, No. 4.  Retrieved September 20, 2004, from http://www.nsdc.org/library/publications/jsd/horsley194.cfm.

Brookfield, S.D. (1986). Understanding and Facilitating Adult Learning.  San Francisco:  Joseey-Bass.

Knowles, M.S.  (1980). The Modern Practices of Adult Education:  From Pedagogy to Andragogy (Rev. Ed.).  Englewood Cliffs:  Cambridge Adult Education.

back to top