VIII. EVALUATING THE
EDUCATION COMPONENT



THE ANNUAL SELF-ASSESSMENT
How The Education Self-Assessment Team Conducts the Self-Assessment
The Education Coordinator's Role in the Self-Assessment

LOCAL PROGRAM EVALUATION EFFORTS
Process and Outcome Evaluations
Collecting Evaluation Information
Program Evaluation Techniques
Evaluation of Program Effectiveness
Using Evaluation Results
Obtaining Support for Making Changes

STRATEGIES FOR SUCCESSFUL EVALUATION

RESOURCES


THE RESPONSIBILITIES OF THE EDUCATION COORDINATOR
FOR EVALUATION


BEFORE READING THIS CHAPTER


SELF-ASSESSMENT: EVALUATING THE EDUCATION
COMPONENT

ANNUAL SELF-ASSESSMENT

1. Have I provided self-assessment training for my staff?
2. Do I provide the self-assessment team with all the information they need?
3. When the self-assessment identifies problems, do I work with staff and parents to address them as soon as possible?

LOCAL PROGRAM EVALUATIONS

4. Do I conduct local program evaluations in addition to the annual self-assessment?
5. Are the Education Component questionnaires written effectively:

a. Do they use specific language?
b. Do they use open-ended questions?
c. Do they ask one question at a time?
d. Do they use familiar terms?
e. Do they avoid questions that begin with "Why"?
f. Do they ask for suggestions?
g. Do we test the questionnaire before using it?
6. Do I decide what to evaluate before collecting information?
7. Do I review existing records before seeking additional information?
8. Do I include staff and parents in program evaluation?
9. Do I make use of program evaluation results?
10. When changes are necessary, do I:
a. make them one or two at a time?
b. include staff and parents in the process?
c. assess the impact of changes?
d. implement changes gradually?
e. provide support and training?
Review your responses, especially those you marked "Needs My Attention," and circle those topics you want to work on. List them below in order of importance to you.

THE ANNUAL SELF-ASSESSMENT

In keeping with Head Start's participatory management philosophy, each program conducts its own annual self- assessment of how successfully it has, complied with the Head Start Performance Standards. This assessment is the operating responsibility of the Policy Council/ Committee of each grantee or delegate agency. The annual self-assessment is required; it provides the local program with valuable information about the quality and impact of its services.

Typically, the Policy Council/Committee enlists the assistance of staff, parents, and representatives from the community in conducting the self-assessment These volunteers are organized into teams to examine the plans and operations of each component. Training is provided by program staff or consultants regarding the content of the Performance Standards and the process used to conduct the self-assessment. Activities, facilities, or plans that are found to be out of compliance or in need of improvement are addressed in a self-assessment report.

"SAVI" is an acronym for self-assessment/validation instrument, a document based on the Performance Standards that is often used by programs to conduct and document the self-assessment; Organized by component the SAVI includes each applicable policy, a cross-reference to the Performance Standards, guidance and suggestions for determining compliance, and a place to indicate whether or not the Head Start program is in compliance with each item. Program use of the SAVI is not required. Programs may use any instrument or process to conduct the self-assessment as long as the Policy Council/Committee maintains operating responsibility.

Once the self-assessment is completed, a plan is developed to address problems that relate to each component. In some. cases the problem: Can and should be addressed immediately; an example of this would be recharging a fire extinguisher. Other problems will require a more detailed review before an improvement plan can be implemented. For example, if the self-assessment team finds the Education Component out of compliance with regard to sex-role stereotyping, the improvement planning might include further examination of the curriculum, teacher attitudes and behaviors, and books and other classroom materials, as well as observation of how the children are encouraged or discouraged with, regard to sex-role stereotyping. The results of this review will then be used to devise a plan to bring the component into compliance. The plan might suggest actions such as purchasing new materials providing training for staff and parents, revising the curriculum, or introducing other techniques and activities.

As with other activities in Head Start, a participatory approach to problem solving is recommended. Staff advisory committees, and parents work together to complete and implement improvement plans.

Head Start grantees are funded on a yearly cycle. Before the program's yearly grant application is approved, the Program Specialist in the regional office responsible for the program validates, the self-assessment and reviews improvement plans and other program documents, including the grant application and budget. If the program is in compliance with the Head Start Performance Standards and no other problems exist, the grant application is approved and funding continues.

HOW THE EDUCATION SELF-
ASSESSMENT TEAM CONDUCTS THE
SELF-ASSESSMENT

The team of staff, parents, and community representatives who review the Education Component are usually trained to use three assessment techniques: observation interviews, and review of written records and documentation. Specific assessment responsibilities are as signed to team members so that the work can be completed within the established time. Self-assessment team members also need to coordinate their findings with teams examining related components.

OBSERVATIONS

Classroom observations are the most time-consuming activity. It is advisable for the Education Coordinator to meet with the team before the classroom observations take place so that team members understand the overall philosophy, goals, and objectives of the component. Ideally, each classroom will be visited, with a pair of observers arriving before the children and staying for the entire session. If possible, each pair of observers should include one person who has had some experience in making classroom observations. Also, it is strongly suggested that parents do not observe in their own child's classroom. It is very difficult for the parent to be objective and their child's presence in the classroom may be too distracting. Observations are confidential; they are shared only with members of the self-assessment team.

Observations should take place on a typical day, rather than when a special event is to take place, such as a party or a field trip. Also, observers should try to keep to their task rather than interacting with the children, because such interaction can change the climate or activities in the classroom. Teachers can explain to the children ahead of time why the visitors are there.

Observers look for examples of how the children are encouraged in their cognitive, social, emotional, and physical growth. They also take note of the kinds of activities occurring, scheduling, materials available, and the physical environment. Throughout the visit, observers will be watching the interactions in the classroom-children with adults, children with children, and adults with adults.

The observation day also includes an inspection of the facility, including hallways, bathrooms, playground, etc. Observers will be looking for potential health and/or safety hazards and evidence of licensing as well as assessing the overall atmosphere of the center.

Members of the education assessment team also observe several home visits. If the program has a home-based option, then observation of a larger sample of home visits is needed. Parents should be asked in advance to grant permission for the team to visit their homes. The focus of these observations is on how the parent and home visitor work together to set goals for the child and on the training provided to the parent by the home visitor.

INTERVIEWS

Classroom and home observations are not the only self- assessment techniques. Team members also will need to interview the Education Component staff and the Education Coordinator to clarify observations and to find out more about non observable parts of the Education Component. Interview questions are used to determine compliance with the Head Start Performance Standards.

Parent interviews also provide valuable information about program operations. These can include one-to-one interviews with classroom volunteers and group inter views with parents at center meetings or Policy Council! Committee sessions. Responsibility for this task may be shared with the team assessing the Parent Involvement Component.

REVIEW OF WRITTEN MATERIALS

Review of written materials begins with a careful reading of the Education Component Plan to determine compliance with the requirements of the Performance Standards and to get a full picture of all aspects of the Education Component. Other records reviewed include the following.

THE EDUCATION COORDINATOR'S
ROLE IN THE SELF-ASSESSMENT

The Education Coordinator assumes several roles in the self-assessment process. These include encouraging staff to be positive and cooperative, conducting staff training on the self-assessment, providing information to the review team or serving as a member, and completing follow-up activities.

ENCOURAGING POSITIVE ATTITUDES

As the leader of the Education Component, the Education Coordinator's attitude toward the self-assessment is extremely important. If the Education Coordinator sees this task as a cumbersome and nonproductive process, then this will be conveyed to staff and parents. However, if the Education Coordinator recognizes that the self assessment is an essential element of the Head Start planning and evaluation cycle, staff and parents will also develop positive attitudes. The Education Coordinator should convey to staff and parents the value of the self assessment in maintaining and improving the overall quality of services provided by the component.

One of the key benefits derived from participating as a member of the self-assessment team is a greater understanding of the total Head Start program. Parents who have been active classroom volunteers are an excellent choice to assess the Education Component. Until they go through the process of looking at how the Education Component implements the Head Start Performance Standards, they. may not realize that there is a purpose for everything that goes on in the classroom.

PREPARING STAFF FOR THE
SELF-ASSESSMENT

Staff should receive training on the self-assessment process and on the techniques that the self-assessment team will be using. Three areas must be discussed.
 

MEETING WITH THE SELF-ASSESSMENT TEAM

The Education Coordinator will also meet with the self-assessment team responsible for reviewing the Education Component. This meeting provides an opportunity to explain the philosophy, goals, and objectives of the component and any terms that may be unfamiliar to someone who doesn't work regularly in a Head Start classroom.

In some programs, the Education Coordinator is asked by the Policy Council/Committee to either serve on the self-assessment team for the Education Component or chair it. Other programs use an existing education advisory committee to conduct the self-assessment of the Education Component.

AFTER THE ASSESSMENT IS OVER

When the self-assessment is complete, the assessment team will prepare a report about noncompliance items and areas in need of improvement. It's a rare Education Component that is doing everything as well as it possibly can. Often the team will also identify compliance and improvement objectives and suggest strategies for improvement. All of this information will be used by the Education Component work group (staff, parents, advisory committee, etc.) to prepare plans for the component, as discussed in Chapter III. Improvement plans will also be incorporated into the annual review and up date of the Education Component Plan.

In some programs, the self-assessment culminates in an exit interview with the assessment team, the Director, and other staff and parents in attendance. This meeting provides an opportunity for the assessment team to offer first-hand comments and suggestions for improving the program. It may be difficult to hear about problems within the Education Component, especially when staff have been working hard to provide a quality program. How ever, important lessons often are learned from the objective reviewers, who can see the program's strengths and needs more clearly than those who work in the program each day.

One final step in the self-assessment process is to share the results with staff. Then congratulate them for their cooperation and continuing efforts to provide quality services to children and families.


LOCAL PROGRAM EVALUATION EFFORTS

In addition to the annual self-assessment, many Head Start programs design and implement other evaluation efforts. Program evaluation activities serve a variety of purposes within the program.

Without some kind of program evaluation, it is difficult to pinpoint the results of the combined efforts of Education Component staff and volunteers. Although the annual self-assessment provides the Education Component with some objective information about compliance and progress, other regular program evaluation activities are needed so that there is an ongoing source of data concerning the program's successes.

PROCESS AND OUTCOME
EVALUATIONS

These two terms are not as complicated as they may sound. Process evaluations look at how a program is implemented and examine compliance, or solicit reactions and suggestions. The annual Head Start self-assessment is a process evaluation: Other examples are staff and parent questionnaires, logs of parent comments, interviews with parents, or end-of-the-year meetings. Process evaluations elicit likes and dislikes, reactions to specific activities, and/or suggestions for changes.

Outcome evaluations are concerned with what the program has accomplished. These evaluations gather information that can be used to measure changes in attitude, knowledge, or practices. The long-term studies of how the Head Start experience has affected children are examples of outcome evaluations, as are evaluations of how teachers apply training experiences to classroom activities.

COLLECTING EVALUATION
INFORMATION

Questionnaires often are used to collect the information needed to conduct a process evaluation. These may be the basis for group discussions, one-to-one interviews, or surveys to be completed at home by parents or staff members. Writing questionnaires effectively is not difficult, but there are several points to keep in mind.

If the questionnaire will be distributed rather than used in a group or one-to-one setting, then clear written instructions are necessary. These can take the form of an introductory letter that also tells the respondents how and why the information is going to be used. Also, the letter should state that completing the questionnaire is voluntary and that names are not required.

PROGRAM EVALUATION TECHNIQUES

A first step in program evaluation is for the Education Coordinator and staff to identify what needs to be evaluated. This might be an area in which there have been complaints, one identified through a records review, a new activity, one the Education Coordinator is thinking about changing, or one that hasn't been looked at in a long time.

The second step is to determine what information is needed to conduct the evaluation, and whether or not it already exists in the Education Component's records or those of another component. Records such as volunteer reports, parent training attendance sheets, agendas and evaluations, minutes, and home visit reports should be reviewed to see if they contain the information needed. For example, an Education Coordinator may want to find out if parents are satisfied with the activities offered by the component. If the records document when participation occurred and how many parents attended, but don't provide the information needed to determine satisfaction, techniques that could be used are a survey or questionnaire sent home with the children, individual interviews with a sample of parents, or a group discussion at a parent meeting. Sample parent evaluations for center-based and home-based options are included in the Resource Papers (VIII- 1-8).

Program evaluation information is gathered from staff through the Education Coordinator's ongoing interactions, through questionnaires or surveys, and through discussions at staff meetings. Staff can provide evaluative information based on their actual experiences in the classroom and their work with parents.

Staff may be asked to evaluate individual component activities, provide an end-of-the-year review, or assess the effectiveness of the support and guidance provided by the Education Coordinator. A sample staff evaluation form is included in the Resource Papers (VIII-9-13).

The overall purpose of evaluation efforts is to find out if the activities of the Education Component are successful. In addition to the questionnaires for staff and parents, there are many other ways to find the answer to this question-use the ones that seem most appropriate. Some suggested evaluation opportunities include the following.
 

EVALUATION OF PROGRAM
EFFECTIVENESS

This type of evaluation is used to assess what the program has accomplished. Year-end assessments of a representative sample of children are often used to determine if the children as a group have progressed in the various developmental areas as planned. They can answer questions, such as, Have language skills improved? Are children more independent? Have their attention spans increased? Program planners review the assessment results and the ongoing observations of staff and parents. Progress or lack of progress in the developmental areas helps the staff to determine whether the program is effective as designed and implemented. Failure to achieve program goals indicates the need to assess the curriculum, staff performance, and the learning environment.

STEPS IN EVALUATING PROGRAM EFFECTIVENESS

Program evaluations are usually conducted every two or three years. The assessment instrument or combination of instruments used must be in line with program goals and relevant to the population served. This means that the items in the instrument match the goals of the program and therefore will collect data on behavior, skills, and knowledge that the program is designed to impact. If the population served is Spanish speaking, the instrument must be one that is available in Spanish (See Chapter IV, the section on "Individualizing the Program" for a discussion of assessment.)

Once the instrument is selected, a representative sample of children is identified (approximately 50 percent in a small program and 25 percent in a large program. The children are given a pre-test at the beginning of the year and a post-test at the end using the same instrument. It may be a good idea to have an independent assessment team conduct the program evaluation-someone from outside the program who may be more objective. At a minimum, those who administer the test should receive some training.

The results of the assessment are analyzed for statistical and programmatic significance. For example, in a program with a small number of children, changes between the pre-test and post-test may not be statistically significant but could still be programmatically significant.

USING EVALUATION RESULTS

One final but important point about program evaluation is that the results should, be used by the staff, parents, and the Education Coordinator to upgrade program quality and to promote the program. Evaluation is not complete until the results have been tallied, analyzed, discussed, and then plugged back into the planning process. Program evaluation results can also be used to:

OBTAINING SUPPORT FOR MAKING
CHANGES

Decisions to make changes in the program sometimes result in position from some staff, parents, or the Director. If the evaluation information documents the need for these changes, they will be easier to sell. Other suggestions for encouraging support for changes include the following.


STRATEGIES FOR SUCCESSFUL EVALUATION


RESOURCES

Head Start Self-Assessment Guidance. Region X, ACYF, Third and Broad Building, 29O1 Third Avenue, Seattle, WA. 1979. Contact the regional office for availability.

This manual was developed to provide Head Start programs with guidance on how to actually conduct the self-assessment. It includes information on selecting and training assessment team members, gathering data, developing the report, and conducting the evaluation.

Morris, Lynn Lyons, and Fitz-Gibbon, Carol Taylor. Program Evaluation Kit. Beverly Hills,
CA: Sage Publications, Inc.

This kit contains eight books written to guide and assist individuals in planning and managing evaluations. These practical, field-tested, step-by-step guides all offer detailed advice, supported by clear definitions and useful procedures explained in nontechnical language. The volumes can be purchased individually or as a complete set.

Taylor, Mary. SAVI: Do It! Salem, UT: Kandor House Publications, Inc., 1984.

This revised manual outlines a field-tested system for parents to use in conducting the annual self-assessment of die program. The manual includes instructions, training agendas, and all the necessary forms for conducting the self-assessment and writing the actual report.


INDEX

Advocacy Assessment, developmental Bilingual/multicultural education Budgeting Child Development Associate (CDA) Program, 7, 160, Classroom management Collaboration, 6, 22, 44, 84-85, 122, 160
Communication Community needs assessment, 45-47
Confidentiality Cross-component coordination, 6, 22, 44, 79-80, 122, 160
Curriculum Decision making, 26-27

Early childhood education, quality

Education advisory committee, 44-45
Education Component, function of, 5, 21, 22
Education Component Plan, development of, annual self-assessment, 43-44 Education Component Plan, implementation of, 10, 7 1-105 Environment Equipment. See Materials and equipment
Evaluation local program, 10, 181-194 Evaluation, staff. See Staff supervision
Evaluation, staff and parent training, 160, 168-169
Goals Handicapping conditions, children with. 6, 8, 84-85, 162. Health Component, function of, 6
Home-based option Home visits Individual Education Plan, 8,84-85, 117, 118
Individualizing Linkages Materials and equipment Networking, 24
Observation, children. See Individualizing
Observation, staff. See Staff supervision
Observation, self-assessment, 187-188
On-site visitation, 24-26, 78, 83-84, 138, 171
Parent involvement Parent Involvement Component, function of, 5-6
Participatory management, 43, 187
Performance Standards Policies and procedures, 113-115
Policy Council/Committee Record keeping Resource Education Coordinator as a, 23-26
Responsibilities of the Education Coordinator, 9-10, 39, 73, 97,
    111, 118, 131, 155, 160-161, 183
Screening, 79-80.
Self-assessment, annual 70.2 (Appendix -B-Head Start Policy Manual: The Parents), 43, 45
Social Services Component, function of, 6
Staff supervision, 10, 22, 129-150 Staffing the Education Component, 23, 59-63 Strategy for Spanish-speaking children, 8-9
Stress management, 28
Time management, 27-28
Training Volunteers