Use this guide to help you determine whether you have the resources and expertise within your initiative to plan and implement a needs assessment and program evaluation, or if it is best to hire a local evaluator to help you conduct these activities. This guide contains useful tips on what to consider in the decisionmaking and selection processes of hiring a local evaluator.

For more information on conducting needs assessments or program evaluations, see the Guide to Conducting a Needs Assessment and Guide to Performance Measurement and Program Evaluation.

About This Series

OVC's series of four technical assistance guides are tools for victim service providers and allied professionals, like you, who want or need to conduct program evaluations or needs assessments. The guides will help you make critical decisions and ensure that you make the best use of your funds to promote the goals and purposes of your program. These four guides were originally developed for OVC and the grantees who received funding to serve victims of human trafficking. The guides have since been adapted for use by other grantees and organizations that provide programs for victims of any type of crime.

Guide to Performance Measurement and Program Evaluation

Prepare goals and objectives, identify performance measures and program outcomes, identify evaluation questions, create a program planning or logic model, select evaluation design, decide on data collection methods, analyze and present data, use evaluation data

Guide to Conducting a Needs Assessment

Design your needs assessment, determine data collection methods, analyze and present data, use the needs assessment results for planning

Guide to Hiring a Local Evaluator

Find a local evaluator, determine questions to ask and what to look for, decide on what to include in job descriptions, find out how to work effectively with evaluators

Guide to Protecting Human Subjects

Read about the law related to protecting human subjects, learn about key issues you need to address when conducting research that involves human subjects, learn about Institutional Review Boards and how to involve them in your research

About This Guide

Overview

This guide addresses four basic questions:

1. How do I know if I should hire a local evaluator?
2. What are the advantages and disadvantages of hiring a local evaluator?
3. What do I look for in a local evaluator?
4. How do I find a local evaluator?

The guide includes a resources section with a list of additional resources available to assist you in selecting a local evaluator and a glossary of key terms and appendixes that include sample tools for hiring a local evaluator.

How To Use the Guide

You may or may not need to use the entire guide. If you are unsure about whether you should hire a local evaluator to assist you with the needs assessment and evaluation of your program, read at least the first three sections. If you know you need to hire a local evaluator but are unsure how to go about it, review the section on finding a local evaluator. If you are looking for tools to help you select a local evaluator, review the appendixes. This guide will help you address whatever needs you may have related to hiring a local evaluator.

Other guides are available in this OVC series to assist you with your needs assessment and program evaluation:

• Guide to Performance Measurement and Program Evaluation.
• Guide to Conducting a Needs Assessment.
• Guide to Protecting Human Subjects.

Should I Hire a Local Evaluator?

Evaluation is crucial to the ultimate success of service provision and is vital to determining whether your program is achieving its goals. During the planning stage, you should assess the evaluation capabilities of your initiative to determine if you need to hire a local evaluator. Exhibit 1 suggests questions to consider when making this determination.

Exhibit 1 Assessing Victim Service Providers and Allied Professionals’ Capabilities for Evaluation
Question	Circle Answer
Does the lead agency have a staff person who knows how to conduct a needs assessment or an evaluation of a comprehensive initiative?	Yes     No
Do any of the key partners in the initiative have a staff person who knows how to conduct a needs assessment or a program evaluation to suit your needs?	Yes     No
Does either a staff person from the lead agency or a key partner know enough about evaluation to conduct a needs assessment or an evaluation with the support of resources (e.g., manuals, toolkits, textbooks, guides, informal partners) that are available in the field?	Yes     No
Do staff have enough time to devote to conducting a meaningful needs assessment or evaluation?	Yes     No

If the majority of your answers to these questions is “no,” then you may need to hire a local evaluator to assist with planning and implementing the needs assessment or evaluation. If the majority of your answers to these questions is “yes,” you may still want to review the sections regarding the competencies, roles, and responsibilities of the evaluator to better determine the level of expertise available within your team.

What Are the Advantages and Disadvantages?

Use your answers from exhibit 1 (see "Should I Hire a Local Evaluator?") to help you decide whether hiring a local evaluator is in the best interest of your program. This section assists you in the decisionmaking process by outlining the advantages, as well as the potential challenges, of hiring a local evaluator. Advantages are presented in exhibit 2.

Exhibit 2 Advantages of Hiring a Local Evaluator
Advantage	Local Evaluator	Victim Service Providers and Allied Professionals
Specialized Knowledge	Possesses specialized knowledge and ability, understands how to assess the needs of a community, documents program outcomes, and collects and analyzes data	Often do not have expertise in the scientific aspects of evaluation
Reduces Burden	Reduces the burden on the program manager and staff who have limited time to devote to an evaluation	Focused on providing services to clients; may not have time to conduct an evaluation
Objective Viewpoint	Remains unbiased about the program under review, identifies the positive and negative aspects of a program, offers suggestions on how best to achieve the program’s goals	Commitment to the work and belief in the effectiveness of the program may result in a biased interpretation of assessment or evaluation results
Higher Level of Credibility	More knowledgeable and objective about the evaluation process, conclusions and recommendations tend to carry more weight with funding institutions, the field, and the general public	Less knowledgeable about evaluation; the field may view the results as less credible
Fresh Perspective	Brings to the table a different way of thinking about program effectiveness that is grounded in the findings from the data, not from previous working relationships	Prior working relationships may limit creativity regarding ways to offer services in a network

Along with the advantages, there are also potential disadvantages to hiring a local evaluator. Two challenges that may arise are expense and fit:

• Expense. Conducting a thorough needs assessment or program evaluation can be a costly and time-consuming process, especially for an evaluator who is not familiar with your community or program. To accelerate the startup process for a local evaluator, hire someone who has the competencies necessary for evaluating comprehensive services, is from your community, and has a basic understanding of crime victim services. If funds for evaluation are included in the program budget award, they can be used to help cover the expense. In the long run, hiring a local evaluator with specific evaluation expertise could be less expensive and produce better results than using existing staff.
• Fit. Hiring an unqualified evaluator or one who does not work well with your organization or partnering organizations can be damaging; such an evaluator may alienate staff, infringe on clients and staff-client relationships, and draw incorrect conclusions about the community and program. Take the time necessary to find a proper match and select the best evaluator for the job. This helps to ensure that the evaluation process is not overly burdensome and that the findings help you answer your research questions about how your program and services are working.

It is important to carefully weigh all potential advantages and disadvantages when deciding whether to hire a local evaluator or to rely on the evaluation capabilities within your organization, or collaborating organizations, to develop and carry out the overall evaluation plan.

What Do I Look For?

If, after careful deliberation, you decide to hire a local evaluator, it is often helpful to organize a hiring committee. The hiring committee should be as large as necessary to adequately represent the various stakeholders, but small enough so that the decisionmaking process is efficient and not overly burdensome. The hiring committee should report back to all of the partners (via periodic e-mails, newsletters, meetings, and so forth) to keep everyone apprised of the selection process.

The rest of this section will guide you in considering the evaluator's competencies and roles and responsibilities, regardless of whether the evaluation is to be performed by project staff or a local evaluator.

Competencies

Each member of the hiring committee should become familiar with the qualifications necessary for a local evaluator. Evaluators hired to conduct a needs assessment or program evaluation should possess the basic skills necessary to—

• Identify data that can be used for planning, managing, and evaluating program policies and practices.
• Use multiple research methods and collect different types of data from various sources.
• Establish data collection and analysis priorities that are informed by the experiences, capabilities, and resources of program staff and participants.
• Reliably interpret data.
• Link data to program development and management.

In addition to these basic skills, evaluators must be willing to work within the realities of your program and its everyday operation. Their skills in communication, team building, group processes, and negotiation are vital to a successful working relationship between the evaluator and the program’s partners.

It is also important to consider the following competencies when hiring a local evaluator:

• Skill in facilitating an evaluation process that includes the experiences of the participants.
• Ability to increase the capability of program staff to conduct and use evaluation findings.
• Willingness to develop and maintain a collaborative working relationship based on cooperation and trust among program staff, managers, stakeholders, and clients.
• Willingness to involve all partners in the evaluation process.
• Ability to design and implement methods to feed evaluation information back to the program for use in program planning, monitoring, and improvement.
• Experience in evaluating the types of program and topical areas in question.
• Experience in conducting evaluations of comprehensive community partnerships involving governmental, nongovernmental, and community-based organizations, if necessary.
• Experience in conducting evaluations with the types of partner and collaborative agencies affiliated with your organization.
• Experience in conducting needs assessments of issues important to the project.
• Experience in working in culturally diverse settings.

Finding an evaluator who is proficient in as many of these competencies as possible is critical. Although the success of the evaluation is not contingent on finding an evaluator with a high level of expertise in each of these competencies, the project and the overall evaluation process will greatly benefit if you take time to find an evaluator with the right skills to meet your needs. Doing so will greatly increase the likelihood of a successful evaluation.

Roles and Responsibilities

After the hiring committee has familiarized itself with the capabilities it needs in a local evaluator, it should define the evaluator’s roles and responsibilities. Involve the evaluator in the evaluation process as early as possible to help you figure out what you need to evaluate, what approach you will use for the evaluation, and how the evaluation will be carried out (e.g., who will do what, when, and with what result). This helps foster a working relationship between the evaluator and the partners and provides the evaluator with a deeper understanding of the goals and objectives of your program.

The evaluator will work with you and your partners to develop an overall evaluation plan and determine how best to accomplish the evaluation’s goals. For example, the evaluator should work with everyone to assess the needs of the community to make sure that programs are designed to best meet the needs of the target population. (See the Guide to Conducting a Needs Assessment in this series for more information on planning and implementing a needs assessment.) The evaluator also will need to ensure that the findings from the needs assessment are used to help develop and refine services. Services should then be assessed to determine how effectively the target population’s needs are met: Are the right services being offered? Are the right organizations present and working on the initiative? Information to help you and the local evaluator design your evaluation, including formulating evaluation questions, is provided in the Guide to Performance Measurement and Program Evaluation in this series.

Findings from the needs assessment will tie directly to program development and refinement; therefore, it may be useful to hire an evaluator who can help with both phases of the evaluation. Doing so may be more cost effective and will help the evaluator to have the bigger picture in mind when it comes time to evaluate services. Involving the evaluator from the start will increase evaluator buy-in and understanding of the entire project. Remember that the local evaluator cannot work in isolation but should collaborate with you and key partners to develop and implement the best needs assessment or evaluation possible.

The evaluator must ensure that the rights of the human subjects of the study (e.g., victims) are protected from any harm that may occur from being involved in the needs assessment or evaluation. Protecting human subjects is something an evaluator knows a great deal about that may be unfamiliar to some of your partners, making this another area in which an evaluator plays a key role. Additional information about protecting human subjects is provided in the Guide to Protecting Human Subjects in this series.

A detailed description of the roles and responsibilities of a local evaluator and the level of expertise needed to carry out each function is provided in appendix A (PDF 62 KB).

How Do I Find One?

Now that you have a better understanding of the type of local evaluator that would be right for your project, you are ready to search for the person to fill that position. This section of the guide provides information on where to look for an evaluator and what to include in a job announcement.

Where To Look

You may choose to use an individual or organization that is referred to you or that you have worked with in the past. If that is not the case, there are several alternate sources for finding an evaluator. The primary sources are provided in exhibit 3, along with the pros and cons of each source.

Exhibit 3 Sources for Finding a Local Evaluator
Source	Pros	Cons
Local College or University	Educational institutions offer a range of expertise via faculty members and graduate students at low to moderate overhead costs. Many small not-for-profit organizations have found low-cost data collection assistance through local colleges or universities. With proper training, graduate students (the more inexpensive option) can easily create instruments and collect and analyze data. Contact college or university departments such as behavioral/social science, criminal justice, social work, or public health to see if they have faculty and/or graduate students to assist you.	Faculty and graduate students often have teaching or schoolwork duties that compete with the time that they have to devote to an evaluation. They may also see this as an opportunity to conduct personal research of interest to them.
Local Independent Researcher	Researchers who work alone or with a minimal staff provide high-level expertise at low overhead cost. They commonly have the flexibility of working onsite, enabling frequent interaction with your initiative’s participants.	Independent researchers do not have the advantage of working within a larger organization that can provide a range of expertise and meet the time commitments necessary to get the job done in a timely manner.
Local Research and Evaluation Organization	Organizations with research and evaluation departments offer a range of expertise and in-house state-of-the-art technology, which allows them to meet multiple needs.	These organizations typically have higher overhead costs.

What To Include in a Job Announcement

The evaluator job announcement should include an overview of your project, the roles and responsibilities of the evaluator, a suggested timeline for completing the work, estimated budget information, identification of preferred skills, and a request for information about the applicant’s background and education. If possible, the hiring committee should prepare the job announcement, but all your initiative’s key partners should review and agree to it. A sample job announcement is provided in exhibit 4.

After the application deadline has passed, the hiring committee will review the applications and select applicants to interview for the job. (Use appendix B (PDF 23.1 KB) as a tool to help you select and interview candidates.) After the committee selects a candidate for the job, it should solicit feedback from all partners. The committee may need to schedule followup interviews to assess the personal style and characteristics of the candidate to ensure that the evaluator who is hired is acceptable to program staff, administrators, and key stakeholders. You should interview more than one applicant in case your first choice is unavailable or does not accept your offer.

Once an offer is accepted, engage the evaluator as soon as possible in planning the needs assessment and/or evaluation. If you have already developed a plan, review the document with the local evaluator and identify any changes needed. After you finalize this plan, you and your evaluator can proceed with implementing your plan.

Conclusion

By conducting a needs assessment and program evaluation, you will determine the extent to which you have built on existing community resources and are meeting the unique needs of your initiative’s target population.

Whether you decide to hire a local evaluator or to conduct the needs assessment or evaluation using personnel from within your initiative, remember that both activities are critical to planning, refining, and sustaining your program.

Resources

Resources

American Evaluation Association (www.eval.org)

The American Evaluation Association is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology evaluation, and many other forms of evaluation. The association provides links to help you find an evaluator and information on training opportunities.

The Evaluation Center, Western Michigan University (http://www.wmich.edu/evalctr/)

The Evaluation Center’s mission is to advance the theory, practice, and use of evaluation. The center’s principal activities include research, development, dissemination, instruction, and national and international leadership in the field of evaluation. The center serves a select group of clients, including school districts, nonprofit agencies, government agencies, foundations, and colleges, helping them to evaluate their programs. The center also provides evaluation training to the field.

Glossary

Data are pieces of specific information collected as part of an evaluation.

Data analysis is the process of applying systematic methods or statistical techniques to compare, describe, explain, or summarize data.

Evaluation plan provides the framework for conducting the evaluation.

Evaluator is an individual trained and experienced in designing and conducting evaluations.

Goals are measurable statements of the desired long-term, global impact of the program. Goals generally address change.

Impacts are the desired long-term effects of the program.

Indicators are measures that demonstrate the extent to which goals have been achieved.

Inputs are the resources that your program uses in its activities to produce outputs and outcomes.

Intermediate outcomes are program results that emerge after immediate outcomes, but before long-term outcomes.

Key partners are organizations that are formal partners of your program either through a memorandum of understanding or another type of formal agreement.

Needs assessment is a process for pinpointing reasons for gaps in performance or a method for identifying new and future performance needs.

Outcome evaluations examine overall program effects. This type of evaluation focuses on goals and objectives and provides useful information about program results.

Outcomes are the intended initial, intermediate, and final results of an activity.

Performance measurement is the ongoing monitoring and reporting of a program’s accomplishments and progress toward preestablished goals.

Process evaluations assess the extent to which the program is functioning as planned.

Program evaluation is a systematic process of obtaining credible information to be used to assess a program and improve it.

Qualitative data are a record of thoughts, observations, opinions, or words. These data are difficult to measure, count, or express in numerical terms.

Quantitative data are numeric information such as a rating of an opinion on a scale, usually from 1 to 5. These data can be counted, measured, compared, and expressed in numerical terms.

Research questions are developed by the evaluator to define the issues that are critical to the evaluation. These questions are worded so that they can be answered using research methods.

Appendixes

• Appendix A: Local Evaluator Roles and Responsibilities (PDF 23.2 KB)
• Appendix B: Interview Questions Checklist (PDF 23.1 KB)