home > Tools
& Resources > Successful
Evaluations
Successful Evaluations
What makes an evaluation successful? Successful evaluations tend to
have four qualities, described below.
- Making the decision to evaluate.
Before getting started, the people involved in the evaluation
need to be in agreement on the evaluation's purpose. They need to agree
on what they are trying to find out and what they plan to do with that
information.
- The Scope of Work is well thought-out.
The Scope of Work (SOW) is a plan for the evaluation. It briefly
answers the big questions: who, what, where, when, and why. The best
SOWs deal with these questions head-on and do not postpone decisions
for the evaluation itself.
- The team is capable. In conducting
an evaluation of a USAID program, USAID CTO's need to go beyond a generically
"qualified" team. The team and the mix of skills it brings need to be
qualified for your evaluation. As you think about what you're trying
to accomplish, you should think about how much country and program specific
knowledge you need, how much independence, and how much you want to
include program participants on the team.
- The results are used. Translating the results of
an evaluation into action depends in part on clarity of purpose. But
even evaluations, which are otherwise well done, are not always used.
Disseminating results effectively requires effective communication.
The way these four elements combine to make a successful evaluation is subtle.
Even the best evaluation team cannot fix problems caused by an unclear purpose
or a vague SOW. These four elements are interdependent, and you need to
get them all right to be successful.
Top
of Page
ยท ADS Requirements
 Making the
Decision to Evaluate
In a project's lifecycle several events might "trigger" an evaluation.
The word trigger should be used cautiously. There is nothing automatic
about conducting an evaluation. Use all three of these sections as tests
for deciding to evaluate in your specific situation. Think through your
potential evaluation in terms of how its purpose would be classified,
whether the management need behind it is associated with positive or negative
signs, and whether it is generated by one of the situations below.
- Performance monitoring indicates unexpected results (positive or negative)
that need to be explained.
- A key management decision must be made and there's inadequate information.
- Annual performance reviews have identified key questions that need
to be answered.
- Customer or partner feedback suggests that there are implementation
problems or unmet needs.
- The contribution of USAID activities to results is questioned.
- Issues of sustainability, cost-effectiveness, or relevance arise.
- The validity of results frameworks hypotheses and critical assumptions
is questioned.
- Recommendations for actions to improve performance are needed.
- Extracting lessons for the benefit of other operating units or for
future programming. (TIPS
#11)
Top of Page
| "The methods used for USAID evaluations are
dominated by the "fly-in" approach having a serious effect on the
quality of evaluations. As things stand now: Scopes ask a team to
come for 4-6 weeks and interview the mission, the activity staff,
and 'representatives' of the local people. There isn't enough time
to get any kind of representative sample. The team frequently tells
the USAID manager pretty much what he/she already knows. —Evaluation
of USAID Evaluations) |
Selecting an Evaluation Team
What are the main issues in contracting a team?
What are some best practices to use when selecting an evaluation team?
How should an evaluation manager support and manage the team during
the evaluation?
What are the main issues to think about when planning project logistics?
Contracting a Team
Sometimes, evaluations will be conducted by teams made up exclusively
of USAID personnel. However, the long term trend at USAID involves hiring
contracted personnel on evaluation teams. These contractors include
USAID/W selected consulting firms, other US-based consulting firms,
host-country consulting firms, or Operating Unit-selected independent
consultants. For these reasons, knowing how to handle contracting tasks
is an important part of managing an evaluation.
General Issues
Contracting processes may take up a significant amount of time.
Delays can have a significant impact because often the evaluation questions
are tied to a decision which has real external constraints (e.g., budget
cycles) which cannot be pushed back.
It is important to know the most appropriate vehicle for contracting
to make the process as quick as possible.
Conflicts of interest. Sometimes the best people to do a job will not
or cannot do the evaluation because by doing so they will preclude themselves
from future work. You also need to make sure that regardless of the contractor's
willingness, you are aware of the general conflict of interest issues
associated with performing the evaluation. Data Ownership
"Rights to data" refers to ownership of that data after the evaluation
has been conducted, as data created through the evaluation is important
beyond its use for the evaluation at hand. It is important to clarify
with the contractor what their rights and responsibilities are with
regard to the data before entering into a contract.
Top of Page
Selecting a Team
Depending on the evaluation and its size, there are a variety of vehicles
through which you might contract personnel. The most common is for a
Mission to use an existing program support contract to bring an existing
contractor on board for the evaluation. This process is relatively quick
because it does not require a competitive bidding process; the Mission
simply issues a task order under an existing contract. Even when such
a vehicle is used, the Mission can request that the contractor subcontract
to a particular individual consultant. Such an arrangement can be an
effective way to get a known consultant on a job, but you should keep
several caveats in mind as discussed below in Team Considerations.
Indefinite Quantity Contracts (IQC) are used to mobilize US-based contractors
relatively rapidly. With IQCs, contractors have already been pre-approved
through a competitive award process for such work. IQCs are much faster
ways to secure contractor support than by issuing a new Request for
Proposals and reviewing bids.
Lastly, in some limited cases, independent consultants might be hired
for short jobs through purchase orders. While purchase orders are relatively
quick, their use is limited, however, because they are intended for
small value goods and services.
Top of Page
|
|
|
