Lessons learned from using the Qualitative Impact Assessment Protocol (QuIP) to assess the contribution of a social protection program in Mozambique

For the last couple of months, Plan Eval has been working on the evaluation of a social protection program using the QuIP methodology. In this blogpost, Pauline Mauclet, Evaluator at Plan Eval, explains what this methodology is all about and reflects on some of the challenges and lessons learned from this evaluation.

The Qualitative Impact Assessment Protocol, commonly referred to as QuIP, is a qualitative evaluation method used to assess the contribution of an intervention, without the use of a counterfactual. In other words, it is part of a wider family of approaches providing an alternative to quantitative impact assessments, which tend to be quite time-consuming and costly, to assess the impact of an intervention.

The method was developed by Bath SDR, a non-profit organization founded by a small team of researchers from the Centre for Development Studies (CDS) at the University of Bath.

In practice, the method assesses the contribution of an intervention by relying on the perceptions of beneficiaries and stakeholders. Therefore, the method consists in asking beneficiaries about the changes, both positive and negative, that they observed in their lives over a certain period of time and to then inquire about the causal factors that might have caused those changes (in their opinion).

In the following paragraphs, I will discuss some of the key features of the QuIP methodology, which help bring robustness and credibility to the research findings. The interesting thing is that most of these features can easily be replicated with other methodologies.

A common issue when asking beneficiaries about a certain benefit they received is that their responses might be biased, meaning that they might not be speaking the truth. Some respondents might for example be inclined to speak very positively about an intervention just to please the interviewer or because they are afraid to lose the benefit if they say anything negative about it. This type of bias is referred to as a response bias. In order to avoid this issue, the QuIP method uses a technique called (Double) Blindfolding. Blindfolding consists in asking the respondent questions without directly mentioning the program or intervention that is being evaluated. With Double Blindfolding, both the respondent and the interviewer are unaware of the intervention that is being evaluated.

In practice, the interview therefore starts with general questions about the changes observed in one’s environment over a certain period of time and then continues with probing questions about the factors that might have caused these changes. The idea is that the respondent would then mention the intervention by him- or herself, without any pressure or expectations.

But what if the respondent doesn’t mention the intervention? In that case, it might mean that the intervention wasn’t that noteworthy or impactful for the respondent, which is an interesting result in itself.

The key advantage of the QuIP method is that by asking general questions which are not focused on the intervention, we open up the possibility for respondents to surprise us. For example, respondents might mention a change which was not anticipated in the intervention’s theory of change. They might also explain how the intervention impacted them, but not in the way that was originally expected. Respondents could also mention other interventions or external factors that brought significant changes in their lives. In other words, the QuIP methodology puts the intervention’s Theory of Change to the test and can be used to refine it.

Now, asking beneficiaires about their perceptions seems nice, but which beneficiaries should we interview? It is impossible to interview everyone, so how do make sure that our results are representatitve and are not just a reflection of the opinion of a small portion of the population?

This is a common issue with qualitative research. Quantitative semi-experiments are able to work around this problem by collecting data from a representative, randomly selected sample of the target population. However, while quantative studies are appropriate to collect “factual” data, they may not be ideal to ask respondents about their experiences and opinions. In those cases, qualitative studies are much more appropriate. So, then, how do we select cases in a way that supports robust and credible generalisation of the results?

In order to rebuff criticisms of “cherry picking” , the QuIP methods favours a transparent and reasoned approach to case selection. Depending on whether a list of beneficiaries exists; whether a theory of change has already been defined; and whether data on outcomes exists and can be used for the case selection, different case selection approaches can be used, as shown in the diagram below (source: Bath SDR)

Case Selection Strategies
(Source: Bath SDR)

Finally, a third feature of the QuIP methodology(not exclusive to the QuIP methodology) is the use of codification to bring transparency and credibility to the analysis process. What is specific to the QuIP methodology is that the codification will focus exclusively on identifying Influence factors and Change factors.

Influence and Change factors
(Source: Bath SDR)

By identifying the different influence factors and change factors, we aim to build causal claims. Note that one change factor can also lead to another change, as shown in the diagram below.

Building Causal Claims
(Source: Bath SDR)

The objective of the codification process is to find stories of change. Through the use of codification, we can present those stories of change visually, while also facilitating internal and external peer review and audit.

Now that we have presented the QuIP methodology, I would like to reflect on some of the challenges and lessons learned from implementing the method for the evaluation of a social protection program in Mozambique.

The evaluation was commissioned by one of Plan Eval’s clients and the research methodology was defined based on the Terms of Reference provided by the client. The evaluation questions included questions related to the changes brought about my the program, but also questions related to the program’s implementation. As a result, our team set up a methodology that included the use of the QuIP methodology, along with a more classical evaluative approach using the OECD DAC Criteria of relevance, effectiveness, efficiency and cohesion. The intervention consisted in cash transfers provided in two parcels to a group of beneficiaries, with a Communication for Development (C4D) component.

In terms of case selection, our initial research design considered the possibility of using beneficiary data to select beneficiaries for the semi-structured interviews. The program had an existing Theory of Change and there was even data available on certain outcomes thanks to a short survey that was conducted by the client to a sample of beneficiaries after reception of each parcel of the cash transfers. Under this scenario, we planned to conduct a Confirmatory analysis stratified by context and outcomes. In practice, this meant that we would use the existing outcome data to select different profiles of beneficiaries to be interviewed in the field. By doing so, we were sure to cover a variety of profiles, while also opening up the possibility of triangulating the qualitative data with the existing quantitative data at the analysis stage.

Unfortunately, we ended up not receiving access to the beneficiary data before the start of the data collection activities. As a result, we had to adapt our case selection approach at the last minute and ended up going for an Opportunistic selection, by location and by beneficiary profile. The beneficiaries were identified and mobilized in the field, with support of the local authorities.

In terms of data collection, we ended up going for the Blindfolding of beneficiaries, without blindfolding the researchers, mainly for practical reasons.

Data Collection activities using the QuIP methodology
(Source: Plan Eval)

In addition to the last-minute change in approach for case selection, another difficulty was that of ensuring the blindfolding of beneficiaries, due to the fact that we conducted in each location both QuIP and non QuIP interviews. In accordancae with the evaluation objectives, the QuIP interviews focused on the contributions and changes brought about by the intervention, while the non QuIP interviews focused on the program’s implementation. By conducting both QuIP and non QuIP interviews in the same location, and considering that beneficiaries were mobilized with the support of local authorities, we had to take a special care to clearly explain to the local authorities the difference between the two types of interviews and to make sure that the respondents to the QuIP interviews weren’t “contaminated” (in other words, that they were informed of the fact that the study aimed to evaluate the social protection program before the start of the interview).

Finally, we observed that it was sometimes difficult to get people to talk during the interviews. People responded to the interview questions, but without providing much detail. This can be problematic for the QuIP methodology, because it may limit our understanding of the real stories of change. As a result, we played around with the format of the interviews and conducted some QuIP interviews in a Focus Group Discussion format in order to see if it helped stimulate the conversation. Additionally, we observed the importance of using open-ended questions to stimulate the conversation and to be patient with respondents, giving them the time to feel enough at ease to open up.

Another important aspect is to make sure that the respondent focuses on his own experience, rather than speaking about the experience of the community and neighbours. Therefore, it is important to remind the person from time to time to talk about their own experience and to focus on the observed changes.

Overall, in terms of lessons learned, I would identify the following elements:

  1. (If possible) Conduct the QuIP and non-QuIP interviews in different locations in order to avoid the risk of “contamination”
  2. Importance of open-ended questions to stimulate conversation
  3. Importance of being patient and letting the respondent speak freely, but reminding the person (when necessary) to talk about their own experience and focusing on observed changes
  4. Encourage respondents to focus on their own experience, rather than the experience of the community, neighbours, etc.
  5. Importance of being well acquainted with the questionnaire BEFORE starting data collection activities

The study is currently at the analysis and reporting phase. Once the study will have been finalized, I will report on any challenges and lessons learned from that stage of the evaluation process.

In the meantime, if you are interested in the results of this evaluation or if you have any questions on the use of the QuIP method, please feel free to contact us by email:

Pauline Mauclet – Evaluator (Plan Eval)

Magdalena Isaurralde – Team Leader (Plan Eval)

Notes from the field – Data collection for impact evaluation in challenging settings

In 2018, PlanEval contributed to the impact assessment of a project supporting smallholder producers of cacao, pepper and coffee in São Tomé and Principe, a two-island state in the Gulf of Guinea. It is Plan Eval’s second project on the African small-island state, and a first for Pauline Mauclet, researcher at Plan since 2017 and Field Coordinator for the project. In this blogpost, she reflects on some of the main challenges of collecting data for a randomized controlled trial and illustrates these challenges with the project in São Tomé and Príncipe.

About the project

Sao Tome and Principe is a small-island state located in the Gulf of Guinea at about 250 kilometers from the West-African coast. The former Portuguese colony has a long history of producing cacao. Under colonization, the Portuguese built large plantation properties, commonly called roças, for the production of cane sugar and later cacao. Although production has known ups and downs both during colonization and after, agriculture has always been the driver of the country’s economy.[1] When organic chocolate started becoming a valuable good at the beginning of the years 2000, the International Fund for Agricultural Development (IFAD) saw an opportunity to boost the country’s economy, which had suffered from the drop in (non organic) cacao prices in the 1990s. That is how the project PAPAFPA, or Participatory Smallholder Agriculture and Artisanal Fisheries Development Programme, came to light. The programme was followed and complemented by the PAPAC project (2004 – 2014). The programme interventions supported the development of an export cooperative (PAPAFPA) and facilitated the production of certified organic family plantations (PAPAC) to increase agricultural productivity, strengthen the existing producers’ associations and enhance access to markets through the provision of training, small infrastructure, as well as financial[2] and managerial support to the cooperatives. After an initial success with the cacao value chain, the programme expanded its activity and applied the same concept to the production of organic pepper and coffee and a second cacao cooperative was created.[3]

Fig. 1: “Agricultura é a base do desenvolvimento” (Agriculture is the basis of development), a slogan written on a wall in a beneficiary community.

Working in the field is rarely an easy task. Field experiments do not enjoy the same level of control as lab experiments, where every observed factor can be carefully controlled and treatment allocation is (usually) randomized. When designing the most appropriate methodology for an evaluation, researchers need to account for uncontrollable and sometimes unobservable factors. In project or programme evaluations for example, the format of the programme itself can prevent randomization, causing additional selection biases which need to be accounted for.

For the evaluation of the PAPAFPA and PAPAC programmes, the research team was faced with the following challenges:

1. Absence of baseline data

Absence of baseline data is a relatively common issue in project and programme evaluations, although more and more organizations are now foreseeing a budget both for the baseline and endline evaluation of their programmes. However suboptimal, the issue may be partially solved through retrospective questions relying on the respondent’s recollection. For this evaluation, the research team designed a questionnaire including questions about the producer’s situation at the end of the project (at the time when the data is being collected), and also about the situation before the implementation of the programme. Note that this exercise is usually easier for beneficiaries than for nonbeneficiary individuals, since the time marker (before/after participation) is much clearer for participants than for non-participants (e.g. what was the situation in, say, 2005?).

2. Programme eligibility criteria and the resulting Selection bias

The immediate beneficiaries of both programmes were the cooperatives which had been created under the PAPAFPA programme. The producers who sold their production to the cooperatives were indirect beneficiaries to the programme. They were selected by the cooperatives through their producers’ organizations based on a set of entry requirements. This implies that the selection of candidates wasn’t randomized, but instead based on observed characteristics, resulting in a non-random treatment allocation and causing a potential selection bias among the group of beneficiaries. If the selection criteria are correlated with the outcome (which they usually are), one needs to define a methodology that will account for the selection bias and make sure that treatment and control groups are comparable. Otherwise, there is no guarantee that the estimated difference in outcome between treatment and control is due to the treatment (and only the treatment), as it could potentially also be explained by the difference in initial selection criteria.

Accepting the hypothesis that selection bias is mostly based on observed characteristics, the most commonly used techniques to account for bias are matching methods. Matching methods, as their name suggests, couple beneficiary and nonbeneficiary units (such as families, households and plots) based on observed characteristics. The technique does not account for any unobserved factors affecting participation.

The IFAD research team chose to use the Propensity-Score-Matching technique, which involves only comparing treatment and control households that are matched according to baseline and target variables. In practice, this included variables linked to the probability of inclusion, along with the level of certain outcome variables (e.g. level of production; level of income) at baseline (pre-project).

3. Incomplete list of beneficiaries and Multiple Treatments

In order to match beneficiary producers with nonbeneficiary ones, the research team needed a complete list of project participants, as well as the treatment received by each. The beneficiary group included producers from the three value chains (cacao, coffee and pepper) who had benefited from both the PAPAFPA and PAPAC programmes, as well as producers who only benefited from the PAPAC programme.

However, not all four cooperatives managed to provide the full list of beneficiary producers, indicating the extent of support received and the producers’ organizations to which they belonged.

4. Finding a sufficiently large and representative control group (and the risk of contamination)

In addition to complete data about the beneficiary group, the research team needed access to a list of communities with similar characteristics to the communities that were exposed to the projects in order to identify producers with a comparable profile to those who received the treatment at baseline. However, at the start of the evaluation such a list didn’t exist and there wasn’t any national farmers’ registry that could serve as a basis to identify potential candidates for the control group.

An additional challenge was related to the programmes’ geographical coverage and the potential spillover effects to neighbouring nonbeneficiary communities. Overall, a total of 108 communities benefited from the two projects. Due to the nature of the programme and its interventions, there is a real possibility of spillover effects from beneficiary communities to the nonbeneficiary neighbouring communities. The islands of São Tomé and Príncipe are small and people know each other well. Most people have family living in a nearby community and our experience in the field showed that people move around a lot from one community to another. These neighbouring communities therefore cannot be considered within the control group, since they might have indirectly benefited from the programmes.

Considering the initial challenges, the IFAD team conducted preliminary visits to the field in order to consolidate the cooperatives’ lists of beneficiaries and identify, together with specialists from the Project Implementation Team and the cooperatives’ leaders, the communities that were eligible to enter the control group. From this initial exercise, only 36 “pure” eligible control communities were identified.[4] Considering that a sufficient number of nonbeneficiary farmers from these 36 communities had to be matched with beneficiary farmers from each of the three value chains in order for the Propensity-Score-Matching to ensure common support [5], there was a real risk of not reaching a sufficiently large control group. However, this hypothesis could only be confirmed or refuted after an initial enumeration exercise among treatment and control communities.

Fig. 2: Field Coordinator Pauline Mauclet (middle) discussing the field logistics with data collection team supervisors Constancio Da Graça Neto (left) and Osvaldo Madre Deus Bastos (right).

As part of the impact assessment and prior to the quantitative data collection, PlanEval’s data collection team conducted a detailed enumeration exercise to obtain a listing all households living in the treatment and control communities (over 5.000 households). Based on this listing, the IFAD team obtained an inventory of all producer households from both the treatment and control communities. The listing also collected basic information on the profile of each producer household, which was then used to perform the matching exercise for each of the three value chains.

The matching exercise turned out successful and a final sample was set up, composed of 1.687 households (799 treated; 800 untreated). In order to guarantee enough common support (max. 6% attrition bias), the data collection team was asked to collect data from a minimum of 700 treated and 800 control observations. Note that some control observations were used for more than one treated observation.

Both the listing exercise and the survey turned out to be extremely challenging considering the local conditions. Our data collection team, supervised by a team of remarkable team supervisors, conducted the listing exercise in the most remote communities of the island, as well as in larger communities such as São João dos Angolares, a vast and confusing community to those who don’t know their way through its small pathways. By the time the data collection team was applying the survey, the rainy season had started and access to the communities got more difficult. Once again, the experience of our data collection team, this time of our drivers, was essential to the evaluation’s success. Muddy roads, steep cliffs and fallen trees. They knew what to expect and were prepared for anything.

Fig. 3: Difficult road conditions.

As if muddy roads and heavy rains weren’t enough, the survey took place during the country’s legislative and municipal election season. Not exactly the most recommended period to conduct a survey, since election campaigners are known to visit the communities and offer alcohol to its inhabitants. If our data collection team got confounded with campaigners from a party that wasn’t supported by the community, there was a risk for things to get violent. Aware of these risks, we decided to provide the data collection team with a neutral white uniform, clearly showing our company’s logo and the name of the programmes PAPAFPA and PAPAC. We also decided, together with the field supervisors, to test out the best time to visit the communities. It turned out that the morning and the early afternoon were most recommendable, as the campaign activities usually took place in the afternoon. The day before and after the election, the data collection activities were suspended as a precautionary measure. At any time, we were ready to suspend the activities if the situation got too heated. However, taking the necessary measures, our data collection team didn’t encounter any difficulties to conduct the survey as a result of the elections.

Fig. 4: Conducting the survey in São Tomé’s most remote villages.

After three months and a half in the field, PlanEval’s data collection team successfully applied the final (approx. 2.5 – 3h long) household questionnaire and provided the IFAD team with the cleaned database. I encourage you to take a look at IFAD’s final evaluation report, which can be accessed through the following link: https://www.ifad.org/en/web/knowledge/publication/asset/41116368  .

[1] Out of a total of 90 potentially eligible communities, 14 were identified as counterfactual communities by more than one cooperative (therefore double counted); 14 other communities overlapped with the domain of other cooperatives; 26 communities received support from a cooperative, but at a low intensity; and 36 communities never benefited from either programmes (the “pure” communities).

[2] The common support condition ensures that treatment observations have comparison observations “nearby” in the propensity score distribution (Heckman, LaLonde, and Smith, 1999).

[3] In recent years, more attention has been given to tourism. The recent discovery of natural resources at its shores has also opened new perspectives for the future.

[4] The programme helped the cooperatives form agreements with international private sector buyers.

[5] Unlike for the production of cacao, pepper was introduced into the country’s agricultural sector by the programme. The production of organic pepper was almost inexistent before PAPAFPA.

Interview with Christos Aivaliotis from the International Consulting Alliance (ICA)

Since August 2017, Plan Eval is a member of the International Consulting Alliance, an international network of consultancy firms in the Development Cooperation field. The network was created with the aim of facilitating long term partnerships between consulting actors located in development cooperation beneficiary countries and highly qualified technical specialist and development experts in different technical sectors. Since its creation in 2012, the network has grown steadily and currently counts 65 Technical and Local Members and over 26.000 experts.

Every year, the ICA holds the Annual ICA Conference. It is an opportunity for Technical and Local Members to meet in person, exchange ideas and discuss future cooperation opportunities. This year, the conference took place in Valencia, Spain. Our Executive Director, Fabrizio Rigout, participated and took the opportunity to sit down with Christos Aivaliotis, Network Manager for the ICA, to discuss the Annual Conference, the network’s annual results and the opportunities it has to offer to a company like Plan Eval.

Fabrizio and Christos Aivaliotis at the Annual ICA Conference, in April 2018.

 FABRIZIO RIGOUT: I’m here this afternoon with Christos. He is the Network Manager for the International Consulting Alliance and we are going to talk a little bit about the conference. So, first of all, how were the results?

CHRISTOS AIVALIOTIS: We’re quite happy this year, there has been a record number of participants in Valencia, Spain. Around 85 consultants coming from more than 30 different countries and representing 50 different organisations.

F: How would you describe Plan Eval’s role within the network?

CH: I would say that there is a lot of room for Plan Eval to expand its activities thanks to the use of the network, especially taking into account your technical expertise and geographical presence in Brazil. This combination offers you a very privileged position inside ICA, as you can work both as the Local Partner for Brazil, but at the same time as a Technical Partner worldwide for motoring and evaluations. As we have discussed already, Plan Eval has the characteristics of a “hybrid” member according to ICA’s internal categorization and can certainly bring on board ICA the best of both worlds – sectorial specialization and geographical expertise.

F: Can you tell us a bit more about the difference between Local Partners and Technical Partners?

CH: Yes, within our network, the International Consulting Alliance, we have two types of member organisations. On the local side, there are organisations that are based in the beneficiary countries where the developing projects take place and with a cross-sector approach, meaning they can work in many different technical areas. While, on the other hand, our technical partners work worldwide but in very specific areas of expertise.

F: What kind of opportunities does ICA offer for consultancy companies?

CH: All our members, either private companies, NGOs or even public parties, are connected through an online cooperation platform that offers them an overview of all open opportunities to work with multi-lateral or bi-lateral donors, through our daily newsletter. While there are many online services providing this, ICA differs in the sense that we are creating a community of development specialists, who can very easily identify not only new funding opportunities but the best available partners (thanks to an automatic match-making system of ICA’s platform) and experts (thanks to the ICA Database of Experts). The vast majority of our members use our platform extensively for networking and business development reasons, but a considerable amount of cooperation still takes place as well during the implementation of their projects through the exchange of expertise, sharing best practices and utilizing the physical presence of our members throughout the world.

F: As a last question, can you give us some examples partnerships that happened between companies because of participation in ICA?

CH: Actually, your timing is great, because we are always collecting the annual statistics around the time of the Annual Conference. For the past year we have had around thirty successfully interacted and implemented projects by ICA partners in all continents and across many different technical areas. A specific example I’d like to mention is a project that was awarded to two of our local partners, without the participation of a technical partner, to implement a World Bank project in China and Pakistan.

As Plan is getting more familiar with the network and its platform, we are eager to increase our involvement within the Alliance. Our presence at the Annual Conference has proven to be very productive. We have had the chance to meet with people from several technical areas such as Education, Health and Public Services and it has been very promising in terms of partnerships for Plan.