Background

The complex emergency in the Federally Administered Tribal Areas of Pakistan (FATA) has resulted in repeated and long term displacements of populations into Khyber Pakhtunkhwa (KP) since 2008This migration crisis reached a peak in April and May 2009, when it was estimated that nearly 3 million people were displaced. Between 2009 and 2011, 66,168 families were displaced from Khyber. In 2013, a further 13,838 families were displaced as a result of a military operation against local militants, and in late 2014, a further 27,867 families were displaced. The situation is compounded by simultaneously occurring military operations in North Waziristan (NWA), which resulted in large displacements into KP, in June 2014. As of January 2015, a total of 277,811 families were living in displacement in KP. Of these, 104,053 originate from Khyber, with the recent influx from the area accounting for 13% of the total. 94,256 IDPs originate from NWA, and the remaining from other agencies in FATA and South Waziristan.[1]

Cesvi, Concern Worldwide and ACTED, which together form the Alliance2015 in Pakistan, have been responding to the humanitarian needs of the affected population with the support of various donors. In April 2014, with funding from ECHO, the Alliance2015 commenced the implementation of their seventh joint project "Integrated humanitarian assistance to address immediate needs of vulnerable communities in Pakistan". The project aims to support the displaced communities in meeting their basic needs and strengthening their resilience to future disasters, through provision of short-term income opportunities, as well as adequate water and sanitation, infrastructure and sound hygiene practices. Specifically the joint intervention is targeting vulnerable communities in the districts of Nowshera, Kohat, Peshawar and DI Khan, which have experienced the strongest influx of displaced people.

By March 2015, the intervention will have provided 203,084 individuals with lifesaving support:24,300 families will have access to new or rehabilitated water and sanitation infrastructure, 11,680 families will have knowledge of sound hygiene practices, 5,850 families will have been engaged with cash-for-work opportunities to support them meet their basic needs upon immediate displacement, as well as endow them with long term skills for income generation. Finally, 490 extremely vulnerable families would have been supported with conditional cash assistance to meet their basic household needs.

Alliance2015 is a partnership of eight like-minded non-government European organizations working in the field of development cooperation and humanitarian assistance. The Alliance members are Cesvi from Italy, Concern Worldwide (Concern) from Ireland, Welthungerhilfe (WHH) from Germany, Hivos from the Netherlands, IBIS from Denmark, People in Need (PIN)from the Czech Republic, ACTED from France and HELVETAS Swiss Intercooperation . Out of the seven Alliance2015 members, four agencies are currently operational in Pakistan ACTED, Cesvi, Concern Worldwide and WHH.

The overall purpose of the evaluation is to evaluate Alliance2015s project entitled Integrated humanitarian assistance to address immediate needs of vulnerable communities in Pakistan, with a particular emphasis on the effectiveness, relevance/appropriateness and efficiency of the intervention. The evaluation will also address certain question related with the connectedness, coherence and coverage of the intervention as per the OECD-DAC criteria.[2]This will allow the consultant to extract lessons learnt and recommendations to enhance the quality of on-going and future programming by Alliance2015 and other operating units involved in humanitarian action. It should capture achievements of the projects results and indicators, and the short and medium term impact of the action. The evaluation will also assess the extent to which the Alliance2015 partnership and cooperation model has contributed to the effectiveness of the project, and so will document the experiences/lessons learned as a result of adopting this approach.

The major questions to look into throughout the evaluation are as follows:

• To what extent does the intervention improve the conditions of affected communities? To what extent does the intervention help reduce negative coping mechanisms by the beneficiaries?
• What has been the progress made towards achievement of the expected outcomes and expected outputs? What were the results achieved? What are the reasons for the achievement or non-achievement?
• Did the intervention reach the targeted (i.e. the most vulnerable) population, including those excluded from other assistance (e.g. WFP, GoP etc.)? Was the methodology used for the beneficiary selection relevant and were communities involved in the process? Was the selection criteria communicated and understood by all members within the community?
• To what extent and how were target communities (men and women) and other key stakeholders involved in the needs assessment, design, and implementation of the response?
• Did the programme have effective monitoring systems in place to measure progress towards results?
• How well did the response mainstream/integrate gender, equality, protection, the environment, capacity building and conflict/cultural sensitivities? What are the possible areas where further improvement is needed?
• How far the action has been successful in building resilience in the targeted communities?
• Was the Alliance 2015 set up appropriately in order to quickly respond to contextual changes which may have arisen during the implementation period and if not, why? Was the type of assistance adapted according to the needs that arose hereof.

• Has the project addressed the real needs?
• How satisfied are the communities with the response? Are there instances of community self-initiatives and mobilization in the project?
• Which components of the intervention (i.e. CFW or WASH) showed greater relevance for the beneficiaries? Why?
• Were actions in the intervention areas in response to the displacement of 2013 and 2014 well enough coordinated to prevent duplication?
• Has the project informed, consulted with, stakeholders, particularly about the standards adopted, programmes to be undertaken and mechanisms available for addressing concerns?
• Has the project established mechanisms to enable beneficiaries and staff to report complaints and seek redress safely?
• Was the action appropriate for the social and cultural context of KP, and the areas of origin of IDPs? Was the action responsive to gender needs, social and cultural values, conditions and practices?

• Were the operational systems put in place by the Alliance2015 effective in ensuring timeliness and cost effectiveness?
• To what extent did the Alliance2015 model enhance the timeliness, quality and effectiveness of response compared to a single agency emergency response, if so how?
• What measures were taken during planning and implementation to ensure that resources are efficiently used?
• Have the outputs been delivered in a timely manner?
• Could the activities and outputs have been delivered with fewer resources without reducing their quality and quantity?
• Sustainability connectedness, coherence, and coverage
• To what extent are the benefits of the project likely to continue after the termination of the project? What are the major factors which influenced the achievement or non-achievement of sustainability of the programme?
• What could be concretely done to ensure sustainability of the action and linkages with other more development oriented operations?
• In a context of recurrent natural disasters, what would be the Alliance 2015 strategy for the next three years in order to enhance the sustainability of the action?
• How sustainable and adequate to the local context were the proposed designs and specifications for hygiene promotion activities and WASH infrastructure (e.g. tube wells)? If not sustainable or appropriate, why?
• Were the WASH installations properly handed over to the Public Health Engineering Department? If not why not?
• Did the projects planning and implementation mechanisms include the involvement of existing Community Organizations? What was the role of existing COs in the project implementation and beyond? If COs were not present in the areas of intervention, were new COs established during the course of the project?
• Were the opinions and attitudes of host populations taken into consideration at the design and implementation stage of the project?
• Do IDPs and hosts have equitable access to water?
• Have all water resources been tested and confirmed safe and suitable for drinking?
• Were humanitarian standards met and humanitarian principles followed? (SPHERE, HAP, Codes of conduct)?
• Did the interventions sufficiently consider social structures and cultural norms of the targeted communities?
• Furthermore the evaluation aims to assess how the members of Alliance2015 worked together and contributed to the harmonization of processes. In addition to the efficiency considerations, it aims to provide insights on the following:
• Did the Alliance2015 model achieve consistency, where required, between members across its relief assistance (harmonization of quality, quantity, targeting criteria, technical approaches etc.)?
• How was gender integrated into strategic planning and project design by Alliance2015 partners? Did the project emphasize the importance of gender equality within the context of overall project design? Were gender considerations woven into all stages of the project cycle?
• How were Alliance2015 staff capacities assessed and definition of capacity needs established for ensuring the success in implementation?
• Did the project activities build successfully on the emergency assistance phase programmes previously implemented by Alliance2015 members? Were lessons and recommendations from the Alliance2015 previous evaluations taken into account?
• How well did the Alliance partners collaborate with local partners and stakeholders and engage local stakeholders during the activity implementation?
• How do local partners assess this collaboration? Has the Alliance identified and used stakeholders local capacities and contributed to build their capacities were needed? Any striking experiences or insights? Did the consortia model offer the opportunity for learning across the agencies? If so, what manner and form did this take?
• How effective and successful was the level of cooperation and communication between the agencies?
• What about the lessons and recommendations from the Alliance2015 previous evaluations (projects in response to militancy conflict and resulting displacement in 2013 and 2014 )? Are there recommendations which have been implemented? If so, how are they working, if not why not?
• Which effect did the enhanced harmonization process have on the implementation by involved partners? What were challenges and benefits of the harmonization?
• How did Alliance 2015 members apply a joint visibility strategy? Is Alliance2015 perceived as a consortium of three individual organizations by target groups, local authorities, other stakeholders ?
• Are there any or some specific observations on the ECHO facilitation and probable role in this process of harmonization?

The evaluation approach to be taken will depend on the nature of the information available to the evaluator and will be developed in detail by the consultancy, and discussed with Alliance2015 partners in its preliminary stages. It will however include at least the following elements:

• Literature review, including review of secondary sources such as UN, NGO and cluster reports, analysis of project monitoring data, PDMS, etc.
• Detailed discussions with Alliance2015 members, ECHO (& other donors), IVAP, relevant clusters and specifically the WASH cluster, government agencies and other INGOs/NGOs focusing on access, coordination and information sharing.
• Field visits to target areas and collection of primary information including detailed discussions with a sample of target beneficiaries - process should be participatory to the extent possible
• Presentation of draft findings / debriefing at the end of the field phase (see also chapter 4: output).
• The writing of a final detailed report and submission to Alliance2015 lead partner for feedback from all Alliance members.
• Incorporation of Alliance2015 feedback in the draft report and submission of the final report
• Discussion and approval of final report with the Alliance2015 members, France.

; It is recommended that Alliance members provide the consultant with data on new IDP arrivals in their area of operations, to also see if the needs were addressed.

The evaluation products that are expected from the evaluation are as follows:

1. An inception report which contains evaluation objectives and scope, description of the evaluation methodology/methodological approach, data collection tools, data analysis methods, key informants/agencies, reassessment of evaluation question, performance indicators, work plan and reporting requirements. It should include a clear evaluation matrix relating all these aspects.

2. A power point presentation of preliminary findings to Alliance 2015 members and key stakeholders including ECHO. The comments made by key stakeholders should inform the draft report.

3. A draft evaluation report which should be delivered with adequate time to allow stakeholder discussion of the findings and formulation of recommendations.

4. Final evaluation report which should be structured as follows:

• Executive summary (maximum 5 pages)
• Programme Description
• Evaluation Purpose
• Evaluation Methodology
• Findings
• Lessons learnt
• Recommendations
• Annexes (including interview list without identifying names for the sake of confidentiality, data collection instruments, key documents consulted, Terms of reference)

The evaluator is expected to identify across the evaluation report examples of best practices in what has worked well and what has not worked well particularly for the following issues:

- Programming and delivery: Process focus. Including mainstreaming gender equality/protection, do not harm principle and accountability. Adherence to the codes

- Nature and quality of the partnerships with the government and other key stakeholders

- Harmonization of approaches among Alliance2015 members

- Logistics, Administration and finance

- Assessments in M&E, Joint Learning opportunities

- Preparedness and development.

The field phase is envisioned to take place in April, 2015.

Approximate days for the evaluation are given in the table below.

Activity

Number of Man Days

1. Inception report and power point presentation of preliminary findings (3)

2. In-country meetings and data collection/field work, followed by draft evaluation report (20)

3. Final Evaluation Report writing (5)

Total: 28

In agreement with Alliance 2015 members, ACTED is taking the lead for this consultancy. The consultant/s will report to the Country Director of ACTED, who will ensure proper information flow and participation of the Alliance2015 colleagues.

• Post-graduate degree in Humanitarian Studies, Disaster Management, Social Sciences and/or related field;
• At least 10 years experience of conducting evaluations of complex emergencies, including and humanitarian programmes;
• Previous experience of conducting evaluations of NGO consortia/alliancesin emergency contexts;
• Familiarity with shelter and NFIs, Cash programming, WASH, including technical knowledge of related infrastructure, and skills development, as well as knowledge of financial and economic analysis;
• Familiarity with international quality and accountability standards applied in emergencies;
• Experience in the use of participatory methodologies and developing gender sensitive evaluation methodologies;
• Competency in gender, equality and rights; resilience and community based disaster risk reduction ;
• Excellent written and spoken communications skills in English. Knowledge of the Pakistani regional context and languages will be an advantage;
• Experience in assessing organizational capacity and gaps and ability to recommend the corrective measures.

A team of two people (one External and one national/local evaluator or resource person) from Pakistan) will also be considered, utilizing one national consultant with good local language and gender specific skills, along with the international consultant.