Indonesia: Impact evaluation support to WSLIC-2 project

IRC’s Christine Sijbesma is providing 65 days of impact evaluation support before completion of the Second Water and Sanitation for Low Income Communities project (WSLIC-2) in Indonesia around December 2008. The project covers 2,000 villages in seven provinces of East Java, West Nusa Tenggara, South Sulawesi, West Java, West Sumatra, South Sumatra and Kepulauan Bangka-Belitung. The objectives of the project are to improve the health status, productivity and quality of life of poor communities in underserved rural villages in project provinces.

The WSLIC-2 project is financed by the Government of Indonesia through a loan from the World Bank, a grant from the Australian government and human resources from the Indonesian departments of Health, Home Affairs, Finance, Education, Public Works and the National Development Planning Board supported by community development, health and technical consultancy teams at district and central levels.

Three phases

Christine did already a first mission in August 2008 to assist with the design of the terms of reference and implementation of impact evaluation in the remaining period of the project. The methods used for this were a desk study of relevant documents, including of the baseline study in four of eight provinces, field visits to two districts (Mojokerto in East Java province and Lombok Timur in NTB province) and in each district two villages with different types of completed and community managed services, consultations with stakeholders within the Indonesian Ministry of Health, World Bank and AusAID and the review of the existing MIS and monitoring systems on project implementation and sustainability, and of public health monitoring through PusKesMas (Sub-district Health Centres).

In Phase II, assistance will be given to the implementation and in Phase III to the data analysis and reporting of the results.

Monitoring based on Methodology for Participatory Assessment

WSLIC-2 has a Monitoring Information System (MIS) that monitors and reports actual versus planned progress and outputs per project quarter, with a summary of the project's sustainability monitoring data. This sustainability monitoring system is based on the use of the Methodology for Participatory Assessment (MPA), which IRC and WSP developed and tested globally and in fieldwork in Indonesia in 2000-2003. Two publications on this can be freely downladed from our site:

- the guide on MPA methodology published in 2003 http://www.irc.nl/page/12947.

- the IRC book “The Best of Two Worlds? Methodology for Participatory Assessment of Community Water Services”, by Christine van Wijk-Sijbesma (2001), published with financial assistance of the Water and Sanitation Program (WSP). http://www.irc.nl/page/1898

This MPA methodology gives quantitative values to qualitative characteristics of the village planning and management processes. Scores reflect, for instance:

• the degree of access for the poorest,
• intermediate and best-off households,
• gender equality in village meetings and decision-making, and
• the degree of good governance in running village service and/or projects, as evidenced by the degree of equity in charging contributions and tariffs, transparency in budgeting and accountability for managing services and finances.

Some outcomes and impacts not included

The project’s MIS can, however, not provide a rigorous evaluation of outcomes and impacts in terms of changes in sanitation and hygiene behaviours, reductions in time and efforts for water collection and excreta disposal, changes in workloads for women and children (girls/boys), impacts on the positions of women and the poor (e.g. to a greater participation in planning and management decisions, organization and training) and impacts on health, socio-economic development and the quality of life of especially poor people.

Suggestions have been made to do that for all villages that will participate in the impact evaluation, also outcome data on sustainability performance will become available. Furthermore, some adjustments of the questionnaire have been proposed to ensure that all access and sustainability aspects will be addressed.