The Control Groups

In El Salvador the evaluation was led by CIAZO, the national NGO specialising in adult literacy with whom the pilot project with COMUS was undertaken. Although CIAZO were involved in the pilot we felt they were also neutral in that the control group used was their own work in other areas which they have built up over five years with 30 different grassroots organisations. They had no vested interest in showing the REFLECT pilot (which is very small scale within their national programme) to have been more successful than the bulk of their own literacy work. Rather they wished to critically review the work in order to help them define their future methodology.

CIAZO chose two types of control group: firstly their other work with grassroots organizations similar to COMUS (one organization in Usulutan - Codecosta, and one in Conchagua) and secondly with their more recent work with local government at the municipal level with UNICEF. It proved difficult to collect all the data from these control groups but sufficient data was found to provide a clear comparison and this was then reinforced through discussions with CIAZO advisers from across the country.

Monitoring

Basic monitoring of the literacy circles took place on a permanent basis through support visits of the COMUS promoters and through the monthly workshops. Each literacy facilitator also maintained a notebook to document key events and debates in their circle.

The Evaluation

In March 1994 CIAZO, COMUS and ACTIONAID worked together to establish a detailed methodology for the evaluation and to field test some formats. Most of the evaluation work was then conducted between March and May 1995 by a team of 2 CIAZO staff (Luis and Oscar) and 3 COMUS staff (Juan, Ovilio, Abdon). Other CIAZO staff helped collect data from the control group (particularly Arantza). The evaluation data was consolidated and analysed in ten days of May with the help of other CIAZO personnel (including Nicolas Foroni, Arantza and Blanca), David Archer (ACTIONAID) and Sandrine Tiller (Latin American Coordinator of the World University Service).

The evaluation team reviewed basic information and statistics and conducted literacy and numeracy tests. A range of structured interviews were designed and various matrices were elaborated which could be completed with learners in a participatory way. All these formats were pre-tested in March and adapted in the light of this pre-testing. A particular effort was made to "close" open-ended questions during this pre-testing period so as to make the forms easier to complete.

The full scope of the evaluation involved: