Appendix 2 - Evaluation procedures
The purpose of this Appendix is to show some simple techniques you can employ to analyze an audience, test message materials, and evaluate the results of communication efforts.
These will be explained under three headings: questionnaires, sampling techniques, and interview procedures.
The simplest kinds of questions you can ask are those to which favor-oppose, can-cannot, yes-no type answers can be given. Fortunately, almost any problem can be broken down into such binary-type alternatives. As you ask questions, keep a record of the audience responses.
The following are some examples of binary-type alternatives:
Some people like one color better than another for certain things. What color would you consider appropriate for a fertilizer sack - flour sack - family planning poster etc.?
After showing a filmstrip which treats the proper method of lifting ground nuts, ask the test subject to show you how it is done.
What exhibits were of interest to you at this year's farm fair?
Another type of questionnaire is a checklist of questions that should be asked when producing materials. The following example lists ten major questions with some sub-points that relate to the production of a leaflet or booklet.
- What is the objective?
- Who is the reader?
- What does the reader think?
Knowledge of subject
- Are the facts adequate?
- Will the reader understand?
- Are the ideas presented workable?
- How will the material be reproduced?
- How will the leaflet or booklet look ?
- How will the material be distributed ?
- Will the reader react as planned?
To obtain a perfect picture of your audience with respect to any yes-no item of information, you would have to record accurate data for every member in it. Alternately, should you interview only two or three individuals, the certainty of obtaining answers which portray the larger group accurately will be extremely low. Two factors bear on the confidence you can place in findings from any sample: (1 ) representativeness, and (2) size.
Any sample is representative only to the extent that every person in the population from which it is drawn has an equal opportunity of being included in it. The following are methods used to attempt to obtain representative or random samples.
Systematic sampling. If it is possible to find prepared lists of the names of members of the population which you wish to investigate, you can simply divide the total number of names by the number required in your sample and draw every nth number. For example, if there are 5,000 names on a list and you want a sample of 100, you simply take every 50th name.
This method is fraught with possibilities for error, however. Printed lists may not be representative of the population you wish to investigate for various reasons. Telephone books may include only the wealthy; church attendance lists may contain only the more socially active; school lists contain only those with school-age children, etc.
Simple randomization. From a complete list of all names of individuals in a village, of all numbered houses or squares in a village, of all villages in a province, of all attendants at a farm fair, etc., you can obtain a random sample by (1) numbering all entries, (2) placing all numbers in a box, (3) shaking the box thoroughly and (4) drawing numbers. After each number is recorded, it must be replaced and the box re-shaken before drawing again. If you can obtain a table of random numbers, this process can be simplified.
Stratified random sampling. Sometimes it is desirable to break large populations into subgroups, each of which has particular characteristics, and then randomize proportionate samples from each of the subgroups. For example, you may wish to be certain that your sample includes high and low income groups, urban and rural families, different religious faiths, etc., in proportion to their numbers in the total population. To obtain a stratified random sample, you will use separate boxes for each group and draw numbers from each box in proportion to the group's percentage of the total.
Though different sub-groups differ in their responses to the questions you ask, your combined sample may better represent the total population than would a random drawing from all members at large.
Assuming a representative sample of the audience has been questioned, the size of the sample selected will determine the level of confidence that you can have in the findings. Statistical techniques can be used to determine appropriate sample sizes. However, in most situations a small sample will provide the information necessary to improve the effectiveness of your instructional materials.
It is not always easy to obtain valid responses to your questions. Sometimes questions are misunderstood. Sometimes, in order to be polite, respondents will give answers they think the interviewer may prefer to hear. Sometimes they may avoid being asked questions when alone, and when accompanied by a friend may tend to report what they think their friend would prefer to hear. Sometimes they may not trust the interviewer, or may question his motives and attempt to trick him. The following practices may help to alleviate some of these difficulties.
- Select your interviewers from among the groups to be questioned. Farmers respond best to farmers, women to women, church folk to church folk, etc.
- Word your interview questionnaire in the vernacular of your audience. Foreign words or alien modes of expression may be offensive or misunderstood. When interviewers have a share in phrasing the questions, they may tend to enter more wholeheartedly and sympathetically into their tasks.
- Carefully train your interviewers to (a) identify the agency sponsoring the study, (b) explain the purpose of the questioning, (c) tell why and how their respondents were selected, and (d) assure their respondents that their answers will be kept confidential.
[Ukrainian] [English] [Russian]