Evaluation

The main evaluation of question generation is based the meta-analysis Systematic comparing of differerent scientific studies on a topic, often reporting an effect size of the combined results. of Rosenshine, Meister, & Chapman (1996) who employed the same criteria as Rosenshine& Meister (1994) for selection of studies.

Grade Level

The study of question generation instruction begins with grade 3 and has been carried out up to grade 9.  he distribution of grade levels for this kind of instruction is: level 3, n=3; level 4; n= 6; level 5; n=4; level 6; n= 9; level 7, n= 4; level 8, n=3; level 9, n=2.  The modal level is grade 6.

Experimenter Tests

The respective effect sizes for Multiple Choice (n = 6), Short-Answer (n = 14) and Summary (n =  3) measures were 0.95, 0.85, and 0.85.

Standardized Tests

The median effect size for 13 studies that used standardized comprehension tests was 0.36.  The median effect sizes for Standardized versus Experimenter Tests are reported in Table 17, broken down by reciprocal teaching and other treatments.  The magnitude of the median effect sizes in Table 17 is approximately the same as those found for reciprocal teaching of multiple strategies.  There is an overlap of studies here so that the similarity is likely a result of common studies.  Of interest is that although there is a positive effect size for Standardized Tests, only 3 out of 13 are statistically significant. Experimenter tests fare better here since 16 out of 19 are statistically significant. Thus the effects of direct instruction of question generation are specific to learning the particular strategy and may not generalize to standardized tests.

Median Effect Sizes for Question Generation

(data from Rosenshine, Meister, & Chapman, 1996)

Summary Evaluation of Question Generation

There is strong empirical and scientific evidence that direct instruction of question generation during reading benefits reading comprehension in terms of memory and answering questions based upon text as well as integrating and identifying main ideas through summarization.  There is mixed evidence that general reading comprehension is improved on standardized comprehension tests.  Question Generation may be best used as a part of a multiple strategy instruction program.