Review item quality on the test

The value of performance data relies on each item assessing the right learning outcome

Content review

  • Does the item measure important learning?
  • Is the content measured worth long-term learning?
  • Does the item measure instructional significant content or skills?
  • Does the student have to think to answer each question? Is it simple recall of information, or do they have to apply critical thinking, problem-solving, or creativity?
  • Is the item overly specific or abstract?
  • Does the question attempt to serve too many purposes? Can you tell from a student response that they understood the behavior or knowledge tested?
  • If a student incorrectly answers an item, will you be able to say what the student did wrong?
  • Is this item cueing an answer to another question?
  • Does this item measure facts, not opinions?
  • Is there anything tricky about the item? Will all students understand the problem?

Format and style review

  • Does the item use a format appropriate for the content, and age of students?
  • Is the question so complicated that most students will not understand the topic?
  • Are the items formatted consistently? For example, vertically or horizontally.
  • Is vocabulary appropriate for the student population tested?
  • Does the item or passage, require too much reading? Is it worth the student’s time?

Stem or the question

  • Does the stem have correct grammar, spelling, and punctuation?
  • Is the stem written so all students should understand the problem?
  • Are there any words or phrases that do not need to be part of the stem?
  • Is all information in stem relative to answering the item? Does the student have to weed out unnecessary information to respond to the item?
  • Is there a better way to phrase the stem?

Correct answers and distractors (incorrect answers)

  • Does the right answer match the key?
  • Are all distractors reasonable and standard errors?
  • Is there too much repetition of phrases in the distractors? For example, repeating the same introductory term in each item answer choice.
  • Does the correct answer item choice vary in position across the assessment?
  • For numerical answers, are the potential solutions in logical or numerical order? Be careful in systems that let you scramble answers for multiple versions of assessment. It may create unfair versions of the item.
  • Is the length of each choice similar?
  • Are there any clues that give away the correct answer, such as silly distractors?

Source:
Haladyna, T. (1997). Writing test items to evaluate higher order thinking. Boston: Allyn and Bacon.

Science assessments

Classroom science assessments

Effective science assessments measure student understanding and learning of concepts. Standards-based questions should work together to create a picture of student understanding.

  1. Identify the important idea of the domain to select standards.
  2. Analyze the skills and concepts within the standards to gauge how they work together to create a learning story.
  3. Broad standards effect the coverage of learning measured in the test. Because of the complementary nature of science standards, it is common to see overlap and cueing across questions.
  4. Standards with many skills need more items or a performance task to allow students to show learning across the skills within a standard.
  5. Science tests easily over-burden a concept or skill. This happens often with popular or interesting concepts.

Quickly review quality of a classroom assessment

Check and review

Quality

  • Provides valuable data and student information
  • Each question measures important content or skills
  • Matches instructional focus and intended learning

Content and skills

  • Each item measures content objectives, and their skills, accurately and appropriately
  • The vocabulary for each item matches instructional practice, and content area specifications
  • Overall test content parallel actual instruction during the learning cycle
  • Required background knowledge is fair
  • Questions free from bias

Thinking

  • Requires students to show thinking needed at this point in the year for Critical, Creative, Problem solving, etc…
  • Age and grade appropriate thinking required