Key Characteristics of Great Surveys & Sample Survey Questions

Key Characteristics of Great Surveys

The following page was created by 2021 SGSO Leadership Working Group Members Djamila Moore – Grow Portland and Scott Feille and Luisa Aviles of Out Teach.

We invite you to contribute your own resources to share with the SGSO Network on the topic of program assessment & sharing results.

Overview

The following is a viewer guide and key points shared by Dr. Mele Wheaton of Stanford University’s Woods Institute for the Environment. Dr. Wheaton presented on key characteristics of creating surveys at the 2020 School Garden Support Organization Leadership Institute. She referenced survey examples from attending SGSOs. The user guide below is meant to outline key elements of her presentation (which has poor audio/visuals). Click the youtube links for short clips from her presentation.

Key Characteristics of Great Surveys

The Survey Response Process

In this clip, Dr. Wheaton starts to analyze Jon Krosnick’s research about using optimal questionnaire design. You will learn more about how respondents approach their questions, and you will begin to develop skills that will help you to manage design issues while developing your surveys:

East to Read
Easy to Analyze
Short (10-15 min. max)
Piloted

Reference:

Jon A. Krosnick, Ph.D. – JonKrosnick

Example Questions: Easy to Read, Analyze and Short Survey

– Q: How engaged do you feel the majority of your students were during lessons presented?
A: Not Engaged | Somewhat Engaged | Engaged | Very Engaged

– Q: How relevant were the lessons presented to the curriculum you were teaching this school year?
A: Non-Relevant | A Little Relevant | Relevant | Super Relevant

– Q: How would you rate your OVERALL experience with our program? If you chose “fair or poor,” please let us know how we can improve.
A: Poor | Fair | Good | Excellent

-What was your (or your classes’) favorite moment this year?

Types of Survey Questions – Open-Ended

In this clip, Dr. Wheaton discusses the first decision you must make when writing a question: Will it be open-ended (allowing respondents to answer in their own words)?

She also presents the formatting decisions that must be made.

Open-ended questions responses usually need to be read and then tabulated or “coded” to determine response rate.

Open-Ended Question Examples

What are some outcomes/experiences you feel your students obtained through the Garden Education program?

What types of support would help you feel more confident helping you to deepen the outcomes for your students in the future? (e.g. curriculum, connection to nonfiction/fiction literature, newsletter tips, lesson kits, materials, training, grab and go kits with an NGSS aligned lesson plan materials available, other)

Types of Survey Questions- Close-Ended

In this clip, Dr. Wheaton defines what closed-ended questions are. It was mentioned to start with the more negative response option to avoid a bias to ‘easily agree’.

Key Characteristics of Great Question

In this clip, Dr. Wheaton starts discussing the formatting decisions that must be made for closed-ended questions, and how to find the hidden variable in the question to write the response choices.

Closed-Ended Questions Example

How confident do you feel teaching your students in the garden after having participated in the fall program? Not confident/ somewhat confident/ very confident/ highly confident

How effective were the NGSS in the Garden Program educators? Not effective/ somewhat effective / very effective / highly effective

Question Types to Avoid

In this clip, Dr. Wheaton explains why to avoid agree/disagree scale questions in your surveys.

She also talks about Acquiescence Bias, or the tendency for respondents to agree with statements.

Reliability Measures for Scaled Responses- Agree/Disagree

In this clip, Dr. Wheaton discusses the question of how to get away from the agree/disagree scale, tapping into the questions of:

What do you really want to know?

What is going to help us improve?

Q&A Related to Reliability and Validity for Scaled Responses

In this clip, Dr. Wheaton answers questions to help participants design better surveys.

Include Balanced Responses

In this clip, Dr. Wheaton discusses the strengths and weaknesses of asking Yes/No and True/False questions.

Scaled Responses – Statements Example:

To what extent did the garden experience do the following:

Helped me create meaningful hands-on activities for students:

NOT AT ALL / A LITTLE/ MODERATELY / A GREAT DEAL / A LOT

Increased my students’ curiosity about the natural environment:

NOT AT ALL / A LITTLE / MODERATELY/ A GREAT DEAL / A LOT

Increased my students’ confidence in being outdoors:

NOT AT ALL / A LITTLE / MODERATELY / A GREAT DEAL / A LOT

Helped me make curriculum more accessible for English Language Learners:

NOT AT ALL / A LITTLE / MODERATELY / A GREAT DEAL / A LOT

Reliability Measures for Scaled Responses – Statements

In this clip, Dr. Wheaton describes how to analyze responses when the questions are written as a statement. And, the importance of writing short questions.

Balanced Responses Example

Did you eat vegetables today or not eat vegetables today?”
Did being in the garden this week make you want to eat vegetables this week or did it make you want to avoid eating vegetables?

Q&A Related to Including Balanced Responses

In this clip, Dr. Wheaton answers questions from the participants and discusses the best practices when using balanced responses.

Begin Scale with More Negative Response Choice

In this clip, Dr. Wheaton explains ‘The Primacy Effect’:

People are more likely to be agreeable and choose first response, especially if positive, which can skew the data.

Use Verbal Labels

In this clip, Dr. Wheaton explains how to reduce task difficulty and respect participants’ time by not combining verbal and numeric response choices. Specific verbal responses provide most comprehensive data.

“How useful were the resources provided in the Garden Program?”

Rather than 1-5 (Not useful to extremely useful), remove number scale and try:

“Not at all useful”
“Slightly useful”
“Moderately useful”
“Very useful”
“Extremely useful”

Bipolar vs. Unipolar Responses

In this clip, Dr. Wheaton explains the difference between bipolar & unipolar responses.

Bipolar response= true neutral point in the middle “neither bad nor good”. Usually best as 7-point scale.
Unipolar response= specific amount from minimum to maximum, with no neutral point. Usually best as 5-point scale.
Respondents usually prefer 5 point scale.
Tailor responses for audience, ie. Picture scales for young children.
Begin scale w/ more negative response choice to reduce primacy effect.

Bipolar=

“Extremely bad / Moderately bad / Slightly bad / Neither good, nor bad / Slightly good / Moderately good / Extremely good”

Unipolar=

“Not at all effective / Slightly effective / Moderately effective / Very effective / Extremely effective”

Reduce Task Difficulty

Strive to make the process of interpreting question answers easy and not complicated. In the example below there are five responses but with the included definitions there are 10 examples. If you really need such detail in your responses you could consider making open-ended questions or creating more specific responses per question.

When tabulating responses for multiple schools, first organize data by school and/or grade.

Then organize by themes, for example:

Comments about hands-on learning.
Importance of time spent outside.
Academic connections.
Social emotional learning.
Requests for specific learning content.
Constructive feedback for instructor.
Observations of student learning.

Reduce Task Difficulty

In this clip, Dr. Wheaton discusses using picture scales for kids.

Picture scales are effective but need to be aware that some students will refuse to choose a ‘frowny face’- want to be agreeable.
Avoid asking kids if they ‘like’ something- better to ask how willing they are to try it.
Simplify question and don’t ask two things at once (Do you like it/have you tried it?)

We invite you to contribute your own resources to share with the SGSO Network on the topic of program assessment & sharing results.