Cultural Responsive Evaluation

The resources on this page have been shared by SGSO Network members, and collectively curated by participants in the 2021 National SGSO Leadership Institute, listed below. We invite you to contribute your own resources to share with the SGSO Network on the topic of program assessment & sharing results.

The following are select video clips, notes, and resources from Dr. Mele Wheaton’s presentation at the 2021 SGSO Virtual Leadership Institute. This presentation on Culturally Responsive Evaluation was part of a week-long working group curating resources related to MEASURING AND SHARING IMPACTS. 
Thanks to Djamila Moore of Grow Portland and Luisa Aviles of OutTeach who worked to document the summary below.
Dr. Mele Wheaton is a Senior Research Scholar at Stanford Woods Institute for the Environment.
 
See more resources related to CULTURALLY RELEVANT IMPACT ASSESSMENT on the Strengthening Equity Promising Practices resource page.

Culturally Responsive Evaluation

Culturally Responsive Evaluation (CRE) 

In this clip, Dr. Wheaton explains that Culturally Responsive Evaluation is more of a holistic approach to evaluation, rather than a prescribed template.

“Culturally responsive evaluation (CRE) is an evaluation approach that places

culture and the community of focus at the center of the evaluation, helps to support community empowerment, and has a goal of social justice” (p2).

McBride, D. (2018)

“CRE is a holistic framework for centering evaluation in culture (Frierson, Hood, Hughes, and Thomas, 2010). 

It rejects culture-free evaluation and recognizes that culturally defined values and beliefs lie at the heart of any evaluative effort” (p282).

Hood, S., Hopson, R.K., & Kirkhart, K.E. (2015)

Important for teaching practices to reflect evaluation approach:

BEETLES- Inclusive, equitable, and culturally relevant learning environments

North American Environmental Education Association- Culturally Relevant Research

Cultural Competence 

In this clip, Dr. Wheaton shares a snapshot of Cultural Competence from the American Evaluation Association.

  • Ensure recognition, accurate interpretation, and respect for diversity and the cultures represented.
  • Include cultural and contextual dimensions.

CRE Characteristics

In this clip, Dr. Wheaton goes in depth on the characteristics of CRE, based on the scholarship of Hood, et al., (2015).

  • Demonstration of cultural competency and understanding of program’s sociocultural contexts.
  • Engagement of stakeholders in trusting and meaningful
    relationships.
  • Inclusion of a diversity of stakeholders with differing status or types of power.
  • Identify purpose and intent of evaluation with stakeholders- formative, process, or summative?
  • Attention to the perspectives of stakeholders in types of
    questions will be asked, how they will be asked, and how they will be prioritized.
  • Evaluation design that captures varied perspectives and responsive to the program context.
  • Map of information sources, time frames for data collection and analysis, and how data will be collected and analyzed.
  • Use of both qualitative and quantitative methods.
  • Consideration of instruments and tools to include checks of cultural bias in language and content.
  • Consideration of culturally appropriate protocol when
    collecting data.
  • Examination of diversity within group during data analysis.
  • Dissemination of results to stakeholders.
  • Community review.
  • Does the program operate in ways that respect local cultural practices?
  • How well does the program connect with the values, and worldviews of participants?
  • “Please choose fruits and vegetables to grow in your school garden that are important to your family, community, or culture: “
  • Provide language translations specific to your community.

Connecting Questions with Audience Needs

In this clip, Dr. Wheaton connects the key aspects of good survey questions (from 2020 presentation) to culturally responsive evaluation approaches.

  • Center prior knowledge/experiences of participants as starting point.
  • Avoid ‘agree/disagree’ response.
  • Specify audience (Teacher? Student? Parent?, etc.) 
  • Open-ended response is more inclusive than scaled response.
  • If using pre/post assessment questions- use same language in both questions to connect directly to focal content.

Grow Portland- Student Garden Planning Survey

Grow Portland- Family Garden Planning Survey (also Spanish version)

Pre-assessment questions for teachers:

  • “What are your experiences with gardens or growing plants?”
  • “What connections do you want students to make to classroom learning?  Home learning?  Community learning?”
  • “What barriers/challenges do you anticipate in the Garden Program?”
  • “What do you hope to accomplish in this program?”
  • “What skills do you want students to practice during the Garden Program?”

Post-assessment questions for teachers:  

  • “What connections did students make to classroom learning?”
  • “What, if any, barriers or challenges did you experience in the Garden Program?”
  • “What skills did students practice in the garden?”

Equitable Evaluation

In this clip, Dr. Wheaton gives a summary of equitable evaluation.

  • Rejects industrial complex of education and evaluation- centers complexity and decolonization.
  • Include diversity of stakeholders with different status and types of power.
  • Examine diversity of responses, not just generalize majority responses- outliers provide important data.  
  • Pay attention to whose voices may be missing.

Evaluation Resources (CRE & Environmental Ed)

Resources: Evolution of terminology

In this clip, Dr. Wheaton acknowledges that culturally relevant evaluation is directly tied to culturally relevant teaching and reviews the evolution of terminology in this area.

 

‘Culturally Relevant Pedagogy’ Ladson-Billings, G. (1994); ‘Culturally Responsive Teaching’ Gay, G. (2010); ‘Culturally Sustaining Pedagogy’ Paris, D. and Alin, S., Eds. (2017)

Understand Needs of Funder Reporting/ Time considerations & equitable evaluations

In this clip, Dr. Wheaton provides tips on designing evaluations that meet the needs of funder reports while balancing and prioritizing community needs.

  • Communicate directly with funders.
  • Pilot survey or other tool- keep it short, easy to read/understand.  
  • Respect participant’s time.  Learn best days/times to ask for feedback.

Beyond Surveys: Culturally-relevant Evaluation Tools

In this clip, an audience member asks about tools other than surveys.  Dr. Wheaton acknowledges that surveys are the least culturally-relevant tool, and provides examples of more inclusive evaluation tools.

 

  • Requires more time and intentional analysis (coding & thematic organization), but creates more equitable and meaningful data.
  • Acknowledges that this is a constant learning process.

 

  • Focus groups
  • Open-ended questions
  • Interviews
  • Listening sessions
  • Historical documents (info/context might be readily available without asking for participant time.)

View the full, unedited Zoom meeting of Dr. Wheaton’s meeting with the leadership institute working group.

Q & A With Dr. Wheaton

Select questions discussed by SGSO Virtual Leadership Institute participants with Mele Wheaton

 

Can you share examples of student voices engaged in evaluations?

My recommendation is to conduct a small student focus group and ask them to talk about their interests/their learning in a program. From there, I would see what themes are occurring and write those into instruments. I would also pilot  any student-focused evaluation questions with a small student group to get feedback on how they are understanding the question. In my experience, hearing from participants/users in crafting program direction or evaluation provides a wealth of information that one wouldn’t normally hear.

 

Can you share child-centered evaluation tools & best practices for evaluating children?

Types of assessment /evaluation tools for children could include concept maps, journaling, observation, questionnaires, and focus groups. The method you choose will be dependent on what you want to learn from your evaluation. 

 

Given that community relationships are key for trust, communication, and equitable evaluations- should we be striving to pass on that institutional knowledge to new staff?  And if so, how?

Yes, I think any part of hiring/onboarding should include that institutional knowledge. I would go about by having new staff accompany seasoned staff to meetings with community partners so new staff can be introduced and begin to develop relationships. I also think it would be helpful to have a seasoned staff member talk with community partners with whom they have close relationships, and ask about what works/doesn’t work for them in maintaining relationships, and make sure that information is passed along. Of course, some part of the relationship will be personnel-based but if an organization has standards for how they conduct positive community partnerships, then I think the passing along of relationships will be more successful.

 

Are there examples of culturally-specific evaluation tools?

My suggestion is to ask the community partner/organization what they think might be appropriate in design and questions if working with cultural groups different from own’s one. For example, in Western-European educational and parenting cultural practices, there is an overwhelming practice to focus on the individual for learning, teaching, and evaluating.  That is not necessarily the case in other cultural groups. So, in that case, developing tools that engage a group of students may be more appropriate. 

The resources on this page have been shared by SGSO Network members, and collectively curated by participants in the 2021 National SGSO Leadership Institute, listed below. We invite you to contribute your own resources to share with the SGSO Network on the topic of program assessment & sharing results.
More Resources on Measuring and Sharing Impact