By Jon Boxerman, Sharon Nelson-Barber, Kimberly Nguyen, and Liz Rechebei
The goal of the NSF INCLUDES: SEAS Islands Alliance’s research is to understand how Indigenous islanders on several U.S. territories and affiliated islands develop science or engineering identities and experience a sense of belonging to the STEM community and to their own cultural heritage.
To accomplish this, we engaged with representatives with lived and nuanced cultural expertise, and together we developed measurement instruments that more accurately and authentically represented core concepts of identity and belonging in participating communities. We hope the process can be a model for creating culturally valid measures that may be more appropriate for the cultural participants in similar research contexts.
Broadening participation in science, technology, engineering, and math (STEM) requires the accurate representation of the diverse perspectives of people from a range of cultural communities. There is great need to develop new and more equitable ways of working, rather than falling back on conventional processes. To create the systems change* needed to broaden STEM participation in the U.S. territories and U.S. affiliated islands, the NSF INCLUDES: SEAS Islands Alliance research team has collaborated with cultural experts from the island nations, to try to understand how to establish evidence-based interventions and develop authentic survey instruments—tools that can accurately and authentically reflect different cultural conceptions of identity and sense of belonging.
The NSF INCLUDES: SEAS Islands Alliance research team is working with marine and geosciences programs at three universities: WestEd at the University of Guam (Mangilao), the University of Illinois Urbana-Champaign at Universidad Interamericana de Puerto Rico (San Germán), and the University of the Virgin Islands (St. Thomas). We are asking and answering the same research questions in each setting using shared measures and common culturally relevant methods and approaches. The team initiated a cultural validation process because we anticipated differences in the surveys and interview protocols across island contexts. We quickly learned that the three cultural communities brought distinct and nuanced perspectives to key social constructs. Although different issues and concerns arose in each research context, for the WestEd research team working with Micronesian students, the topic of race was salient and stood out.
Developing a Process for Culturally Validating Measures Locally
The research team was aware that words and meanings often carry different connotations across distinct contexts. The research team proceeded according to two essential values:
- Shared measures need to be clear and feel appropriate to the people using them;
- Measurement instruments must be valid in the contexts where they are used and must invite authentic and meaningful interaction among participants.
Local experts—community members fluent in the local cultures and languages—have been essential to our work, serving as “research liaisons” who shared their cultural knowledge and experiences to validate the study instruments for our diverse populations.
The research team downloaded identity and sense of belonging surveys from the STEM Learning and Research Center website. The research liaisons evaluated the cultural validity of these measures. Over one summer, the research team and liaisons met online to discuss insights and to collaboratively refine and revise the instruments. For example, a research liaison might flag an item and add a comment to the shared document. The concern was discussed in a meeting and a group decision made by the full research team including the liaisons, as to whether to revise or drop the item. The revised item was reviewed by the research liaisons once more before being deployed.
An early version of one interview item asked the commonly used census-type open-ended question: “What is your race?” Some Chamorro students chose not to identify a race; some respondents asked for clarification: “Do you mean ethnicity?” And some Chamorro students shared their nationality instead, for example, by identifying their race as Chamorro American. By contrast, a faculty member from Puerto Rico explained that participants often self-identify ethnically as Hispanic and racially as White. Such unanticipated answers are illustrating how deeply context-dependent the concepts of race and ethnicity are, indicating how they are locally defined and understood.
As this cultural validation process unfolded, our Chamorro liaison, Dr. Liz Rechebei, was not surprised that some students either chose not to identify a race or were not sure about the question and did not respond at all. She informed us that in Guam, the social construct “race/ethnicity” was fundamentally problematic because “race” is not a meaningful identifier; it is a superficial term, seldom used among many Chamorros and/or Pacific Islanders in Guam. This led the research team to replace the race/ethnicity question with the more productive open-ended query: “How do you identify yourself?” The open-ended question allows participants to affirm language they ordinarily use to reflect their conception of identity.
Challenges and Lessons Learned
Developing validated research instruments in collaboration with expert community members shed light on the fundamental need to center local cultural contours and perspectives to design inclusive instruments that help communicate how to effectively broaden participation in STEM. When we moved away from checkboxes and categories and toward open-ended constructed response items, we found it challenging to aggregate survey data to create shared measures. The checkboxes did not work. Had we followed that practice, we would have gathered inauthentic information. Collecting authentic information generated by partners, elders, and other knowledgeable community members far outweighed the need to force-fit people into boxes defined by outsiders.
We had underestimated the extent to which cultural validation would be needed for this project, which required more cost and time than anticipated. The cultural validation process was possible because we drew on relationships with local knowledge keepers who consulted for the project. In the end, similar measures were developed for each of the three island settings.
In the context of this NSF INCLUDES program, census-style type questions had the potential to undermine the goals of the research. Take, for instance, the label “Asian/Pacific Islander,” a broad and inaccurate categorization that essentializes two very distinct groups into a single checkbox. Given that our research goal is to understand how Indigenous islanders develop science or engineering identities and experience a sense of belonging to the STEM community and to their own cultural heritage, it was necessary to tap into nuanced cultural expertise in our partner communities. If appropriate checkboxes were unavailable when respondents answered questions, they could have concluded the program “was not a good fit.”
Moving Forward
By sharing our process for developing and culturally validating shared measures, we hope to refocus the broadening participation in STEM conversation on the appropriateness of the methods and instruments. Just because a survey question has worked in one community, there is no guarantee that it will mean the same thing or will be interpreted the same way in another context within the same community. The research community must consider the practice of creating culturally valid measures by carefully listening and responding to culture and language and by encouraging local measure-validation processes.
To learn more about the NSF INCLUDES SEAS Islands Alliance, visit our website. Find us on Twitter @SEAS_Islands.
| The NSF INCLUDES: SEAS Islands Alliance research team is a collaboration between WestEd and the University of Illinois Urbana-Champaign (UIUC). The team works on different islands, with distinctly different cultures and observations. The WestEd research team works with some students from the geographic region known as Micronesia, which includes Guam, Northern Mariana Islands, Republic of Palau, Republic of the Marshall Islands, and the Federated States of Micronesia and is led by Sharon Nelson-Barber and includes Jon Boxerman, Kimberly Nguyen, Ramon Rechebei (Palauan), and Liz Rechebei (Chamorro). The UIUC research team is working with students in Puerto Rico and the U.S. Virgin Islands. The UIUC team is led by Rodney Hopson and includes Manuel Pérez-Troncoso, María Serrano Abreu, José Santiago Román, Howard Forbes Jr., and Jarvon Stout. |
*By “systems change” the NSF INCLUDES: SEAS Islands Alliance means rebuilding the structures and processes of tribal, state, and federal entities, for the purposes of culturally revitalizing and sustaining teaching and learning practices and resources and aligning these efforts with education and economic recovery.