Qualitative Methods

The National Science Foundation recommends use of qualitative methods in data collection (see the 2002 Evaluation Handbook). Use of both quantitative and qualitative data collection serves to triangulate findings, i.e. to substantiate outcomes. Most commonly used methods that are feasible in CISE REU include individual interviews and focus groups. This section is not intended to replace consultation with expert in qualitative methodology, but instead to inform and guide qualitative design among CISE REU sites.

What are the goals of the focus group and interviews:

  • Triangulation of attitudes and perspective of the REU
  • More depth than survey can provide
  • Fill gaps in our understanding of the students
  • Is powerful conveyor of student experience

What are the goals of the focus group and interviews:

  • Focus group intent is lab oriented, and focused on team process, group outcomes
  • Interviews individual is personally centered on their experience (learning, fit, future plans)

Individual Interviews – By conducting interviews, we assume that the opinions and experiences of participants are meaningful, and that their perspectives impact the project success. Interviews may be structured (no deviation in questions among interviewees), semi-structured (some variation based upon the interviewer’s discretion), or unstructured (usually in depth interviews based on the interviewees subjective experience). Interviews are an effective means to explore students experience, perspective, expectation, and what is salient to them. Interviews allow depth of exploration and provide insight beyond a survey questionnaire. Disadvantages to conducting interviews are that they are time consuming, require extensive interviewer training, and lengthy data analysis process. Consult with social science departments on your campus to locate faculty expertise and assistance.

Focus Groups – Focus groups enable in depth information to be collected, similar to individual interviews, with an advantage of providing efficiency in that they are conducting in small groups of 8 – 10 participants. As with interviews, focus groups allow for insight into the subjective experiences and reflections of participants, with the catalyst being group interaction. However, focus group facilitators should be trained in delivery and in group dynamics, to catalyze open participation from everyone and minimize the dominance of a small minority in the group. If group peer pressure is of concern at your site, focus groups are not recommended. Ideally, the facilitator is not a faculty advisor to the students.

Considerations

  • Since most CISE REU PIs are not familiar with qualitative methods, consulting social science faculty with this expertise is recommended.
  • Interviewers and focus group facilitators should not be faculty advisors to students, but an objective and trained third party.
  • Data analysis is time consuming; software is available (see Software Packages).
  • Regard for personal privacy is essential, therefore strict levels of informed consent and confidentiality need to be employed.

The following questions can serve as guidelines for qualitative protocols among CISE REU sites.Semi-structured Group Questions (probes may follow to prompt discussion):

Semi-structured Group Questions (probes may follow to prompt discussion)

  1. What did they learn, and how do they know they learned it? [brainstorm and pick top 3]
    • In particular, what computer science concepts or skills did you learn that you did not know before?
    • What resources helped you learn them?
  2. How have their views towards research evolved through this experience? [What characteristics do you think a good researcher needs to have (and describe what they mean)]
  3. How have their views towards grad school evolved, if they have changed?
  4. Describe what the lab climate was like in your groups. [faculty and peer support, social interaction]
  5. What recommendations do they have that would have been helpful for your lab project?

Semi-structured Individual Questions (probes may follow to explore responses)

  1. What about doing a summer REU appealed to you?
    • What did you like about the program?
    • What concerns do you have about the program?
  2. Describe what your summer experience was like:
  3. List and discuss the top 3 things you learned (inside and outside of discipline).
  4. How is your research socially relevant and how important is that to you?
  5. What are your short term and long term academic and professional goals?
    • How do you think the REU will contribute to these goals?

[This section is an adaptation from the ADVANCE portal, the NSF User-Friendly Handbook for Project Evaluation, and Assessing Campus Diversity Initiatives (Garcia, et al.)]