Data Collection Methods
There are a variety of ways that REU sites can collect information on participants. The scope of this section is to provide an overview of primary ways of gathering information from site participants to measure the intended outcomes and to also address the political issues that are involved.
Before Data Collection
- Be sure to obtain necessary permission and clearance
- Institutional Research approval of studies with Human Subjects
- Consent from participants
- Inform Institutional Research of any changes in the study
- Be sensitive to the needs of participants
- Anonymity will be limited in small samples such as REU sites
- Confidentiality of personal information
- Cause as little disruption as possible
- Be sure that the data is handled and analyzed by an adequately trained, objective and unbiased individual
All CISE REU sites rely on surveys to measure student attitudes and experiences. However, the designs of the studies vary. The National Science Foundation recommends use of mixed-method approach to data collection (see the 2002 Evaluation Handbook, p. 46). When investigating human behaviors and attitudes, use of a myriad of collection methods enriches the study by minimizing the weaknesses of any one method. Use of both quantitative and qualitative data collection serves to triangulate findings, i.e. to substantiate outcomes.
A typical mixed-method approach in a CISE REU site might consist of the following data collection methods: surveys, individual student interviews, focus groups, knowledge tests, observations from faculty, and document reviews (of student presentations, posters, papers). Each of these are described below.
Surveys – Surveys are easy and cost effective methods of data collection. Most universities provide web based software for delivery of surveys (such as Survey Share, or Survey Monkey). If not, accounts can be quickly established for reasonable rates. Surveys provide standardization, descriptive data, coverage of REU constructs (research and computing acquisition, self efficacy, commitment to computing), and ease in analysis. A pre – post design is strongly encouraged to allow measurement of program impact, meaning that the same survey is administered to students at the beginning of the REU and again at the end of the REU. A disadvantage of note is self-report bias, and may lack depth and context.
Individual Interviews – By conducting interviews, we assume that the opinions and experiences of participants are meaningful, and that their perspectives impact the project success. Interviews may be structured (no deviation in questions among interviewees), semi-structured (some variation based upon the interviewer’s discretion), or unstructured (usually in depth interviews based on the interviewees subjective experience). Interviews are an effective means to explore students experience, perspective, expectation, and what is salient to them. Interviews allow depth of exploration and provide insight beyond a survey questionnaire. Disadvantages to conducting interviews are that they are time consuming, require extensive interviewer training, and lengthy data analysis process. Consult with social science departments on your campus to locate faculty expertise and assistance.
Focus Groups – Focus groups enable in depth information to be collected, similar to individual interviews, with an advantage of providing efficiency in that they are conducting in small groups of 8 – 10 participants. As with interviews, focus groups allow for insight into the subjective experiences and reflections of participants, with the catalyst being group interaction. However, focus group facilitators should be trained in delivery and in group dynamics, to catalyze open participation from everyone and minimize the dominance of a small minority in the group. If group peer pressure is of concern at your site, focus groups are not recommended. Ideally, the facilitator is not a faculty advisor to the students.
Knowledge Tests – A test of the students’ knowledge of computing and research skills is an effective measure of gains made from participating in the REU. Tests are generally perceived as credible, and provide an objective measurement. The content being assessed will depend upon the particular research areas addressed at each site and the instrument must be carefully evaluated to ensure that it is valid and reliable. Self report (via survey) on knowledge gains is an acceptable measure.
Observations – Faculty observations can provide valuable insight into the behaviors of students and how the REU site is operating. Observations identify unexpected events and outcomes and are easy to collect. However, observations are subjective and may not necessarily apply to all participants.
Document Review – Tracking the documents produced during the REU are an easy and cost effective method of triangulation. Any submitted papers, posters, and presentations can be archived as evidence of outcomes. Most REU sites culminate the experience with student presentations, which can be recorded (with student permission) and archived. Slides can also be maintained. Documents provide a mechanism for describing student research progress and achievement.
Longitudinal Study – Following up with participants after their REU experience can provide meaningful information, such as whether or not students actually pursue graduate studies. However, cost and time are a consideration upon termination of funding. While longitudinal surveys are not required, they are recommended. Short and cost effective surveys can be delivered online to former student participants, in order to obtain computing retention information.
This section is an adaptation from the ADVANCE portal, the NSF User-Friendly Handbook for Project Evaluation, and Assessing Campus Diversity Initiatives (Garcia, et al.)]