Skip to main content

Advertisement

SpringerLink
Log in

Examining a Comprehensive College Transition Program: An Account of Iterative Mixed Methods Longitudinal Survey Design

  • Research and Practice
  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

There are few accounts in the higher education literature of mixing methods at the survey design stage and very little guidance targeting higher education researchers and practitioners who want to implement a mixed methods approach to design survey tools. This article explores an eight-step, iterative, mixed methods approach for creating a longitudinal, multi-institutional survey to assess how participation in a comprehensive college transition program is related to students’ psychosocial and academic outcomes. In the context of a college transition program study, our mixed methods strategy to develop a survey instrument included initial qualitative data collection and review of psychosocial constructs, psychometric pilot, baseline survey, focus groups, case study research, cognitive interviews, follow-up pilot, and follow-up survey. This article makes a unique contribution to higher education research, providing a potential model for others seeking guidance in mixing methods at the study design and instrument development stage. Major lessons learned from the mixed methods survey design process are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. A mixed method process (i.e. sometimes referred to as a triangulation process), where focus groups, cognitive interviews and other techniques used in the development of closed-ended items has been cited as far back as Lazarsfeld (1944), but more recently in higher education literature by Ouimet et al. (2004) with the creation of the NSSE (National Survey of Student Engagement). However, Creswell et al. (2003), Dillman et al. (2014), and Jick (1979) have arguably been some of the most frequently referenced work on how closed-end items can be created through this kind of process. The kind of feedback, as well as the data sources used to construct items through a mixed methods process, can minimize item ambiguity, increase item accuracy, while enhancing item measurement variability. In prior research by Ouimet et al. (2004), focus groups and cognitive interviews provided critical information about survey usefulness and effectiveness. According to this literature, these procedures are underutilized as they require expertise and additional resources like time and funding support.

References

  • Bandura, A. (1969). Principles of behavior modification. New York: Holt, Rinehart & Winston.

    Google Scholar 

  • Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191.

    Article  Google Scholar 

  • Barnhardt, C., Sheets, J., & Pasquesi, K. (2015). You expect what? Students’ perceptions as resources in acquiring commitments and capacities for civic engagement. Research in Higher Education, 56(6), 622–644.

    Article  Google Scholar 

  • Bollen, K., & Hoyle, R. (1990). Perceived cohesion: A conceptual and empirical examination. Social Forces, 69(2), 479–504.

    Article  Google Scholar 

  • Bourdieu, P. (1977). Outline of a theory of practice (Vol. 16). Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Bourdieu, P. (1986). The forms of capital. In J. G. Richardson (Ed.), Handbook of theory and research for the sociology of education. New York: Greenwood.

    Google Scholar 

  • Cabrera, N. (2011). Using a sequential exploratory mixed method design to examine racial hyperprivilege in higher education. New Directions for Institutional Research, 151, 77–91.

    Article  Google Scholar 

  • Callie, T., & Cheslock, J. (2008). The hiring and compensation practices of business school deans. The Review of Higher Education, 32(1), 25–49.

    Article  Google Scholar 

  • Chesborough, R. (2011). College students and service: A mixed methods exploration of motivations, choices, and learning outcomes. Journal of College Student Development, 52(6), 687–705.

    Article  Google Scholar 

  • Choi, N. (2005). Self-efficacy and self-concept as predictors of college students’ academic performance. Psychology in the Schools, 42(2), 197–205.

    Article  Google Scholar 

  • Creswell, J. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Sage: Thousand Oaks, CA.

    Google Scholar 

  • Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook on mixed methods in the behavioral and social sciences (pp. 209–240). Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. New York: Wiley.

    Google Scholar 

  • Garmezy, N., & Masten, A. S. (1991). The protective role of competence indicators in children at risk. In E. M. Cummings, A. L. Greene, & K. H. Karraker (Eds.), Life-span developmental psychology: Perspectives on stress and coping (pp. 151–174). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Gasiewski, J., Eagan, M., Garcia, G., Hurtado, S., & Chang, M. (2012). From gatekeeping to engagement: A multicontextual, mixed method study of student academic engagement in introductory STEM courses. Research in Higher Education, 53(2), 229–261.

    Article  Google Scholar 

  • Griffin, K., & Museus, S. (2011). Application of mixed-methods approaches to higher education and intersectional analyses. New Directions for Institutional Research, 151, 15–26.

    Article  Google Scholar 

  • Hausmann, L. R., Schofield, J. W., & Woods, R. L. (2007). Sense of belonging as a predictor of intentions to persist among African American and White first-year college students. Research in Higher Education, 48(7), 803–839.

    Article  Google Scholar 

  • Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121.

    Article  Google Scholar 

  • Hora, M., & Anderson, C. (2012). Perceived norms for interactive teaching and their relationship to instructional decision-making: A mixed methods study. Higher Education, 64, 573–592.

    Article  Google Scholar 

  • Hurtado, S., & Carter, D. F. (1997). Effects of college transition and perceptions of the campus racial climate on Latino college students’ sense of belonging. Sociology of Education, 70, 324–345.

    Article  Google Scholar 

  • Inkelas, K. (2007). National survey of living-learning programs. Report of findings: 2007. Retrieved from http://drum.lib.umd.edu/handle/1903/8392.

  • Inkelas, K. K., Vogt, K., Longerbeam, S., Owen, J., & Johnson, D. (2006). Measuring outcomes of living-learning programs: Examining college environments and student learning and development. Journal of General Education, 55(1), 40–76.

    Article  Google Scholar 

  • Ivankova, N., & Stick, S. (2007). Students’ persistence in a distributed doctoral program in educational leadership in higher education: A mixed methods study. Research in Higher Education, 48(1), 93–135.

    Article  Google Scholar 

  • Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24(4), 602–611.

    Article  Google Scholar 

  • Johnson, B., & Onwuegbuzie, A. (2004). Mixed methods research: A research paradigm whose time as home. Educational Researcher, 33(7), 14–26.

    Article  Google Scholar 

  • Johnson, B., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133.

    Article  Google Scholar 

  • Lazarsfeld, P. F. (1944). The controversy over detailed interviews: An offer for negotiation. Public Opinion Quarterly, 8, 38–60.

    Article  Google Scholar 

  • Leech, N., & Onwuegbuzie, A. (2009). A typology of mixed methods research designs. Quality & Quantity, 43, 265–275.

    Article  Google Scholar 

  • Lent, R. W., Brown, S. D., & Hackett, G. (2002). Social cognitive career theory. Career Choice and Development, 4, 255–311.

    Google Scholar 

  • Litwin, M. S., & Fink, A. (1995). How to measure survey reliability and validity (Vol. 7). Thousand Oaks: Sage.

    Book  Google Scholar 

  • Masten, A., & Obradović, J. (2006). Competence and resilience in development. Annals of the New York Academy of Sciences, 1094(1), 13–27.

    Article  Google Scholar 

  • Masten, A., & Obradović, J. (2008). Disaster preparation and recovery: Lessons from research on resilience in human development. Ecology and Society, 13(1), 9.

    Article  Google Scholar 

  • Mertes, S., & Jankoviak, M. (2016). Creating a college-wide retention program: A mixed methods approach. Community College Enterprise, 22(1), 9–27.

    Google Scholar 

  • Ouimet, J. A., Bunnage, J. C., Carini, R. M., Kuh, G. D., & Kennedy, J. (2004). Using focus groups, expert advice, and cognitive interviews to establish the validity of a college student survey. Research in Higher Education, 45(3), 233–250.

    Article  Google Scholar 

  • Pajares, F., & Schunk, D. H. (2001). Self-beliefs and school success: Self-efficacy, self-concept, and school achievement. Perception, 11, 239–266.

    Google Scholar 

  • Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16(4), 385–407.

    Article  Google Scholar 

  • Ray, A., & Margaret, W. (Eds.). (2003). PISA Programme for international student assessment (PISA) PISA 2000 technical report: PISA 2000 technical report. Paris: OECD Publishing.

    Google Scholar 

  • Ruthig, J., Haynes, T., Stupnisky, R., & Perry, R. (2009). Perceived academic control: Mediating the effects of optimism and social support on college students’ psychological health. Social Psychology of Education, 12(2), 233–249.

    Article  Google Scholar 

  • Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19(4), 460–475.

    Article  Google Scholar 

  • Smith, B. L., MacGregor, J., Matthews, R., & Gabelnick, F. (2004). Learning communities: Reforming undergraduate education. San Francisco: Jossey-Bass.

    Google Scholar 

  • Soldner, M., Rowan-Kenyon, H., Inkelas, K. K., Garvey, J., & Robbins, C. (2012). Supporting students’ intentions to persist in STEM disciplines: The role of living-learning programs among other social-cognitive factors. The Journal of Higher Education, 83(3), 311–336.

    Article  Google Scholar 

  • Strayhorn, T. L. (2015). Factors influencing Black males’ preparation for college and success in STEM majors: A mixed methods study. Western Journal of Black Studies, 39(1), 45–63.

    Google Scholar 

  • Tashakkori, A., & Teddlie, C. (2010). Putting the human back in “human research methodology”: The researcher in mixed methods research. Journal of Mixed Methods Research, 4(4), 271–277.

    Article  Google Scholar 

  • Teddlie, C., & Tashakkori, A. (2010). Overview of contemporary issues in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Sage handbook of mixed methods in social and behavioral research (Vol. 2, pp. 1–44). Thousand Oaks: Sage.

    Google Scholar 

  • Torres, V. (2006). A mixed method study testing data-model fit of a retention model for Latino/a students at urban universities. Journal of College Student Development, 47(3), 299–318.

    Article  Google Scholar 

  • Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.

    Google Scholar 

  • Vuong, M., Brown-Welty, S., & Tracz, S. (2010). The effects of self-efficacy on academic success of first-generation college sophomore students. Journal of College Student Development, 51(1), 50–64.

    Article  Google Scholar 

  • Wells, R., Kolek, E., Williams, E., & Saunders, D. (2015). “How we know what we know”: A systematic comparison of research methods employed in higher education journals, 1996–-2000 v. 2006–2010. Journal of Higher Education, 86(2), 171–195.

    Google Scholar 

  • Willis, G. (1999). Reducing survey error through research on the cognitive and decision processes in surveys. Short course presented at the 1999 meeting of the American Statistical Association. Research Triangle Institute.

  • Willis, G. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications.

    Book  Google Scholar 

  • Young, A., & Fry, J. D. (2008). Metacognitive awareness and academic achievement in college students. Journal of the Scholarship of Teaching and Learning, 8(2), 1–10.

    Google Scholar 

  • Zimmerman, M. A., & Arunkumar, R. (1994). Resiliency research: Implications for schools and policy. Social Policy Report, 8(4), 1–18.

    Article  Google Scholar 

Download references

Acknowledgements

This work would not have been possible without the important contributions, input, and guidance of our research partners at USC, AIR, and ISU. We are especially indebted to Matthew Soldner and Mark Masterton of AIR for the foundational work they contributed to in the development and administration of the surveys described in this article.

Author information

Authors and Affiliations

  1. Rossier School of Education, University of Southern California, Waite Phillips Hall, Room 701, 3470 Trousdale Parkway, Los Angeles, CA, 90089, USA

    Darnell Cole, Joseph A. Kitchen & Adrianna Kezar

Authors
  1. Darnell Cole
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joseph A. Kitchen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adrianna Kezar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joseph A. Kitchen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cole, D., Kitchen, J.A. & Kezar, A. Examining a Comprehensive College Transition Program: An Account of Iterative Mixed Methods Longitudinal Survey Design. Res High Educ 60, 392–413 (2019). https://doi.org/10.1007/s11162-018-9515-1

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-018-9515-1

Keywords

Search

Navigation