multi method qualitative research examples

• Search Menu
• Sign in through your institution
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Numismatics
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Social History
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Acquisition
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Religion
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Meta-Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Legal System - Costs and Funding
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Restitution
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business Strategy
• Business History
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Social Issues in Business and Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Systems
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Research Methodology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Management of Land and Natural Resources (Social Science)
• Natural Disasters (Environment)
• Pollution and Threats to the Environment (Social Science)
• Social Impact of Environmental Issues (Social Science)
• Sustainability
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• Ethnic Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Politics and Law
• Politics of Development
• Public Policy
• Public Administration
• Qualitative Political Methodology
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Disability Studies
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

11 Designing Multimethod Research

Albert Hunter is currently professor of sociology at Northwestern University and Director of the Urban Studies Program where he is affiliated with the Institute for Policy Research and the Transportation Center. His degrees are from Cornell (BA) and the University of Chicago (MA,PhD). He has previously taught at the University of Chicago, Wesleyan University, the University of Rochester and had visiting appointments at The London School of Economics, Yale University, the University of Edinburgh, and the University of Paris VIII-St. Denis. He is the author of numerous articles and books and most of his research is multimethod including two major multi-institutional multi-site collective projects conducted at the Center for Urban Affairs and Policy research – The Reactions to Crime Project, and the Changing Relations Project on Ethnic Diversity. His books include Symbolic Communities: The Persistence and Change of Chicago’s Local Communities (1974) two books on “multimethod research” (with John Brewer), Multimethod Research: A Synthesis of Styles (1989) and Foundations of Multimethod Research (2006); a book on The Rhetoric of Social Research: Understood and Believed (1990), and most recently a book on “civil society” titled Pragmatic Liberalism: Constructing a Civil Society (with Carl Milofsky, 2007). He continues research on the “symbolic ecology” of cities including a comparative study of neighborhood response to gangs, and a comparative study of elite and poor suburbs of Chicago. He is also engaged in a ongoing comparative study of civil society in the US and the UK. He has served as Editor of Urban Affairs Review, Chair of the Community Section of the ASA, and Chair of the Plan Commission of the City of Evanston.

John D. Brewer is Professor of Sociology Emeritus at Trinity College in Hartford Connecticut. Previously he taught at the University of California, Los Angeles, York University in Toronto, Canada, and Wesleyan University in Middletown, Connecticut and has held a visiting appointment at the University of Connecticut School of Law. He received his BA,. MA, and Ph.D degrees from the University of Chicago, where he was a Woodrow Wilson Fellow. He has studied formal organizations, written about problems in organization research and theory, and served in the elected position of Secretary to the American Sociological Association’s Section on Organizations and Occupations. He is co-author (with Albert Hunter, SAGE, 1989) Multimethod Research: A Synthesis of Styles, and co-author (with Albert Hunter, SAGE, 2006) Foundations of Multmethod Research: Synthesizing Styles.

• Published: 19 January 2016
• Cite Icon Cite
• Permissions Icon Permissions

The chapter begins with the “spirit of multimethod research,” which refers to an openness to serendipity and humble recognition that all methods have strengths and weaknesses and that by combining different methods one has compensating strengths leading to more credible results in the face of a series of skeptics’ questions. “Multimethods” involves combining any different methods while” mixed methods” more specifically focuses on combining qualitative and quantitative methods. Typologies of multimethod research from the literature based on characteristics and sequencing of methods themselves are briefly explored. The focus of the chapter is on the design of multimethod research in different subfields elucidating early historical exemplars and current developments. The selective and nonexhaustive subfields include words and deeds, rational actor, contextual effects, biomedical-social, evaluation, historical/comparative, and disaster research. Much of the chapter uses a positivist framework but concludes with a discussion of how multimethods are relevant for postpositive rhetorical and narrative frames.

Preamble: The Spirit of Multimethod Research

In addressing the question of research design we do not provide a “cookbook” or textbook normative approach to design where all is codified in a sequence of steps such that by doing A then B and combining with C one will produce D, an optimal outcome of research findings. We would rather stress the “art” of multimethod research—recognizing that good science exists to the degree it addresses a limited but profound set of questions. We have referred to these as the skeptics’ questions: an overarching concern with content, not a fetishsized concern with form—what some have labelled scientism .

We must not have an overreified notion of what science is but rather a more realistic empirical understanding of how science is actually done. As repeated studies in the sociology of science, or science studies suggest, it is better to pay attention to what practitioners say they really did, not what the methods textbooks say one should do ( Denzin, 1970 ). As Paul Lazarsfeld (1945) , an early practitioner of multimethod as well as survey research, stressed, “A good methodological investigation is not concerned with what a science should be: it tries to clarify what a science is and how it obtains its results” (p. vii; see also Jahoda, Lazarsfeld, & Zeisel, 1933/1971 ).

If one is to learn by studying then perhaps it is best to study exemplars, reports on research itself, not rules. This same spirit we suggest may carry over into the teaching and mentoring of science education. In the hard sciences (physics, chemistry, biology, zoology) one does not have courses in methods per se; rather one acquires methods through the older apprentice model of working in labs, and as a research assistant one learns methods by doing research, not reading about how one should do research. This is a distinction we will see shortly is akin to distinguishing between “saying versus doing” or “words versus deeds.” Methods are what one does—not to say there are no limits and boundaries, but these are few and easily communicated as ideals—reliability and reproducibility, openness, no fabrication or fraud, and so on. Beyond that, researchers should show their data, explain how they obtained their data, how they analyzed their data, and consider alternative interpretations, summarizations, and generalizations as well as limitations and threats to validity. The objective is to build an argument that is consistent (logical), corresponding (to the data), and convincing (believable).

The Skeptics’ Questions

Research methods, in short, must address a few very basic questions true of all scientific investigations, and these normative ideals ask that research be systematic, open, reproducible, reliable, valid, and believable after scrutiny with a healthy skepticism.

Elsewhere we have enumerated some of these basic skeptics’ questions as follows:

Absolute theoretical adequacy —to what degree does a theory explain the full set of empirical findings?

Relative theoretical adequacy —how well does a theory explain data compared to other possible theories?

Questions of methodological bias —how might the data to be explained be influenced by the particular method(s) used to obtain the data?

Questions of measurement and conceptualization—how well have the concepts of a theory been measured?

Questions of causal inference —how well have research methods addressed the cause-and-effect relationships of a theory’s hypotheses

Questions of generalizability —how generalizable are the findings of research given the methods used?

Questions of realism versus simplification—how much realism versus simplification do the research’s methods provide? ( Brewer & Hunter, 1989 , 2006 )

All research, whatever the method, should address these questions, and different methods may address one or another of the questions better than others. The multimethods we suggest may be particularly helpful in addressing more of them, thereby reducing the skepticism about the research results or, conversely, increasing their credibility.

The overarching purpose of our conceiving and writing Multimethod Research ( Brewer & Hunter, 1989 ) over some three decades ago was to overcome the continuing and at times acrimonious debates between practitioners of different techniques of research, especially but not exclusively between quantitative and qualitative approaches in the social sciences. It was a disciplinary concern as much as a narrow methodological one—with an overall objective of reasserting the centrality of the old Baconian generic goal of an eclectic but rigorous method of science itself.

The earlier debates were useful for self-reflection and continuing refinement of different techniques highlighting the limitations noted by critics as well as the strengths asserted by proponents often in hyperbolic rhetoric on both sides. Our goal was to discipline but not stifle the debates and to bring about a tolerance and mutual understanding of a unified endeavor—advancing the singular overarching method of science itself.

What Do We Mean By “Design”?

Design means (for us) a number of different things. Design may, on the one hand, be the outcome of prescriptive planning as in research proposals where specification and selection of such elements as units of investigation, universes, sampling frames, sampling techniques, measurement, data transformation, and modes of analysis are thoughtfully defined in detail. On the other hand design may be discerned in post hoc pattern recognition of what has been an unfolding, evolutionary, pragmatic adaptation in the research process. In short, design can consciously occur before starting the research, or the process can occur as the research unfolds in an organic, opportunistic, and serendipitous fashion. We try to encompass both of these meanings—the enacted and the emergent—in our discussion of design. We suggest that most research progresses as a combination of some preplanning coupled with judicious emergent decision-making as the research is carried out. We would encourage a spirit of a competently trained pragmatic openness to serendipity; in short, the idea of “design” in designing multimethod research is more of a Darwinian notion of evolution and adaptation rather than an overreliance on an authoritative “Intelligent Designer.”

The evolution of multimethod designs, as with all methods, can occur at two levels—at the micro level with the opportunistic unfolding and combining of specific techniques of data collection and analysis present in a single research project, and at the macro level of a discipline as a whole as Mario Small (2011) demonstrated in his comprehensive review of multimethod research in sociology as he traced the unfolding natural history of specific combinations of different types of methods over recent decades. The multimethod spirit is one that is open to new, innovative, and at times unanticipated techniques.

Prepared Contingency

One way individuals learn different methods is through the study of prior exemplars and also through an apprenticeship in mentoring relationships during the actual conduct of research itself. An exposure to, understanding of, and capacity to use different methods is an ongoing learning experience throughout one’s career, a learning process greatly aided by an openness to a multimethod perspective. Some individuals may develop varying degrees of mastery of a variety of techniques, say for data collection or data analysis, while others may become highly specialized in this or that singular technique. The former person may craft a multimethod piece of research on his or her own, while the latter may still participate in multimethod research through a division of labor within a collective research project. Such was the case for example with the Reactions to Crime Project carried out by Northwestern University’s Center for Urban Affairs and Policy Research in the 1970s and 1980s ( Hunter & Maxfield, 1980 ). But, to work, the collective multimethod research project must have a collective spirit that recognizes the strengths and limitations of the different methods, a spirit that does not just tolerate but emphasizes and appreciates the need for rigor and quality in the application of different methods in the overall scientific enterprise. In sum, multimethod research does not disparage monomethod research and in fact encourages more reflexive and refined sophistication in the mastering of each method with the goal of enacting the highest quality competent research. At the same time, the multimethod spirit encourages a “design for serendipity” recognizing that opportunistic and creative insights come to the well-prepared and reflective mind.

Methods Are What Researchers Do

Case studies in the ethnography of science have shown repeatedly that actual research is different than any ideal, and, we suggest, this is true of multimethod research as well. The oft told story of W. I. Thomas (1923) discovering a packet of letters thrown out in an alley and seeing them as a potential data source for studying Polish peasants who had immigrated to America is perhaps apocryphal but underscores the need for openness to diverse data and methods. We are not interested in prescription or preaching with respect to multimethod research. It is an approach we highly recommend when and where appropriate. Monomethod research is at times under certain conditions and contexts the way to go. We definitely prefer well-executed monomethod research to less competent multimethod research. The use of multimethods alone does not exonerate nor should it be a cloak for incompetence.

Multimethod Research Design

Multimethod research may be broadly defined as the practice of employing two or more different methods or styles of research within the same study or research program rather than confining the research to the use of a single method ( Brewer & Hunter, 1989 , 2006 ). Unlike mixed method research, it is not restricted to combining qualitative and quantitative methods but rather is open to the full variety of possible methodological combinations.

As there are numerous methods available to social researchers, and a variety of ways in which these methods can be combined, the topic of multimethod research design involves foremost asking (a) which methods are combined with which other methods and (b) how are the different methods deployed and implemented in relation to one another in the research process.

Typologies of Mixing

Prior commentators on multimethod research have produced a number of typologies reflecting the “what” and “how” of combining or mixing different distinct methods. We will briefly explore a few of these as developed by others. We note that most commonly in the literature “mixed methods” is used to refer to mixing quantitative and a qualitative method, while “multimethod” more broadly refers to mixing of two or more methods—regardless of whether they are qualitative or quantitative. In short, mixed method is a subset of multimethod.

One of the more formal and systematic typologies is that by Janice Morse (2003) who began with the basic qualitative quantitative distinction that she equated with a deductive and inductive drive, respectively. She then paired each with either a second quantitative or qualitative method and then subdivided each of those four cases in turn into a simultaneous or sequential timing of the deployment of the methods. The result was an eight-fold typology that she then analyzed in more depth as to their contribution to theory.

Johnson and Turner (2003) derived a typology based on the principal method of data collection by first distinguishing among the methods of questionnaires, interviews, focus groups, tests, observation, and secondary data. They then considered that each of these methods may in and of itself use mixed data collection strategies that could include two quantitative, two qualitative, or one qualitative and one quantitative. The first two “mixes” would be multimethod, while the third would be defined as an example of “mixed methods.” Finally, they suggested the six different methods themselves may be combined to create a “continuum” of a variety of methods and varied qualitative/quantitative combinations.

We will look at two other typologies that are more inductive in their derivation by Jerry Jacobs (2005) and Mario Small (2011) , respectively. Let us begin by examining research published recently by Jerry Jacobs in the American Sociological Review , the official journal of the American Sociological Association. This seems a promising way to lay out the basic dimensions of multimethod and mixed methods research design as now practiced in the discipline with which we are most familiar. Jacobs has helpfully published a brief survey of the papers published between 2003 and 2005, a period when he was editor of the American Sociological Review . He reported that a quarter of the papers he accepted for publication (17 out of 65) involved multiple method research, by which he means studies that “draw on data from more than one source and present more than one type of analysis” ( Jacobs, 2005 ).

Jacobs (2005) found several different combinations of methods in this group of studies. He noted that these multiple method studies “often but not always combined quantitative and qualitative data.” First, statistical surveys were combined with qualitative interviews. The interviews might either precede the survey to refine its questions or be conducted later to increase understanding and deepen interpretation of the survey data. Second, there were multiple quantitative approaches: for example, a field experiment followed later by a survey of the same subjects to test and compare with the experimental results; analysis of survey data from 30 countries to bolster a hypothesis derived from aggregate data from those countries. Finally, historical analyses employed qualitative and quantitative analyses of historical documents and archival data, and quantitative analysis of survey data were available as well as qualitative interview data from still-living participants. These different sorts of data were sometimes used concurrently to provide a more complete picture and sometimes in sequence to form and test hypotheses.

To put Jacobs’s (2005) report in perspective we have examined two other samples of the American Sociological Review , one roughly 20 years earlier (1984 and 1985) and the second from 2011 to the present. A striking contrast in these different time periods is that while Jacobs identified 17 multiple method studies in the 2003–2005 period, there were 10 such studies during 1984 and 1985, and 32 during 2011 and 2012. Clearly there is a linear trend of increasing publication of multimethod studies within sociology.

Mario Small (2011) in an Annual Review of Sociology article similarly developed an inductive typology based on an extensive review of the sociological literature. He too first noted the burgeoning number of studies that have come to use multiple methods in recent decades. He also noted that the usual combined qualitative/quantitative meaning of the category is unnecessarily restrictive as different qualitative methods and different quantitative methods may also be combined, and that furthermore the distinction of what constitutes quantitative and qualitative is itself a “fuzzy” distinction.

He began his typology by distinguishing between data collection and data analysis, recognizing, for example, that one could have a qualitative analysis of quantitative data and conversely a quantitative analysis of qualitative data. He then proceeded to elucidate examples of these varying types. He also considered the epistemology of multimethod research, which he noted is heavily rooted in pragmatism, and the primary motivations of researchers, which are either confirmation or complementarity. He further explored questions of sequencing and nesting in data collection and cross-over and integration in data analysis. Finally he noted the enduring questions of commensurability and specialization among methods. As he concluded:

the challenges of mixed methods research reflect those of sociology writ large, a discipline whose core methodological pluralism has produced, over its history, periods of conflict and cooperation, but few of lasting resolution. . . . Mixed method projects provide both the challenge and the opportunity for researchers to resolve some of the ambiguities that result from pluralism. ( Small, 2011 , p. 79)

An Inductive Typology of Fruitful Substantive Arenas for Multimethod Social Science Research

The previous discussion of multimethod research highlights typologies of different combinations of methods that are derived from characteristics of the methods themselves—such as the basic quantitative qualitative distinction, or relative significance as to which method has primacy in the research versus which plays a supportive role, or the mere sequencing and timing as to which comes first and which follows. These typologies are valuable for giving some coherence and organization to the rapidly expanding accumulation of multimethod studies. One must be careful, however, to not overreify these typologies and potentially advance them as normatively prescriptive strategies for conducting multimethod research. All research, multimethod included, is, we contend, a combination of science and art, design and serendipity, thoughtful planning and pragmatic opportunism.

We use here a more inductive approach following the lead of Jacobs (2005) and Small (2011) and develop a typology of multimethod research that begins not from first principles about methods but from different generalized types of substantive questions that have already been explored in the literature using multimethod research. The types of substantive questions are varied, and some that we identify, like that exploring the differences between words and deeds, have a long history in the literature, while others, such as combing biomedical markers and interview data, are relatively recent in origin. We present this typology as neither exhaustive nor necessarily mutually exclusive and do so in the hopes of presenting categories that will be useful in organizing and bringing some order to the field. But, in contrast to most multimethod typologies, we offer these less as prescriptive normative guides and more as sensitizing exemplars. As one prepares to conduct a piece of research and thinks about how to possibly incorporate a multimethod style, one can ask substantively, not just methodologically: What kind of research question is this? In such a case, one is being more true to the old adage that research questions should be driven by theoretical substance not methodological proclivities. Substance should dictate methods, not the reverse. This is an admonition shared widely across science in general, not just the social sciences.

For example, a recent article in Science ( Joppa et al., 2013 ) reporting a survey study of 400 biologists using canned modeling programs in the substantive area of “species distribution modeling” quotes one scientist: “The research question and the data should be king, with an approach being selected on the basis that it is appropriate to both the research question and the data rather than the research question and the data being selected to fit the approach which a person knows how to use.” While another biologist echoed a multimethod thesis: “We don’t need fancier software, we need people who understand ecology and the importance of multiple types of data. . . The key is the ability to think in ecological terms”(p. 815).

Substantive Arenas

The following are a few of these types of substantive arenas often using multimethod research that we have begun to explore and that we briefly elaborate in the following discussion:

Words and Deeds

Rational action, contextual effects, biomedical social research, evaluation research.

Historical/Comparative

Disaster Research

Words are what people say and are windows into what they think, while deeds are what people do—their behavior. We are most interested in the latter, because that is what has direct consequences for us and that is what we are most interested in anticipating so we may adapt our own behavior accordingly.

We acknowledge that words may be thought of as a form of behavior, so in this light we are comparing two different forms of behavior with one another. The words, which give the participants’ “meanings,” are closer to thoughts and ideas, or, using a more social psychological vocabulary, they are “attitudes” and “opinions” (things in people’s heads) and so “words and deeds” are linked to the “attitude behavior” distinction. Attitudes are seen as “predispositions” that may or may not match “dispositions” or behavior itself.

In comparing words and deeds, the approach is most often to compare prior words to subsequent deeds and to ascertain the degree of agreement or divergence between the two. The classic case of “cognitive dissonance” is but one type of divergence between word and deed.

Words as data can be gathered in a variety ways that often require direct interaction: they are “voiced” and may take the form of responses to questions on a survey instrument, recorded phone conversations, or written documents.

By contrast, deeds as data ideally require direct observation of behavior and so often require different methods than those used to gather “words.” Hence the multimethod approach is routinely found in such studies where two different data sets are compared.

Richard Lapiere’s (1934) work is generally cited as being among the earliest multimethod studies of the relationship between words and deeds or, in his words, attitudes and actions. In 1930 Lapiere conducted a field experiment consisting of 251 visits across the United States to assess the responses of hotel and restaurant proprietors to a young Chinese couple’s requests for service. Only once was the couple turned away. He followed up this action research component six months later with an attitude component, a mail survey asking the proprietors whether or not they would “accept members of the Chinese race” at their establishments. Contrary to their prior actions, very few of the proprietors expressed an accepting attitude ( Lapiere, 1934 ).

Lapiere’s work has proven to be exemplary in several ways. First, it demonstrated that individuals’ actions and attitudes could not necessarily be inferred directly from one another. Second, it demonstrated a methodology for measuring attitudes and actions separately and then empirically determining their interrelationships. Finally, the study has inspired and informed later research, particularly in the study of discrimination but also in the wider theoretical area of attitude–behavior relations (e.g., Ajzen & Fishbein, 1977 ; Ajzen, 2005). Pager and Quillian (2005) provided an overview of this more recent and more broadly theoretical work in the introduction to their own research, which is evocatively titled “Walking the Walk? What Employers Say Versus What They Do.” It is a sophisticated contemporary analogue of Lapiere’s study. They investigated employment discrimination based on the applicants’ race and criminal record.

The study was conducted in two stages. In the first “employers’ responses to job applicants were measured in real employment settings using an experimental audit methodology. . . . The preferences of employers were measured based on the number of call-backs to each of the applicants” ( Pager & Quillian, 2005 , p. 362). “The findings of the audit showed large and significant effects of both race and criminal record on employment opportunities” (p. 362). The second stage of the study was a telephone survey carried out several months later in which employers were asked to express their hiring preferences for hiring offenders. They found that employers who in the survey expressed “a greater likelihood of hiring ex-offenders were no more likely to do so in practice, and that large differences in racial hiring preferences were found in the experiment, but not in the survey” (p. 362).

Pager and Quillian (2005) concluded that “these comparisons suggest that employer surveys—even those using an experimental design to control for social desirability bias—may be insufficient for drawing conclusions about the actual level of hiring against stigmatized groups” (p. 355).

A central, and some say defining, assumption in classical economic theory is that individuals act rationally in pursing their own self-interest. From an economic perspective, words may be seen as a verbal expression of preferences, as illustrated in surveys of marketing research such as when people are asked which brand of detergent they prefer. The ultimate behavior or “deed” one is interested in, however, is: “Which detergent do they buy?” “What do they spend their money on?” According to Paul Samuelson (1948) , who developed the theory of revealed preferences, preferences are revealed in the behavior of how money is spent. Economists often eschew any other indicator of preferences other than that. In short, revealed preference is

an economic theory of consumption behavior which asserts that the best way to measure consumer preferences is to observe their purchasing behavior. Revealed preference theory works on the assumption that consumers have considered a set of alternatives before making a purchasing decision. Thus, given that a consumer chooses one option out of the set, this option must be the preferred option. ( Investopedia, 2014 )

One of the critics of the revealed preference theory states that “Instead of replacing ‘metaphysical’ terms such as ‘desire’ and ‘purpose’ ” they “used it to legitimize them by giving them operational definitions” ( Wade, 2004 , p. 958). Thus in psychology, as in economics, the initial, quite radical operationalist ideas eventually came to serve as little more than a “reassurance fetish” for mainstream methodological practice.

A common critique of this rational economic approach is that it is often based on sophisticated modeling relying on monomethod data points of either an experimental type, as in “prisoners’ dilemma” research, or monetized aggregated market data. In a circular self-referential manner, monetary expenditure is the “revealed preference” of economic researchers about how best to measure rationality itself. When the same propositions are explored with multimethods, different types of data, and other measures, researchers are often able to more realistically address the simplifying assumptions of such models and provide more robust analyses. For example, the seminal work of Amos Tversky and Daniel Kahneman (1974 , 2011) speaks to a variety of different methods that suggest that economic decision-making may be other than rational. Their research begins from a psychological experimental approach but is elaborated to include in-depth interviews, aggregate market data, and historical examples. Their results suggest that individuals operate in their decision-making by the use of heuristics or rules of thumb that are often less than optimally rational in economic terms or payoffs.

Mark Granovetter’s (1985) work on the socially embedded nature of economic transactions again relied on different data and methods that led to a reconsideration of the “rational individual” assumptions of traditional economics. The study of social networks required the addition of different methods and types of data beyond that provided by either aggregated economic or individualistic experimental methods. Studying the impact of people’s embeddedness in social contexts and social networks has become both a conceptual and a methodological addition to prior research on individualized and aggregated data methods and has been used to study everything from obesity to getting a job or a loan from a bank ( Granovetter, 1985 ; Uzzi, 1999 ; Cohen-Cole & Fletcher, 2008 ).

All social behavior occurs in a specific context or situation, and observed patterns of behavior may be generalized from one context to another, leading to more generalized understanding, or may be seen to be limited to specific contexts. The idea of seeing individuals’ behaviors as related to their context is exemplified by W. I. Thomas’s (1923) older notion of “the definition of the situation” and in more current discussions of structural effects and in empirical analyses employing hierarchical linear modeling. In short, individuals’ behaviors may not be simply a product of their individual attributes and attitudes but may be “influenced” by the fact that they are embedded in socially specific time and space contexts such as networks of friends and kin, institutions of school, church, and work, or small and large communities from neighborhoods to nations. A current example of such a research area is the discussion of “neighborhood effects” on such specific outcomes as academic achievement, mental and physical health, delinquency, and employment ( Sampson, Morenoff, & Gannon-Rowley, 2002 ).

The study of such contextual effects often requires an explicit multimethod perspective in that one is simultaneously gathering data on two different types of units—individuals and the collective context in which they are embedded. Researchers attempt to assess the degree to which individual attributes versus collective characteristics (such as “race” or “class,” for example) have similar or different effects on people’s behavior. A typical structural effects question would ask: “Do poor people in rich neighborhoods behave differently than poor people in poor neighborhoods?”

Some subsets or types of contexts whose effects have been explored include the following:

Geo-political units that may vary in scale from neighborhood to cities and metro areas to states/provinces and nations. (See the voluminous research on neighborhood effects summarized in Sampson et al., 2002 ).

Organizational/institutional contexts such as schools, firms, agencies ( Scott & Meyer, 1994 ).

Small group/social network/social tie contexts such as friendship, coworkers, family, and kin.

This is what theorist Georg Simmel (1971) referred to as “social circles” and what Gary Fine (2012) documented in his numerous ethnographic studies of what he calls “tiny publics,” ranging from boys’ Little League teams to mushroom hunters to restaurant kitchens.

The idea of contextual effects is that one is collecting data at different scales, levels, or units of analysis and these different types of data require different methods of data collection. For example, at the collective level, there is census or economic data on geopolitical units or archival data on organizations. These collective levels of data are then coupled with individual-level data obtained through surveys, interviews, observation, or records.

Examples of structural effects are seen in the early work of reference group researchers such as Robert Merton, Samuel Stouffer, and Herbert Hyman among others ( Merton & Lazersfeld, 1950 ). The classic case is from Stouffer et al.’s (1949) , The American Soldier , where they discovered that satisfaction with the promotion system in different branches of the military depended on the varying rates of promotion between the branches (Air Force vs, Marines). The counterintuitive outcome was that higher rates of promotion produced less satisfaction with the system (Air Force) with lower rates producing greater satisfaction (Marines). James A. Davis, Spaeth, and Huson (1961) later elaborated these into systematic comparisons of individual- and collective-level variables producing a “typology of effects:” (a) Additive effects in which both individual-level and collective-level variables operate to combine their effects; (b) interaction effects in which the effect of either the individual-level or the collective-level variable is different depending on the value of the other; and (c) spurious effects in which the effect of either the individual-level or the collective-level variable disappears once one has “controlled” for the other level. More recent techniques of hierarchical linear modeling and their variants similarly attempt to disentangle these multimethod data to uncover contextual effects ( Raudenbush & Bryk, 2001 ).

There is an increasing interdisciplinary perspective on healthcare-related issues that has led to an explosion of multimethod research using combined methods of collecting biomedical data, social, and psychological data of individuals. From blood samples to mouth swabs to more detailed use of medical charts and the numerous “test results” they may contain, such data can be coupled to interview, survey, or field observational data on individuals. Such research is often motivated to finding causes or the etiology of disease or “cures” that may mitigate the consequences. Beyond causal analyses and questions of social causes of “illness,” or the reverse causation of medical conditions affecting social and psychological outcomes, lies the measurement and conceptual question of validity of medical conditions and symptoms. Such assessments may use multimethods research to actually identify and diagnose or define a syndrome or medical condition, for example posttraumatic stress disorder.

Such research has become especially significant with increasing use of genetic markers and some researchers attempts, perhaps too often naively, to link specific genes to specific behaviors—thereby short-circuiting what are likely much more complex processes at work. One is reminded of the early attempts of Cesare Lombroso to see a causal link between skull sizes and shapes and William Sheldon’s “biological body types” of ectomorphs, mesomorphs, and endomorphs and predilections to engage in criminal/delinquent behavior.

The multimethod perspective of this research extends beyond different types of data and data collection to include various analytic frameworks and strategies. Often relying on very large data sets of tens of thousands of cases, the analytic strategies point to “statistically significant” correlations between genetic markers and health or behavioral outcomes, even when the amount of variance explained is miniscule by comparison to the vast amount of unexplained variance still left to be explained by nongenetic factors. Such results are too readily overly interpreted from a “genetic determinist” argument by some, even when the authors of the research caution against such interpretations.

What a multimethod perspective brings to such questions is a need for other types of data and analyses that would fill in the black box of mechanisms and multifaceted chains of contingent causation that might link the genetic marker to some specific outcome. The perennial question of correlation and causation becomes entangled in the nature versus nurture debate. Jeremy Freese has perhaps been one of the leading sociological spokespersons calling for more cautionary conclusions based on multimethod strategies, especially when they are advanced in consideration of policy recommendations and programs. A recent research article in Science reporting a link between specific genetic markers and “educational outcomes” cautioned “the main lesson of the research, experts say, should be that attributing cultural and socioeconomic traits to genes is a dicey enterprise. “ ‘If there is a policy implication, it’s that there’s even more reason to be skeptical of genetic determinism’, says Jeremy Freese of Northwestern University in Evanston Illinois” ( Saey, 2013 ).

How does multimethod research help address genetic/social issues? Research aimed at relating genetics and social behavior is, of course, inherently multimethod in the broadest sense of the term because it employs methods derived from biology to study the genetic components and social science methods for the social components. Moreover, there are multiple biological research techniques and designs (e.g., blood vs. saliva DNA tests; twin, sibling, adoption, and molecular genetic studies). However, as sociological research interest has grown in the past two decades, the area has also become more multimethod in the general sense of employing varieties of social science methods in conjunction with one another and also as well with varieties of genetic data.

Survey research is the primary data collecting method currently employed in many social genetic studies. It allows for large sample studies that may be put to wide use. For example, the National Longitudinal Study of Adolescent Health was started through the University of North Carolina–Chapel Hill by Bearman, Udry, and Harris

to capture as much information as possible about the social circumstance, friendship networks and family conditions of 21,000 teenagers in 132 schools, from grades 7 through 12. The survey included a disproportionate number of twins, fraternal and identical, full and half siblings, and adopted children. . . Follow-up interviews were conducted a year later. Then, for the third wave. . . (in 2002), 2500 siblings were asked for DNA samples (via cheek swabs). ( Shea, 2009 , p. B6)

A fourth wave collected DNA with saliva samples for all respondents.

Data from surveys such as this have been employed to investigate the possible main and interaction effects of genetic characteristics on social behavior and are often supplemented by additional methods to improve upon the surveys. Guo, Elder, Cai, and Hamilton (2009) , for example, tested the hypothesis that “the genetic contribution to adolescent drinking depends on the drinking behavior of their friends” (p. 355) with data from a sample of clusters of siblings and their friends from the larger National Longitudinal Study of Adolescent Health. They reinterviewed respondents at home, beyond the survey, to construct the clusters of friends. A special problem in determining friends’ drinking behavior is that there may be a projection effect when respondents themselves are asked to report their friends’ behavior. To avoid this possible bias, the researchers identified and then directly interviewed the respondents’ friends.

In another study, Guo et al. (2009) addressed the issue of selection bias in studying the influence of friends on behavior in their study of peer impacts on attitudes and drinking behavior. The study was a survey of a sample of roommates in a large university. The issue was that adolescents in particular tend to choose as friends people similar to themselves ( Guo et al., 2009 ). To address this issue, the researchers introduced an experimental component to their study by separately studying students who had been randomly assigned to roommates by the university housing office, a procedure that in their words “avoids the confounding effects of residential choice” (p. 4). The authors report that “This is the first time that gene-environmental interactions have been investigated using randomly assigned environmental influences” (p. 6). They conclude that the “genetic contribution” to adolescent drinking depends on the drinking behavior of their friends. That is, friend behavior moderates the genetic contribution to alcohol use.

Multimethod research has found a ready application in the burgeoning area of evaluation research. Evaluation research emerged explicitly from a policy-oriented set of questions about the effectiveness of programs designed to deal with specific social problems ( Rossi, Lipsey, & Freeman, 2004 ). The call for evaluation of such programs has been heightened under the current antitax political context of doing away with programs that cannot demonstrate their effectiveness and more stringently their efficiency in a cost/benefit calculus. Evaluation research is therefore explicitly geared to providing data and analysis that can inform decision-making about supporting or negating specific programs. As a rubric in rational decision-making quantification is seen to be particularly valued as able to provide objective, universalistic, summary, comparative criteria by which to judge such programs. Coupled with the “natural” quantification provided by a “money” calculus, costs and benefits can be rendered commensurable and evaluated accordingly ( Espeland & Stevens, 1998 ).

An additional more refined consideration beyond assessing efficacy and efficiency is that of equity. Here the question becomes: “Who bears the cost, and who reaps the benefit?” The answer to this question may require different kinds of data and still different methods of research to distinguish how the costs and benefits are differentially distributed.

This logic of evaluation research has had a number of consequences. One is to focus on outcomes, more explicitly outcomes that are quantifiable, and even more explicitly quantifiable outcomes that can be given a dollar value. Second, the most valued research design is the true experiment with pre- and posttest measurement of outcomes (or “dependent” variables) coupled with random assignment to control and experimental groups that “do” or “do not” get the treatment of the program (the independent variable). These “ideal” research designs are often difficult to implement for any number of cost, time, practical, and ethical reasons, and this has led to a variety of quasi-experimental designs that relax one or another of the criteria ( Campbell & Stanley, 1963 , Cook & Campbell, 1979 ; Cook, 2002 ; Shadish, Cook, & Campbell, 2002 ). The actual implementation and process of carrying out the program are often left relatively unexplored in such designs. This is the point at which more qualitative and ethnographic methods are brought to bear in a fuller understanding of the evaluation of programs ( Cook, Shadish, & Wong, 2008 ). Such qualitative research (e.g., in-depth interviews and field observation) may provide an understanding of the “causal process,” the detailed concatenated sequential “mechanisms” by which certain outcomes were obtained. Furthermore, such qualitative research may heighten the possibility of serendipitous discovery of unanticipated outcomes not previously considered, especially given a specific focus on monetized quantitative measurable outcomes.

As a result of the these considerations, evaluation research has increasingly come to incorporate a multimethod perspective—combining a variety of methods that more robustly evaluate programs in terms of process and outcomes and varieties of methods both within and between qualitative and quantitative methods. The “approximations” to what some see as the ideal of randomized experimental designs in “quasi-experiments come with appropriate caveats as to their limitations ( Cook & Campbell, 1979 ), and as Cook et al. (2008) later note, the use of different methods in combination have utility for answering a variety of different questions in evaluation research.

A number of major experimental social policy programs have been evaluated using multimethod research. For example researchers at the Seattle Income Maintenance Experiment and the Denver Income Maintenance Experiment randomly assigned poor people to various regimes and levels of welfare expenditure and assessed through multiple methods, such as surveys, interviews, diaries, and various specific outcomes in the life and welfare of recipients ( US Department of Health and Human Services, 1983 ). More recently, Move to Opportunity Programs have been evaluated with a variety of methods to assess the effects of moving poor families out of concentrated inner-city public housing to better neighborhoods and assessing various outcomes such as health, employment and educational performance of children ( Briggs, Popkin, & Goering, 2010 ).

Greene, Caracelli, and Graham (1989) developed a systematic rationale and prescription for the use of multimethods in evaluation research for five distinct purposes: (a) triangulation for convergence and corroboration, (b) complementarity for elaboration and enhancement, (c) development for helping to inform one method by another, (d) initiation for discovering paradox and contradiction, and (e) expansion for extending breadth and range. These were assessed empirically by looking at 53 evaluation studies and resulted in recommendations for ideal mixing of particular combinations of qualitative and quantitative methods. The combinations took into account their similarity and difference in phenomena studied, paradigm used, primacy of one over the other, integration or independence, and timing. One of the key distinctions noted was between product (outcome)-oriented evaluation and process-oriented evaluation and the finding that those that evaluated both product and process constituted the majority of mixed method studies.

Comparative Historical Causation, Trends, and Character Types

Our initial formulation of the multimethod perspective in multimethod research ( Brewer & Hunter, 1989 ) clearly called for a broad conception of multimethods beyond the narrow question of measurement as in Campbell and Fiske’s (1959) landmark article on triangulation. Multimethods we suggested should apply to all stages of the research to include for, example, multiple theoretical perspectives, and also different analytical strategies.

Historical causation

The question of varying analytical strategies has emerged in a number of arenas of social science but perhaps no more clearly than in the field of comparative/historical analysis. This becomes especially apparent in addressing the question of causation and its complexities ( Abbott, 1990 ). Should one pursue a variable-based statistical linear approach of a large numbers of cases isolating the impact of causal variables on selected outcomes, or should one pursue a small- N comparative analysis based on detailed exposition of selected cases using a logic of similarity/difference and presence/absence and being sensitive to sequence and timing in trying to isolate necessary and sufficient causes ( Skocpol & Somers, 1980 ; Skocpol, 1984 )? This is where Andrew Abbott’s (1997) reflections on the early Chicago School stressing “contingency and context” suggest the need for multimethod strategies. Abbott distinguished, for example, between those comparative historians who rely on variable analyses of large- N studies of nation/states statistically parsing out the effects of independent variables on selected dependent variables of interest versus those who rely on a small- N more comparative case-based analysis of specific events that have unfolded at varying points in time.

Among comparative historical analysts, Charles Ragin (1987) has developed a synthetic approach that reflects a multimethod merger of qualitative and quantitative methods. He uses the logic of Boolean algebra and “fuzzy sets” that produces a rigorous quantifiable assessment of varying combinations of qualitative characteristics that permit one to assess the contribution of necessary and sufficient causes for given outcomes. In short, he marries the rich detailed knowledge of qualitative data and small- N comparative logic with larger N data sets and develops quantitative statistical assessments of the likelihood of different combinations of contexts and conditions producing specified outcomes. Walton and Ragin (1990) have applied this technique to a study of which combinations of factors contributed to countries experiencing middle-class riots in response to International Monetary Fund policies. James Mahoney and Dietrich Rueschemeyer (2003) posed similar concerns in their landmark compilation Comparative Historical Analysis in the Social Sciences . Focused especially on questions of causation such as necessary and sufficient causes and temporal order, sequence, and duration in societal level events—such as revolutions and depressions—they were concerned with the different logics and modes of analysis in this emergent field. Specifically addressing the concerns that a multimethod strategy is designed to address, they noted

First of all, across the relevant disciplines, methodological disagreements of varying intensities have emerged between qualitative and quantitative approaches. In comparative social science this is represented by disputes between comparative historical researchers and cross-national statistical researchers who work with large numbers of cases. (p. 16)

However, they note that the advocates of large- N and small- N methods are more recently “acknowledging that there is a place for both in the cycle of research” (p. 17). For example, methodologists report on iterative research programs in which comparative historical research supplements the initial findings of statistical studies and vice versa. They quote King, Kehone, and Verba’s book Designing Social Inquiry (1994) to the effect that “the differences between the quantitative and qualitative traditions are only stylistic and are methodologically and substantively unimportant. All good research can be understood—and is indeed best understood—to derive from the same underlying logic of inference” (p. 4). Mahoney and Rueschemeyer (2003) conclude with an invocation to the spirit of multimethod research “comparative historical researchers do their best research when they remain open to the use of diverse methodologies and analytic tools” (p. 24).

Debates over causal analysis in comparative and historical social science are often predicated on documenting broad historical trends of social change such as urbanization, industrialization, bureaucratization, and secularization, all of which are often captured in theories of modernity ( Moore, 1958 , 1966 ). Such studies—both historical analyses of changes over time and comparative analyses of different societies considered to be at different stages in an historical linear development from primitive to modern—often rely on a variety of data and different methods to demonstrate these broad trends of social change. We are not debating here the “Western bias” of theories of modernity or their assumptions of linear societal development (see Wallerstein, 1974 , 2004 on world systems theory and others for such questions and critiques), nor are we entering the debate as to whether or not such trends can be variously couched as upward positive “progress” versus downward declining trajectories, or combinations in various cyclical rise and fall theories. Rather we are highlighting here the degree to which such studies often rely on a multimethod accumulation of diverse sets of data as the basis of their assertions. We will briefly look at two such empirically based landmark exemplars, David McClelland’s (1961) research reported in The Achieving Society and Robert Putnam’s (2001) more recent research on the decline of civil society reported in Bowling Alone .

As a social psychologist, McClelland (1961) became fascinated with the psychological trait that he identified as “the need for achievement” or nAch as it became known, which is an internalized motive of people to continue to strive for ever greater accomplishments. Drawing on the work of the sociologist Max Weber (1905) and his landmark study of the rise of modern capitalism documented in The Protestant Ethic and the Spirit of Capitalism , McClelland saw this need for achievement as a cultural variable that socialized individuals from different cultures into varying needs for achievement. In his experimental research, he documented the ways in which this need for achievement could be manipulated and heightened or depressed under varying conditions. But he extended his line of research far beyond the laboratory to include other data and other methods, such as analyses of artifacts of different cultures over time and content analysis of myths and stories from different cultures as to their emphasis on “achievement.” He correlated these with data from still other methods such as governmental statistics and showed, for example, a correlation between socialization into need for achievement as indicated by children’s stories and GDP as an outcome variable reinforcing one of Weber’s central tenants of the Spirit of Capitalism .

Robert Putnam (2001) begins his analysis of the decline of civil society in America by focusing on the questions of “trust” and “trustworthy,” which are seen to rest ultimately on the idea of keeping one’s word—meaning one’s subsequent behavior is in line with one’s prior verbal commitment. Trust is not just an attribute of individual dyadic relationships but may be generalized not only to people in general, including even strangers, but to institutions as well, as in people’s trust in government. The idea of civil society, that is, voluntary social relationships among individuals who have minimal social ties and may even be strangers to one another, is predicated on this element of a presumption of trust. To explore this realm of informal voluntary social ties, that is, civil society, which is distinct from primary ties of friends and kin and more formal ties governed by laws and the institutions of social control of the state ( Shils, 1957 ), requires a variety of different methods and data sources. This is clearly demonstrated by Putnam in his contemporary classic, Bowling Alone: The Collapse and Revival of American Community . In this summary volume, he argues there has been a decline and transformation of the civil sphere in American life. In building his argument he presents a variety of data derived from different methods, including survey research on participation in voluntary associations, attitudes toward government, and generalized trust of others, as well as historical and archival data such as organizational memberships, voting records, and content analyses of media. As he states in his methodological appendix in Bowling Alone, “My primary strategy, as explained in Chapter 1 , has been to triangulate among as many independent sources of evidence as possible, following the model of researchers into global warming” (p. 315). Overall he finds there is a decline in people’s expressed trust of major traditional institutions in society –from government, to religion to business—which is seen to be the result of broken promises in the conduct of these institutions due to incompetence and corruption that undermines their legitimacy. It is the sheer volume and variety of data derived from different methods that makes his argument and analysis so persuasive. This is in part because he is able to provide a litany of data that support his argument and also because of the variety and types of data through which he is able to demonstrate a subtle and nuanced analysis that shows exceptions, qualifications, and limitations that increase the credibility of his overall argument. In short, he comes across as an objective researcher rather than a true believer polemicist.

Character types

C. Wright Mills (1959) in The Sociological Imagination roots the understanding of social life in the intersection of biography and history. Combining these two elements has been a continuing concern in social science research and has often involved different data and methods—a multimethod perspective—to capture and tease out this critical intersection. We have already briefly looked at macro-level multimethod approaches to history in considering causation and trends. Cultural shifts over time have social consequences and vice versa, not the least in producing different emerging types of characters as a consequence.

With respect to biography, there is an old tradition within sociology of producing life histories or characterizations of a type of person: a composite summarization of data obtained from numerous discrete individuals. These include The Unadjusted Girl ( Thomas, 1923 ), The Jack-roller ( Shaw, 1930/2013 ), The Professional Thief ( Sutherland, 1937 ), The Hobo ( Anderson, 1923 ), and others. James Bennett (1981) wrote of this approach as a typification or characterization of what theorists like Robert Merton (1968) would call statuses and roles, that is, structural positions that are distinct from the more individualistic psychology of personality. The composite character was a fictionalized distillation and amalgamation, with data selectively organized and summarized, but it was all true in the sense that all of the facts were verified and valid or, as one commentator put it, “nothing in here was made-up” (Howard Becker in personal communication at Rhetoric of Research Conference Northwestern University, May 1980). This is one type of biography that may result from an analysis of numerous discrete individual biographies. By contrast, the focus on discrete individual biographies—the archetypal literary biography such as James Boswell’s The Life of Samuel Johnson (1998) or William Manchester’s (1978)   American Caesar: Douglas MacArthur, 1880–1964— are in effect case studies of a single individual while the composite biography constitutes the case itself (see Ragin and Becker’s What Is a Case? [1992]). Here, different methods and different types of data would be employed to produce the distinction between an individual personality and a social type or character.

The former would more likely involve archives and perhaps interviews of those who knew the focal person, while the latter would more likely draw on interviews and observations of numerous individuals representative of that type. An example of the latter is Martin Jankowski’s (1991) development of the summary gang character defined by defiant individualism (as opposed to a specific individual’s personality). He synthesized and distilled the characteristics of a typical gang member based on his decade-long participant observation of 29 gangs in three cities. A focus on character requires a multimethod strategy that incorporates cultural (e.g., ideas, beliefs) and structural (e.g., network) data about the milieu within which character types are formed. Context produces similarity that is distilled in similar character traits. Fitting individuals into history is a multimethod strategy of placing different types of characters into different historical trends and causal processes. The life histories are typified characters who are the outcomes of these processes.

Disaster research is one of the earliest sociological research areas to develop an explicitly multimethod approach. The impetus was the immediacy and transience of the events being studied.

On December 5, 1917, a munitions ship exploded in Halifax Harbor, inflicting severe damage on the city and injury to its inhabitants. Close to 2,000 people were killed and 9,000 more injured (in all about 22% of the city’s population). Samuel Henry Prince, who was then serving as an episcopal priest in Halifax, was both a participant and an observer of this explosion and its results. He later recorded his observations in his Columbia University doctoral dissertation, in what is generally regarded as the first sociological study of disaster ( Prince, 1920 ). Prince introduced his study with a methodological note that has resounded in the design of disaster research ever since:

The whole field. . . is a virgin subject in sociology. Knowledge will grow scientific only after the most faithful examination of many catastrophes. But it must be realized that the data of greatest value is left sometimes unrecorded and fades rapidly from the special memory. Investigation is needed immediately after the event. (p. 22)

In short, disaster studies to capture these fleeting memories and events must be based on quick response and firsthand research.

Following the disastrous experiences of World War II and the onset of the Cold War ( Form & Nosow, 1958 ), a number of quick-response disaster research programs were established at the National Opinion Research Center and elsewhere (see Barton, 1969 ). In 1963 the Disaster Research Center was established to more regularly monitor disasters with field and survey research. It has conducted over 600 such studies since its inception. Thus, in contrast to the 1917 Halifax explosion, the September 11, 2001, attack on New York City’s World Trade Center, while equally unexpected, had trained and experienced observers from the Disaster Research Center ready to respond almost immediately.

However, the routinization of disaster research readiness does not routinize the disastrous events themselves or the research into them. For example, one of the lead Disaster Research Center 9/11 researchers writes in her dissertation:

Direct observation of ongoing emergency activities was particularly valuable in this case precisely because the event itself seriously hampered record-keeping and clouded the memory of some emergency responders. With key decision makers often unable to recollect what was happening literally from one moment to the next, the ability to actually be present as decisions were made was critical for later efforts to reconstruct events. . . also facilitated rapport in later face-to-face interviews and added validity to information obtained in those conversations. To contend with the. . . research challenges. . . I draw upon the variety of data sources and employ multiple qualitative strategies—among them direct exploratory observation, primary data collection and analysis, as well as in-depth face-to-face and telephone interviews—to triangulate emergency response information. ( Kendra & Wachtendorf, 2003 , pp. 41–42)

More generally, Stallings (1997) has suggested that

The “challenge” of disaster research. . . is the lack of time between the occurrence of a disaster and the fielding of research: lack of time to develop theory and hypotheses; lack of time to develop research instruments; lack of time to decide which events are worthy of study. (p. 9)

This highly specialized area of research highlights another key factor that a multimethod perspective has proven be helpful in addressing the time element and a need for pragmatic quick response to an often unanticipated research situation and the need to address the multiple audiences and consumers of research.

One example of this pragmatic adaptation in a manmade disaster was the “experimental/survey” research of Bobo, Zubrinsky, Johnson, and Oliver (1994) in Los Angeles during the time of the Rodney King riot following the “not guilty” verdict of police officers captured on tape beating Mr. King. A survey had been launched days prior to the riot, which, among other things, had a series of items focused on race relations. After the riots the survey continued, and the researchers had a “natural experiment” or quasi-experimental design of before and after measures with the riot as the experimental variable. This piece of research beautifully represents two key aspects of the spirit of multimethod research—a pragmatic adaptive response to the serendipity of the research situation and a melded multimethod design combining survey and experimental logics.

The fact that “disaster research” often has multiple purposes for different audiences is another factor that makes multimethod research particularly attractive as a purposeful design. This was clearly the case, for example, in Kai Erikson’s (1976) research following the Buffalo Creek disaster in West Virginia reported in Everything in Its Path: Destruction of Community in the Buffalo Creek Flood . Erikson was initially approached by a law firm involved in the litigation of the case on behalf of residents whose homes and hamlets had been wiped out by the floodwaters released by a failed earthen dam built by a mining company. Erikson’s multimethod research relied predominantly on in-depth interviews with displaced residents and, perhaps most poignantly and empathetically, firsthand field observations of the everyday disrupted lives of residents of the hills and hollows of the area. Documenting the effects of the disaster for the legal audience may have been the initial impetus of the research, but other audiences included welfare services and various governmental agencies. Drawing on officially compiled statistics and archives, including the history of actions by the mining company, permitted Erikson’s multimethod research to both draw from and be of use to multiple sources and audiences. The simple profound conclusion for the sociology of disasters was to qualify the commonly stated proposition that disasters create a heightened sense of community in response to the shared fate. Erikson’s research concluded that if the disaster was sufficiently severe, as in the case of Buffalo Creek, it could in fact destroy community.

Another example of disaster research employing multimethods is Eric Klinenberg’s (2003) study of the Chicago summer heat wave of 1995, which resulted in the death of hundreds of Chicagoans, especially the elderly in certain poor minority neighborhoods. Again, relying on official statistics, in-depth interviews, and field observations Klinenberg concluded that the deaths were differentially distributed by neighborhood and that a critical factor was the variable density of networks of elderly found in two adjacent neighborhoods—dense in a Latino community that had relatively fewer deaths and relatively sparse in an adjacent Black community that had greater deaths. The relative isolation or connectedness affected whether or not others were there to check on how the elderly were faring in the heat and to offer assistance if needed. In a broadened autopsy of the disaster, Klinenberg’s research also explored the response and programs in place of various city agencies and departments, from police and fire to health and welfare, and concluded that they too bore some culpability.

One policy implication of the research has now been instituted by the mayor’s office: repeated public service announcements during subsequent heat waves for residents of the city to contact and check on the welfare of elderly kin and neighbors and for city employees such as police and fire and welfare workers to do the same. Klinenberg’s (2003) research has also been important for thinking about the quality of multimethod research and raising the question: Does the use of multimethods ipso facto produce better research results? Critiques of Klinenberg’s research by Duneier (2004 , 2006 ) and others generated some controversy and debate as to the adequacy and “thinness” of the field research he conducted. The implication is that time and effort spent on collecting other types of data might have been better spent in a greater immersion in the field providing more “thick description” rather than the more superficial observations reported. The lesson is, in short, that multimethods may be valuable but that simply having more methods and different data does not ensure greater validity if the methods themselves are inadequately employed.

Disaster research often attempts to distinguish between what is natural, as in a natural disaster, and what is the result of human agency and in the latter case to parse out “blame” and accountability This research gets readily translated into legal conceptual frameworks of culpability. Recent legal cases, for example, have held meteorologists “guilty” for having failed to predict and warn residents of impending storms or geologists for failure to warn of earthquakes ( Erikson, 1994 ; Freudenburg, Gramling, Laska, & Erikson, 2009 ).

The art of translating scientific and social science research results to multiple audiences was exemplified in the case of research on Love Canal, the toxic chemical site in Upstate New York where different publics, which included fellow scientists, evacuated former residents eager to learn if they could safely return to their homes and public officials and governmental decision-makers all eager for a simplified “yes or no” recommendation. Different data obtained through different methods, and perhaps equally significant, from different disciplines proved useful in communicating an understanding of the facts of the case and their implications for what people might decide to do ( Hunter, 1986 ).

Rhetoric, Narrative, and Postpositive Postmodern Multimethods: Turn, Turn, Turn

Science is a social process, and social science is social in both form and substance. The idea that methods of systematic research are employed to discover truth may be a central and noble teleological goal, but it is an oversimplified, reified, and idealized conception of what we actually do. We are not mere scientific automatons programmed to follow fixed procedures for probing reality, like the Mars rovers (Spirit, Opportunity, and Curiosity), but rather we are active social agents who talk to one another, read one another’s work, and debate and argue about the direction, meaning, and credibility of one another’s research and our assertions about the “truth” of what we have observed. This social constructionist perspective on science has a number of implications often defined as postmodern or postpositivist critique. Two of the more significant ones are the role of “the new rhetoric” and the “narrative” emphasis that sees science as a collection of stories. We briefly explore elements of rhetoric and narrative in turn.

The Rhetorical Turn

The social and rhetorical aspect of science is evidenced in a multimethod perspective with respect to the following:

The social goal of science is to convince others of the tentative truth of one’s assertions about questions posed. The tentativeness arises for a number of reasons, such as method limitations and historical and spatial limitations. These result in varying degrees of generalizability—from modest, limited claims of local truths to broader claims and heroic ones of universal truths. The convincing part entails the art of persuasion and being attentive to one’s audiences and the criteria by which they will judge the validity of one’s assertions.

Multimethods permit a variety of different questions to be posed about a given phenomenon of interest. For example, is the question/assertion one of descriptive fact or causal explanation (correlation vs. causation), one dealing with macro structural/cultural phenomena or the micro level of agency and intent, one focused on process and outcome, or one concerned with understanding alone or prescriptive policy implications?

Diverse others may have a variety of criteria by which they evaluate and are convinced of the truth value of one’s assertions. Multimethods allow one to address these different criteria. Different methods are varyingly adept at addressing each of the skeptic’s questions, and beyond these there may be other criteria in addition—some more general and some more specific that go to ontological assumptions: for example debates over the best mode of analysis in studying social change from “interrupted time series with switching replications” ( Cook & Campbell, 1979 ) to event history analysis ( Allison, 1984 ) to patterned sequences of nomothetic narrative ( Abbott 1995 ).

Multimethods address diverse criteria and answer a variety of skeptic’s questions, thereby becoming more convincing to more people and more types of people, as different types of texts and rhetorics are employed. These others may range from fellow academics concerned with the skeptics’ questions to policy planners concerned with cost/benefit analyses of effectiveness and efficiency to the broader public concerned with issues of equity and ethics. The variety of rhetorical arguments contained in a multimethod perspective may range from quantitative survey results to numbers to a sample of personal accounts to historical archives.

The rediscovery of rhetoric as a central component of scientific argument and social research is based on this social aspect of science. From these rhetorical perspective methods and the varying rules, procedures, techniques, and norms of science we derive tools for building an argument about the link between observations and data about the world and one’s ideas or theories. Scientific method narrowly construed is concerned about “persuading skeptics” by satisfying them that one has followed these rules in the research. According to The British Strong Programme ( Bloor, 1976/1991 ; Mulkay 1979 ). science is, from this perspective, subsumed under rhetoric as a method or “tool” of argumentation and persuasion. The “strong” assertion is that it is all rhetoric through and through. Hunter (1990) in The Rhetoric of Social Research sees social science research as a set of rhetorical relationships among the researcher, the subjects, and the audience tied together by texts or research reports.

Not only are there “conventions” of science that must be followed to be convincing, but the reports of research, from published articles to PowerPoint presentations, must be similarly stylized to persuade specific audiences. The multimethod approach, which is a “synthesis of styles” of research ( Brewer and Hunter, 1989 ), lends itself to being more persuasive (believable) not merely due to a greater number of methods but to their addressing different criteria of credibility. As poet John Donne noted about An Obscure Writer

Philo, with twelve years study hath been grieved This is to be understood as when will he be believed. ( Donne, 1986 , p. 210)

Margarete Sandelowski (2003) in Tables or Tableaux? The Challenges of Writing and Reading Mixed Methods Studies distinguished between method and methodology and also paradigms and techniques and added, “A major—and arguably the most important—criterion in evaluating the merits of a study lies in the ability of writers to persuade readers of its merits in their research reports (p. 321). Invoking Fish (1980), she directly addressed the rhetoric of multimethods by noting that qualitative and quantitative belong to different interpretive communities. She considered this an “aesthetic criteria, including the sense of rightness and comfort readers experience that is crucial to the judgment they make about the validity of a study.” She therefore explores the need for” mixed media for mixed methods for mixed audiences” (p. 335).

The issue of different genres of research reports are seen in two chapters from Hunter’s edited volume (1990)— Joseph Gusfield’s (1990) comparison of Liebow’s (1967)   Tally’s Corner based on qualitative research versus Blau and Duncan’s (1978) quantitative analysis of the American Occupational Structure —and in Marjorie DeVault’s (1990) comparison of Kanter’s (1993)   Men and Women of the Corporation versus Krieger’s (1983)   The Mirror Dance , both dealing with women’s roles and identities bur the former with a traditional organizational analysis and the latter with a more feminist perspective. To paraphrase DeVault: same subject, different methods. Quoting Wolfer (1991), Sandelowski (2003) noted that “different aspects of reality lend themselves to different methods of inquiry” (p. 327), and, she added, “there is no uniform paradigm-method link, there is a method reality link” (p. 327).

Sandelowski (2003) further suggests the mixing may have one of two purposes “to achieve a fuller understanding of a target phenomenon and to verify one set of findings against the other or a comprehensive kaleidoscopic sense of understanding versus truth or validity ‘representation’.” She posed a great question: What is mixed? And what kind of mixing occurs? Through focusing on the qualitative quantitative distinction she recognizes the ambiguity in distinguishing between them and the difficulty of crafting research reports incorporating both in a convincing manner.

Johnson and Turner’s (2003) chapter in the Handbook of Mixed Methods uses the key idea of “trustworthy,” which they equate with validity and which in the title of Hunter’s book would be “believable.” They say, “Valid research is plausible, credible, trustworthy, and, therefore, defensible. . . . We treat the terms valid and trustworthy as synonyms” (p. 300; italics in original). This may be directly linked to multimethods in that we are prone to trust the many over the few, if confirmatory, and to have if not “mistrust” then at least doubt over the disconfirming or specification of differences due to method itself ( Lever, 1978 ).

The rhetorical turn is closely allied with another postmodern humanistic approach to social science research and that is the idea that what we do is basically tell a story—create a narrative. The iconic corollary to multimethod research from the narrative perspective is like the “Rashomon effect,” the stories of a single event or phenomenon told from the perspectives of different witnesses and participants. This, of course, raises the larger phenomenological question—is it many versions of one event or many different events? We operate from the assumption that the real world does exist (physicalism) and yet recognize that our knowledge of that real world is a varied and imperfect product of the observations we make of it (relativism).

As Andrew Abbott (1992) has observed in his landmark paper “From Causes to Events: Notes on Narrative Positivism”:

In the last decade, a number of writers have proposed narrative as the foundation for sociological methodology. By this they do not mean narrative in terms of words as opposed to numbers and complexity as opposed to formalization. Rather, they mean narrative in the more generic sense of process, or story. . . . In the context of contemporary empirical practice, such a conception is revolutionary. Our normal methods parse social reality into fixed entities with variable qualities. They attribute causality to the variables –hypostatized social characteristics—rather than to agents; variables do things, not social actors. Stories disappear. (p. 428)

The key trope ( Booth, 1961/1983 ) that governs narratives of science, including social science, is that of a “quest.” It is a quest that begins with a question leading to a search using research to explore the unknown. It is a purposeful journey to find a treasured goal, to go out into that real world and make the unknown known—in a word to acquire knowledge. One hopes that the knowledge one acquires has some validity that it is “true” and that what one believes to be true about the phenomena bears some close approximation to reality. From a pragmatic perspective—it is valid if the knowledge works, if it allows one to accomplish what one wishes to get done; if it doesn’t work it is useless ( Dewey, 1938 ).

Some might think the narrative is merely the story about how one does science—that it is not the doing of science itself. We suggest that the narrative is in fact a critical part of the doing of science itself, and the stories we tell about what we do serve to reflexively construct and reconstruct the actual quest itself.

As noted by Scott Baker (1990) :

In studying the discourse of scientific communities, in fact, we find these communities employing multiple rhetorics as often as they use strict singular logics. . . . We note moreover, that scientists do not exclusively depend upon, or even follow, the strict guidelines of their field’s logic or method. They convince each other and the lay public that their theories are reliable by means of persuasion not provided for or sanctioned by the accepted methodology. . . these communication strategies are outside the formal rules of method, [yet] they do persuade. (pp. 233–234)

But the postmodernist idea of narrative is more than just telling a story about how science is conducted; narrative is seen to be an ontological aspect of how we as “sensate human beings” “make sense” of the real world. Abbott (1997) claims we do so by taking into consideration “context and contingency,” and Ragin (1987) suggests we tell narratives of constellations of characteristics defining our objects of study (our cases) as they alter through time. To highlight a quote presented previously that focused on the mechanism or the black box of explanation ( Guo & Adkins, 2008 ):

It should be pointed out that significant statistical findings alone are rarely, if ever, considered proof of a link between a genetic variant and a human complex phenotype. This contrasts with the usual practice in social sciences. Repeated significant results in social sciences showing a connection between, for example, parental education and children’s education attainment are often considered sufficient evidence for such a connection. The credibility of the evidence is not only from the replicated statistical results but also from real-life observation. Drawing from personal experiences, most people would probably agree that a higher level of parental education would lead to a higher level of education in children on average. Such confirmation from life experiences is not available for interpreting genetic findings. Genotypes are not visible in everyday life. To develop a credible story that supports statistical findings, other evidence is needed, such as those from animal studies and biochemical studies. (p. 224; italics added)

The Narrative Moral of Multimethod Research Design

We return to the opening of our discussion about multimethod research design, just as Ulysses eventually returned to Greece after a meandering Odyssey, and that is to stress the spirit of multimethod research with which we began. True believers of one or another method, positing different ontological assumptions, will continue to debate the appropriateness of different methods, as they should in the full spirit of free inquiry. But in the further spirit of multimethod research such assumptions should be open to challenge, and a tolerance for entertaining alternative assumptions should be considered. The spirit of humility, not hubris, and the recognition of limitations in methods are more likely to advance the cause of science than any authoritative dictates of idealized methods. Context and contingency apply to research itself, and understanding the broader social context and the pragmatic contingent decisions made in the conduct of research must be taken into accounts. What we claim to know is dependent on how we came to know it. And the multimethod perspective still holds out the promise of closer approximations to truth about reality or, at a minimum, more moderated contingent claims to truth that reflect the reality of science itself.

We previously stated our criteria of good scientific research that it be

Consistent—logically consistent, not random or contradictory

Corresponding—data and ideas linked in measurement

Convincing—rhetorically persuasive

To this we add another—that it be “competent”

Multimethod research is not a magic bullet to truth but a style that still demands rigor and reflection in addressing the skeptic’s central question: “How do you know?”

Discussion Questions

Can one consider mixed methods research a subset of multimethod research?

What parts of reality and the world does multimethod research focus on?

In what ways is multimethod research frequently superior to single or monomethod research?

Is multimethod research multidisciplinary?

What are some classic multimethod studies provided in this chapter, and what makes them important?

Suggested Websites

http://www.sociology.northwestern.edu/people/faculty/albert-hunter.html

Information about the authors of this chapter.

Abbott, A. ( 1990 ). Conceptions of time and events in social science methods: Causal and narrative approaches.   Historical Methods: A Journal of Quantitative and Interdisciplinary History , 23 (4), 140–150.

Google Scholar

Abbott, A. ( 1992 ). From causes to events notes on narrative positivism.   Sociological Methods & Research , 20 , 428–455.

Abbott, A. ( 1995 ). Sequence analysis: New methods for old ideas.   Annual Review of Sociology , 21 , 93–113.

Abbott, A. ( 1997 ). Of time and space: The contemporary relevance of the Chicago School.   Social Forces , 75 , 1149–1182.

Ajzen, I. , & Fishbein, M. ( 1977 ). Attitude-behavior relations: A theoretical analysis and review of empirical research.   Psychological Bulletin , 84 (5). doi: 10.1037/0033-2909.84.888

Allison, P. D. ( 1984 ). Event history analysis: Regression for longitudinal event data . Newbury Park, CA: Sage.

Google Preview

Anderson, N. ( 1923 ). The hobo: The sociology of homeless men . Chicago, IL: University of Chicago Press.

Baker, S. ( 1990 ). Reflection, doubt, and the place of rhetoric in postmodern social theory.   Sociological Theory , 8 , 212–245.

Barton, A. H. ( 1969 ). Communities in disaster: A sociological analysis of collective stress situations . Garden City, NJ: Doubleday.

Bennett, J. ( 1981 ). Oral history and delinquency: The rhetoric of criminology . Chicago, IL: University of Chicago Press.

Blau, P. M. , & Duncan, O. D. ( 1967 ). American occupational structur e. New York, NY: Free Press.

Bloor, D. ( 1991 ). Knowledge and social imagery . Chicago, IL: University of Chicago Press.

Bobo, L. , Zubrinsky, C. L. , Johnson, J. H. Jr. , & Oliver, M. L. ( 1994 ). Public opinion before and after a spring of discontent. In M. Baladassare (Ed.), The Los Angeles riots: Lessons for the urban future (pp. 103–133). Boulder, CO: Westview.

Booth, W. C. ( 1983 ). The rhetoric of fiction . Chicago: University of Chicago Press.

Boswell, J. ( 1998 ). Life of Johnson . Edited by R. W. Chapman. Oxford, UK: Oxford University Press.

Brewer, J. , & Hunter, A. ( 1989 ). Multimethod research: A synthesis of styles . Newbury Park, CA: Sage.

Brewer, J. , & Hunter, A. ( 2006 ). Foundations of multimethod research: Synthesizing styles . Thousand Oaks, CA: Sage.

Briggs, X. S. , Popkin S. J. , & Goering J. M. ( 2010 ). Moving to opportunity: The story of an American experiment to fight ghetto poverty . New York, NY: Oxford University Press.

Campbell, D. T. , & Fiske, D. W. ( 1959 ). Convergent and discriminant validation by the multitrait-multimethod matrix.   Psychological Bulletin , 56 , 81.

Campbell, D. T. , & Stanley, J. C. ( 1963 ). Experimental and quasi-experimental designs for research . Chicago, IL: Rand McNally.

Cohen-Cole, E. , & Fletcher, J. M. ( 2008 ). Is obesity contagious? Social networks vs. environmental factors in the obesity epidemic.   Journal of Health Economics , 27 , 1382–1387. doi: 10.1016/j.healeco.2008.04.005

Cook, T. D. ( 2002 ). Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them.   Educational Evaluation and Policy Analysis , 24 (3), 175–199.

Cook, T. D. , & Campbell, D. T. ( 1979 ). Quasi- experimentation: Design and analysis issues for field settings . Boston, MA: Houghton Mifflin.

Cook, T. D. , Shadish, W. R. , & Wong, V. C. ( 2008 ). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons.   Journal of Policy Analysis and Management , 27 , 724–750. doi: 10.1002/pam.20375

Davis, J. A. , Spaeth, J. L. , & Huson, C. ( 1961 ). A technique for analyzing the effects of group composition.   American Sociological Review , 26 , 215–225.

Denzin, N. K. ( 1970 ). The research act in sociology: A theoretical introduction to sociological methods . Piscataway, NJ: Aldine.

DeVault, M. L. ( 1990 ). Women write sociology: Rhetorical strategies. In A. Hunter (Ed.), The rhetoric of social research: Understood and believed , (pp. 97–110.). New Brunswick, NJ: Rutgers University Press.

Dewey, J. ( 1938 ). Logic: The theory of inquiry . New York, NY: Holt, Rinehart and Winston.

Donne, J. ( 1896 ). Epigrams. In E. K. Chambers (Ed.) Poems of John Donne (Vol II, p. 210). London: Lawrence & Bullen.

Duneier, M. ( 2004 ). Scrutinizing the heat: On ethnic myths and the importance of shoe leather.   Contemporary Sociology , 2 , 139–150.

Duneier, M. ( 2006 ). Ethnography, the ecological fallacy, and the 1995 Chicago heat wave.   American Sociological Review , 71 , 679–688.

Erikson, K. T. ( 1976 ). Everything in its path: Destruction of community in the Buffalo Creek flood . New York, NY: Simon and Schuster.

Erikson, K. ( 1994 ). A new species of trouble: The human experience of modern disasters . New York, NY: W. W. Norton.

Espeland, W. N. , & Stevens, M. L. ( 1998 ). Commensuration as a social process.   Annual Review of Sociology , 24 , 313–343.

Fine, G.A. ( 2012 ). Tiny publics: A theory of group action and culture . Ithaca, NY: CUP Services.

Form, W. H. , & Nosow, S. ( 1958 ). Community in disaster . New York, NY: Harper.

Freudenburg, W. R. , Gramling, R. , Laska, S. & Erikson, K. ( 2009 ). Catastrophe in the making: The engineering of Katrina and the disasters of tomorrow . Washington, DC: Island Press.

Granovetter, M. ( 1985 ). Economic action and social structure: The problem of embeddedness.   American Journal of Sociology , 91 , 481–510.

Greene, J. C. , Caracelli, V. J. , & Graham, W. F. ( 1989 ). Toward a conceptual framework for mixed-method evaluation designs.   Educational Evaluation and Policy Analysis , 11 , 255–274.

Guo, G. , & Adkins, D. E. ( 2008 ). How is a statistical link established between a human outcome and a genetic variant?   Sociological Methods Research , 37 , 201–226.

Guo, G. , Elder, G. H. , Cai, T. , & Hamilton, N. ( 2009 ). Gene–environment interactions: Peers’ alcohol use moderates genetic contribution to adolescent drinking behavior.   Social Science Research , 38 , 213–224.

Guo, G. , Hardie, J. H. , Owen, C. , Daw, J.K. , Fu, Y. , Lee, H . . . Duncan, G. ( 2009 ). DNA collection in a randomized social science study of college peer effects.   Sociological Methodology , 39 , 1–29.

Gusfield, J. R. ( 1990 ). Two genres of sociology: A literary analysis of the American occupational structure and tally’s corner. In A. Hunter (Ed.), The rhetoric of social research: Understood and believed , (pp. 62–96.). New Brunswick, NJ: Rutgers University Press.

Hunter, A. ( 1986 ). Love Canal emergency declaration area: Proposed habitability criteria study: Vol. 1. Technical Review Committee . New York, NY: US Environmental Protection Agency, US Department of Health and Human Services/Centers for Disease Control, New York State Department of Health, New York State Department of Environmental Conservation.

Hunter, A. (Ed.). ( 1990 ). The rhetoric of social research: Understood and believed . New Brunswick, NJ: Rutgers University Press.

Hunter, A. , & Maxfield, M. G. ( 1980 ). Reactions to Crime Project: Vol. 1. Methodological overview of the Reactions to Crime Project . Evanston, IL: Northwestern University, Center for Urban Affairs.

Investopedia. (2014). Revealed preference. Retrieved from http://www.investopedia.com/terms/r/revealed-preference.asp

Jacobs, J. A. ( 2005 ). Multiple methods in ASR.   Footnotes: Newsletter of the American Sociological Review , 33 (9). Retrieved from http://www.asanet.orgwww.asanet.org/footnotes/dec05/indextwo.html .

Jahoda, M , Lazarsfeld, P. F. , & Zeisel, H , ( 1971 ). Marienthal: The sociography of an unemployed community . New Brunswick, NJ: Transaction. (Originally published in 1933)

Jankowski, M. S. ( 1991 ). Islands in the street: Gangs and American urban society . Berkeley: University of California.

Joppa, L. N. , McInerny, G. , Harper, R. , Salido, L. , Takeda, K. , O’Hara, K. , . . . Emmott, S. ( 2013 ). Troubling trends in scientific software use.   Science , 340 , 814–815.

Johnson, B. , & Turner, L. A. ( 2003 ). Data collection strategies in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research , (pp. 297–319). Thousand Oaks, CA: Sage.

Kahneman, D. ( 2011 ). Thinking, fast and slow . New York, NY: Farrar, Strauss & Giroux.

Kanter, R. M. ( 1993 ). Men and women of the corporation . New York, NY: Basic Books.

Kendra, J. M. , & Wachtendorf, T. ( 2003 ). Elements of resilience after the World Trade Center disaster: Reconstituting New York City’s Emergency Operations Centre.   Disasters , 27 (1), 37–53. doi:10.1111/1467-7717.00218

King, G. , Keohane, R. O. , & Verba, S. ( 1994 ). Designing social inquiry: Scientific inference in qualitative research . Princeton, NJ: Princeton University Press.

Klinenberg, E. ( 2003 ). Heat wave: A social autopsy of disaster in Chicago . Chicago, IL: University of Chicago Press.

Krieger, S. ( 1983 ). The mirror dance: Identity in a women’s community . Philadelphia, PA: Temple University Press.

LaPiere, R. T. ( 1934 ). Attitudes vs. actions.   Social Forces , 13 , 230–237.

Lazarsfeld, P. F. ( 1945 ) Foreward to Ernest Greenwood, Experimental Sociology . New York, NY: King’s Crown Press.

Lever, J. ( 1978 ). Sex differences in the complexity of children’s play and games.   American Sociological Review , 43 , 471–483.

Liebow, E. ( 1967 ). Tally’s corner: A study of Negro streetcorner men . New York, NY: Rowman & Littlefield.

Mahoney, J. , & Rueschemeyer, D. (Eds.). ( 2003 ). Comparative historical analysis in the social sciences . Cambridge, UK: Cambridge University Press.

Manchester, W. ( 1978 ). American Caesar: Douglas McArthur . New York, NY: Dell.

McClelland, D. C. ( 1961 ). Achieving society . Princeton, NJ: Van Nostrand.

Merton, R. K. (Ed.). ( 1968 ). Social theory and social structure . New York, NY: Free Press.

Merton, R. K. , & Lazersfeld, P. F. (Eds.). ( 1950 ). Continuities in social research: Studies in the scope and method of “The American Soldier.” New York, NY: Free Press.

Mills, C. W. ( 1959 ). The sociological imagination . New York, NY: Oxford University Press.

Moore, B. ( 1958 ). Political power and social theory: Six studies . Cambridge, MA: Harvard University Press.

Moore, B. ( 1966 ). Social origins of dictatorship and democracy: Lord and peasant in the making of the modern world . Boston, MA: Beacon Press.

Morse, J. M. ( 2003 ). Principles of mixed methods and multimethod research design. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 189–208). Thousand Oaks, CA: Sage.

Mulkay, M. J. ( 1979 ). Science and the sociology of knowledge . London: Allen & Unwin.

Office of Income Security Policy, Office of the Assistant Secretary for Planning and Evaluation, US Department of Health and Human Services. ( 1983 ). Overview of the Seattle–Denver income maintenance experiment final report . Washington, DC: Author.

Pager, D. , & Quillian, L. ( 2005 ). Walking the talk? What employers say versus what they do.   American Sociological Review , 70 , 355–380. doi: 10.1177/000312240507000301

Prince, S. H. ( 1920 ). Catastrophe and social change, based upon a sociological study of the Halifax disaster . Studies in History, Economics, and Public Law 212. New York, NY: Columbia University Press.

Putnam, R. D. ( 2001 ). Bowling alone: The collapse and revival of American community . New York, NY: Simon and Schuster.

Ragin, C. C. ( 1987 ). The comparative method: Moving beyond qualitative and quantitative strategies . Berkeley: University of California Press.

Ragin, C. C. , & Becker, H. S. (Eds.). ( 1992 ). What is a case? Exploring the foundations of social inquiry . Cambridge, UK: Cambridge University Press.

Raudenbush, S. W. , & Bryk, A. S. ( 2001 ). Hierarchical linear models: Applications and data analysis methods . Advanced Quantitative Techniques in the Social Sciences 1. Thousand Oaks, CA: Sage.

Rossi, P. H. , Lipsey, M. W. , & Freeman, H. E. ( 2004 ). Evaluation: A systematic approach . Thousand Oaks, CA: Sage.

Saey, T. H. (2013, June 29). Gene weakly linked to education level. Science News . Retrieved from https://www.sciencenews.org/article/genes-weakly-linked-education-level

Sampson, R. J. , Morenoff, J. D. , & Gannon-Rowley, T. ( 2002 ). Assessing “neighborhood effects”: Social processes and new directions in research.   Annual Review of Sociology , 28 , 443–478.

Samuelson, P. ( 1948 ). Economics . New York, NY: McGraw-Hill.

Sandelowski, M. ( 2003 ). Tables or tableaux? The challenges of writing and reading mixed methods studies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research , (pp. 321–350). Thousand Oaks, CA: Sage.

Scott, W. R. , & Meyer, J. W. (Eds.). ( 1994 ). Institutional environments and organizations: Structural complexity and individualism . Thousand Oaks, CA: Sage.

Shadish, W. R. , Cook, T. D. , & Campbell, D. T. ( 2002 ). Experimental and quasi-experimental designs for generalized causal inference . Boston, MA: Houghton Mifflin.

Shaw, C. R. ( 2013 ). The jack-roller . Eastford, CT: Martino. (Originally published in 1930)

Shea, C. ( 2009 ). The nature–nurture debate, redux: Genetic research finally makes its way into the thinking of sociologists.   The Chronicle of Higher Education , 54 (31), B6.

Shils, E. ( 1957 ). Primordial, personal, sacred and civil ties: Some particular observations on the relationships of sociological research and theory.   The British Journal of Sociology , 8 (2), 130–145.

Simmel, G. ( 1971 ). On individuality and social forms . Chicago, IL: University of Chicago Press.

Skocpol, T (Ed.). ( 1984 ). Vision and method in historical sociology . New York, NY: Cambridge University Press.

Skocpol, T. , & Somers, M. ( 1980 ). The uses of comparative history in macrosocial inquiry.   Comparative Studies in Society and History , 22 (2), 174–197.

Small, M. L. ( 2011 ). How to conduct a mixed methods study: Recent trends in a rapidly growing literature.   Annual Review of Sociology , 37 , 57–86. doi: 10.1146/annurev.soc.012809.102657

Stallings, R. A. ( 1997 ). Sociological theories and disaster studies . Newark, DE: Disaster Research Center.

Stouffer, S. A. , Lumsdaine, A. A. , Lumsdaine, M. H. , Williams R. M. Jr. , Smith, M. B. , Janis, I. L. , . . . & Cottrell L. S. Jr. ( 1949 ). Studies in social psychology in World War II: Vol. 2. The American soldier: Combat and its aftermath . Oxford, UK: Princeton University Press.

Sutherland, E. H. (Ed.) ( 1937 ). The professional thief . Chicago, IL: University of Chicago Press

Thomas, W. I. ( 1923 ). The unadjusted girl . Oxford, UK: Little, Brown and Co.

Tversky, A. , & Kahneman, D. ( 1974 ). Judgment under uncertainty: Heuristics and biases.   Science , 185 , 1124–1131.

Uzzi, B. ( 1999 ). Embeddedness in the making of financial capital: How social relations and networks benefit firms seeking financing.   American Sociological Review , 64 , 481–505.

Wade, H. D. ( 2004 ). On operationalisms and economics.   Journal of Economic Issues , 38 (4), 953–968.

Wallerstein, I. M. ( 1974 ). The modern world-system: Vol. 1. Capitalist agriculture and the origins of the European world-economy in the sixteenth century . New York, NY: Academic Press.

Wallerstein, I. M. ( 2004 ). World-systems analysis: An introduction . Durham, NC: Duke University Press.

Walton, J. , & Ragin, C. ( 1990 ). Global and national sources of political protest: Third world responses to the debt crisis.   American Sociological Review , 55 , 876–890.

Weber, M. (1905). The protestant ethic and the spirit of capitalism . New York, NY: Routledge. 2001 .

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Qualitative Data Analysis

• Choosing QDA software
• Free QDA tools
• Transcription tools
• Social media research
• Mixed and multi-method research
• Network Diagrams
• Publishing qualitative data
• Student specialists

General Information

For assistance, please submit a request .  You can also reach us via the chat below, email [email protected] , or join Discord server .

If you've met with us before,  tell us how we're doing .

Stay in touch by signing up for our Data Services newsletter .

Service Desk and Chat

Bobst Library , 5th floor

Staffed Hours: Fall 2024

Mondays:  12pm - 5pm         Tuesdays:  12pm - 5pm         Wednesdays:  12pm - 5pm         Thursdays:  12pm - 5pm         Fridays:  12pm - 5pm        

Data Services closes for winter break at the end of the day on Friday, Dec. 22, 2023. We will reopen on Wednesday, Jan. 3, 2024.

Learning Mixed Methods

• Start-to-Finish Mixed Methods with MAXQDA
• Tutorials and Videos
• Suggested literature: Schoonenboom, J., Johnson, R.B. How to Construct a Mixed Methods Research Design. Köln Z Soziol 69, 107–131 (2017). https://doi.org/10.1007/s11577-017-0454-1 First step: Justify your research plan previously. Why is it relevant to integrate qualitative and quantitative methods for answering your research objectives? Then, select a research design and appropriate methods to gather information.
• Suggested resource: MAXQDA Document Variables tab Second Step: organize your qualitative sources of information with relevant variables. Afterward, identify interviews with the associated date and place, sociodemographic information of interviewees. Identify to which part of the study corresponds the information gathered, exploratory, triangulation, or confirmatory.
• Suggested resource Data Services Data Management Planning Third step: If you are dealing with large sets of text or numeric information. Use data cleaning and codebook procedures.
• Suggested resource: QDA resources at Data Services Fourth Step: Conduct the relevant Qualitative Analysis. Write preliminary Conclusions.
• Suggested literature: Cox V. (2017) Exploratory Data Analysis. In: Translating Statistics to Make Decisions. Apress, Berkeley, CA. https://doi.org/10.1007/978-1-4842-2256-0_3 Fifth Step: Conduct an in-depth exploratory analysis in the case of your quantitative data. Write preliminary conclusions.
• Suggested resource: NYU Data Services Quantitative resources Sixth Step: Do the quantitative relevant analysis. Write preliminary conclusions.
• Suggested resource: General Information about Mixed Methods in MAXQDA. Seventh Step: Quantify themes/codes in your qualitative analysis and do an exploratory analysis according to variables. Write preliminary conclusions. 
• Suggested resource: Exporting Data to Excel and SPSS with MAXQDA Eight step: if relevant, export themes and variables to do a quantitative analysis from qualitative data. Write preliminary conclusions.
• Suggested literature: Fielding, N. G. (2012). Triangulation and Mixed Methods Designs: Data Integration With New Research Technologies. Journal of Mixed Methods Research, 6(2), 124–136. https://doi.org/10.1177/1558689812437101 Ninth Step: compare the conclusions from each analysis by seeking similarities and differences. 
• Suggested literature: Bergman, M. M. (2008). 7 quality of inferences in mixed methods research: calling for an integrative framework. In Bergman, M. M. (Ed.), Advances in mixed methods research (pp. 101-119). SAGE Publications Ltd, https://www.doi.org/10. Tenth step: conclude according to validity and reliability criteria that match your research paradigm.
• Telling a Complete Story with Qualitative and Mixed Methods Research - A conversation with Dr. John W. Creswell A Sage video interview with one of the leading scholars in Mixed Methods.
• SAGE project planner A step-by-step decision-making process for planning your research project.
• Sage Methods Maps A map for choosing qualitative and quantitative research designs and analysis.
• MAXQDA’s Mixed Methods Features A brief overview of MAXQDA's Mixed-methods functions
• Introduction to Mixed Methods Functions MAXQDA NYU Data services presentation about the Mixed Methods functions in MAXQDA.
• Creswell, John W., "Steps in Conducting a Scholarly Mixed Methods Study" (2013). DBER Speaker Series. 48. https://digitalcommons.unl.edu/dberspeakers/48 A primer to mixed methods research designs and workflows.
• Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of mixed methods research, 1(2), 112-133. The purpose of this article is to examine definitions of mixed methods.
• DeCuir-Gunby, J., & Schutz, P. (2017). Developing a mixed methods proposal: A practical guide for beginning researchers. SAGE Publications, Inc. https://www.doi.org/10.4135/9781483399980 An introductory guide for making mixed-methods proposals
• Schoonenboom, J. (2018). Designing mixed methods research by mixing and merging methodologies: A 13-step model. American Behavioral Scientist, 62(7), 998-1015. In this article, a 13-step model is presented that researchers can follow in designing their own multimethodology research.
• Tashakkori, A., & Teddlie, C. (2010). SAGE handbook of mixed methods in social & behavioral research (2nd ed.). SAGE Publications, Inc. https://www.doi.org/10.4135/9781506335193 A book about different approaches in mixed-methods research.
• Seawright, J., & Gerring, J. (2008). Case selection techniques in case study research: A menu of qualitative and quantitative options. Political research quarterly, 61(2), 294-308. A qualitative and quantitative guide to case selection
• << Previous: Social media research
• Next: Network Diagrams >>
• Last Updated: Oct 16, 2024 5:28 PM
• URL: https://guides.nyu.edu/QDA

• Open access
• Published: 15 December 2015

Qualitative and mixed methods in systematic reviews

• David Gough 1  

Systematic Reviews volume  4 , Article number:  181 ( 2015 ) Cite this article

24k Accesses

42 Citations

26 Altmetric

Metrics details

Expanding the range of methods of systematic review

The logic of systematic reviews is very simple. We use transparent rigorous approaches to undertake primary research, and so we should do the same in bringing together studies to describe what has been studied (a research map) or to integrate the findings of the different studies to answer a research question (a research synthesis). We should not really need to use the term ‘systematic’ as it should be assumed that researchers are using and reporting systematic methods in all of their research, whether primary or secondary. Despite the universality of this logic, systematic reviews (maps and syntheses) are much better known in health research and for answering questions of the effectiveness of interventions (what works). Systematic reviews addressing other sorts of questions have been around for many years, as in, for example, meta ethnography [ 1 ] and other forms of conceptual synthesis [ 2 ], but only recently has there been a major increase in the use of systematic review approaches to answer other sorts of research questions.

There are probably several reasons for this broadening of approach. One may be that the increased awareness of systematic reviews has made people consider the possibilities for all areas of research. A second related factor may be that more training and funding resources have become available and increased the capacity to undertake such varied review work.

A third reason could be that some of the initial anxieties about systematic reviews have subsided. Initially, there were concerns that their use was being promoted by a new managerialism where reviews, particularly effectiveness reviews, were being used to promote particular ideological and theoretical assumptions and to indirectly control research agendas. However, others like me believe that explicit methods should be used to enable transparency of perspectives driving research and to open up access to and participation in research agendas and priority setting [ 3 ] as illustrated, for example, by the James Lind Alliance (see http://www.jla.nihr.ac.uk/ ).

A fourth possible reason for the development of new approaches is that effectiveness reviews have themselves broadened. Some ‘what works’ reviews can be open to criticism for only testing a ‘black box’ hypothesis of what works with little theorizing or any logic model about why any such hypothesis should be true and the mechanisms involved in such processes. There is now more concern to develop theory and to test how variables combine and interact. In primary research, qualitative strategies are advised prior to undertaking experimental trials [ 4 , 5 ] and similar approaches are being advocated to address complexity in reviews [ 6 ], in order to ask questions and use methods that address theories and processes that enable an understanding of both impact and context.

This Special Issue of Systematic Reviews Journal is providing a focus for these new methods of review whether these use qualitative review methods on their own or mixed together with more quantitative approaches. We are linking together with the sister journal Trials for this Special Issue as there is a similar interest in what qualitative approaches can and should contribute to primary research using experimentally controlled trials (see Trials Special Issue editorial by Claire Snowdon).

Dimensions of difference in reviews

Developing the range of methods to address different questions for review creates a challenge in describing and understanding such methods. There are many names and brands for the new methods which may or may not withstand the changes of historical time, but another way to comprehend the changes and new developments is to consider the dimensions on which the approaches to review differ [ 7 , 8 ].

One important distinction is the research question being asked and the associated paradigm underlying the method used to address this question. Research assumes a particular theoretical position and then gathers data within this conceptual lens. In some cases, this is a very specific hypothesis that is then tested empirically, and sometimes, the research is more exploratory and iterative with concepts being emergent and constructed during the research process. This distinction is often labelled as quantitative or positivist versus qualitative or constructionist. However, this can be confusing as much research taking a ‘quantitative’ perspective does not have the necessary numeric data to analyse. Even if it does have such data, this might be explored for emergent properties. Similarly, research taking a ‘qualitative’ perspective may include implicit quantitative themes in terms of the extent of different qualitative findings reported by a study.

Sandelowski and colleagues’ solution is to consider the analytic activity and whether this aggregates (adds up) or configures (arranges) the data [ 9 ]. In a randomized controlled trial and an effectiveness review of such studies, the main analysis is the aggregation of data using a priori non-emergent strategies with little iteration. However, there may also be post hoc analysis that is more exploratory in arranging (configuring) data to identify patterns as in, for example, meta regression or qualitative comparative analysis aiming to identify the active ingredients of effective interventions [ 10 ]. Similarly, qualitative primary research or reviews of such research are predominantly exploring emergent patterns and developing concepts iteratively, yet there may be some aggregation of data to make statements of generalizations of extent.

Even where the analysis is predominantly configuration, there can be a wide variation in the dimensions of difference of iteration of theories and concepts. In thematic synthesis [ 11 ], there may be few presumptions about the concepts that will be configured. In meta ethnography which can be richer in theory, there may be theoretical assumptions underlying the review question framing the analysis. In framework synthesis, there is an explicit conceptual framework that is iteratively developed and changed through the review process [ 12 , 13 ].

In addition to the variation in question, degree of configuration, complexity of theory, and iteration are many other dimensions of difference between reviews. Some of these differences follow on from the research questions being asked and the research paradigm being used such as in the approach to searching (exhaustive or based on exploration or saturation) and the appraisal of the quality and relevance of included studies (based more on risk of bias or more on meaning). Others include the extent that reviews have a broad question, depth of analysis, and the extent of resultant ‘work done’ in terms of progressing a field of inquiry [ 7 , 8 ].

Mixed methods reviews

As one reason for the growth in qualitative synthesis is what they can add to quantitative reviews, it is not surprising that there is also growing interest in mixed methods reviews. This reflects similar developments in primary research in mixing methods to examine the relationship between theory and empirical data which is of course the cornerstone of much research. But, both primary and secondary mixed methods research also face similar challenges in examining complex questions at different levels of analysis and of combining research findings investigated in different ways and may be based on very different epistemological assumptions [ 14 , 15 ].

Some mixed methods approaches are convergent in that they integrate different data and methods of analysis together at the same time [ 16 , 17 ]. Convergent systematic reviews could be described as having broad inclusion criteria (or two or more different sets of criteria) for methods of primary studies and have special methods for the synthesis of the resultant variation in data. Other reviews (and also primary mixed methods studies) are sequences of sub-reviews in that one sub-study using one research paradigm is followed by another sub-study with a different research paradigm. In other words, a qualitative synthesis might be used to explore the findings of a prior quantitative synthesis or vice versa [ 16 , 17 ].

An example of a predominantly aggregative sub-review followed by a configuring sub-review is the EPPI-Centre’s mixed methods review of barriers to healthy eating [ 18 ]. A sub-review on the effectiveness of public health interventions showed a modest effect size. A configuring review of studies of children and young people’s understanding and views about eating provided evidence that the public health interventions did not take good account of such user views research, and that the interventions most closely aligned to the user views were the most effective. The already mentioned qualitative comparative analysis to identify the active ingredients within interventions leading to impact could also be considered a qualitative configuring investigation of an existing quantitative aggregative review [ 10 ].

An example of a predominantly configurative review followed by an aggregative review is realist synthesis. Realist reviews examine the evidence in support of mid-range theories [ 19 ] with a first stage of a configuring review of what is proposed by the theory or proposal (what would need to be in place and what casual pathways would have to be effective for the outcomes proposed by the theory to be supported?) and a second stage searching for empirical evidence to test for those necessary conditions and effectiveness of the pathways. The empirical testing does not however use a standard ‘what works’ a priori methods approach but rather a more iterative seeking out of evidence that confirms or undermines the theory being evaluated [ 20 ].

Although sequential mixed methods approaches are considered to be sub-parts of one larger study, they could be separate studies as part of a long-term strategic approach to studying an issue. We tend to see both primary studies and reviews as one-off events, yet reviews are a way of examining what we know and what more we want to know as a strategic approach to studying an issue over time. If we are in favour of mixing paradigms of research to enable multiple levels and perspectives and mixing of theory development and empirical evaluation, then we are really seeking mixed methods research strategies rather than simply mixed methods studies and reviews.

Noblit G. Hare RD: meta-ethnography: synthesizing qualitative studies. Newbury Park NY: Sage Publications; 1988.

Google Scholar  

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59.

Article   PubMed   PubMed Central   Google Scholar  

Gough D, Elbourne D. Systematic research synthesis to inform policy, practice and democratic debate. Soc Pol Soc. 2002;2002:1.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance 2015. BMJ. 2015;350:h1258

Candy B, Jone L, King M, Oliver S. Using qualitative evidence to help understand complex palliative care interventions: a novel evidence synthesis approach. BMJ Support Palliat Care. 2014;4:Supp A41–A42.

Article   Google Scholar  

Noyes J, Gough D, Lewin S, Mayhew A, Michie S, Pantoja T, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. 2013;66:11.

Gough D, Oliver S, Thomas J. Introduction to systematic reviews. London: Sage; 2012.

Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1:28.

Sandelowski M, Voils CJ, Leeman J, Crandlee JL. Mapping the mixed methods-mixed research synthesis terrain. J Mix Methods Res. 2012;6:4.

Thomas J, O’Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3:67.

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.

Oliver S, Rees R, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Exp. 2008;11:72–84.

Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Qual Saf. 2015. 2014-003642.

Brannen J. Mixed methods research: a discussion paper. NCRM Methods Review Papers, 2006. NCRM/005.

Creswell J. Mapping the developing landscape of mixed methods research. In: Teddlie C, Tashakkori A, editors. SAGE handbook of mixed methods in social & behavioral research. New York: Sage; 2011.

Morse JM. Principles of mixed method and multi-method research design. In: Teddlie C, Tashakkori A, editors. Handbook of mixed methods in social and behavioural research. London: Sage; 2003.

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29–45.

Harden A, Thomas J. Mixed methods and systematic reviews: examples and emerging issues. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in the social and behavioral sciences. 2nd ed. London: Sage; 2010. p. 749–74.

Chapter   Google Scholar  

Pawson R. Evidenced-based policy: a realist perspective. London: Sage; 2006.

Book   Google Scholar  

Gough D. Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Med. 2013;11:22.

Download references

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, University College London, London, WC1H 0NR, UK

David Gough

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to David Gough .

Additional information

Competing interests.

The author is a writer and researcher in this area. The author declares that he has no other competing interests.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Gough, D. Qualitative and mixed methods in systematic reviews. Syst Rev 4 , 181 (2015). https://doi.org/10.1186/s13643-015-0151-y

Download citation

Received : 13 October 2015

Accepted : 29 October 2015

Published : 15 December 2015

DOI : https://doi.org/10.1186/s13643-015-0151-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Mixed methods research: what it is and what it could be

• Open access
• Published: 29 March 2019
• Volume 48 , pages 193–216, ( 2019 )

Cite this article

You have full access to this open access article

• Rob Timans 1 ,
• Paul Wouters 2 &
• Johan Heilbron 3  

136k Accesses

124 Citations

12 Altmetric

Explore all metrics

A Correction to this article was published on 06 May 2019

This article has been updated

Combining methods in social scientific research has recently gained momentum through a research strand called Mixed Methods Research (MMR). This approach, which explicitly aims to offer a framework for combining methods, has rapidly spread through the social and behavioural sciences, and this article offers an analysis of the approach from a field theoretical perspective. After a brief outline of the MMR program, we ask how its recent rise can be understood. We then delve deeper into some of the specific elements that constitute the MMR approach, and we engage critically with the assumptions that underlay this particular conception of using multiple methods. We conclude by offering an alternative view regarding methods and method use.

Similar content being viewed by others

Mixed Methods

Avoid common mistakes on your manuscript.

The interest in combining methods in social scientific research has a long history. Terms such as “triangulation,” “combining methods,” and “multiple methods” have been around for quite a while to designate using different methods of data analysis in empirical studies. However, this practice has gained new momentum through a research strand that has recently emerged and that explicitly aims to offer a framework for combining methods. This approach, which goes by the name of Mixed Methods Research (MMR), has rapidly become popular in the social and behavioural sciences. This can be seen, for instance, in Fig.  1 , where the number of publications mentioning “mixed methods” in the title or abstract in the Thomson Reuters Web of Science is depicted. The number increased rapidly over the past ten years, especially after 2006. Footnote 1

Fraction of the total of articles mentioning Mixed Method Research appearing in a given year, 1990–2017 (yearly values sum to 1). See footnote 1

The subject of mixed methods thus seems to have gained recognition among social scientists. The rapid rise of the number of articles mentioning the term raises various sociological questions. In this article, we address three of these questions. The first question concerns the degree to which the approach of MMR has become institutionalized within the field of the social sciences. Has MMR become a recognizable realm of knowledge production? Has its ascendance been accompanied by the production of textbooks, the founding of journals, and other indicators of institutionalization? The answer to this question provides an assessment of the current state of MMR. Once that is determined, the second question is how MMR’s rise can be understood. Where does the approach come from and how can its emergence and spread be understood? To answer this question, we use Pierre Bourdieu’s field analytical approach to science and academic institutions (Bourdieu 1975 , 1988 , 2004 , 2007 ; Bourdieu et al. 1991 ). We flesh out this approach in the next section. The third question concerns the substance of the MMR corpus seen in the light of the answers to the previous questions: how can we interpret the specific content of this approach in the context of its socio-historical genesis and institutionalization, and how can we understand its proposal for “mixing methods” in practice?

We proceed as follows. In the next section, we give an account of our theoretical approach. Then, in the third, we assess the degree of institutionalization of MMR, drawing on the indicators of academic institutionalization developed by Fleck et al. ( 2016 ). In the fourth section, we address the second question by examining the position of the academic entrepreneurs behind the rise of MMR. The aim is to understand these agents’ engagement in MMR, as well as its distinctive content as being informed by their position in this field. Viewing MMR as a position-taking of academic entrepreneurs, linked to their objective position in this field, allows us to reflect sociologically on the substance of the approach. We offer this reflection in the fifth section, where we indicate some problems with MMR. To get ahead of the discussion, these problems have to do with the framing of MMR as a distinct methodology and its specific conceptualization of data and methods of data analysis. We argue that these problems hinder fruitfully combining methods in a practical understanding of social scientific research. Finally, we conclude with some tentative proposals for an alternative view on combining methods.

A field approach

Our investigation of the rise and institutionalization of MMR relies on Bourdieu’s field approach. In general, field theory provides a model for the structural dimensions of practices. In fields, agents occupy a position relative to each other based on the differences in the volume and structure of their capital holdings. Capital can be seen as a resource that agents employ to exert power in the field. The distribution of the form of capital that is specific to the field serves as a principle of hierarchization in the field, differentiating those that hold more capital from those that hold less. This principle allows us to make a distinction between, respectively, the dominant and dominated factions in a field. However, in mature fields all agents—dominant and dominated—share an understanding of what is at stake in the field and tend to accept its principle of hierarchization. They are invested in the game, have an interest in it, and share the field’s illusio .

In the present case, we can interpret the various disciplines in the social sciences as more or less autonomous spaces that revolve around the shared stake in producing legitimate scientific knowledge by the standards of the field. What constitutes legitimate knowledge in these disciplinary fields, the production of which bestows scholars with prestige and an aura of competence, is in large part determined by the dominant agents in the field, who occupy positions in which most of the consecration of scientific work takes place. Scholars operating in a field are endowed with initial and accumulated field-specific capital, and are engaged in the struggle to gain additional capital (mainly scientific and intellectual prestige) in order to advance their position in the field. The main focus of these agents will generally be the disciplinary field in which they built their careers and invested their capital. These various disciplinary spaces are in turn part of a broader field of the social sciences in which the social status and prestige of the various disciplines is at stake. The ensuing disciplinary hierarchy is an important factor to take into account when analysing the circulation of new scientific products such as MMR. Furthermore, a distinction needs to be made between the academic and the scientific field. While the academic field revolves around universities and other degree-granting institutions, the stakes in the scientific field entail the production and valuation of knowledge. Of course, in modern science these fields are closely related, but they do not coincide (Gingras and Gemme 2006 ). For instance, part of the production of legitimate knowledge takes place outside of universities.

This framework makes it possible to contextualize the emergence of MMR in a socio-historical way. It also enables an assessment of some of the characteristics of MMR as a scientific product, since Bourdieu insists on the homology between the objective positions in a field and the position-takings of the agents who occupy these positions. As a new methodological approach, MMR is the result of the position-takings of its producers. The position-takings of the entrepreneurs at the core of MMR can therefore be seen as expressions in the struggles over the authority to define the proper methodology that underlies good scientific work regarding combining methods, and the potential rewards that come with being seen, by other agents, as authoritative on these matters. Possible rewards include a strengthened autonomy of the subfield of MMR and an improved position in the social-scientific field.

The role of these entrepreneurs or ‘intellectual leaders’ who can channel intellectual energy and can take the lead in institution building has been emphasised by sociologists of science as an important aspect of the production of knowledge that is visible and recognized as distinct in the larger scientific field (e.g., Mullins 1973 ; Collins 1998 ). According to Bourdieu, their position can, to a certain degree, explain the strategy they pursue and the options they perceive to be viable in the trade-off regarding the risks and potential rewards for their work.

We do not provide a full-fledged field analysis of MMR here. Rather, we use the concept as a heuristic device to account for the phenomenon of MMR in the social context in which it emerged and diffused. But first, we take stock of the current situation of MMR by focusing on the degree of institutionalization of MMR in the scientific field.

The institutionalization of mixed methods research

When discussing institutionalization, we have to be careful about what we mean by this term. More precisely, we need to be specific about the context and distinguish between institutionalization in the academic field and institutionalization within the scientific field (see Gingras and Gemme 2006 ; Sapiro et al. 2018 ). The first process refers to the establishment of degrees, curricula, faculties, etc., or to institutions tied to the academic bureaucracy and academic politics. The latter refers to the emergence of institutions that support the autonomization of scholarship such as scholarly associations and scientific journals. Since MMR is still a relatively young phenomenon and academic institutionalization tends to lag scientific institutionalization (e.g., for the case of sociology and psychology, see Sapiro et al. 2018 , p. 26), we mainly focus here on the latter dimension.

Drawing on criteria proposed by Fleck et al. ( 2016 ) for the institutionalization of academic disciplines, MMR seems to have achieved a significant degree of institutionalization within the scientific field. MMR quickly gained popularity in the first decade of the twenty-first century (e.g., Tashakkori and Teddlie 2010c , pp. 803–804). A distinct corpus of publications has been produced that aims to educate those interested in MMR and to function as a source of reference for researchers: there are a number of textbooks (e.g., Plowright 2010 ; Creswell and Plano Clark 2011 ; Teddlie and Tashakkori 2008 ); a handbook that is now in its second edition (Tashakkori and Teddlie 2003 , 2010a ); as well as a reader (Plano Clark and Creswell 2007 ). Furthermore, a journal (the Journal of Mixed Methods Research [ JMMR] ) was established in 2007. The JMMR was founded by the editors John Creswell and Abbas Tashakkori with the primary aim of “building an international and multidisciplinary community of mixed methods researchers.” Footnote 2 Contributions to the journal must “fit the definition of mixed methods research” Footnote 3 and explicitly integrate qualitative and quantitative aspects of research, either in an empirical study or in a more theoretical-methodologically oriented piece.

In addition, general textbooks on social research methods and methodology now increasingly devote sections to the issue of combining methods (e.g., Creswell 2008 ; Nagy Hesse-Biber and Leavy 2008 ; Bryman 2012 ), and MMR has been described as a “third paradigm” (Denscombe 2008 ), a “movement” (Bryman 2009 ), a “third methodology” (Tashakkori and Teddlie 2010b ), a “distinct approach” (Greene 2008 ) and an “emerging field” (Tashakkori and Teddlie 2011 ), defined by a common name (that sets it apart from other approaches to combining methods) and shared terminology (Tashakkori and Teddlie 2010b , p. 19). As a further indication of institutionalization, a research association (the Mixed Methods International Research Association—MMIRA) was founded in 2013 and its inaugural conference was held in 2014. Prior to this, there have been a number of conferences on MMR or occasions on which MMR was presented and discussed in other contexts. An example of the first is the conference on mixed method research design held in Basel in 2005. Starting also in 2005, the British Homerton School of Health Studies has organised a series of international conferences on mixed methods. Moreover, MMR was on the list of sessions in a number of conferences on qualitative research (see, e.g., Creswell 2012 ).

Another sign of institutionalization can be found in efforts to forge a common disciplinary identity by providing a narrative about its history. This involves the identification of precursors and pioneers as well as an interpretation of the process that gave rise to a distinctive set of ideas and practices. An explicit attempt to chart the early history of MMR is provided by Johnson and Gray ( 2010 ). They frame MMR as rooted in the philosophy of science, particularly as a way of thinking about science that has transcended some of the most salient historical oppositions in philosophy. Philosophers like Aristotle and Kant are portrayed as thinkers who sought to integrate opposing stances, forwarding “proto-mixed methods ideas” that exhibited the spirit of MMR (Johnson and Gray 2010 , p. 72, p. 86). In this capacity, they (as well as other philosophers like Vico and Montesquieu) are presented as part of MMR providing a philosophical validation of the project by presenting it as a continuation of ideas that have already been voiced by great thinkers in the past.

In the second edition of their textbook, Creswell and Plano Clark ( 2011 ) provide an overview of the history of MMR by identifying five historical stages: the first one being a precursor to the MMR approach, consisting of rather atomised attempts by different authors to combine methods in their research. For Creswell and Plano Clark, one of the earliest examples is Campbell and Fiske’s ( 1959 ) combination of quantitative methods to improve the validity of psychological scales that gave rise to the triangulation approach to research. However, they regard this and other studies that combined methods around that time, as “antecedents to (…) more systematic attempts to forge mixed methods into a complete research design” (Creswell and Plano Clark 2011 , p. 21), and hence label this stage as the “formative period” (ibid., p. 25). Their second stage consists of the emergence of MMR as an identifiable research strand, accompanied by a “paradigm debate” about the possibility of combining qualitative and quantitative data. They locate its beginnings in the late 1980s when researchers in various fields began to combine qualitative and quantitative methods (ibid., pp. 20–21). This provoked a discussion about the feasibility of combining data that were viewed as coming from very different philosophical points of view. The third stage, the “procedural development period,” saw an emphasis on developing more hands-on procedures for designing a mixed methods study, while stage four is identified as consisting of “advocacy and expansion” of MMR as a separate methodology, involving conferences, the establishment of a journal and the first edition of the aforementioned handbook (Tashakkori and Teddlie 2003 ). Finally, the fifth stage is seen as a “reflective period,” in which discussions about the unique philosophical underpinnings and the scientific position of MMR emerge.

Creswell and Plano Clark thus locate the emergence of “MMR proper” at the second stage, when researchers started to use both qualitative and quantitative methods within a single research effort. As reasons for the emergence of MMR at this stage they identify the growing complexity of research problems, the perception of qualitative research as a legitimate form of inquiry (also by quantitative researchers) and the increasing need qualitative researchers felt for generalising their findings. They therefore perceive the emergence of the practice of combining methods as a bottom up process that grew out of research practices, and at some point in time converged towards a more structural approach. Footnote 4 Historical accounts such as these add a cognitive dimension to the efforts to institutionalize MMR. They lay the groundwork for MMR as a separate subfield with its own identity, topics, problems and intellectual history. The use of terms such as “third paradigm” and “third methodology” also suggests that there is a tendency to perceive and promote MMR as a distinct and coherent way to do research.

In view of the brief exploration of the indicators of institutionalisation of MMR, it seems reasonable to conclude that MMR has become a recognizable and fairly institutionalized strand of research with its own identity and profile within the social scientific field. This can be seen both from the establishment of formal institutions (like associations and journals) and more informal ones that rely more on the tacit agreement between agents about “what MMR is” (an example of this, which we address later in the article, is the search for a common definition of MMR in order to fix the meaning of the term). The establishment of these institutions supports the autonomization of MMR and its emancipation from the field in which it originated, but in which it continues to be embedded. This way, it can be viewed as a semi-autonomous subfield within the larger field of the social sciences and as the result of a differentiation internal to this field (Steinmetz 2016 , p. 109). It is a space that is clearly embedded within this higher level field; for example, members of the subfield of MMR also qualify as members of the overarching field, and the allocation of the most valuable and current form of capital is determined there as well. Nevertheless, as a distinct subfield, it also has specific principles that govern the production of knowledge and the rewards of domination.

We return to the content and form of this specific knowledge later in the article. The next section addresses the question of the socio-genesis of MMR.

Where does mixed methods research come from?

The origins of the subfield of MMR lay in the broader field of social scientific disciplines. We interpret the positions of the scholars most involved in MMR (the “pioneers” or “scientific entrepreneurs”) as occupying particular positions within the larger academic and scientific field. Who, then, are the researchers at the heart of MMR? Leech ( 2010 ) interviewed 4 scholars (out of 6) that she identified as early developers of the field: Alan Bryman (UK; sociology), John Creswell (USA; educational psychology), Jennifer Greene (USA; educational psychology) and Janice Morse (USA; nursing and anthropology). Educated in the 1970s and early 1980s, all four of them indicated that they were initially trained in “quantitative methods” and later acquired skills in “qualitative methods.” For two of them (Bryman and Creswell) the impetus to learn qualitative methods was their involvement in writing on, and teaching of, research methods; for Greene and Morse the initial motivation was more instrumental and related to their concrete research activity at the time. Creswell describes himself as “a postpositivist in the 1970s, self-education as a constructivist through teaching qualitative courses in the 1980s, and advocacy for mixed methods (…) from the 1990s to the present” (Creswell 2011 , p. 269). Of this group, only Morse had the benefit of learning about qualitative methods as part of her educational training (in nursing and anthropology; Leech 2010 , p. 267). Independently, Creswell ( 2012 ) identified (in addition to Bryman, Greene and Morse) John Hunter, Allen Brewer (USA; Northwestern and Boston College) and Nigel Fielding (University of Surrey, UK) as important early movers in MMR.

The selections that Leech and Creswell make regarding the key actors are based on their close involvement with the “MMR movement.” It is corroborated by a simple analysis of the articles that appeared in the Journal of Mixed Methods Research ( JMMR ), founded in 2007 as an outlet for MMR.

Table 1 lists all the authors that have published in the issues of the journal since its first publication in 2007 and that have either received more than 14 (4%) of the citations allocated between the group of 343 authors (the TLCS score in Table 1 ), or have written more than 2 articles for the Journal (1.2% of all the articles that have appeared from 2007 until October 2013) together with their educational background (i.e., the discipline in which they completed their PhD).

All the members of Leech’s selection, except for Morse, and the members of Creswell’s selection (except Hunter, Brewer, and Fielding) are represented in the selection based on the entries in the JMMR . Footnote 5 The same holds for two of the three additional authors identified by Creswell. Hunter and Brewer have developed a somewhat different approach to combining methods that explicitly targets data gathering techniques and largely avoids epistemological discussions. In Brewer and Hunter ( 2006 ) they discuss the MMR approach very briefly and only include two references in their bibliography to the handbook of Tashakkori and Teddlie ( 2003 ), and at the end of 2013 they had not published in the JMMR . Fielding, meanwhile, has written two articles for the JMMR (Fielding and Cisneros-Puebla 2009 ; Fielding 2012 ). In general, it seems reasonable to assume that a publication in a journal that positions itself as part of a systematic attempt to build a research tradition, and can be viewed as part of a strategic effort to advance MMR as a distinct alternative to more “traditional” academic research—particularly in methods—at least signals a degree of adherence to the effort and acceptance of the rules of the game it lays out. This would locate Fielding closer to the MMR movement than the others.

The majority of the researchers listed in Table 1 have a background in psychology or social psychology (35%), and sociology (25%). Most of them work in the United States or are UK citizens, and the positions they occupied at the beginning of 2013 indicates that most of these are in applied research: educational research and educational psychology account for 50% of all the disciplinary occupations of the group that were still employed in academia. This is consistent with the view that MMR originated in applied disciplines and thematic studies like education and nursing, rather than “pure disciplines” like psychology and sociology (Tashakkori and Teddlie ( 2010b ), p. 32). Although most of the 20 individuals mentioned in Table 1 have taught methods courses in academic curricula (for 15 of them, we could determine that they were involved in the teaching of qualitative, quantitative, or mixed methods), there are few individuals with a background in statistics or a neighbouring discipline: only Amy Dellinger did her PhD in “research methodology.” In addition, as far as we could determine, only three individuals held a position in a methodological department at some time: Dellinger, Tony Onwuegbuzie, and Nancy Leech.

The pre-eminence of applied fields in MMR is supported when we turn our attention to the circulation of MMR. To assess this we proceeded as follows. We selected 10 categories in the Web of Science that form a rough representation of the space of social science disciplines, taking care to include the most important so-called “studies.” These thematically orientated, interdisciplinary research areas have progressively expanded since they emerged at the end of the 1960s as a critique of the traditional disciplines (Heilbron et al. 2017 ). For each category, we selected the 10 journals with the highest 5-year impact factor in their category in the period 2007–2015. The lists were compiled bi-annually over this period, resulting in 5 top ten lists for the following Web of Science categories: Economics, Psychology, Sociology, Anthropology, Political Science, Nursing, Education & Educational Research, Business, Cultural Studies, and Family Studies. After removing multiple occurring journals, we obtained a list of 164 journals.

We searched the titles and abstracts of the articles appearing in these journals over the period 1992–2016 for occurrences of the terms “mixed method” or “multiple methods” and variants thereof. We chose this particular period and combination of search terms to see if a shift from a more general use of the term “multiple methods” to “mixed methods” occurred following the institutionalization of MMR. In total, we found 797 articles (out of a total of 241,521 articles that appeared in these journals during that time), published in 95 different journals. Table 2 lists the 20 journals that contain at least 1% (8 articles) of the total amount of articles.

As is clear from Table 2 , the largest number of articles in the sample were published in journals in the field of nursing: 332 articles (42%) appeared in journals that can be assigned to this category. The next largest category is Education & Educational Research, to which 224 (28 percentage) of the articles can be allocated. By contrast, classical social science disciples are barely represented. In Table 2 only the journal Field Methods (Anthropology) and the Journal of Child Psychology and Psychiatry (Psychology) are related to classical disciplines. In Table 3 , the articles in the sample are categorized according to the disciplinary category of the journal in which they appeared. Overall, the traditional disciplines are clearly underrepresented: for the Economics category, for example, only the Journal of Economic Geography contains three articles that make a reference to mixed methods.

Focusing on the core MMR group, the top ten authors of the group together collect 458 citations from the 797 articles in the sample, locating them at the center of the citation network. Creswell is the most cited author (210 citations) and his work too receives most citations from journals in nursing and education studies.

The question whether a terminological shift has occurred from “multiple methods” to “mixed methods” must be answered affirmative for this sample. Prior to 2001 most articles (23 out of 31) refer to “multiple methods” or “multi-method” in their title or abstract, while the term “mixed methods” gains traction after 2001. This shift occurs first in journals in nursing studies, with journals in education studies following somewhat later. The same fields are also the first to cite the first textbooks and handbooks of MMR.

Taken together, these results corroborate the notion that MMR circulates mainly in nursing and education studies. How can this be understood from a field theoretical perspective? MMR can be seen as an innovation in the social scientific field, introducing a new methodology for combining existing methods in research. In general, innovation is a relatively risky strategy. Coming up with a truly rule-breaking innovation often involves a small probability of great success and a large probability of failure. However, it is important to add some nuance to this general observation. First, the risk an innovator faces depends on her position in the field. Agents occupying positions at the top of their field’s hierarchy are rich in specific capital and can more easily afford to undertake risky projects. In the scientific field, these are the agents richest in scientific capital. They have the knowledge, authority, and reputation (derived from recognition by their peers; Bourdieu 2004 , p. 34) that tends to decrease the risk they face and increase the chances of success. Moreover, the positions richest in scientific capital will, by definition, be the most consecrated ones. This consecration involves scientific rather than academic capital (cf. Wacquant 2013 , p. 20) and within disciplines these consecrated positions often are related to orthodox position-takings. This presents a paradox: although they have the capital to take more risks, they have also invested heavily in the orthodoxy of the field and will thus be reluctant to upset the status quo and risk destroying the value of their investment. This results in a tendency to take a more conservative stance, aimed at preserving the status quo in the field and defending their position. Footnote 6

For agents in dominated positions this logic is reversed. Possessing less scientific capital, they hold less consecrated positions and their chances of introducing successful innovations are much lower. This leaves them too with two possible strategies. One is to revert to a strategy of adaptation, accepting the established hierarchy in the field and embarking on a slow advancement to gain the necessary capital to make their mark from within the established order. However, Bourdieu notes that sometimes agents with a relatively marginal position in the field will engage in a “flight forward” and pursue higher risk strategies. Strategies promoting a heterodox approach challenge the orthodoxy and the principles of hierarchization of the field, and, if successful (which will be the case only with a small probability), can rake in significant profits by laying claim to a new orthodoxy (Bourdieu 1975 , p. 104; Bourdieu 1993 , pp. 116–117).

Thus, the coupling of innovative strategies to specific field positions based on the amount of scientific capital alone is not straightforward. It is therefore helpful to introduce a second differentiation in the field that, following Bourdieu ( 1975 , p. 103), is based on the differences between the expected profits from these strategies. Here a distinction can be made between an autonomous and a heteronomous pole of the field, i.e., between the purest, most “disinterested” positions and the most “temporal” positions that are more pervious to the heteronomous logic of social hierarchies outside the scientific field. Of course, this difference is a matter of degree, as even the works produced at the most heteronomous positions still have to adhere to the standards of the scientific field to be seen as legitimate. But within each discipline this dimension captures the difference between agents predominantly engaged in fundamental, scholarly work—“production solely for the producers”—and agents more involved in applied lines of research. The main component of the expected profit from innovation in the first case is scientific, whereas in the second case the balance tends to shift towards more temporal profits. This two-fold structuring of the field allows for a more nuanced conception of innovation than the dichotomy “conservative” versus “radical.” Holders of large amounts of scientific capital at the autonomous pole of the field are the producers and conservators of orthodoxy, producing and diffusing what can be called “orthodox innovations” through their control of relatively powerful networks of consecration and circulation. Innovations can be radical or revolutionary in a rational sense, but they tend to originate from questions raised by the orthodoxy of the field. Likewise, the strategy to innovate in this sense can be very risky in that success is in no way guaranteed, but the risk is mitigated by the assurance of peers that these are legitimate questions, tackled in a way that is consistent with orthodoxy and that does not threaten control of the consecration and circulation networks.

These producers are seen as intellectual leaders by most agents in the field, especially by those aspiring to become part of the specific networks of production and circulation they maintain. The exception are the agents located at the autonomous end of the field who possess less scientific capital and outright reject this orthodoxy produced by the field’s elite. Being strictly focused on the most autonomous principles of legitimacy, they are unable to accommodate and have no choice but to reject the orthodoxy. Their only hope is to engage in heterodox innovations that may one day become the new orthodoxy.

The issue is less antagonistic at the heteronomous side of the field, at least as far as the irreconcilable position-takings at the autonomous pole are concerned. The main battle here is also for scientific capital, but is complemented by the legitimacy it brings to gain access to those who are in power outside of the scientific field. At the dominant side, those with more scientific capital tend to have access to the field of power, agents who hold the most economic and cultural capital, for example by holding positions in policy advisory committees or company boards. The dominated groups at this side of the field will cater more to practitioners or professionals outside of the field of science.

Overall, there will be fewer innovations on this side. Moreover, innovative strategies will be less concerned with the intricacies of the pure discussions that prevail at the autonomous pole and be of a more practical nature, but pursued from different degrees of legitimacy according to the differences in scientific capital. This affects the form these more practical, process-orientated innovations take. At the dominant side of this pole, agents tend to accept the outcome of the struggles at the autonomous pole: they will accept the orthodoxy because mastery of this provides them with scientific capital and the legitimacy they need to gain access to those in power. In contrast, agents at the dominated side will be more interested in doing “what works,” neutralizing the points of conflict at the autonomous pole and deriving less value from strictly following the orthodoxy. This way, a four-fold classification of innovative strategies in the scientific field emerges (see Fig.  2 ) that helps to understand the context in which MMR was developed.

Scientific field and scientific innovation

In summary, the small group of researchers who have been identified as the core of MMR consist predominantly of users of methods, who were educated and have worked exclusively at US and British universities. The specific approach to combining methods that is proposed by MMR has been successful from an institutional point of view, achieving visibility through the foundation of a journal and association and a considerable output of core MMR scholars in terms of books, conference proceedings, and journal articles. Its origins and circulation in vocational studies rather than classical academic disciplines can be understood from the position these studies occupy in the scientific field and the kinds of position-taking and innovations these positions give rise to. This context allows a reflexive understanding of the content of MMR and the issues that are dominant in the approach. We turn to this in the next section.

Mixed methods research: Position-taking

The position of the subfield of MMR in the scientific field is related to the position-takings of agents that form the core of this subfield (Bourdieu 1993 , p. 35). The space of position takings, in turn, provides the framework to study the most salient issues that are debated within the subfield. Since we can consider MMR to be an emerging subfield, where positions and position takings are not as clearly defined as in more mature and settled fields, it comes as no surprise that there is a lively discussion of fundamental matters. Out of the various topics that are actively discussed, we have distilled three themes that are important for the way the subfield of MMR conveys its autonomy as a field and as a distinct approach to research. Footnote 7 In our view, these also represent the main problems with the way MMR approaches the issue of combining methods.

Methodology making and standardization

The first topic is that the approach is moving towards defining a unified MMR methodology. There are differences in opinion as to how this is best achieved, but there is widespread agreement that some kind of common methodological and conceptual foundation of MMR is needed. To this end, some propose a broad methodology that can serve as distinct marker of MMR research. For instance, in their introduction to the handbook, Tashakkori and Teddlie ( 2010b ) propose a definition of the methodology of mixed methods research as “the broad inquiry logic that guides the selection of specific methods and that is informed by conceptual positions common to mixed methods practitioners” (Tashakkori and Teddlie 2010b , p. 5). When they (later on in the text) provide two methodological principles that differentiate MMR from other communities of scholars, they state that they regard it as a “crucial mission” for the MMR community to generate distinct methodological principles (Tashakkori and Teddlie 2010b , pp. 16–17). They envision an MMR methodology that can function as a “guide” for selecting specific methods. Others are more in favour of finding a philosophical foundation that underlies MMR. For instance, Morgan ( 2007 ) and Hesse-Biber ( 2010 ) consider pragmatism as a philosophy that distinguishes MMR from qualitative (constructivism) and quantitative (positivist) research and that can provide a rationale for the paradigmatic pluralism typical of MMR.

Furthermore, there is wide agreement that some unified definition of MMR would be beneficial, but it is precisely here that there is a large variation in interpretations regarding the essentials of MMR. This can be seen in the plethora of definitions that have been proposed. Johnson et al. ( 2007 ) identified 19 alternative definitions of MMR at the time, out of which they condensed their own:

[MMR] is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purpose of breath and depth of understanding and corroboration. Footnote 8

Four years later, the issue is not settled yet. Creswell and Plano Clark ( 2011 ) list a number of authors who have proposed a different definition of MMR, and conclude that there is a common trend in the content of these definitions over time. They take the view that earlier texts on mixing methods stressed a “disentanglement of methods and philosophy,” while later texts locate the practice of mixing methods in “all phases of the research process” (Creswell and Plano Clark 2011 , p. 2). It would seem, then, that according to these authors the definitions of MMR have become more abstract, further away from the practicality of “merely” combining methods. Specifically, researchers now seem to speak of mixing higher order concepts: some speak of mixing methodologies, others refer to mixing “research approaches,” or combining “types of research,” or engage in “multiple ways of seeing the social world” (Creswell and Plano Clark 2011 ).

This shift is in line with the direction in which MMR has developed and that emphasises practical ‘manuals’ and schemas for conducting research. A relatively large portion of the MMR literature is devoted to classifications of mixed methods designs. These classifications provide the basis for typologies that, in turn, provide guidelines to conduct MMR in a concrete research project. Tashakkori and Teddlie ( 2003 ) view these typologies as important elements of the organizational structure and legitimacy of the field. In addition, Leech and Onwuegbuzie ( 2009 ) see typologies as helpful guides for researchers and of pedagogical value (Leech and Onwuegbuzie 2009 , p. 272). Proposals for typologies can be found in textbooks, articles, and contributions to the handbook(s). For example, Creswell et al. ( 2003 , pp. 169-170) reviewed a number of studies and identified 8 different ways to classify MMR studies. This list was updated and extended by Creswell and Plano Clark ( 2011 , pp. 56-59) to 15 typologies. Leech and Onwuegbuzie ( 2009 ) identified 35 different research designs in the contributions to Teddlie and Tashakkori (2003) alone, and proposed their own three-dimensional typology that resulted in 8 different types of mixed methods studies. As another example of the ubiquity of these typologies, Nastasi et al. ( 2010 ) classified a large number of existing typologies in MMR into 7”meta-typologies” that each emphasize different aspects of the research process as important markers for MMR. According to the authors, these typologies have the same function in MMR as the more familiar names of “qualitative” or “quantitative” methods (e.g., “content analysis” or “structural equation modelling”) have: to signal readers of research what is going on, what procedures have been followed, how to interpret results, etc. (see also Creswell et al. 2003 , pp. 162–163). The criteria underlying these typologies mainly have to do with the degree of mixing (e.g., are methods mixed throughout the research project or not?), the timing (e.g., sequential or concurrent mixing of methods) and the emphasis (e.g., is one approach dominant, or do they have equal status?).

We find this strong drive to develop methodologies, definitions, and typologies of MMR as guides to valid mixed methods research problematic. What it amounts to in practice is a methodology that lays out the basic guidelines for doing MMR in a “proper way.” This entails the danger of straight-jacketing reflection about the use of methods, decoupling it from theoretical and empirical considerations, thus favouring the unreflexive use of a standard methodology. Researchers are asked to make a choice for a particular MMR design and adhere to the guidelines for a “proper” MMR study. Such methodological prescription diametrically opposes the initial critique of the mechanical and unreflexive use of methods. The insight offered by Bourdieu’s notion of reflexivity is, on the contrary, that the actual research practice is fundamentally open in terms of being guided by a logic of practice that cannot be captured by a preconceived and all-encompassing logic independent of that practice. Reflexivity in this view cannot be achieved by hiding behind the construct of a standardized methodology—of whatever signature—it can only be achieved by objectifying the process of objectification that goes on within the context of the field in which the researcher is embedded. This reflexivity, then, requires an analysis of the position of the researcher as a critical component of the research process, both as the embodiment of past choices that have consequences for the strategic position in the scientific field, and as predispositions regarding the choice for the subject and content of a research project. By adding the insight of STS researchers that the point of deconstructing science and technology is not so much to offer a new best way of doing science or technology, but to provide insights into the critical moments in research (for a take on such a debate, see, for example, Edge 1995 , pp. 16–20), this calls for a sociology of science that takes methods much more seriously as objects of study. Such a programme should be based on studying the process of codification and standardization of methods in their historical context of production, circulation, and use. It would provide a basis for a sociological understanding of methods that can illuminate the critical moments in research alluded to above, enabling a systematic reflection on the process of objectification. This, in turn, allows a more sophisticated validation of using—and combining—methods than relying on prescribed methodologies.

The role of epistemology

The second theme discussed in a large number of contributions is the role epistemology plays in MMR. In a sense, epistemology provides the lifeblood for MMR in that methods in MMR are mainly seen in epistemological terms. This interpretation of methods is at the core of the knowledge claim of MMR practitioners, i.e., that the mixing of methods means mixing broad, different ways of knowing, which leads to better knowledge of the research object. It is also part of the identity that MMR consciously assumes, and that serves to set it apart from previous, more practical attempts to combine methods. This can be seen in the historical overview that Creswell and Plano Clark ( 2011 ) presented and that was discussed above. This reading, in which combining methods has evolved from the rather unproblematic level (one could alternatively say “naïve” or “unaware”) of instrumental use of various tools and techniques into an act that requires deeper thinking on a methodological and epistemological level, provides the legitimacy of MMR.

At the core of the MMR approach we thus find that methods are seen as unproblematic representations of different epistemologies. But this leads to a paradox, since the epistemological frameworks need to be held flexible enough to allow researchers to integrate elements of each of them (in the shape of methods) into one MMR design. As a consequence, the issue becomes the following: methods need to be disengaged from too strict an interpretation of the epistemological context in which they were developed in order for them to be “mixable,”’, but, at the same time, they must keep the epistemology attributed to them firmly intact.

In the MMR discourse two epistemological positions are identified that matter most: a positivist approach that gives rise to quantitative methods and a constructivist approach that is home to qualitative methods. For MMR to be a feasible endeavour, the differences between both forms of research must be defined as reconcilable. This position necessitates an engagement with those who hold that the quantitative/qualitative dichotomy is unbridgeable. Within MMR an interesting way of doing so has emerged. In the first issue of the Journal of Mixed Methods Research, Morgan ( 2007 ) frames the debate about research methodology in the social sciences in terms of Kuhnian paradigms, and he argues that the pioneers of the emancipation of qualitative research methods used a particular interpretation of the paradigm-concept to state their case against the then dominant paradigm in the social sciences. According to Morgan, they interpreted a paradigm mainly in metaphysical terms, stressing the connections among the trinity of ontology, epistemology, and methodology as used in the philosophy of knowledge (Morgan 2007 , p. 57). This allowed these scholars to depict the line between research traditions in stark, contrasting terms, using Kuhn’s idea of “incommensurability” in the sense of its “early Kuhn” interpretation. This strategy fixed the contrast between the proposed alternative approach (a “constructivist paradigm”), and the traditional approach (constructed as “the positivist paradigm”) to research as a whole, and offered the alternative approach as a valid option rooted in the philosophy of knowledge. Morgan focuses especially on the work of Egon Guba and Yvonne Lincoln who developed what they initially termed a “naturalistic paradigm” as an alternative to their perception of positivism in the social sciences (e.g., Guba and Lincoln 1985 ). Footnote 9 MMR requires a more flexible or “a-paradigmatic stance” towards research, which would entail that “in real-world practice, methods can be separated from the epistemology out of which they emerged” (Patton 2002 , quoted in Tashakkori and Teddlie 2010b , p. 14).

This proposal of an ‘interpretative flexibility’ (Bijker 1987 , 1997 ) regarding paradigms is an interesting proposition. But it immediately raises the question: why stop there? Why not take a deeper look into the epistemological technology of methods themselves, to let the muted components speak up in order to look for alternative “mixing interfaces” that could potentially provide equally valid benefits in terms of the understanding of a research object? The answer, of course, was already seen above. It is that the MMR approach requires situating methods epistemologically in order to keep them intact as unproblematic mediators of specific epistemologies and, thus, make the methodological prescriptions work. There are several problems with this. First, seeing methods solely through an epistemological lens is problematic, but it would be less consequential if it were applied to multiple elements of methods separately. This would at least allow a look under the hood of a method, and new ways of mixing methods could be opened up that go beyond the crude “qualitative” versus “quantitative” dichotomy. Second, there is also the issue of the ontological dimension of methods that is disregarded in an exclusively epistemological framing of methods (e.g., Law 2004 ). Taking this ontological dimension seriously has at least two important facets. First, it draws attention to the ontological assumptions that are woven into methods in their respective fields of production and that are imported into fields of users. Second, it entails the ontological consequences of practising methods: using, applying, and referring to methods and the realities this produces. This latter facet brings the world-making and boundary-drawing capacities of methods to the fore. Both facets are ignored in MMR. We say more about the first facet in the next section. With regard to the second facet, a crucial element concerns the data that are generated, collected, and analysed in a research project. But rather than problematizing the link between the performativity of methods and the data that are enacted within the frame of a method, here too MMR relies on a dichotomy: that between quantitative and qualitative data. Methods are primarily viewed as ways of gathering data or as analytic techniques dealing with a specific kind of data. Methods and data are conceptualised intertwiningly: methods too are seen as either quantitative or qualitative (often written as QUANT and QUAL in the literature), and perform the role of linking epistemology and data. In the final analysis, the MMR approach is based on the epistemological legitimization of the dichotomy between qualitative and quantitative data in order to define and combine methods: data obtain epistemological currency through the supposed in-severable link to certain methods, and methods are reduced to the role of acting as neutral mediators between them.

In this way, methods are effectively reduced to, on the one hand, placeholders for epistemological paradigms and, on the other hand, mediators between one kind of data and the appropriate epistemology. To put it bluntly, the name “mixed methods research” is actually a misnomer, because what is mixed are paradigms or “approaches,” not methods. Thus, the act of mixing methods à la MMR has the paradoxical effect of encouraging a crude black box approach to methods. This is a third problematic characteristic of MMR, because it hinders a detailed study of methods that can lead to a much richer perspective on mixing methods.

Black boxed methods and how to open them

The third problem that we identified with the MMR approach, then, is that with the impetus to standardize the MMR methodology by fixing methods epistemologically, complemented by a dichotomous view of data, they are, in the words of philosopher Bruno Latour, “blackboxed.” This is a peculiar result of the prescription for mixing methods as proposed by MMR that thus not only denies practice and the ontological dimensions of methods and data, but also casts methods in the role of unyielding black boxes. Footnote 10 With this in mind, it will come as no surprise that most foundational contributions to the MMR literature do not explicitly define what a method is, nor that they do not provide an elaborative historical account of individual methods. The particular framing of methods in MMR results in a blind spot for the historical and social context of the production and circulation of methods as intellectual products. Instead it chooses to reify the boundaries that are drawn between “qualitative” and “quantitative” methods and reproduce them in the methodology it proposes. Footnote 11 This is an example of “circulation without context” (Bourdieu 2002 , p. 4): classifications that are constructed in the field of use or reception without taking the constellation within the field of production seriously.

Of course, this does not mean that the reality of the differences between quantitative and qualitative research must be denied. These labels are sticky and symbolically laden. They have come, in many ways, to represent “two cultures” (Goertz and Mahony 2012 ) of research, institutionalised in academia, and the effects of nominally “belonging” to (or being assigned to) one particular category have very real consequences in terms of, for instance, access to research grants and specific journals. However, if the goal of an approach such as MMR is to open up new pathways in social science research, (and why should that not be the case?) it is hard to see how that is accomplished by defining the act of combining methods solely in terms of reified differences between research using qualitative and quantitative data. In our view, methods are far richer and more interesting constructs than that, and a practice of combining methods in research should reflect that. Footnote 12

Addressing these problems entices a reflection on methods and using (multiple) methods that is missing in the MMR perspective. A fruitful way to open up the black boxes and take into account the epistemological and ontological facets of methods is to make them, and their use, the object of sociological-historical investigation. Methods are constituted through particular practices. In Bourdieusian terms, they are objectifications of the subjectively understood practices of scientists “in other fields.” Rather than basing a practice of combining methods on an uncritical acceptance of the historically grown classification of types of social research (and using these as the building stones of a methodology of mixing methods), we propose the development of a multifaceted approach that is based on a study of the different socio-historical contexts and practices in which methods developed and circulated.

A sociological understanding of methods based on these premises provides the tools to break with the dichotomously designed interface for combining methods in MMR. Instead, focusing on the historical and social contexts of production and use can reveal the traces that these contexts leave, both in the internal structure of methods, how they are perceived, how they are put into practice, and how this practice informs the ontological effects of methods. Seeing methods as complex technologies, with a history that entails the struggles among the different agents involved in their production, and use opens the way to identify multiple interfaces for combining them: the one-sided boxes become polyhedra. The critical study of methods as “objects of objectification” also entices analyses of the way in which methods intervene between subject (researcher) and object and the way in which different methods are employed in practice to draw this boundary differently. The reflexive position generated by such a systematic juxtaposition of methods is a fruitful basis to come to a richer perspective on combining methods.

We critically reviewed the emerging practice of combining methods under the label of MMR. MMR challenges the mono-method approaches that are still dominant in the social sciences, and this is both refreshing and important. Combining methods should indeed be taken much more seriously in the social sciences.

However, the direction that the practice of combining methods is taking under the MMR approach seems problematic to us. We identified three main concerns. First, MMR scholars seem to be committed to designing a standardized methodological framework for combining methods. This is unfortunate, since it amounts to enforcing an unnecessary codification of aspects of research practices that should not be formally standardized. Second, MMR constructs methods as unproblematic representations of an epistemology. Although methods must be separable from their native epistemology for MMR to work, at the same time they have to be nested within a qualitative or a quantitative research approach, which are characterized by the data they use. By this logic, combining quantitative methods with other quantitative methods, or qualitative methods with other qualitative methods, cannot offer the same benefits: they originate from the same way of viewing and knowing the world, so it would have the same effect as blending two gradations of the same colour paint. The importance attached to the epistemological grounding of methods and data in MMR also disregards the ontological aspects of methods. In this article, we are arguing that this one-sided perspective is problematic. Seeing combining methods as equivalent to combining epistemologies that are somehow pure and internally homogeneous because they can be placed in a qualitative or quantitative framework essentially amounts to reifying these categories.

It also leads to the third problem: the black boxing of methods as neutral mediators between these epistemologies and data. This not only constitutes a problem for trying to understand methods as intellectual products, but also for regarding the practice of combining methods, because it ignores the social-historical context of the development of individual methods and hinders a sociologically grounded notion of combining methods.

We proceed from a different perspective on methods. In our view, methods are complex constructions. They are world-making technologies that encapsulate different assumptions on causality, rely on different conceptual relations and categorizations, allow for different degrees of emergence, and employ different theories of the data that they internalise as objects of analysis. Even more importantly, their current form as intellectual products cannot be separated from the historical context of their production, circulation, and use.

A fully developed exposition of such an approach will have to await further work. Footnote 13 So far, the sociological study of methods has not (yet) developed into a consistent research programme, but important elements can be derived from existing contributions such as MacKenzie ( 1981 ), Chapoulie ( 1984 ), Platt ( 1996 ), Freeman ( 2004 ), and Desrosières ( 2008a , b ). The work on the “social life of methods” (e.g., Savage 2013 ) also contains important leads for the development of a systematic sociological approach to method production and circulation. Based on the discussion in this article and the contributions listed above, some tantalizing questions can be formulated. How are methods and their elements objectified? How are epistemology and ontology defined in different fields and how do those definitions feed into methods? How do they circulate and how are they translated and used in different contexts? What are the main controversies in fields of users and how are these related to the field of production? What are the homologies between these fields?

Setting out to answer these questions opens up the possibility of exploring other interesting combinations of methods that emerge from the combination of different practices, situated in different historical and epistemological contexts, and with their unique set of interpretations regarding their constituent elements. One of these must surely be the data-theoretical elements that different methods incorporate. The problematization of data has become all the more pressing now that the debate about the consequences of “big data” for social scientific practices has become prominent (Savage and Burrows 2007 ; Levallois et al. 2013 ; Burrows and Savage 2014 ). Whereas MMR emphasizes the dichotomy between qualitative and quantitative data, a historical analysis of the production and use of methods can explore the more subtle, different interpretations and enactments of the “same” data. These differences inform method construction, controversies surrounding methods and, hence, opportunities for combining methods. These could then be constructed based on alternative conceptualisations of data. Again, while in some contexts it might be enlightening to rely on the distinction between data as qualitative or quantitative, and to combine methods based on this categorization, it is an exciting possibility that in other research contexts other conceptualisations of data might be of more value to enhance a specific (contextual) form of knowledge.

Change history

06 may 2019.

Unfortunately, figure 2 was incorrectly published.

The search term used was “mixed method*” in the “topic” search field of SSCI, A&HCI, and CPCI-SSH as contained in the Web of Science. A Google NGram search (not shown) confirmed this pattern. The results of a search for “mixed methods” and “mixed methods research” showed a very steep increase after 1994: in the first case, the normalized share in the total corpus increased by 855% from 1994 till 2008. Also, Creswell ( 2012 ) reports an almost hundred-fold increase in the number of theses and dissertations with mixed methods’ in the citation and abstract (from 26 in 1990–1994 to 2524 in 2005–2009).

Retrieved from https://uk.sagepub.com/en-gb/eur/journal-of-mixed-methods-research/journal201775#aims-and-scope on 1/17/2019.

In terms of antecedents of mixed methods research, it is interesting to note that Bourdieu, whose sociology of science we draw on, was, from his earliest studies in Algeria onwards, a strong advocate of combining research methods. He made it into a central characteristic of his approach to social science in Bourdieu et al. ( 1991 [1968]). His approach, as we see below, was very different from the one now proposed under the banner of MMR. Significantly, there is no mention of Bourdieu’s take on combining methods in any of the sources we studied.

Morse’s example in particular warns us that restricting the analysis to the authors that have published in the JMMR runs the risk of missing some important contributors to the spread of MMR through the social sciences. On her website, Morse lists 11 publications (journal articles, book chapters, and books) that explicitly make reference to mixed methods (and a substantial number of other publications are about methodological aspects of research), so the fact that she has not (yet) published in the JMMR cannot, by itself, be taken as an indication of a lesser involvement with the practice of combining methods. See the website of Janice Morse at https://faculty.utah.edu/u0556920-Janice_Morse_RN,_PhD,_FAAN/hm/index.hml accessed 1/17/2019.

Bourdieu ( 1999 , p. 26) mentions that one has to be a scientific capitalist to be able to start a scientific revolution. But here he refers explicitly to the autonomy of the scientific field, making it virtually impossible for amateurs to stand up against the historically accumulated capital in the field and incite a revolution.

The themes summarize the key issues through which MMR as a group comes “into difference” (Bourdieu 1993 , p. 32). Of course, as in any (sub)field, the agents identified above often differ in their opinions on some of these key issues or disagree on the answer to the question if there should be a high degree of convergence of opinions at all. For instance, Bryman ( 2009 ) worried that MMR could become “a ghetto.” For him, the institutional landmarks of having a journal, conferences, and a handbook increase the risk of “not considering the whole range of possibilities.” He added: “I don’t regard it as a field, I kind of think of it as a way of thinking about how you go about research.” (Bryman, cited in Leech 2010 , p. 261). It is interesting to note that Bryman, like fellow sociologists Morgan and Denscombe, had published only one paper in the JMMR by the end of 2016 (Bryman passed away in June of 2017). Although these papers are among the most cited papers in the journal (see Table 1 ), this low number is consistent with the more eclectic approach that Bryman proposed.

Johnson, Onwuegbuzie, and Turner ( 2007 , p. 123).

Guba and Lincoln ( 1985 ) discuss the features of their version of a positivistic approach mainly in ontological and epistemological terms, but they are also careful to distinguish the opposition between naturalistic and positivist approaches from the difference between what they call the quantitative and the qualitative paradigms. Since they go on to state that, in principle, quantitative methods can be used within a naturalistic approach (although in practice, qualitative methods would be preferred by researchers embracing this paradigm), they seem to locate methods on a somewhat “lower,” i.e., less incommensurable level. However, in their later work (both together as well as with others or individually) and that of others in their wake, there seems to have been a shift towards a stricter interpretation of the qualitative/quantitative divide in metaphysical terms, enabling Teddlie and Tashakkori (2010b) to label this group “purists” (Tashakkori and Teddlie 2010b , p. 13).

See, for instance, Onwuegbuzie et al.’s ( 2011 ) classification of 58 qualitative data analysis techniques and 18 quantitative data analysis techniques.

This can also be seen in Morgan’s ( 2018 ) response to Sandelowski’s ( 2014 ) critique of the binary distinctions in MMR between qualitative and quantitative research approaches and methods. Morgan denounces the essentialist approach to categorizing qualitative and quantitative research in favor of a categorization based on “family resemblances,” in which he draws on Wittgenstein. However, this denies the fact that the essentialist way of categorizing is very common in the MMR corpus, particularly in textbooks and manuals (e.g., Plano Clark and Ivankova 2016 ). Moreover, and more importantly, he still does not extend this non-essentialist model of categorization to the level of methods, referring, for instance, to the different strengths of qualitative and quantitative methods in mixed methods studies (Morgan 2018 , p. 276).

While it goes beyond the scope of this article to delve into the history of the qualitative-quantitative divide in the social sciences, some broad observations can be made here. The history of method use in the social sciences can briefly be summarized as first, a rather fluid use of what can retrospectively be called different methods in large scale research projects—such as the Yankee City study of Lloyd Warner and his associates (see Platt 1996 , p. 102), the study on union democracy of Lipset et al. ( 1956 ), and the Marienthal study by Lazarsfeld and his associates (Jahoda et al. 1933 ); see Brewer and Hunter ( 2006 , p. xvi)—followed by an increasing emphasis on quantitative data and the objectification and standardization of methods. The rise of research using qualitative data can be understood as a reaction against this use and interpretation of method in the social sciences. However, out of the ensuing clash a new, still dominant classification of methods emerged, one that relies on the framing of methods as either “qualitative” or “quantitative.” Moreover, these labels have become synonymous with epistemological positions that are reproduced in MMR.

A proposal to come to such an approach can be found in Timans ( 2015 ).

Bijker, W. (1987). The social construction of bakelite: Toward a theory of invention. In W. Bijker, T. Hughes, T. Pinch, & D. Douglas (Eds.), The social construction of technological systems: New directions in the sociology and history of technology . Cambridge, MA: MIT press.

Google Scholar  

Bijker, W. (1997). Of bicycles, bakelites, and bulbs: Toward a theory of sociotechnical change . Cambridge, MA: MIT Press.

Bourdieu, P. (1975). La spécifité du champ scientifique et les conditions sociales du progrès de la raison. Sociologie et Sociétés, 7 (1), 91–118.

Article   Google Scholar  

Bourdieu, P. (1988). Homo academicus . Stanford, CA: Stanford University Press.

Bourdieu, P. (1993). The field of cultural production . Cambridge, UK: Polity Press.

Bourdieu, P. (1999). Les historiens et la sociologie de Pierre Bourdieu. Le Bulletin de la Société d'Histoire Moderne et Contemporaine/SHMC, 1999 (3&4), 4–27.

Bourdieu, P. (2002). Les conditions sociales de la circulation internationale des idées. Actes de la Recherche en Sciences Sociales, 145 (5), 3–8.

Bourdieu, P. (2004). Science of science and reflexivity . Cambridge, UK: Polity.

Bourdieu, P. (2007). Sketch for a self-analysis . Cambridge, UK: Polity.

Bourdieu, P., Chamboredon, J., & Passeron, J. (1991). The craft of sociology: Epistemological preliminaries . Berlin, Germany: De Gruyter.

Book   Google Scholar  

Brewer, J., & Hunter, A. (2006). Multimethod research: A synthesis of styles . London, UK: Sage.

Bryman, A. (2009). Sage Methodspace: Alan Bryman on research methods. Retrieved from http://www.youtube.com/watch?v=bHzM9RlO6j0 . Accessed 3/7/2019.

Bryman, A. (2012). Social research methods . Oxford, UK: Oxford University Press.

Burrows, R., & Savage, M. (2014). After the crisis? Big data and the methodological challenges of empirical sociology. Big Data & Society, 1 (1), 1–6.

Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56 (2), 81–105.

Chapoulie, J. (1984). Everett C. Hughes et le développement du travail de terrain en sociologie. Revue Française de Sociologie, 25 (4), 582–608.

Collins, R. (1998). The sociology of philosophies: A global theory of intellectual change . Cambridge, MA: Harvard University Press.

Creswell, J. (2008). Research design: Qualitative, quantitative, and mixed methods approaches . Thousand Oaks, CA: Sage.

Creswell, J. (2011). Controversies in mixed methods research. In N. Denzin & Y. Lincoln (Eds.), The Sage handbook of qualitative research . Thousand Oaks, CA: Sage.

Creswell, J. (2012). Qualitative inquiry and research design: Choosing among five approaches . Thousand Oaks, CA: Sage.

Creswell, J., & Plano Clark, V. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

Creswell, J., Plano Clark, V., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Denscombe, M. (2008). Communities of practice a research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2 (3), 270–283.

Desrosières, A. (2008a). Pour une sociologie historique de la quantification - L’Argument statistique I . Paris, France: Presses des Mines.

Desrosières, A. (2008b). Gouverner par les nombres - L’Argument statistique II . Paris, France: Presses des Mines.

Edge, D. (1995). Reinventing the wheel. In D. Edge, S. Jasanof, G. Markle, J. Petersen, & T. Pinch (Eds.), Handbook of science and technology studies . Thousand Oaks, CA: Sage.

Fielding, N. (2012). Triangulation and mixed methods designs data integration with new research technologies. Journal of Mixed Methods Research, 6 (2), 124–136.

Fielding, N., & Cisneros-Puebla, C. (2009). CAQDAS-GIS convergence: Toward a new integrated mixed method research practice? Journal of Mixed Methods Research, 3 (4), 349–370.

Fleck, C., Heilbron, J., Karady, V., & Sapiro, G. (2016). Handbook of indicators of institutionalization of academic disciplines in SSH. Serendipities, Journal for the Sociology and History of the Social Sciences, 1 (1) Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/1 . Accessed 10/10/2018.

Freeman, L. (2004). The development of social network analysis: A study in the sociology of science . Vancouver, Canada: Empirical Press.

Gingras, Y., & Gemme, B. (2006). L’Emprise du champ scientifique sur le champ universitaire et ses effets. Actes de la Recherche en Sciences Sociales, 164 , 51–60.

Goertz, G., & Mahony, J. (2012). A tale of two cultures: Qualitative and quantitative research in the social sciences . Princeton: Princeton University Press.

Greene, J. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2 (1), 7–22.

Guba, E., & Lincoln, Y. (1985). Naturalistic inquiry . Thousand Oaks, CA: Sage.

Heilbron, J., Bedecarré, M., & Timans, R. (2017). European journals in the social sciences and humanities. Serendipities, Journal for the Sociology and History of the Social Sciences, 2 (1), 33–49 Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/5 . Accessed 10/10/2018.

Hesse-Biber, S. (2010). Mixed methods research: Merging theory with practice . New York, NY: Guilford Press.

Jahoda, M., Lazarsfeld, P., & Zeisel, H. (1933). Die Arbeitslosen von Marienthal. Psychologische Monographen, 5 .

Johnson, R., & Gray, R. (2010). A history of philosophical and theoretical issues for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Johnson, R., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1 (2), 112–133.

Law, J. (2004). After method: Mess in social science research . London, UK: Routledge.

Leech, N. (2010). Interviews with the early developers of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Leech, N., & Onwuegbuzie, T. (2009). A typology of mixed methods research designs. Quality & Quantity, 43 (2), 265–275.

Levallois, C., Steinmetz, S., & Wouters, P. (2013). Sloppy data floods or precise social science methodologies? In P. Wouters, A. Beaulieu, A. Scharnhorst, & S. Wyatt (Eds.), Virtual knowledge . Cambridge, MA: MIT Press.

Lipset, S., Trow, M., & Coleman, J. (1956). Union democracy: The internal politics of the international typographical union . Glencoe, UK: Free Press.

MacKenzie, D. (1981). Statistics in Britain: 1865–1930: The social construction of scientific knowledge . Edinburgh, UK: Edinburgh University Press.

Morgan, D. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1 (1), 48–76.

Morgan, D. (2018). Living with blurry boundaries: The values of distinguishing between qualitative and quantitative research. Journal of Mixed Methods Research, 12 (3), 268–276.

Mullins, N. (1973). Theories and theory groups in contemporary American sociology . New York, NY: Harper & Row.

Nagy Hesse-Biber, S., & Leavy, P. (Eds.). (2008). Handbook of emergent methods . New York, NY and London, UK: Guilford Press.

Nastasi, B., Hitchcock, J., & Brown, L. (2010). An inclusive framework for conceptualizing mixed method design typologies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Onwuegbuzie, A., Leech, N., & Collins, K. (2011). Toward a new era for conducting mixed analyses: The role of quantitative dominant and qualitative dominant crossover mixed analyses. In M. Williams & P. Vogt (Eds.), Handbook of innovation in social research methods . London, UK: Sage.

Patton, M. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

Plano Clark, V., & Creswell, J. (Eds.). (2007). The mixed methods reader . Thousand Oaks, CA: Sage.

Plano Clark, V., & Ivankova, N. (2016). Mixed methods research: A guide to the field . Thousand Oaks, CA: Sage.

Platt, J. (1996). A history of sociological research methods in America: 1920–1960 . Cambridge, UK: Cambridge University Press.

Plowright, D. (2010). Using mixed methods – Frameworks for an integrated methodology . Thousand Oaks, CA: Sage.

Sandelowski, M. (2014). Unmixing mixed methods research. Research in Nursing & Health, 37 (1), 3–8.

Sapiro, G., Brun, E., & Fordant, C. (2018). The rise of the social sciences and humanities in France: Iinstitutionalization, professionalization and autonomization. In C. Fleck, M. Duller, & V. Karady (Eds.), Shaping human science disciplines: Institutional developments in Europe and beyond . Basingstoke, UK: Palgrave.

Savage, M. (2013). The ‘social life of methods’: A critical introduction. Theory, Culture and Society, 30 (4), 3–21.

Savage, M., & Burrows, R. (2007). The coming crisis of empirical sociology. Sociology, 41 (5), 885–899.

Steinmetz, G. (2016). Social fields, subfields and social spaces at the scale of empires: Explaining the colonial state and colonial sociology. The Sociological Review, 64(2_suppl). 98-123.

Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (Eds.). (2010a). Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2010b). Overview of contemporary issues in mixed methods. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Chapter   Google Scholar  

Tashakkori, A., & Teddlie, C. (2010c). Epilogue. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2011). Mixed methods research: Contemporary issues in an emerging field. In N. Denzin & Y. Lincoln (Eds.), The SAGE handbook of qualitative research (4th ed.). Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2008). Foundations of mixed methods research – Integrating quantitative and qualitative approaches in the social and behavioral sciences . Thousand Oaks, CA: Sage.

Timans, R. (2015). Studying the Dutch business elite: Relational concepts and methods . Doctoral dissertation, Erasmus University Rotterdam, the Netherlands.

Wacquant, L. (2013). Bourdieu 1993: A case study in scientific consecration. Sociology, 47 (1), 15–29.

Download references

Acknowledgments

This research is part of the Interco-SSH project, funded by the European Union under the 7th Research Framework Programme (grant agreement no. 319974). Johan Heilbron would like to thank Louise and John Steffens, members of the Friends Founders’ Circle, who assisted his stay at the Princeton Institute for Advanced Study in 2017-18 during which he completed his part of the present article.

Author information

Authors and affiliations.

Erasmus Centre for Economic Sociology (ECES), Erasmus University Rotterdam, Rotterdam, Netherlands

Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, Netherlands

Paul Wouters

Erasmus Centre for Economic Sociology (ECES), Rotterdam and Centre européen de sociologie et de science politique de la Sorbonne (CESSP-CNRS-EHESS), Paris, France

Johan Heilbron

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rob Timans .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Timans, R., Wouters, P. & Heilbron, J. Mixed methods research: what it is and what it could be. Theor Soc 48 , 193–216 (2019). https://doi.org/10.1007/s11186-019-09345-5

Download citation

Published : 29 March 2019

Issue Date : 01 April 2019

DOI : https://doi.org/10.1007/s11186-019-09345-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Data, field analysis
• Mixed methods research
• Multiple methods
• Reflexivity
• Sociology of science
• Find a journal
• Publish with us
• Track your research

Research Design Review

A discussion of qualitative & quantitative research design, working with multiple methods in qualitative research: 7 unique researcher skills.

Multi-method research enables the qualitative researcher to study relatively complex entities or phenomena in a way that is holistic and retains meaning.  The purpose is to tackle the research objective from all the methodological sides.  Rather than pigeonholing the research into a series of IDIs, focus groups, or observations, the multi-method approach frees the researcher into total immersion with the subject matter.

 A multi-method approach to conduct case-centered research requires sufficient time and resources – in terms of financial and human support – as well as unique skills on the part of the researcher. A researcher adept at single-method research – e.g., an IDI study to examine employee attitudes toward new company policies, a focus group study concerning the drinking habits among teenagers – is not necessarily equipped with the appropriate skills for conducting multi-method studies. Here are seven important skills required of the researcher who plans to use multiple methods to conduct case-centered – case study or narrative – research:

• Experience & expertise in different qualitative research methods – IDIs, group discussions, observation, content analysis.
• Exceptional organizational skills, e.g., the ability to coordinate/stage the various elements of the research design.
• Exceptional time management skills, e.g., the ability to allocate a reasonable time frame for each step.
• Wherewithal to obtain the necessary permissions to gain access to observation venues, activities, documents.
• Ability to relinquish control, allowing the case or the narrative to steer the direction of the investigation.
• Ability to accept many different points of view.
• Ability to notice the sequence of events as well as the physical & substantive context of information across all methods.

Image captured from: http://www.dailyartmuse.com/2010/08/11/dryden-wells-ceramic-multiples-imply-movement/

Share this:

• Click to share on Reddit (Opens in new window)
• Click to share on Twitter (Opens in new window)
• Click to share on LinkedIn (Opens in new window)
• Click to share on Facebook (Opens in new window)
• Click to share on Tumblr (Opens in new window)
• Click to email a link to a friend (Opens in new window)
• Click to print (Opens in new window)
• Pingback: 10 Distinctive Qualities of Qualitative Research | Research Design Review
• Pingback: A Multi-method Approach in Qualitative Research | Research Design Review
• Pingback: The Focus Group Method: 18 Articles on Design & Moderating | Research Design Review

Leave a comment Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed .

• Already have a WordPress.com account? Log in now.
• Subscribe Subscribed
• Copy shortlink
• Report this content
• View post in Reader
• Manage subscriptions
• Collapse this bar

The Ultimate Guide to Qualitative Research - Part 1: The Basics

• Introduction and overview
• What is qualitative research?
• What is qualitative data?
• Examples of qualitative data
• Qualitative vs. quantitative research
• Introduction

What is a mixed methods design?

Triangulation in mixed methods research, types of mixed methods research designs, using atlas.ti for mixed methods research.

• Qualitative research preparation
• Theoretical perspective
• Theoretical framework
• Literature reviews
• Research question
• Conceptual framework
• Conceptual vs. theoretical framework
• Data collection
• Qualitative research methods
• Focus groups
• Observational research
• Case studies
• Ethnographical research
• Ethical considerations
• Confidentiality and privacy
• Power dynamics
• Reflexivity

What is mixed methods research?

When starting the research process, researchers sometimes think they have to decide whether qualitative research or quantitative research is more appropriate for their research design. However, the more important question is whether the methods they employ in data collection and analysis sufficiently capture the phenomenon they want to study. In some cases, answering this question requires using multiple methods of research.

Mixed methods research is a research paradigm that involves collecting qualitative data and quantitative data on the same object of inquiry. Researchers who employ mixed methods research synthesize qualitative findings with quantitative findings to achieve a better understanding.

Let's look at the established research paradigms, then mixed methods research, why it's useful, and which research methods complement each other. Then we'll examine how ATLAS.ti can help you execute a mixed methods design.

Mixed methods research is followed out of the need to understand concepts or phenomena at a deep level. A standalone quantitative study or qualitative study can provide great insight. Still, one method alone may not be able to capture all knowledge necessary to fully understand a topic or issue.

Those who conduct mixed methods research acknowledge the importance of pursuing both qualitative and quantitative research to achieve more complete results. However, this is not simply an issue of collecting more data just for its own sake. Mixed methods design is purposeful in carefully crafting research questions and employing appropriate research methods to essentially fill in the gaps of knowledge surrounding a particular research inquiry.

To determine which methods and data can address particular research needs, let's look at the capabilities of and differences between qualitative and quantitative data collection .

Qualitative and quantitative data

Researchers are often quick to make conclusions about whether qualitative research is better than quantitative research or vice versa. The reality is that quantitative and qualitative data can both look at the world in different ways that are useful at various points of a research inquiry. Qualitative and quantitative research are established research paradigms precisely because they provide relevant insights with the appropriate research design, data collection, and analysis.

One of the main goals of qualitative research is to generate a description of a social phenomenon. When something is difficult to quantify, it needs to be broken down into more constituent elements that are, by themselves, easier to perceive. In educational evaluation, for example, it is difficult to evaluate good academic writing with just a single score alone. Writing teachers employ a rubric to measure writing by a number of aspects which may include argumentation, organization, and cohesion.

Qualitative methods of research tend to collect data for an analysis that is capable of generating frameworks of constituent elements. Such a framework can then be used in subsequent research, evaluation, or decision-making processes. Researchers can collect qualitative data from observations , interviews , or records searches. Qualitative data analysis then aims to identify patterns and themes frequently appearing in the collected data.

The efficacy of experimental drugs in clinical trials, for example, is seldom easy to measure through quantitative methods alone. Qualitative research methods are often employed to determine a research participant's well-being, emotional state of mind, and other factors to help researchers decide the overall success of their clinical trials.

Quantitative research

If qualitative methods describe a concept or phenomenon, quantitative methods employ the resulting framework to measure that concept or phenomenon. Quantitative research methodology takes the theories generated from qualitative findings to collect quantitative data that can be used to measure a concept or phenomenon at scale.

Ultimately, numbers and values inform decision-making processes in many contexts. Quantitative results are useful in research areas where precision is valued or required. Still, they are also used in social and behavioral research to numerically describe phenomena that may not appear to be naturally quantifiable.

Mixing methods

Quantitative and qualitative strands of research are often pitted against each other for various reasons. Researchers might shun qualitative data collection as it is often time-consuming. In contrast, quantitative data collection is often critiqued for its reductive power (i.e., reducing ambiguous concepts into simplistic numerical values). Many scholarly disciplines, as a result, tend to prefer one research paradigm over the other (e.g., chemistry tends toward quantitative data collection, while anthropology tends toward qualitative data collection).

In the long run of any sufficiently complex research inquiry, however, it is seldom necessary to remain confined to one research approach. The main objective of scientific research is to organize knowledge through theories about the world around us. As a result, researchers employ mixed methods to combine theory generation in qualitative research with confirmatory testing in quantitative research to ultimately produce a robust theory and new knowledge.

However, research studies that combine qualitative and quantitative methods for the sake of having multiple methods of data collection and analysis are not as persuasive or impactful as true mixed methods studies where research methods are purposefully chosen to achieve a better understanding.

An example of mixed methods research

The objective of mixed methods research designs is to employ different inquiry components under one larger study. However, it might be easier to think of mixed methods research designs as having at least one qualitative study and one quantitative study, each with related but ultimately separate research questions . Examining a mixed methods research design in this way might make it easier to understand the need for pursuing multiple methods in certain cases.

• Consider the following example:

Remote work performance and job satisfaction

- RQ1: How have work outputs at XYZ Company changed since the shift to fully remote work?

- RQ2: What perceptions do remote workers at XYZ Company have about the shift to fully remote work?

In general terms, the goal of the study is to examine the efficacy of remote work in comparison to traditional, in-office work at one company. Actually determining this efficacy requires looking at the phenomenon of remote work through different methods.

As a result, one possible mixed methods study might look at the performance metrics of the company. Research question 1 (RQ1) is posed to conduct a quantitative research study that collects data on possibly quantifiable concepts related to work (e.g., amount of sales generated, number of new clients acquired). In this case, the researchers collect quantitative data to compare post-remote work performance to pre-remote work performance and determine if productivity has changed over time.

While this is a useful angle to examine remote work, it does not tell the whole story. After all, if people at Company XYZ are more or less productive than before, what are the reasons that explain this change? To address research question 2 (RQ2), researchers collect qualitative data on the level of satisfaction employees have with their jobs. Qualitative data from interviews with employees can be used to determine which aspects of their job they find satisfying or not.

With all the data collected, mixed methods researchers can combine the initial quantitative results and the initial qualitative results to form a deeper understanding of their topic of inquiry. In this case, if the quantitative data shows that worker productivity has suffered since the switch to remote work, the qualitative data might illuminate the aspects of remote work that employees don't like.

Other mixed methods research examples

While there are many different forms of mixed-methods research, the research approach is generally the same across mixed-methods research designs. A mixed methods research design is likely to require researchers to collect quantitative and qualitative data relevant to an overarching topic that necessitates examination from different methods. A couple of examples are:

Literacy development among children

RQ1: What is the rate of literacy development among children at ABC School based on scores from a standardized reading test?

RQ2: What are the instructional practices common in classrooms with high-performing students on standardized reading tests?

Market research for a new computer model

RQ1: How much time does it take to complete a series of tasks on an experimental computer model compared to a comparable computer model?

RQ2: What factors do potential customers take into consideration when buying a new computer?

Notice that qualitative and quantitative data pursue related but ultimately different aspects of the phenomena under study. As a result, the discrete inquiries in a mixed methods study will most likely employ different methods to collect data.

Conduct research from conception to analysis with ATLAS.ti.

Download a free trial of ATLAS.ti to see how our software can aid you in the research process.

Researchers do not employ mixed methods research just for the sake of having different methods in one research inquiry. The objective behind mixing methods is to generate new knowledge and strengthen understanding of that knowledge by examining it from different angles. This is a concept in research called triangulation, which refers to affirming a given location based on measures taken from different points. The equivalent notion in research is that viewing the same object of inquiry from multiple angles will provide a more reliable understanding of that object.

To further understand the utility of a mixed methods approach, imagine you and your friends are looking at a merry-go-round. You can only see one part of it at any one time, while other parts are obscured from your view. On the other hand, if your friends are positioned to see the merry-go-round from different angles, your combined observations can capture a more complete picture of the object you are studying.

Mixed methods research relies on multiple research methods, data sets, or theoretical approaches to assemble a more comprehensive picture of a concept or phenomenon. Especially in qualitative research or social science research, any set of findings can be considered more credible if they are supported with evidentiary data that comes from different perspectives.

Method triangulation

Method triangulation involves combining qualitative and quantitative methods together to study different but related aspects. In this respect, quantitative and qualitative research study the same phenomenon to lend support to each method's findings. Note that the goal of triangulated mixed methods research is not to simply use multiple methods to arrive at the same answer but to generate a better understanding of a phenomenon that one method alone cannot sufficiently capture.

In this case, method triangulation is a useful concept for a mixed methods researcher because it requires them to acknowledge the strengths and weaknesses of each particular research method. At scale, quantitative methods cannot capture concepts that are unquantifiable (e.g., beauty, convenience). In contrast, qualitative methods often do not conduct data collection at scales necessary to make generalizations about phenomena. Integrating quantitative and qualitative research components under the same mixed methods design ensures a comprehensive examination of a phenomenon that one method alone cannot accomplish.

Ethnography provides ample opportunities to pursue method triangulation. Data collection in ethnographic research often involves collecting qualitative data through observations and interviews . In contrast, data analysis can assess quantitative data by identifying patterns in behavior and perspectives and determining their frequencies.

Another example is a mixed methods study that examines patient outcomes at a hospital. Initial qualitative results might come from field notes from observations of doctors and nurses and interview data with patients. The quantitative findings might come from conducting a statistical analysis of the money and resources used for each patient observed or interviewed to determine whether the expenditure is commensurate with the patient outcomes achieved.

A standalone quantitative study might look only at the financial aspects of health care, while a qualitative study might do better at examining the social and emotional aspects. Conducting both of these studies in tandem can help researchers determine actionable insights for streamlining health care services while maintaining satisfactory standards of care.

Data triangulation

Mixed methods research usually depends on method triangulation, but it's important to identify other forms of triangulation that can strengthen the findings in any research. A study that relies on data triangulation looks at different sets of data. For example, an educational researcher might examine student outcomes at different schools or at the same school but at different times. Data triangulation is useful in affirming that the findings in one context are applicable across other contexts.

Theory triangulation

Another kind of triangulation less commonly associated with mixed methods research deals with analyzing data using different theories. A sequential research design, for example, may use the initial quantitative results from a survey study to generate a conceptual framework for the analysis of a subsequent qualitative study. At the same time, existing theories may also be employed in that analysis to compare and contrasts the kinds of insights and outcomes that each may produce.

Theory generation in mixed methods research

Many forms of research seek to generate or develop a theoretical framework to understand the object of inquiry. There are two common forms of theory generation, and both can manifest in the research questions that are posed in any study.

Research questions can either be exploratory, which try to define or gain a greater understanding of a phenomenon, or confirmatory, which try to test a theory or hypothesis regarding that phenomenon. With some exceptions, exploratory research questions call for collecting qualitative data , while confirmatory research questions require quantitative data .

In that respect, common mixed methods designs combine qualitative and quantitative components to generate a theory and either strengthen or challenge that theory, respectively. To understand what that theory generation looks like when employing mixed methods, we need to examine some of the different kinds of mixed methods research designs.

Look at your data from all angles with ATLAS.ti.

Analyze your data quickly and efficiently with ATLAS.ti. Click here for a free trial.

Data collection and analysis in mixed methods research depends on the research design you adopt. Ultimately, it might be easy to think about the different research designs in terms of the timing of the discrete inquiries within a mixed methods inquiry.

Concurrent triangulation design

A study that collects quantitative and qualitative data simultaneously is a common form of mixed methods design to achieve triangulation. The goal of a concurrent triangulation design is to observe the object of inquiry from multiple methods.

For example, imagine an educational researcher who wants to examine the efficacy of an after-school reading program. The researcher can then pursue two concurrent studies, one that qualitatively observes the reading program in action between educators and students and another that quantitatively tests students' reading comprehension. Over time, the researcher can draw correlations between improvements in test scores and any observations of the students in the program.

Exploratory sequential design

Another way to look at mixed methods research is with the idea that data collection and analysis are cyclical and evolve as new knowledge is generated. Researchers might undertake an exploratory sequential design if they don't yet know the aspects of a concept or phenomenon they want to test. In short, they need to conduct a qualitative study first in order to generate a conceptual framework to apply in a subsequent quantitative study.

Exploratory sequential design is useful in market research, for example, to identify the potential needs and preferences of prospective customers. Focus group research with a group of target customers can inquire about what they are looking for when choosing from a line of products. The researcher can take the initial qualitative findings to inform the design of a subsequent survey study that can confirm the extent to which the preferences of the focus group are reflected in the larger market.

Researchers can also conduct a quantitative study to preface observations in a qualitative study. Imagine that an educational researcher is adopting mixed methods approaches when examining learning outcomes among schools within a given geographical area. They might start by examining test scores published by these schools, using the initial quantitative results to determine where students are struggling and might need intervention. The resulting qualitative study might conduct observations in struggling schools to determine potential shortcomings in teaching and learning.

Concurrent nested design

This research design involves conducting multiple inquiries at the same time for the purpose of using one inquiry to strengthen the other. In a mixed methods approach, concurrent nested design places one research paradigm within another (e.g., a quantitative study within a qualitative study).

Sequential transformative design

This is a mixed methods research design with a critical or social justice orientation, meaning that the research is ultimately conducted to challenge the understanding of existing theory or produce meaningful social change, respectively. In either case, a sequential mixed methods research design can have a transformative effect by employing one study to create the rationale for a second critical or social justice research inquiry.

As you employ multiple research methods for a single mixed methods research design, you might find that your data collection will involve large sets of data, presenting a challenge in managing all that information in an orderly manner. Whether you are conducting research through qualitative data collection, quantitative data collection, or both, ATLAS.ti can help you organize and analyze your data. A robust mixed methods approach requires systematic organization of your data collection to ensure efficient and insightful analysis.

Document groups

Data in ATLAS.ti is stored in documents, which can be classified by the data type they contain. ATLAS.ti allows you to analyze text, images, video, audio , and more, and each document's data type is marked in the Document Manager for easy organization.

However, you may also need to divide your documents by type of study or method employed. In that case, you can use Document Groups in ATLAS.ti to label your documents so your project has categories for quantitative and qualitative data, interviews and focus groups, observations and test scores. Documents can belong to multiple document groups, allowing for easy organization of documents into multiple categories.

Once you have fully coded your data , it might be a challenge to narrow down your analysis to the relevant data you're looking for. If you have to sift through large numbers of documents, the Query Tool can help you look for the most relevant quotations based on the codes you have applied to your data.

Global filters

Studies that employ mixed methods research can accumulate such vast amounts of qualitative and quantitative data that it might become cumbersome for the human eye to keep track of it all manually. Even the most organized project in ATLAS.ti can have thousands of documents or hundreds of codes, making it a challenge to find the right data.

In ATLAS.ti, you can set a global filter using any of the elements of your project. For example, if you have a document group labeled " interviews ," you can set a global filter for that document group, which will lead ATLAS.ti to only show the documents in that group.

Working with both qualitative and quantitative software

ATLAS.ti has a number of tools that provide visualizations to help illustrate quantitative findings. However, you may find that other software, such as Microsoft Excel or SPSS, can help you further analyze and visualize the quantitative research components in your study. As a result, ATLAS.ti allows you to export your analysis into a Microsoft Excel spreadsheet. The Code Co-Occurrence Analysis and Code-Document Analysis tools, for example, can export their resulting tables into Microsoft Excel, which includes tools for deeper statistical analysis or for creating other kinds of data visualizations.

ATLAS.ti projects can also be exported as syntax files that can be imported into other statistical analysis software such as SPSS and R. These files convert qualitative data into quantitative data for further statistical analyses, regressions, and quantitative visualizations. Researchers can fully realize the convergence between qualitative and quantitative research when using multiple software platforms to conduct their analysis.

From data collection to data analysis, rely on ATLAS.ti.

Start with a free trial of our software to conduct your mixed methods research.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Download Free PDF

Multimethod qualitative research

2020, Qualitative Research

Related papers

Qualitative and multi-method research in political science has undergone a remarkable transformation. After a long period of relative quiescence, in roughly the past 15 years this branch of methodology has experienced a resurgence, and a considerable body of new research has emerged. Under this methodological umbrella we find diverse tools, including small-N comparative analysis, the comparative case-study method, ethnography, constructivism, and interpretivism. Against the backdrop of a larger movement in political science advocating greater methodological pluralism, new organizations have been established to institutionalize scholarly communication and training. Moreover, mainstream scholarly outlets have come to recognize the importance of this approach, and publications in this tradition have increasingly appeared in leading journals and from leading academic presses. Many qualitative/multi-method scholars seek to maintain strong connections with quantitative methodologists. At the same time, just as occurred with the formation some years ago of the APSA organized section for quantitative methods – with the goal of creating an arena in which scholars in that tradition could closely engage in one another’s work; so qualitative/multi-method scholars see substantial need for separate organizations to help sustain their own approach.

Theory and Society

Methodological Innovations

Qualitative health research, 2005

Qualitative Inquiry, 2010

PERSPECTIVES ON POLITICS (Vol. 10, No. 4): 935-953, 2012

While acknowledging the many forms and contributions of multi-method research (MMR), we examine the costs of treating it as best practice and as a means to neutralize method-specific weaknesses and increase external validity for findings. Focusing on MMR that combines some type of qualitative analysis with statistical or formal approaches, we demonstrate that error-reduction and cross-validation are not feasible where methods are not sufficiently similar in their basic ontologies and their conceptions of causality. In such cases, MMR may still yield important benefits (such as uncovering related insights or improving the coding of variables) but these can be readily obtained through collaboration among scholars specializing in single-method research (SMR). Such scholars often set the standards for the application of particular methods and produce distinctive insights that can elude researchers concerned about competently deploying different methods and producing coherent findings. Thus, the unchecked proliferation of multi-method skill sets risks forefeiting the benefits of SMR and marginalizing idiographically-oriented qualitative research that fits less well with formal or quantitative approaches. This would effectively subvert the pluralism that once gave impetus to MMR unless disciplinary expectations and professional rewards are predicated on a more balanced and nuanced understanding of what various forms of SMR and MMR bring to the table.

Scientific trends: modern challenges. Volume 2, 2021

RELITTI DALL'ITALIA TIRRENICA NEL MEDITERRANEO OCCIDENTALE (fine IV secolo a.C. - I d.C.). ARCHEOLOGIA E ARCHEOMETRIA DELLE ANFORE.. Con la collaborazione di Iannis Iliopoulos, Claudio Capelli, Domenico Michele Surace, Andrea Razza, 2021

Proceedings of the Edinburgh Mathematical Society, 2004

INDIAN JOURNAL OF LEGAL REVIEW, 2024

GHGT6 Paper, 2002

Annales de l’APLAES, 2020

Journal of Entrepreneurship and Management, 2018

Bioscience Reports, 2005

Environment International, 2021

Revista de Salud Pública, 2005

The Egyptian Journal of Medical Microbiology, 2018

Journal of Educational Psychology, 1974

PLoS Genetics, 2012

Communicable diseases intelligence, 2024

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

• Triangulation in Qualitative Research: Methods and Benefits

Discover key triangulation methods in qualitative research and how they strengthen data reliability and insights.

Making sense of heaps of research can sometimes feel like torture. Not even the research enthusiasts find it thrilling to squint at data for hours, thinking, “ Wait… am I truly seeing the full picture? ”

Luckily, triangulation in qualitative research can save you from the agony. Read on to learn how it cross-checks your research from multiple angles, making it more precise and reliable.

And if you’re more into the “how” than the “why,” our AI-powered research assistant can help you out. Marvin easily stores, analyzes, and connects the dots across all your customer research. Try Marvin free and see for yourself.

What is Triangulation in Qualitative Research?

Triangulation sounds fancy. But in reality, it’s just about not putting all your research eggs in one basket. Instead of relying on one method (which could miss the mark), you use different approaches to get a more reliable result:

• Collect data from various users
• Try multiple research methods
• Ask fellow researchers to weigh in

Human behavior is messy, and qualitative research dives right into it. That’s why you need more than one angle to understand what’s going on.

When it comes to qualitative research, triangulation takes different routes to see if they all lead to the same destination. The goal is to confirm that your findings are well-rounded and reliable.

What is the Purpose of Triangulation in Qualitative Research?

Customer analysis is critical when designing a product. You need to know what your users truly think, feel, and experience. And no one wants shaky, one-sided conclusions. That’s where triangulation comes in — to:

• Reduce the risk of bias or misunderstandings
• Strengthen your findings by viewing them from different perspectives

Example of Triangulation in Qualitative Research

Let’s say you’re conducting user research for a new app feature. Triangulation could involve a combination of:

• Method 1: Interview users
• Method 2: Observe their behavior using screen recordings
• Method 3: Analyze feedback from your support team about common complaints

In the end, you get a more complete, reliable picture of your users’ needs. And that brings you much closer to building something that truly solves their problems.

Types of Triangulation in Research

The example above is methodological triangulation in action. But that’s just one of the four types of triangulation.

Let’s break them down to help you pick the right one for your next research project.

1. Methodological Triangulation

Here, you mix up different methods to investigate the same research question. Maybe you combine user interviews with usability testing. You can also conduct focus groups and online surveys.

The goal of this triangulation research method is to cross-check your results from different angles. One method might highlight user frustration. But when you combine it with another one that reveals their behavior patterns, you get the full story.

2. Data Triangulation

This one’s all about variety. Data triangulation in qualitative research involves gathering feedback from different times, places, or people. It makes you less likely to miss important details.

For example, if you’re running a study, you can collect feedback from three groups: early adopters, casual users, and support teams. Each group gives you a different perspective on the same product or service.

3. Investigator Triangulation

Investigator or researcher triangulation brings in multiple researchers. They analyze the data from their perspectives. You compare their interpretations. In short, you add extra eyes to your work and end up with fewer blind spots.

This method can eliminate personal bias and give a more objective view of the findings. (Plus, collaboration is always more fun, right?)

4. Theory Triangulation

Picture this one as if trying on glasses with different theoretical lenses. Let’s say you’re researching user behavior. One lens might be behavioral psychology; another might be design thinking. Each view lets you see your data from a new angle.

By using different theories, you avoid squeezing your data into one box. It keeps your mind open — and who knows? You might stumble on insights you didn’t expect.

Advantages of Qualitative Research Triangulation

Triangulation adds depth and confidence to your research. It helps you feel less overwhelmed by data and more assured in the insights you pull. Here’s why:

• More reliable insights: Triangulation prevents feedback from getting all twisted up by the end. No more “ Did I get this right? ” vibes — just solid, trustworthy results.
• Reduced researcher bias: When using triangulation, you’re less likely to let your assumptions color the results. You get a more balanced and objective view.
• Richer, deeper data: Different methods or perspectives give you more context. Instead of a surface-level understanding of what users do, you grasp why they do it.
• Stronger findings: You’re hard to argue with when your conclusions rely on multiple methods or sources. This makes it easier to win over stakeholders.
• Flexible research design: You can mix and match methods to fit your project’s needs, gaining more control over your research process.

How to Implement Triangulation in Qualitative Research

With triangulation, you want to cover every angle and leave no stone unturned. Follow these steps to triangulate your way to deeper insights.

1. Define Your Research Objective

Get crystal clear on your research question. What are you trying to learn about your users? This will help you pick the best combination of methods and sources to answer your question from all angles.

2. Choose Your Triangulation Strategy and Methods

Which type of triangulation do you want to use? You might pick just one or a combination of a few, depending on your research question.

For example, if you’re using methodological triangulation, you can combine qualitative methods (interviews or focus groups) with quantitative methods (surveys). If you’ve chosen data triangulation, sources can be different user types, time periods, or locations.

No matter what methods or sources you use, they should serve your research objective and cross-verify your findings to uncover deeper insights.

3. Collect and Analyze the Data

Start by collecting data from each method or source you’ve chosen. 

As you analyze, ensure each method addresses the same research question — this will let you spot patterns or discrepancies across sources and make the next step easier.

4. Compare and Cross-Check Your Findings

The actual comparison is the essence of triangulation. Are there any contradictions? Are you seeing the same patterns across the board and getting well-rounded, reliable insights?

Contradictions can reveal nuances you might have missed or point to areas where different user groups or methods provide unique perspectives.

5. Collaborate With Other Researchers

If you choose investigator triangulation, you can now bring other researchers to interpret the data. 

Multiple minds can spot biases or assumptions one person might miss, making your research even more objective.

6. Draw and Communicate Conclusions

Finally, your insights are stronger and more credible because they’ve been validated from different angles. When presenting your findings, highlight how triangulation strengthened the results.

Now that you’ve learned how triangulation can consolidate your research findings, why not make it easier with Marvin? Our AI-powered research repository takes the heavy lifting out of cross-checking data.

Need to pull insights from different sources and analyze them in one place without missing a beat? Create a free account to see how Marvin helps you get the most out of your qualitative research.

Limitations and Challenges of Implementing Triangulation

Despite its benefits, triangulation isn’t all rainbows and perfectly cross-checked data. Watch out for the following challenges you might face when implementing it:

• Takes more time and resources: Triangulation is no small task. You’ll need to budget more time and resources to pull it off. Watch out for extended deadlines and bigger research teams!
• Limitations come into play: Each method might have limitations or quirks, making it tough to line up the results. Sometimes, when your interview data contradicts your survey results, it’s challenging to piece it all together without pulling your hair out.
• Consistency is hard to achieve: One researcher might interpret the data differently than another. Or maybe the way you phrased questions in a survey doesn’t quite match the interview format. Consistency becomes slippery when using different methods or collaborating with multiple researchers.
• Information overload may creep in: More data isn’t always better, leaving you feeling like drowning in a sea of insights. Getting lost in the details is easy, but sticking to the plan will help avoid analysis paralysis.

Best Practices for Triangulation in Research

Like with any good strategy, there are ways to do triangulation right to get the full benefits without the headaches:

1. Focus on Complementary Methods or Data Sources

The goal is to cover different dimensions of the same issue, not duplicate efforts. Therefore, pick the most suitable methods instead of using as many as possible.

For instance, pairing user interviews with usability testing works because interviews reveal user opinions, while usability testing shows how they actually behave.

2. Prioritize Quality Over Quantity

Two or three well-chosen methods that give you rich, reliable insights will often be more effective than spreading yourself thin with too many.

Instead of going wide, go deep. Focus on gathering high-quality data that truly addresses your research question.

3. Maintain a Balance Between Consistency and Flexibility

Triangulation’s strength is the ability to adapt. Balance consistency with flexibility to tweak your approach when necessary.

Don’t be afraid to pivot if one approach isn’t delivering the expected insights or new information suggests a better path.

4. Don’t Ignore Contradictions—Explore Them

Contradictory results can often point to deeper insights. They may reveal nuances or hidden factors you hadn’t considered.

Dig deeper into those contradictions instead of trying to reconcile everything neatly—they might tell you something valuable.

Frequently Asked Questions (FAQs)

Ready to jump head-first into triangulation? Check out these FAQs first:

Can Triangulation be Used in All Qualitative Research Studies?

You can use triangulation in most qualitative research studies, but it’s not always necessary. With complex research or ambiguous user behavior, it makes sense to use multiple methods or data sources. However, it may not be worth the extra work for smaller, more focused studies.

What Are the Ethical Considerations When Using Triangulation in Qualitative Research?

You need to be sure that you’re transparent with participants about how their data will be used across these methods. Also, keep privacy in mind. If you’re pulling data from multiple sources, be clear on how you’re storing and handling that data securely.

How Do You Ensure Consistency When Using Multiple Methods?

Consistency doesn’t mean everything has to be exactly the same. Still,  your methods should be comparable enough to draw meaningful insights when cross-checking the results. The key is to set up a clear framework from the start. Your questions, protocols, and data collection techniques should align even if they spread across different methods.

There you have it — triangulation in all its multi-angled glory. You can mix methods, gather data from different sources, or team up with other researchers. However you do it, it’s the ace up your sleeve for uncovering deeper insights and minimizing bias.

Before you go forth and triangulate, remember that covering all those angles doesn’t have to be a pain. Marvin helps with triangulation in qualitative data analysis through:

• Centralized data: Store interviews, surveys, support tickets, and sales calls in one place.
• AI-powered insights: Automatically connect the dots between multiple data sources.
• Effortless comparisons: Easily cross-check findings from different methods.

Book a demo today and see how Marvin supports your research by simplifying data collection, organization, and analysis. Bring all your insights in one place to focus on what actually matters — cross-checking your data and getting deeper insights through more efficient triangulation.

Categories:

• Leading the Way
• Marvin Customers
• Marvin Events
• Talk to Your Users

• Cari Murray
• Latest posts

Cari Murray is director of marketing & partnerships for HeyMarvin, a UX research repository that makes it super simple to talk to your customers and design products they love. She's been telling powerful brand stories for 20 years.

• Survey Analysis Methods Explained: Qualitative, Quantitative, and Mixed
• How to Craft UX Research Questions [Tips and Examples]
• Master User Research Methods: Qualitative vs. Quantitative Techniques

withemes on instagram

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews

Patricia j lucas, janis baird, catherine law, helen m roberts.

• Author information
• Article notes
• Copyright and License information

Corresponding author.

Received 2006 Oct 9; Accepted 2007 Jan 15; Collection date 2007.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

The inclusion of qualitative studies in systematic reviews poses methodological challenges. This paper presents worked examples of two methods of data synthesis (textual narrative and thematic), used in relation to one review, with the aim of enabling researchers to consider the strength of different approaches.

A systematic review of lay perspectives of infant size and growth was conducted, locating 19 studies (including both qualitative and quantitative). The data extracted from these were synthesised using both a textual narrative and a thematic synthesis.

The processes of both methods are presented, showing a stepwise progression to the final synthesis. Both methods led us to similar conclusions about lay views toward infant size and growth. Differences between methods lie in the way they dealt with study quality and heterogeneity.

On the basis of the work reported here, we consider textual narrative and thematic synthesis have strengths and weaknesses in relation to different research questions. Thematic synthesis holds most potential for hypothesis generation, but may obscure heterogeneity and quality appraisal. Textual narrative synthesis is better able to describe the scope of existing research and account for the strength of evidence, but is less good at identifying commonality.

The inclusion of qualitative data in systematic reviews is an area of ongoing methodological development [ 1 - 3 ], with particular problems arising for reviews attempting to synthesise quantitative with qualitative data. The Cochrane qualitative methods group [ 2 ] suggests four areas in which development is needed; (1) searching, (2) critical appraisal, (3) synthesis/summary, and (4) loss of research context. This paper aims to contribute to development in the synthesis of qualitative and quantitative data. Alternative models and vocabularies of synthesis are emerging [ 3 - 9 ], but standard methods for combining different data types from the qualitative and quantitative research traditions have not yet been agreed [ 8 ].

Innovative methods are often developed during the course of research, but in general, papers report methods only briefly. As a result, the material that could inform learning is more often to be found in filing cabinets than in journals. In this paper we aim to distinguish between "the trivial and non-trivial points of divergence" p.31 [ 4 ] by providing worked examples of two methods of evidence synthesis (thematic and textual narrative) tested in one systematic review.

A systematic review of lay views about infant size and growth was undertaken as part of a series of interlinked reviews examining the evidence for associations between early growth and a number of later outcomes. The systematic review of views included both qualitative and quantitative studies.

Study methods and findings are reported in greater detail elsewhere [ 10 - 13 ]. Standard systematic review methods were employed, following guidance from the Centre for Reviews and Dissemination [ 14 ] and from an advisory group with backgrounds in public health, paediatrics, infant nutrition, qualitative and quantitative methods, systematic reviewing, and including representatives from user groups. Twelve databases were searched using terms for growth, height, weight and infancy as well as appropriate methodological terms. 2,694 abstracts were retrieved, from which 19 studies met the inclusion criteria for the review.

Two researchers independently extracted findings by interrogating each study using the following questions developed from the aims of the review:

1. What is healthy growth/size?

2. How important is growth/size to participants?

3. What concepts are used to define healthy growth/size?

4. How do participants assess growth/size?

5. Where does growth lie among priorities for child health?

6. What information influences views/behaviour?

7. Who influences views/behaviour?

Directly reported participant data (e.g. verbatim quotations or scores on attitudinal scales) and author interpretations were recorded separately, to retain the richness or 'thickness' of the contributing data. 'Thickness' in this context refers to the kinds of relatively detailed descriptions and contextual material which help the reader to make judgements about the trustworthiness of the data, particularly when applying it to different contexts [ 15 , 16 ]. Study characteristics and quality assessment were summarised (for examples see Table 3 ). There is vigorous debate on whether qualitative research can be assessed using standard quality criteria, or whether this process is contrary to the nature of qualitative enquiry [ 17 ]. While the controversy on the use of critical appraisal in systematic reviews including qualitative data lies beyond the scope of this article, with views ranging from those who believe that critical appraisal is core to qualitative synthesis [ 18 ] to those who, like Barbour [ 19 ] consider that critical appraisal of qualitative research can be reductionist, it is notable that there is general agreement that a checklist approach to critical appraisal can bring its own problems, particularly in relation to transparency in assessing interpretative work. We took the view that applying quality criteria rigidly would be likely to exclude relevant studies that had failed to comply with a particular reporting regime. Thus, all studies meeting our inclusion criteria listed were included and quality appraisal was used at the data synthesis stage contributing to strength of evidence.

Example study summaries

* Special Supplemental Nutrition Program for Women, Infants and Children

Two methods were proposed for synthesis of findings, textual narrative and thematic, both of which the advisory group agreed were appropriate to our needs. The first, the textual narrative approach, involves a commentary reporting on study characteristics, context, quality, and findings, using the scope, differences and similarities among studies were used to draw conclusions across the studies, whilst the second, the thematic approach, groups data into the themes. Given the relatively small number of studies located, it was feasible to test both methods. Findings from the review are provided briefly for illustration, but the focus of this paper is on the process of synthesis and a comparison of methods used. The two reviews ran in tandem, as the thematic review needed time for response and comparison between reviewers.

Worked Example 1 – Textual Narrative Synthesis

Factors identified by the research team from the research literature as likely to affect views on infant growth were used to define a number of sub-groups. These were:

1. Relationship between participant and infant (e.g. mothers, other family members, health professionals, unrelated others)

2. Weight status of participant

3. Ethnicity of participant

4. Age of infant

5. Views about infants considered 'high risk' at birth i.e. those born too small or too early, or who were placed in a neonatal intensive care unit (NICU)

6. Weight/growth status of infant after birth

7. Mode of infant feeding (breast fed, bottle fed, weaned)

Using agreed versions of quality appraisal and extracted data a textual narrative synthesis was undertaken by a single researcher (PL). Each study within a sub-group was described in a commentary reporting on study characteristics, context, quality, and findings. The scope, differences and similarities among studies were used to draw conclusions across the studies (the synthesis). Drawing conclusions across studies was not always possible due to study heterogeneity and lack of data. A worked example of the process is shown in Table 1 .

Stepwise textual narrative synthesis

Findings – Textual Narrative Synthesis

We noted that unrelated members of the public tended to prefer infants of mid-range body sizes, but the evidence to support this observation was thin. Families of children with poor growth were acutely aware of growth as a problem; they monitored growth and discussed it with others. They desired "normal" growth in their child, and looked for ways that they could interpret the infant's growth as normal (for example finding members of the extended family who were of similar body shape). The most common method of assessing size in all sub-groups was by comparison with others, although the use of growth charts and physical measurement were also important for those with children with poor growth including babies born too small or too early. However, growth and size in themselves were low among concerns about such 'high risk' babies. The predominance of those with 'high risk' infants may explain our conclusion that growth was low among priorities for mothers of younger infants (aged 0–3 and 3–6 months). Among older children (more than 12 months) with poor growth there was concern among parents. Parents wanted to see good growth in their children, but they also considered love, attention, good health and good diet as important.

We judged that we had insufficient data to draw conclusions about the views of family members other than mothers, health professionals, or to compare the views of participants of different weight, ethnicity, or toward breast versus bottle fed infants.

Worked Example 2 – Thematic Synthesis

Thematic synthesis was undertaken by two researchers, LA and PL. Findings from all studies were collated under the 7 questions used in data extraction. Each researcher independently conducted a thematic analysis using these findings. On initial discussion of themes, researchers judged that there was repetition between the data extraction questions, and that data referred to four broad areas of enquiry:

1. Understanding healthy growth/size

2. Assessment of growth/size

3. Concerns about growth/size

4. Influences on views, behaviour, interpretations of growth/size

Data and themes were grouped into these areas and emerging themes were then considered for relevance, presence across studies, 'thickness' and duplication. This process was repeated until researchers were satisfied that all data could be interpreted within these themes and an agreed version reached. A worked example of the process is shown in Table 2 .

Stepwise thematic synthesis

Findings – Thematic Synthesis

Across the thematic synthesis the predominant concern of participants was normality. This was seen through the creation of norms of growth and models to explain difference. This was conducted across physical, observable characteristics, but included physical unobservable (such as underlying health status) and non physical (such as emotional care) dimensions. Where growth differed from the norm and a plausible explanation could not be found, for example among families of those with faltering growth [ 20 ], growth became an important concern for parents.

Data from across studies could be usefully combined in this method, for example in listing all the sources of influence on behaviour or views found. Family, other parents and friends, information from the infant themselves, health professionals, clothing sizes, magazines, books, radio, TV and their religious beliefs were all important to some, but the relative importance of these could not be explored.

Strengths and limitations of our study

While the data extraction and thematic synthesis was undertaken by two researchers working independently, only one of these researchers (employed to work on the qualitative aspect of the review) worked on the narrative synthesis with a second researcher discussing the work as it progressed. Whether the findings might be different with more than one researcher working on both syntheses, or researchers not involved in the data extraction doing the syntheses, or the syntheses being carried out in a different order, are themselves research-able (if rather expensive) questions, as is the issue of whether the immersion of one researcher in the data at every stage a strength (as we believe it to be) or a source of bias.

Reassuringly, the conclusions to which these analyses led us about lay perspectives were largely similar across the thematic and textual narrative synthesis. Whether using a different research team, or a larger number of reviewers, would have produced different results is itself a researchable question. However, in this case conclusions from both analyses were dominated by importance of having babies that were a 'normal' size, leading to interest in monitoring of growth in a number of ways and, sometimes, to concern that there was an underlying problem leading to 'abnormal' growth. While the general conclusions were the same, the process and the implications of the two types of synthesis differed.

Strengths and Weaknesses of Textual Narrative Synthesis Methods

A textual narrative approach typically groups studies into more homogenous groups. This technique has been particularly successful in synthesising different types of research evidence (e.g. qualitative, quantitative, economic). Examples include a number of reviews carried out by the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) [ 21 - 23 ], reviews of tobacco use and exposure to tobacco smoke [ 24 ], reviews of ultrasound in pregnancy [ 25 ] and of communication between health care professionals and patients about prescribing [ 26 ].

In our review, the textual synthesis proved a useful way to describe difference in the included studies, making explicit the diversity in study designs and contexts. The textual narrative review also described gaps in the literature, both by showing where evidence was absent and by making an evaluation of the strength of evidence in different areas. Using this method enabled us to comment on, for example, the ethnic uniformity of participants, and the lack of evidence collected regarding mode of feeding.

However, transparency remained a problem. For example, decisions about which sub-groups to use for synthesis of individual studies rely on judgements, albeit ones which can be informed by the scientific literature and by lay views. While we sought to make the decision making process clear, interpretation and judgement, which are not fully susceptible to external scrutiny, lie at the heart of the process.

Strengths and Weaknesses of Thematic Synthesis

The strengths of the thematic synthesis lie in its potential to draw conclusions based on common elements across otherwise heterogeneous studies. This synthesis is potentially more accessible for the reader than a textual synthesis. Conclusions from this thematic synthesis fulfil an important research aim of qualitative research in generating hypotheses, an area to which traditional systematic reviews are poorly suited [ 27 ].

However, pooling findings in the thematic synthesis risks masking the shortcomings of the individual studies that make up the review. Although descriptions of study characteristics and quality appraisal were presented alongside synthesised findings, the synthesis process obscured these in the conclusions. We believe that further debate about the reliability of this approach would be useful. On the one hand, the hypotheses that emerge from this synthesis draw on a broader body of views than any single study (as in a meta-analysis) and may therefore increase reliability; on the other, we risk making strong conclusions based on a group of studies none of which is in itself reliable on the grounds of quality or diversity of context. This method may also be poor at examining contradictions, as well as commonalities, in the data and at highlighting gaps in the evidence.

The selection of synthesis method for systematic reviews such as this may depend on the aims of the synthesis. For the purpose of generating future research hypotheses, the thematic synthesis appears to hold the greatest potential; describing common themes and providing a possible structure for new research. In contrast, the textual narrative synthesis might be better suited to reviews which aim to describe the existing body of literature; identifying the scope of what has been studied, the strength of evidence available, and gaps that need to be filled.

Competing interests

The authors declare no competing interests.

Authors' contributions

CL, JB, HR, obtained funding for the study. All authors were responsible for the concept and design of the study. PL, and HR carried out the review work with assistance from all other authors. PL, LA & HR were responsible for the interpretation of findings. PL and HR produced the first and subsequent drafts of the paper, all authors were responsible for critical revision of the manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1471-2288/7/4/prepub

Acknowledgments

Acknowledgements.

We would like to thank our advisory group for their input to the project, especially Paul Dieppe for chairing it, Sandy Oliver and David Jones for methodological advice and Phyll Buchanan for the additional lay input. Jos Kleijnen assisted CL, JB and HR in obtaining funding for the study and provided methodological advice. This project was funded by the Department of Health in the UK, and we thank them for their support. The views expressed in this report are those of the authors and not necessarily those of the Department of Health.

Contributor Information

Patricia J Lucas, Email: [email protected].

Janis Baird, Email: [email protected].

Lisa Arai, Email: [email protected].

Catherine Law, Email: [email protected].

Helen M Roberts, Email: [email protected].

• Petticrew M, Roberts H. Systematic Reviews in the Social Sciences. A Practical Guide. Oxford, UK, Blackwell Publishing; 2006. [ Google Scholar ]
• CQMG Cochrane Qualitative Methods Group. 2007. http://www.joannabriggs.edu.au/cqrmg/index.html
• Popay J, Roberts H, Sowden A, Pettticrew M, Arai L, Rodgers M, Britten N. Guidance on the conduct of narrative synthesis in systematic reviews. http://www lancs ac uk/fass/projects/nssr/ 2007.
• Dixon-Woods M, Agarwal S, Young B, Jones DR, Sutton AJ. Integrative approaches to qualitative and quantitative evidence. London, Health Development Agency; 2004. [ Google Scholar ]
• Harden A. Extending the boundaries of systematic reviews to integrate different types of study: examples of methods developed within reviews of young people's health. In: Popay J, editor. Moving beyond effectiveness in evidence synthesis. London, National Institute for Health & Clinical Excellence; 2006. pp. 15–30. [ Google Scholar ]
• Campbell R, Britten N, Pound P, Donovan J, Morgan M, Pill R, Pope C. Using meta-ethnography to synthesise qualitative research. In: Popay J, editor. Moving beyond effectiveness in evidence synthesis. London, National Institute for Health & Clinical Excellence; 2006. pp. 75–82. [ Google Scholar ]
• Dixon-Woods M, Bonas S, Booth A, R JD, Miller T, Sutton AJ, Shaw RL, Smith J, A, Young B. How can systematic reviews incorporate qualitative research? A critical perspective. Qualitative Research. 2005;6:27–44. doi: 10.1177/1468794106058867. [ DOI ] [ Google Scholar ]
• CQMG Methodological Issues Arising from the Inclusion of Qualitative Evidence in Systematic Reviews. http://www lancs ac uk/fass/ihr/research/public/cochrane htm. 2007.
• Shaw RL, Booth A, Sutton AJ, Miller T, Smith J, A, Young B, Jones DR, Dixon-Woods M. Finding qualitative research: an evaluation of search strategies. BMC Medical Research Methodology. 2004;4 doi: 10.1186/1471-2288-4-5. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Baird J, Fisher D, Lucas P, Kleijnen J, Roberts H, Law C. Being Big or Growing Fast; a systematic review of size and growth in infancy and later obesity. British Medical Journal. 2005;331:929–934. doi: 10.1136/bmj.38586.411273.E0. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Baird J, Lucas P, Kleijnen J, Fisher D, Roberts H, Law C. Defining optimal infant growth for lifetime health: a systematic review of lay and scientific literature. 2005. http://www.mrc.soton.ac.uk/index.asp?page=176
• Fisher D, Baird J, Payne L, Lucas P, Kleijnen J, Roberts H, Law C. Are infant size and growth related to burden of disease in adulthood? A systematic review of literature. International Journal of Epidemiology. 2006;35:1196–1210. doi: 10.1093/ije/dyl130. [ DOI ] [ PubMed ] [ Google Scholar ]
• Lucas P, Arai L, Baird J, Kleijnen J, Law C, Roberts H. A systematic review of lay views about infant size and growth. Archives of Disease in Childhood. 2007. [ DOI ] [ PMC free article ] [ PubMed ]
• NHS Centre for Reviews and Dissemination . Undertaking systematic reviews of research on effectiveness: CRD's guidance for those carrying out or commissioning reviews. 4 (2nd Edition) CRD; 2001. [ Google Scholar ]
• Arai L, Popay J, Roen K, Roberts H. It might work in Oklahoma but will it work in Oakhampton? What does the effectiveness literature on domestic smoke detectors tell us about context and implementation? Injury Prevention. 2005;11:148–151. doi: 10.1136/ip.2004.007336. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of qualitative literature in health services research. Qualitative Health Research. 1998;8:341–351. doi: 10.1177/104973239800800305. [ DOI ] [ PubMed ] [ Google Scholar ]
• Dixon-Woods M, Bonas S, Booth A, R JD, Miller T, Sutton AJ, Shaw RL, Smith J, A, Young B. How can systematic reviews incorporate qualitative research? A critical perspective. Qualitative Research. 2006;6:27–44. doi: 10.1177/1468794106058867. [ DOI ] [ Google Scholar ]
• Attree P, Milton B. Critically appraising qualitative research for systematic reviews: defusing the methodological cluster bombs. Evidence & Policy. 2006;2:109–126. [ Google Scholar ]
• Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? British Medical Journal. 2001;322:1115–1117. doi: 10.1136/bmj.322.7294.1115. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Thomlinson EH. The lived experience of families of children who are failing to thrive. Journal of Advanced Nursing. 2002;39:537–545. doi: 10.1046/j.1365-2648.2002.02322.x. [ DOI ] [ PubMed ] [ Google Scholar ]
• Shepherd J, Garcia J, Oliver S, Harden A, Rees R, Brunton G, Oakley A. Barriers to, and facilitators of the health of young people: A systematic review of evidence on young people's views and on intervention in mental health, physical activity and health eating. Volume 2: Complete Report. London, EPPI-Centre, Social Science Research Unit, Institute of Education; 2002. http://eppi.ioe.ac.uk [ Google Scholar ]
• Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A. Vol. 58. J Epidemiol Community Health; 2004. Applying systematic review methods to studies of people's views: an example from public health research. pp. 794–800. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Oliver S, Harden A, Rees R, Shepherd J, Brunton G, Garcia J, Oakley A. Vol. 11. Evaluation; 2005. An emerging framework for including different types of evidence in systematic reviews for public policy; p. 446. [ DOI ] [ Google Scholar ]
• Hopkins DP, Briss PA, Ricard CJ, Husten CG, Carande-Kulis VG, Fielding JE, McKenna JW, Sharp DJ, Harris JR, Wollery TA, Harris KW. Reviews of evidence regarding interventions to reduce tobacco use and exposure to environmental tobacco smoke. American Journal of Preventive Medicine. 2001;20:16–66. doi: 10.1016/S0749-3797(00)00297-X. [ DOI ] [ PubMed ] [ Google Scholar ]
• Garcia J, Bricker L, Henderson J, Martin M, Mugford M, Nielson J, Roberts T. Women's views of pregnancy ultrasound: A systematic review. Birth. 2002;29:225–250. doi: 10.1046/j.1523-536X.2002.00198.x. [ DOI ] [ PubMed ] [ Google Scholar ]
• Cox K, Stevenson F, Britten N, Dundar Y. A Systematic review of communication between patients and health care professionals about medicine-taking and prescribing. King's College London, GKT Concordance Unit, Guys' King's and St Thomas' School of Medicine; 2003. [ Google Scholar ]
• Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ. Conducting a critical interpretative synthesis of the literature on access to healthcare by vulnerable groups. BMC Medical Research Methodology. 2006;6 doi: 10.1186/1471-2288-6-35. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Baughcum AE, Burklow KA, Deeks CM, Powers SW, Whitaker RC. Maternal feeding practices and childhood obesity: a focus group study of low-income mothers. Archives of Pediatrics & Adolescent Medicine. 1998;152:1010–1014. doi: 10.1001/archpedi.152.10.1010. [ DOI ] [ PubMed ] [ Google Scholar ]
• Baughcum AE, Powers SW, Johnson SB, Chamberlin LA, Deeks CM, Jain A, Whitaker RC. Vol. 22. J Dev Behav Pediatr; 2001. Maternal feeding practices and beliefs and their relationships to overweight in early childhood; pp. 391–408. [ DOI ] [ PubMed ] [ Google Scholar ]
• Hewat RJ, Ellis DJ. Vol. 2. Midwifery; 1986. Similarities and differences between women who breastfeed for short and long duration; pp. 37–43. [ DOI ] [ PubMed ] [ Google Scholar ]
• May KM. Vol. 23. Pediatr Nurs; 1997. Searching for normalcy: mothers' caregiving for low birth weight infants; pp. 17–20. [ PubMed ] [ Google Scholar ]
• McCann JB, Stein A, Fairburn CG, Dunger DB. Eating habits and attitudes of mothers of children with non-organic failure to thrive. Archives of Disease in Childhood. 1994:234–236. doi: 10.1136/adc.70.3.234. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Pridham KF. Information needs and problem solving behavior of parents of infants. Birth Defects: Original Article Series. 1984;20:125–165. [ PubMed ] [ Google Scholar ]
• Sturm LA, Drotar D, Laing K, Zimet GD. Mothers' beliefs about the causes of infant growth deficiency: is there attributional bias? Journal of Pediatric Psychology. 1997;22:329–344. doi: 10.1093/jpepsy/22.3.329. [ DOI ] [ PubMed ] [ Google Scholar ]
• Hall WA, Shearer K, Mogan J, Berkowitz J. Vol. 27. MCN Am J Matern Child Nurs; 2002. Weighing preterm infants before & after breastfeeding: does it increase maternal confidence and competence? pp. 318–326. [ DOI ] [ PubMed ] [ Google Scholar ]
• Rajan L, Oakley A. Vol. 6. Midwifery; 1990. Low birth weight babies: the mother's point of view; pp. 73–85. [ DOI ] [ PubMed ] [ Google Scholar ]
• Reifsnider E, Allan J, Percy M. Mothers' explanatory models of lack of child growth. Public Health Nursing. 2000;17:434–442. doi: 10.1046/j.1525-1446.2000.00434.x. [ DOI ] [ PubMed ] [ Google Scholar ]
• Sherratt F, Johnson A, Holmes S. Responding to parental concerns at the six-month stage. Health Visitor. 1991;64:84–86. [ PubMed ] [ Google Scholar ]
• Smith MP. Postnatal concerns of mothers: an update. Midwifery. 1989;5:182–188. doi: 10.1016/S0266-6138(89)80005-1. [ DOI ] [ PubMed ] [ Google Scholar ]
• Kramer MS, Barr RG, Leduc DG, Boisjoly C, Pless IB. Maternal psychological determinants of infant obesity. Development and testing of two new instruments. Journal of Chronic Diseases. 1983;36:329–335. doi: 10.1016/0021-9681(83)90118-2. [ DOI ] [ PubMed ] [ Google Scholar ]
• Brown MM. An exploration of parental concerns about preterms and full term infants during the first nine months of life. University of California, Los Angeles, USA; 1981. pp. 1–122. [ Google Scholar ]
• Vehvilainen-Julkunen K. The function of home visits in maternal and child welfare as evaluated by service providers and users. Journal of Advanced Nursing. 1994;20:672–678. doi: 10.1046/j.1365-2648.1994.20040672.x. [ DOI ] [ PubMed ] [ Google Scholar ]
• Rand CSW, Wright BA. Continuity and change in the evaluation of ideal and acceptable body sizes across a wide age span. International Journal of Eating Disorders. 2000;28:90–100. doi: 10.1002/(SICI)1098-108X(200007)28:1&#x0003c;90::AID-EAT11&#x0003e;3.0.CO;2-P. [ DOI ] [ PubMed ] [ Google Scholar ]
• Rand CSW, Wright BA. Thinner females and heavier males: Who says? A comparison of female to male ideal body sizes across a wide age span. International Journal of Eating Disorders. 2001;29:45–50. doi: 10.1002/1098-108X(200101)29:1&#x0003c;45::AID-EAT7&#x0003e;3.0.CO;2-I. [ DOI ] [ PubMed ] [ Google Scholar ]
• Birgenaeu CT. Body Image in Infancy. Adult body weight-related biases applied to infants. Univeristy of Massachesetts Boston, USA; 2001. pp. 1–127. [ Google Scholar ]
• View on publisher site
• PDF (263.1 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 04 November 2024

A multiple biomolecules-based rapid life detection protocol embedded in a rover scientific subsystem for soil sample analysis

• Akib Zaman 1 , 2 ,
• Fardeen Ashraf 2 ,
• Haseena Khan 3 ,
• Faria Noshin Ahona 3 ,
• Oliullah Samir 4 ,
• Asif Mahmud Rayhan 4 ,
• Sadia Nur Nazifa 2 ,
• Hafsah Mahzabin Chowdhury 2 &
• Md. Mahbubur Rahman 2  

Scientific Reports volume  14 , Article number:  26645 ( 2024 ) Cite this article

Metrics details

• Astrobiology

The study of whether life exists, is extinct, or not depends on various sophisticated experimental studies, as many different signatures of life can be used. The experimental procedures that can be performed to identify life can be further restricted by time, resources, and mobility constraints. Therefore, any research analyzing the presence of extraterrestrial life must be precise and unambiguous. This research focuses on the objective of the extraterrestrial life detection domain and seeks to provide an efficient protocol that can produce life detection decisions based on empirical data obtained through chemical analysis under time and resource-constrained conditions. While the majority of existing frameworks in this field are designed to identify biomolecules, our goal is to accomplish the same with minimal operational expense and mission complexity. We argue that the thoughtful integration of multiple biomolecular detections with lesser complexity and a robust framework can improve overall mission performance by satisfying the necessary time and resource constraints. In this study, a rapid multiple biomolecules-based life detection protocol (MBLDP-R) from soil samples is developed from scratch and embedded in an operational scientific rover subsystem targeted for planetary analysis missions. The study uses artificial biomolecule samples and simulated extraterrestrial environments to illustrate the suggested protocol’s end-to-end process. First, we list a few significant biomolecules, including lipids, proteins, carbohydrates, nucleic acids, ammonia, and pigments. Then, a weighted qualitative test scoring is applied to sort out the best test method for the finally selected biomolecules which are used as operational analogue to showcase the protocol’s in-situ analysis and decision-making capabilities. Based on the suitable biomolecules, a scientific exploration subsystem is developed, and the implemented protocol is built to perform onboard sample analysis. Evaluation results show that: (1) the proposed MBLDP-R protocol could effectively predict the classes with an average f1-score of 98.65% (macro) and 90.00% (micro), (2) the area under the Receiver Operating Characteristics (AUC-ROC) curve shows the sample categories to be correctly predicted 92% of the time (97% for Extant, 88% for Extinct, and 92% in the case of NPL), and (3) the protocol is time-efficient with an average completion time of 17.60 min, demonstrating the protocol’s rapid nature in detecting biosignatures in soil samples. The research outcome yields useful additional data for related future studies, particularly in the design of scientific frameworks for mission-specific requirements with limited resources while also serving as a reference point for constraint evaluation methods for similar systems.

Introduction

Knowing whether there is life beyond Earth has been a vital question for humanity for a long time. Several scientific exploration missions have recently been set for Mars since it is most similar to the Earth in the Solar System 1 , 2 . Soil, also known as pedosphere, serves as a habitat for various life forms and is a rich source of organic matter 3 . Newer science missions 4 , 5 to Mars are beginning to concentrate on soil and rock composition, in addition to examining the geology and environmental aspects of the red planet. And to do this, scientists choose to sample or analyse Martian soil and rocks. For instance, launched in July 2020, NASA’s Perseverance rover collects soil (regolith) and rock core samples and caches them in a specific coordinate that will be sent back to Earth for analysis in terrestrial labs. Simultaneously it searches for biosignatures in-situ using several onboard scientific equipment (PIXL, MEDA, SHERLOC) 4 . Similarly, the Tianwen-1 mission, launched in July 2020, aimed to learn more about the features of the surface and subsurface layers of Martian soil as well as the composition and types of rocks present there 5 .

Similarly, biomolecules are direct products of any cell or living organism and are essential components to test for the presence of life 6 . Studies 7 , 8 , 9 , 10 demonstrated the effectiveness of using a single class of biomolecule, such as a protein, nucleic acid, or other biomolecules, as successful biosignature detection mechanisms. However, dependency on a single biomolecule detection has several limitations, including uncertainty of the presence of that particular biomolecule and generation of false positives while classifying the presence of life in a soil sample. We assume that a life-detection framework, containing a global decision mechanism based on the local detection results of multiple biomolecules might be useful to address this issue.

Interestingly, in the future human exploration mission (already planned by NASA in 2031) 11 , 12 , there might be instances where life detection will be necessary for a remote hazardous area where humans cannot explore physically. This type of exploration is expected to be conducted by a mobile rover capable of onboard evaluation and instant result generation since storing soil samples for a long time in an extraterrestrial environment can be difficult. To add on, the rover might not even make it out of the mission due to extreme environmental and geological challenges. In this type of use case, time will be a massive constraint along with the installation of lighter and simpler equipment, sacrificing the perks of installing heavy equipment for conducting complex experiments of detecting biomolecules like ATP, Nucleic acid, etc.

Thus, we contend that by meeting the requisite time and resource restrictions, a multi-biomolecule life detection approach with lighter assay and complexity can address the hypothetical corner cases where resource conservation and maneuverability are critical. Consequently, this research targets the extraplanetary life detection domain and aims to provide a simple yet effective operational framework to make life detection decisions from soil samples based on empirical data achieved through chemical analysis in less time and resources. To achieve the research’s objective, the methodological phases enumerated (a) Development of an efficient and rapid on-site life-detection protocol from soil samples integrating multiple biomolecules; (b) Development of a mechanical rover subsystem for soil sample collection and classification for implementing the proposed biosignature detection protocol; and (c) Implementation of a novel evaluation methodology to assess the performance of the protocols that are used for biosignature detection from soil samples.

In this study, we develop a Multiple Biomolecules-based Rapid Life Detection Protocol (MBLDP-R) from soil samples with the integration of protein, carbohydrate, and ammonium ions. Firstly, we compile a list of potentially significant biomolecules, including protein, carbohydrate, nucleic acid, ammonia, pigment, and lipid. Then, we determine the optimum test methods for the selected biomolecules using a weighted qualitative test-scoring approach. The selection of biomolecules and their chosen detection methods considered in this study are not constant in terms of all life detection strategies. Rather the study attempts to create a baseline procedure that demonstrates the identification of biomolecules and their detection mechanism selection criteria for any future development of similar mechanisms. Using information extracted from the designated biomolecules in the earlier phase, the structure of the proposed protocol is developed. A three-layered decision tree is at the heart of the protocol’s architecture. Based on the presence and absence of the selected biomolecules in the collected sample, the tree generates life presence results as outcomes. Furthermore, a subsystem for scientific investigation is constructed along with a fully functional Mars rover prototype 13 , and onboard sample analysis is implemented using the developed MBLDP-R to evaluate the results in real-life test cases. The artificial samples used in the evaluation process are modified adhering to the rules of the University Rover Challenge (URC), one of the premium global Robotics Challenges organized by the Mars Society at the Mars Desert Research Station (MDRS), Southern Utah, USA. We also have demonstrated the evaluation results in the Science mission of the URC. Evaluation results indicate that: (1) the proposed MBLDP-R protocol accurately predicts classes with an average f1-score of 98.65% (macro) and 90.00% (micro); (2) the sample categories are correctly predicted 92% of the time (97% Extant, 88% Extinct, and 92% in the case of NPL); and (3) the protocol is time-efficient with an average completion time of 17.60 min, demonstrating the rapid nature of the protocol to detect biosignatures in soil samples.

This paper is divided into five sections. The following section, section " Literature review ", describes the prior work in this field. The research methodology is described in section " Methodology ", which includes the protocol development and the rover subsystem for scientific exploration. Evaluation of the samples using real-life samples is described in section " Evaluation ". Finally, section " Discussion and conclusion " summarizes the contribution of this work along with the limitations and future scopes.

Literature review

Several studies have been conducted to find the presence of life in soil samples. Some of these are directly intended for the exploration of Mars, and others have focused on finding a generic solution for the detection of life. A summary of the related works is included in Table  1 .

Recko et al. 14 developed a robotic platform-based soil sample collection technique with a unique sample collection method and suitability test based on humidity and UV light observations before collecting the sample. However, the study did not cover the possible techniques of soil sample analysis for detecting biosignatures. Neveu et al. 15 proposed a theoretical framework to guide the design of investigations to detect microbial life within the practical constraints of robotic space missions. They discussed extracting features related to life detection and the methodology for detecting them. However, this basic framework assumed the probability of microbial life being either above or below a rejection threshold. This measurement can be improved in a realistic scenario by quantifying a set of distinct numerical probability values. Kite et al. 16 also described a framework with a proposition that flying life-detection missions as hypothesis tests will maximize scientific value. In contrast, a negative detection can also become scientifically important. Similar to this, a theoretical framework was proposed by Sharukh et al. 17 , highlighting the significance of the companion mini-rover system for deep, narrow scientific research. The outcomes of these proposed investigations are based on a theoretical framework or hypothesis that can be further investigated by creating robotic subsystems to evaluate any procedure using empirical data sets.

On the contrary, Kiflen et al. 9 developed a UV-C spectrometer based on Ribonucleic Acid (RNA)/ Deoxyribonucleic Acid (DNA) extraction method to detect nucleic acid in the soil sample. On a similar note, Goordial et al. 8 used multiple low-cost techniques to detect viable extant microorganisms and nucleic acid from the soil sample in an environment. Similarly, Mojarro et al. 18 developed the Search for Extra-Terrestrial Genomes (SETG) equipment to isolate and identify the sequence of nucleic acids from extant or preserved life on Mars. This instrument blends nucleic acid extraction and nanopore sequencing. However, automation of these multi-steps incorporating sophisticated critical steps for successful DNA extraction, such as desalting, competitive binding, DNA-protein separation, etc., is resource exhaustive and costly. In another study, Mora et al. 10 created a Microchip Electrophoresis Laser-Induced Fluorescence (ME-LIF) instrument to receive and analyse a liquid soil sample in an automated module. This was a foundational development of microchip electrophoresis instruments for potential life-detection missions. However, this computerised module was developed to evaluate liquid soil samples based on protein (amino acid) detection. Similarly, Abrahamsson et al. 7 prominently marked amino acid as a significant bio-molecule for life detection and proposed a novel automated chiral amino acid analysis method. These studies described various individual techniques of soil sample analysis.

Moreover, several missions and studies develop instruments and techniques to support future planetary expeditions by exploring Mars-like drilling tools for subsurface samples 19 , earth volcanic samples to model martian geology 20 , mission simulations in lava terrains to evaluate equipments 21 , and creating Mars analog environments to replicate surface condition 22 . Few potential future missions target biomolecule detection, such as the assessment of Enceladus’s 23 (a moon of Saturn) ocean environment, plume composition, and biosignatures, as well as the exploration of metabolic products and signs of life while assessing the habitability and surface properties of Europa 24 (a moon of Jupiter). Continuing the exploration for correct biomarker detection Aerts et al. 25 proposed several techniques DNA, amino acids, lipids focusing on extracting the required biomolecule from samples and include High Performance Liquid Chromatography (HPLC), Gas Chromatography, Polymerase Chain Reaction (PCR), and acid digestion. Similarly, Gómez-Elvira et al. 26 proposed detection protocols based on protein interactions and outlined a microarray-based instrument design that can operate in a space environment. While these studies contribute to scientific advancement, their computational complexity, time requirements, and financial demands constrain the rapid prototyping of in-field systems capable of delivering reasonable results under limited time and resource conditions. In this work, We aim to demonstrate a protocol for decision-making based on multiple assay outcomes, optimized for limited time and resources. We also propose a weight-based qualitative method to establish trade-offs among constraints (see section " Phase 4: qualitative test scoring for selection of test methods "), which can be extended to other situations by adjusting the weights accordingly. While not directly comparable to studies using realistic samples, weight-based qualitative analysis with synthetic samples serves as a valuable operational analogue to demonstrate our protocol for in-field decision-making based on multiple assay results.

Methodology

In this section, we explain the methodology of MBLDP-R and the phase-wise development procedure. Figure  1 illustrates the overview of the methodological framework. Firstly, a handful of important biomolecules such as protein, carbohydrate, nucleic acid, ammonia, pigment, and lipid are listed. The listed biomolecules are filtered to meet the URC 2021 guidelines. Then, we develop a multilevel decision tree-based protocol analysing the correlation among the selected biomolecules (Figs.  2 and 3 ). A weighted qualitative analysis is carried out across all the accepted techniques for identifying the chosen biomolecules in order to determine the test score for each method and identify which test method is optimal for each biomolecule. Then, the best test methods are implemented and integrated with the scientific exploration subsystem for the constructed rover PHOENIX (Fig.  4 a). Finally, an exhaustive evaluation is carried out using empirical soil samples to evaluate the performance of the proposed protocol.

Research framework.

Development of MBLDP-R

The multiple biomolecules-based life detection protocol, MBLDP-R, is developed with the accumulation of works conducted in 4 phases (1) Potential list of biomolecules, (2) Selection of biomolecules based on requirement analysis, (3) Development of the protocol structure, and (4) Qualitative test scoring for the selection of the best test methods.

Phase 1: potential list of Biomolecules

Biomolecules are the chemical building blocks of living things, whereas biosignatures are observable signs of life in the environment. Despite some overlap, these two ideas are separate and have different functions in biology. A biosignature refers to any specific component, molecule, substance, or feature that can indicate past or present life and is distinct from an abiogenic (non-biological) background 27 . On the other hand, Biomolecules refer to essential organic compounds that play critical roles in various biological activities, including lipids, proteins, carbohydrates, and nucleic acids 28 . Making the distinction between possible and irrefutable biosignatures, the value of a biosignature is evaluated not only by the likelihood that life created it but also by the improbability of nonbiological processes producing it 29 . Biomolecules, when detected in specific contexts, serve as a subset of biosignatures, providing chemical evidence of life. For instance, chemical biosignatures such as, Molecular fossils provide evidence of past life as they are degradation products of biomolecules 27 . Major biomolecules have been considered a sign of extraterrestrial life, such as pigmented microorganisms 30 . However, since many biomolecules can be produced abiotically or due to other natural occurrences, one can argue against them being the concrete sign of life’s existence 31 .

In this work, we consider six major biomolecule protein, carbohydrate, ammonia, nucleic acid, lipid, and pigment as potential biomolecules for creating the multiple biomolecules-based biosignature detection protocol. Though not a conventional biomolecule, ammonia is considered here due to its derivation from the metabolism of amino acids and other biomolecules which contain nitrogen and is produced in soil from bacterial processes.

Protein . Extensive research circling archaeological remains dating from roughly 6000 years from today has pointed towards the correspondence between bimolecular preservation and the relationship between proteins, primarily collagen and mineral components, the durability of which depends chiefly on the elements of the soil 32 . If the thermal history of the burial environment is considered the deciding factor, relevant studies and models suggest that from warmer environments, proteins may provide opportunities to recover genetic information, as the diagenesis is controlled by slow, chemical decay of the organic phase 33 .

Ammonia . Mineralisation of nitrogen takes place in two steps, namely, ammonification and hydrolysis. In the first step, organic nitrogen is converted to ammonia, which in turn converts to ammonium ions in the presence of water in the second step. Ammonia, once formed, quickly escapes from the soil into the air. The positively charged ammonium ion is attracted to the negatively charged soil particles and thus acts as an exchangeable cation in soil 34 . The process is given below:

Ammonification of organic nitrogen.

\({\text{R }}--{\text{ N}}{{\text{H}}_{\text{2}}}+{{\text{H}}_{\text{2}}}{\text{O}} \to {\text{N}}{{\text{H}}_{\text{3}}}+{\text{R}} - {\text{OH}}+{\text{energy}}\)

Second step

Hydrolysis of ammonia.

\({\text{N}}{{\text{H}}_{\text{3}}}+{{\text{H}}_{\text{2}}}{\text{O}} \to {\text{N}}{{\text{H}}_{\text{4}}}^{+}+{\text{ O}}{{\text{H}}^ - }\)

Carbohydrate . Carbohydrates play a key role in biological recognition processes 35 , 36 . Carbohydrates can be classified as monosaccharides, oligosaccharides and polysaccharides. Monosaccharides such as galactose and mannose are the chief forms of carbohydrates produced by microorganisms, while arabinose and xylose are formed by plant tissue and roots 37 , 38 . These carbohydrates are the byproducts of microbial metabolism and have vital roles to play in the formation and stabilisation of soil structure 33 .

Nucleic Acid. Nucleic acids are considered an important mark of living cells and are the basic molecules of life 39 . They are naturally made polymers consisting of nucleotides that can store, encode, transmit and express genetic information 40 , 41 . Deoxyribonucleic acid (DNA) from a mummified Egyptian kid was found in a paper by Paabo 42 , which provides the first evidence of the presence of DNA in archaeological remains (radiocarbon dated to nearly 2,500 years). This further strengthens the selection of nucleic acid for the consideration of extinct or extant life.

Pigments . Pigments are found in high amounts in plants 43 . The largest two sources of pigments are plants and microorganisms 44 , 45 . Bacterial pigments can be obtained from soils that are harvested commercially for a range of purposes, including their use in medical, food, cosmetic, and textile fields 46 . These pigments, which are responsible for the appearance of colours in higher plants, are classified into several groups: chlorophylls, flavonoids (chalcones, anthocyanins, flavonols, flavones), carotenoids (xanthophylls, carotenes) and betalains (betacyanin, betaxanthin) 47 . Moreover, prokaryotic (cyanobacteria) and eukaryotic cells (cyanelles, red algae, and cryptomonads) contain phycobilins, a class of water-soluble photosynthetic pigments, in the cytoplasm or the stroma of the chloroplast 48 .

Lipids . Lipid is one of the three most abundant biomolecules found in animal tissues, the other two being protein and carbohydrate 43 . Lipids also make up between 2 and 20% of dry weights of bacteria 49 . Up to 20% of soil humus exists in the form of lipids 50 .

Phase 2: selected biomolecules based on requirement analysis

Requirement analysis is conducted emphasising two factors: (a) rules of URC 2021 and (b) compatibility of the test equipment for the biomolecule with the rover subsystem.

In URC 2021 48 51 , the Science mission was designed to find out the presence of life in given soil samples. The goal was to categorise the samples into three significant classes (a) Extinct, (b) Extant and (c) No Presence of Life where:

Extinct: Fossilized life that is no longer metabolizing 51 .

Extant: Life that is metabolizing, or died recently enough for the biomolecules to still be intact 51 .

No Presence of life: No existence of life, inanimate/abiotic objects 51 .

URC 2021 set various constraints in the case of the detection of biomolecules. One of the foremost constraints was the use of hazardous substances like concentrated nitric acid (HNO 3 ), sulphuric acid (H 2 SO 4 ) etc., to conduct biomolecule detection tests. Moreover, the time limitation of 30 min for test completion played a major role in the case of selecting the biomolecules. Besides, the rover needs to traverse from the start gate to the sample location, collect the data from the samples, and pass it back to the base station. Thus, an onboard payload with the necessary testing capability (Fig.  5 ) is required to be developed, which will also be adept in sending necessary data back to the base station. Additionally, the weight of the total rover system with the scientific exploration subsystem must be less than 50 Kg, which restricts test designs based on heavy equipment. Finally, the budget limit for the whole system was US dollars 18,000. An analysis of all six potential biomolecules based on the requirements is made to find out suitable biomolecules for the protocol.

Traditional spectrophotometric methods considered for detecting the likelihood of protein presence include the Biuret test, Ninhydrin test, Xanthoproteic test, Millon’s test and many more 52 . Although some of these traditional methods are not feasible considering time constraints and test preparations, some methods such as Biuret, Ninhydrin, and Xanthoproteic tests can be done under a time range of 1 to 10 min while applying a water bath as a heating source under a temperature range of 60 to 100 degrees 52 , 53 . More modern methods of protein detection include the use of various mass spectrometers such as IT-LIT, ToF-ToF, Q-Q-ToF, Q-Q-Q, FT-ICR, and QQ-LIT 54 . However, considering the factors of requirement analysis, such as the complexity of the equipment required for the tests, compatibility with the rover system and budget constraint of URC 2021, the traditional methods seems to be more suitable than the modern methods. On a similar note, methods of detecting ammonia from soil samples may include Schloesing’s methods, Baumann’s method or Russel’s methods 55 . Ammonia can also be detected by using a dilute sodium hydroxide solution and heating it. The resultant ammonia gas can be detected by its distinct pungent smell or by using a damp red litmus paper, which turns blue if the gas is present. This method is both simple and fast based on the type of heating source (less than a minute for direct heat). At present, various types of ammonia gas detecting sensors are available such as metal oxide-based sensors, conducting polymer sensors, tunable diode laser absorption spectroscopy (TDLAS), electrochemical sensors, surface acoustic wave sensors, field-effect transistor (FET) sensors, etc 56 . Easy detection procedures and readily available sensors make ammonia a potential biomolecule in these circumstances. On the other hand, for identifying the possibility of the presence of carbohydrates, several rapid tests based on specific colour reactions are available such as Molisch’s test, Anthrone test, Iodine test for glycans (starch, glycogen), Seliwanoff’s test for ketoses, Benedict’s test, Fehling’s test, and Picric acid test for reducing sugars Mucic acid test for galactose, Bial’s test for pentoses and Barfoed’s test to distinguish between monosaccharides from reducing disaccharides 57 . The majority of these test methods are fairly less complex and also time-efficient. Additionally, the equipment for conducting these tests is simple, lightweight and compatible with the rover system. Based on the conditions of requirement analysis, all three biomolecules: carbohydrates, ammonia and protein have the essential characteristics to be integrated as potential candidates for the protocol.

On the other hand, the nucleic acids or the microbial cells must be isolated from the soil particles and humic substances, which otherwise obscures nearly any observation, before the analysis of nucleic acid (DNA, RNA) from soil samples 58 . This multi-step approach of detection methods further complicates the implication of the nucleic acid detection system embedded in the rover. Moreover, automation of these multi-steps incorporating sophisticated key steps for successful DNA extraction, such as desalting, competitive binding, DNA-protein separation etc., is resource exhaustive and costly. In addition, extraction of DNA from different sources like a spore is generally a time-consuming process employed in the detection of nucleic acids from Martian or Mars-like soils, as well 18 and biological reagents used in this process are a concern in the context of URC protection regulations. Moreover, 30 min, the time set for the competition, is not enough for the rover to manoeuvre, collect the sample and use the DNA extraction kits for sample analysis. In the case of pigment detection, paper chromatography 59 , 60 is a prominent traditional method. Even though the technique can be applied to a variety of pigment detection tasks, its complexity as an onboard testing mechanism for the rover subsystem is increased because of its multi-step procedure. Moreover, modern techniques use High-Performance Liquid Chromatography (HPLC), a complex multi-step process that involves pigment extraction and HPLC analysis. The HPLC method can separate photosynthetic pigments like chlorophylls and carotenoids, etc 61 , 62 , 63 . The equipment needed for HPLC is sophisticated, non-modifiable and expensive. The equipment of HPLC is sophisticated, non-modifiable and very costly. A portable HPLC machine weighs up to 8 kg 64 , which is not very efficient considering the 50 Kg weight limit of the total rover system. Thus, it is certainly not possible to include pigment detection in the multiple biomolecule-based protocol. On the other hand, conventional methods for lipid quantification rely on solvent extraction and either gravimetric assays or chromatographic determination, which are time-consuming multi-step procedures that depend on the creation of the required test environments 65 . Thus, the complexity level of these methods is not suitable for the rover system and it is not convenient to execute the methods in an external environment. In contrast, colourimetric methods for lipid quantification constitute an attractive choice due to their fast response and simplified sample handling; however, they also usually require other preliminary steps, such as cell disruption and lipid extraction 66 , that are not convenient based on compatibility and complexity. Furthermore, other nonconventional spectroscopic methods include infrared spectroscopy, nuclear magnetic resonance spectroscopy, Raman spectroscopy, fluorescence spectroscopy and dielectric spectrometry 66 , which involve costly equipment. The above analysis shows that the tests of nucleic acid, pigment and lipid detection do not fulfil the minimum criteria of the requirement analysis. Thus, they are not considered for further analysis. Based on the above-mentioned discussions, protein, carbohydrate and ammonium ions are selected to develop a multiple biomolecule-based biosignature detection protocol for the soil samples.

Phase 3: development of the multilayered decision-tree Framework

Using information compiled from the selected biomolecules (protein, carbohydrate and ammonium ion) mentioned in 3.1.2, a multilevel decision tree-based framework is developed to integrate the results of multiple biomolecular detection for classifying an unknown soil sample. A detailed architecture of the framework is illustrated in Fig.  2 . The tree consists of 3 decision layers. Each of these layers contains unit blocks. This unit blocks work as decision breakpoints for the tree. From the topmost layer, the most weighted biomolecule unit blocks are present. In this research, the most weighted biomolecule for life presence making a decision is deemed to be protein. Then the carbohydrate and ammonia unit blocks are present in the later layers of the tree, respectively. Each layer of the tree is connected with a binary decision selector. These selectors help the protocol in branching out its decision from the topmost layer of the tree to the bottom. With each passing layer, the plausible decision-making from the protocol increases by an exponential of two.

Considering the structure of a unit block, each block contains a definite biomolecule as its domain, where a number of chemical analysis tests, related to the domain, are also present as the blocks cell. Each of these cells are weighted. The description of calculating the weight score based on a six-principle qualitative analysis can be found in the subsequent subsection " Phase 4: qualitative test scoring for selection of test methods " in detail. The weighted calculation of these cells determines which cell will be considered to be passed on to the binary decision selector for chemical analysis. Each unit block calculates the weight of each cell present within it and passes the most weighted cell to the binary decision selector. The binary decision selector then uses the chemical analysis test present in the past cell and based on its result passes a binary result (true/false) in the next layer. This unit block structure is followed for all the blocks of the tree until the leaf unit blocks of the last layer. The leaf unit blocks also pass their best weighted cell to binary decision selector but unlike passing it onto another layer, these selectors provide life expectancy decisions based on the flow of data on the tree.

Architecture of the multiple biomolecule-based framework for soil sample detection.

Based on the presence and absence of the selected biomolecules in the collected sample, the tree generates life expectancy results as outcomes. The presence of protein is examined at the very first layer of the tree. Following that, the tree looks for carbohydrates, and lastly, ammonium ions. There are eight potential results of the proposed protocol. When all three types of biomolecules are present, the result is classified as Extant; if none are present, the classification is NPL. All living things have proteins, which cannot be found in an abiotic environment. Moreover, one important class of biomolecules that can be discovered in subsurface fossils is protein 33 . Similarly, microorganisms, plant tissue, or tree roots all contribute to the presence of carbohydrates in soil samples 67 . Fossilized samples of protozoa, mollusks, arthropods, and plants also contain carbohydrates 68 . Furthermore, ancient sedimentary rocks contain carbohydrates as well 69 . Even though both proteins and carbohydrates can be found as fossil molecules, carbohydrates exhibit greater resistance to decay than proteins, suggesting a greater likelihood of their retention in fossil records or absorption into fossil fuels 70 . Hence, the presence of protein always provides an Extant result for us independent of the other two sorts of molecules. The framework gives priority to the existence of carbohydrates in the absence of protein. Regardless of whether ammonium ion (NH4+) is present, the framework always produces the result “Extinct” if carbohydrates are present but protein is absent. Although ammonium ion (NH4+) can be generated abiotically or biotically, it can also be found in living things 71 . Because of this, any one of the three outcomes (Extant, Extinct, and NPL) cannot be determined by the presence of ammonium ion alone. This ambiguous outcome is treated as NPL by the framework to prevent difficulties.

Phase 4: qualitative test scoring for selection of test methods

Various tests exist for detecting proteins, carbohydrates, and ammonium ions. However, the requirement analysis discussed in 3.1.2 does not allow all the variants of the tests to be conducted in an onboard laboratory as a rover subsystem. Thus, six principles are extracted from the requirement analysis, and the principles are assigned a numerical weight according to priority based on requirement analysis factors. After that, a weighted test scoring based on these principles is conducted to select the best suitable test methods for the different biomolecules. Selected principles for qualitative analysis are as follows:

Hazardous Substances (HS). Any substance that can potentially affect safety and cause health hazards is deemed hazardous. Properties that can compromise the safety of the users and the components include flammability, explosiveness, toxicity, and the ability to oxidize. Concentrated H 2 SO 4 , HCl, NaOH, H 2 O 2 , Br are some examples of hazardous substances. For strict safety purposes, hazardous substances are not allowed in the on-site tests ( Labels: Require HS-0 , Don’t Require HS-1; Weight = 1 ).

Time (T). The amount of time required to conduct an experiment and to get the result ( Labels: > than 25 min-0 , <= 25 min − 1 , Weight = 0.9 ).

Color Identification (CI). It defines the recognition of colours of the components of a chemical reaction once it reaches the desired stage. Although colour identification alone is not a reliable procedure in the identification of compounds due to the effect chemical impurities have on the colour of compounds, this technique essentially provides a head start to the identification process ( Labels: Distinct colour-1 , No Distinct Colour-0 , weight = 0.8 ).

Unit Compound Identification (UCI). Unit compound (UC) identification such as amino acids for proteins and monosaccharides for carbohydrates is important as it ensures the feasibility of the test for all variants of that particular biomolecule ( Labels: UC identified-1 , UC Not Identified-0 , weight = 0.7 ).

Direct Heat Contact (DHC). Direct heat contact refers to the process of test completion over a heating component rather than using a water bath. Although a water bath can provide more precise control over the heat generated for the tests, DHC is more suitable for a rapid and spill-free approach. Usage of heating components with direct contact of testing beakers also removes the complications of using water bath heating mechanism on a rover body. Regarding the safety of the rover, non-flame-generating heating components are considered for DHC. Thus, it is more convenient to use direct heat using a heating pad for a rover subsystem ( Labels: Functional under DHC-1 , Not Functional under DHC-0 , weight = 0.5 ).

False Positive (FP). While identifying a specific compound, the presence of certain compounds can cause a positive result while the actual compound is absent. This positive result turns out to be a False Positive that is not desired ( Labels: Doesn’t Give False Positive − 1 , Gives False Positive – 0 , weight = 0.3 ).

Modern instrumental methods are far more sensitive and accurate in terms of detecting or quantifying certain biomolecules than traditional spectrophotometric methods. However, these modern instruments can be expensive and difficult to manipulate for certain mission scenarios as discussed in the previous subsection. Considering the budget constraints given by the URC and the feasibility of detecting these biomolecules on rover bodies through on-board analysis, mostly traditional spectrophotometric detection methods have been considered in this research.

Some of the most popular test methods for the detection of proteins, carbohydrates and ammonium ions are considered for the scoring procedure. The tests are scored according to the characteristics of the test procedures in the six selected scoring principles. They are Biuret, Xanthoproteic, Millon’s and Ninhydrin test for protein detection; Benedict’s, Seliwanoff’s, Barfoed’s, Molisch’s and Iodine/Lugol’s tests for carbohydrate detection; and Schloesing’s method, direct distillation with magnesium ion and Litmus strip tests for ammonium detection. The above-mentioned tests are conducted in the Chemistry laboratory of MIST and the results are documented to determine the score of the principles. Test scoring results of the mentioned methods are shown in Table  2 . The total score for any test has been calculated using the formula:

\(K\,=\,{\text{Set of Principles }}={\text{ }}\left( {{\text{HS}},{\text{ T}},{\text{ CI}},{\text{ UCI}},{\text{ DHC}},{\text{ FP}}} \right)\)

It is evident (Table  2 ) that although the Biuret test can detect the presence of peptide bonds in a substantial peptide bond concentrated protein sample, it fails to detect the unit compound amino acids (#4 UCI) in the soil specimen. Moreover, ammonium sulfate can often generate colored complexes or interfere with color development resulting in false positives (#6 FP) . On a similar note, Millon’s test cannot identify the unit compound (#4) , amino acids in the soil sample despite having a quick completion time of 2–3 min (#2 T) . Moreover, compounds like salicylic acid and phenolic compounds can give rise to a false positive (#6) in this method. Both Biuret and Millon’s tests get a score of 2.7 (Table  2 ). In contrast, detection of amino acids by both Ninhydrin and Xanthoproteic tests can be stated as indirect protein assay methods. However, the prime component for the nitration reaction in the Xanthoproteic test is concentrated HNO 3 which is a hazardous substance (#1 HS) and requires careful handling. The use of the Ninhydrin test is time-efficient with the detection time only between 3 and 4 min (#2) . It can also easily detect amino acids (#4) indicating the likelihood of protein presence in the soil sample. Thus, the Ninhydrin test is selected with a total test score of 4.2 while the Xanthoproteic test is the nearest one with a test score of 3.2 (Table  2 ).

In the case of carbohydrate detection, Seliwanoff’s reagent contains highly concentrated HCl (#1) that may compromise the safety of the rover. Moreover, it detects aldose and ketose but is unable to detect monosaccharides (#4) and the use of a water bath (#5 DHC) is a must in this method. Similarly, Iodine/Lugol’s solution test can only detect polysaccharides (#4) and cannot identify monosaccharides. Although Barfoed’s method can successfully detect monosaccharides (#4) , direct heat contact (#5) is not convenient for the test. Again, the concentrated sulfuric acid (#1) required to conduct Molisch’s test for the detection of carbohydrates is highly corrosive and is potentially explosive in its concentrated form. In addition, the presence of some organic acids (citric acid, lactic acid, oxalic acid, formic acid etc.) can give false-positive (#6) results in this method. Benedict’s test identifies monosaccharides (#4) in a soil sample and does not involve any corrosive ingredients (#1) in the process. Considering all the principles, Benedict’s test achieves the highest test score of 4.2 than Barfoed’s (3.7), Iodine/Lugol’s (3), Seliwanoff’s (2.0) or Molisch’s (1.7) test (Table  2 ). Thus, Benedict’s test is selected for the detection of carbohydrates. As previously mentioned, not all types of protein and carbohydrates are detected by the selected assays. The presence of these detection tests rather predicts the likelihood that the sample includes carbs and protein.

In the case of ammonium ion detection, the first Schloesing’s method requires strong sodium hydroxide. It, therefore, is discontinued because this strong alkali even at low temperatures gradually decomposes the organic nitrogen compounds giving rise to ammonium ions. Hydrochloric acid (#1) needed in the second Schloesing’s method removes approximately 60–70% of the added ammonium ions 55 . This method requires the distillation of the soil directly with magnesia, and the amount of ammonium obtained by distillation is dependent on the duration of the distillation, which is proportional to the amount of water and soil used and, on the amount of heat applied. Therefore, it takes more time (#2) to get the ammonium ions by this process. Moreover, these two methods do not give coloured identification (#3 CI) during the test. The use of hazardous substances and the length of time required to obtain the ammonium ions are drawbacks of the first two methods resulting in a test score of 1.2 and 1.8 respectively (Table  2 ). On the other hand, in the Litmus Strip test sodium hydroxide solution is mixed directly with the soil sample and then heated. The ammonium ions present in the sample then turns into ammonia gas and this gas turns the red litmus paper blue. With the highest score of 3.5 (Table  2 ) among all the ammonium ion detection methods, the litmus strip test method is selected for the detection.

The weighted analysis system for selecting assays is a dynamic component of the proposed protocol, as the weight criteria can be adjusted based on mission requirements and constraints to better align with a positive development stage later on. For instance, this study’s demonstrated case illustrates the protocol’s usage in URC. The URC mission criteria, as previously stated, required strict time and resource limits while discouraging the use of hazardous compounds. Therefore, the hazardous substance weight (HS) and time weight (T) are given higher priority than the others in this weighted analysis. The color weight is also emphasized due to the availability of color-detecting sensors, rather than more precise detectors that would reduce false positives. Similarly, in other extraterritorial mission scenarios, mission planners can dynamically adapt their in-situ experimentation plan using the weighted analysis even before the development phase begins. For example, in missions focused on rapid testing, the time weight (T) can be heavily prioritized to ensure speed, while exploratory missions requiring more precise results can reduce T and increase the false positive (FP) weight to enable more thorough analyses with fewer false positives. Similarly, in resource-limited missions, the color identification weight (CI) and the device heating control weight (DHC) can be elevated to favor simpler methods and heating mechanisms, whereas missions demanding precise control or advanced molecular detection can adjust these weights accordingly, ensuring the protocol aligns with varying mission priorities and constraints.

Development of the Rover Scientific Exploration Subsystem

The rover subsystem is developed to be attached to the main chassis to provide a tool suitable for collecting soil samples from remote and difficult terrains.

Design and fabrication

For evaluating the proposed protocol, a mechanical payload is designed as shown in (Fig.  3 ). The subsystem has four degrees of freedom (DOF) mechanism. There are three main parts of the developed subsystem such as (a) manipulator (b) sample collection chamber and (c) reagent mechanism. The body of the mechanical payload is made of stainless steel and aluminum sheets and is capable of collecting 4 soil samples at a time and conducting tests for protein (Ninhydrin), carbohydrate (Benedict), and ammonium ion (Litmus strip test) concurrently.

Design of the rover subsystem.

Manipulator

As depicted in Fig.  4 b, the subsystem is able to gather soil samples using a three-degrees-of-freedom (DOF) manipulator to which a custom-made drill bit is mounted as the end effector. The drill bit is composed of mild steel and measures roughly 100 mm in length and 35 mm in diameter. The stainless steel plates are supported by linear rods with a diameter of 8 millimeters and linear bearings with a 3D-printed housing to ensure smooth motion. Two 8 mm-diameter square-threaded lead screws are used to drive the manipulator to collect samples of soil from the ground surface and then place them in the beaker for further processing. The lead screws are driven by two separate stepper motors (NEMA 17) A 5*8 coupler is used to connect lead screws to stepper motors. The manipulator may move along the X and Z axes using a lead screw mechanism, while the end effector spins along the Z axis to gather soil samples. A liquid sanitization mechanism is included to prevent cross-contamination. Following each sample collection, the end effector releases a flow of 3% hydrogen peroxide solution.

Sample collection chamber

The sample collection chamber comprises of 12 beakers made of borosilicate glass, a 2.5 mm thick aluminum frame, and a heat source. To gather samples, the frame adjacent to the beakers can move back and forth along the Y-axis. As illustrated in the diagram, samples are placed along the Y-axis and concurrent biomolecules testing are planned along the X-axis (Fig.  5 b). As a heat source capable of generating 3,600 W/hour, a tailored heat pad is employed to provide the necessary heat for the testing.

(a) Complete rover system (PHOENIX) with the attached scientific subsystem. (b) Collection of soil samples through the subsystem.

Reagent flow mechanism

Three containers carrying the chemicals for the chosen tests, a litmus device for the ammonium test, and a framework for a single DOF tube make up the reagent mechanism. The containers are filled with Benedict’s reagent (carbohydrate test), Ninhydrin reagent (protein test), Sodium Hydroxide solution (ammonium ion test), and litmus paper strips in accordance with the selected tests (ammonium ion test). Reagents are delivered by three 12-volt DC Diaphragm Metering Micro Short Motor Water Pumps.

Detection mechanism

Four NEMA 17 stepper motors alongside the drivers are used to mobilize the motions of the subsystem. The whole rover system is controlled manually by a controller remote using a 915 MHz communication frequency, which has a range of more than 1 km. As shown in Fig.  6 , the rover at first moves to the desired sample location and collects the sample using the manipulator. Using the motion of the beaker-holding frame along the y-axis, the samples collected are stored in three rows of a dedicated column of the collection chamber. Then, moving the manipulator along the x-axis the rover can collect up to four samples and store them in each column. After collecting samples for all four columns, the reagent flow mechanism adds the regents to the beakers by sliding down the frame along the x-axis to reach all the columns as shown in Fig.  5 a. The heat source provides the required amount of heat. Then, by using a servomotor, a SS bar with three litmus strips is held on top of the beakers of row three as shown in Fig.  5 b. The visual color change of rows one and two and the colour of litmus paper of row three gives the status of protein, carbohydrate and ammonium to the samples respectively. The observation is noted using USB camera feedback from the base station. Finally, the output of the three results is used as the input to the proposed MBLDP-R to classify the sample.

(a) Setup of test methods. (b) Onboard sample analysis in the developed subsystem.

The evaluation study is conducted in the chemistry laboratory of MIST and at a mock science mission field. The tests are conducted in a series of 10 events. At each event, 4 of the samples are placed in various locations of the mock science field. The gathered data from the field are transmitted back to the main base and the samples are predicted using the proposed protocol as a backbone algorithm. The field test videos are available at the deposited official System Acceptance Review (SAR) video 72 of the team.

Preparation of samples

The soil samples are collected from a local site. To enhance or decrease the bioload, however, essential alteration is made. The samples are given the URC 2021 criteria labels of Extant, Extinct, and No Presence of Life (NPL). The “Extant” samples are made up of rich dirt. These samples have any unwanted rock particles removed. In addition, several of these samples have additional bioloads added to them (dextrose, albumin, etc.). According to the specific instructions of the URC science judges and organizing committeeg 51 , the No Presence of Life (NPL) samples are made by baking the soil for more than 24 h at 500. Two steps are taken to prepare the Extinct samples. Phase one involves heating the gathered samples at 500 degrees for more than 24 h to destroy any biosignatures that might have been present. Phase two involves creating an Extinct status by combining the samples with crushed fishbones.

Evaluation using mock Science Mission setup.

Sensitivity analysis of the colorimetric reactions

An evaluation study is conducted to measure the sensitivity of the tests. The selected Ninhydrin test of this research is sensitive enough to detect up to 0.001 pmol of amines, tested from 15 sets of protein samples. This quantity yields a reading at 570 nm of 0.028 absorbance units and the typical reagent blank is 0.02 units using the standard assay parameters for the monitoring test. Amines present less than the lower limit in the soil sample will not produce any colors hence will not be detected by the rover scientific exploration subsystem and hence will produce a False Negative Output.

Study procedure

The evaluation is conducted in a mock science field of MIST. A total of 10 events are conducted where 4 samples are used in each event. In a single event, the rover has to traverse 4 sample locations (Fig.  6 ) and conduct the onboard sample analysis using the developed subsystem. The evaluation is conducted using 40 samples consisting of 18 Extant, 12 Extinct, and 10 NPL samples.

The study indicates that the methodology was 90% accurate (36 out of 40) in classifying samples into intended classes. In the case of Extant samples, the technique attains 94.44% accuracy (17 out of 18). Furthermore, the method’s accuracy is 83.33% (10 of 12) for Extinct samples and 90% (9 of 10) for NPL samples. However, accuracy is the most suited performance indicator for binary classes instead of numerous classes 73 . Thus, performance indicators like F1-score, precision, and recall are calculated per class to address this issue. Figure  7 depicts a confusion matrix used to evaluate class identification performance. Moreover, Micro and Macro averaging, widely acknowledged and regularly used in multiclass classification research, are utilized to assess the overall classification performance of MBLDP-R.

Confusion matrix of the evaluation study of the protocol.

Precision (See Eq. 2) is the ratio of true positive (TP) elements to positively predicted units (column sum of the predicted positives). It expresses the proportion of units deemed positive by the model, and, they are positive. In other words, precision reflects our confidence in the model’s accurate sample prediction. Recall is the ratio of True Positive elements to positively classified units (row sum of the actual positives). The prediction effectiveness of the model for the positive class is quantified by recall (see Eq. 3). Intuitively, it assesses the model’s capacity to locate all positive units in the dataset. F1-score evaluates the effectiveness of a classification model beginning with the confusion matrix; it combines precision and recalls’ metrics using the idea of harmonic mean. F1-score (See Eq. 4) is the weighted average of precision and recall, with a maximum score of 1 (100.00) and a minimum score of 0 (0.00). The proportional contribution of precision and recall equals the F1-score. The harmonic mean can determine the optimal trade-off between the two quantities.

Thus, for any class k among the three classes (Extant, Extinct and NPL):

The macro-averaged measures of precision, recall, and F1-score are the simple average overall classes with equal weight to each incident type while micro average measures are based on the cumulative number of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN) per studied type 73 .

Class wise performance of the protocol is shown in Fig.  7 . Among the three classes, Extant samples are better predicted by the protocol with an F1-score of 97.14. The protocol slightly outperforms in the case of Extinct Class prediction (F1-score: 83.33) than NPL class (F1-score: 85.71). The micro average for recall, precision, and F1-score in the proposed is 89.25, 88.38, and 88.72, respectively. Since this is a single labelled multi-classification problem, the micro average’s precision and recall are both equal to the protocol’s accuracy of 90%, from which the harmonic mean, or micro F1 score, is calculated as a similar value (Table  3 ).

The area under the curve (AUC) of receiver operating characteristics (ROC) curve (AUC-ROC) is a performance metric that is based on a varying threshold value. ROC is a probability curve and the area under the curve (AUC) measures separability. In summary, the AUC metric announces the capability of the protocol in distinguishing the classes. AUC ranges from 0 to 1 and higher value of AUC depicts better model (1 depicts a perfect model). Mathematically, it can be created by plotting true positive rate (TPR) i.e., sensitivity or recall vs. false positive rate (FPR) (1- Specificity ( \(\:\frac{TN}{TN+FP})\) ) on varying threshold values 74 . AUC for the curve of the Extant class (0.97) demonstrates the highest score, followed by NPL (0.92) and Extinct (0.88) with a small margin between them. Overall, the micro and macro average of the protocol is 0.92 which depicts that MBLDP-R can correctly predict the class of an unknown sample in 92% of the instances.

Intriguingly, the average time to complete the tests is 17.6 min, whereas the least and maximum times are 15.20 and 19.45 min, respectively. Extant samples require 17.3 min, whereas Extinct samples require 16.8 min, and NPL samples require 18.2 min. In addition, negative tests require an average of 18.4 min longer. Positive protein tests require additional processing time. All three yes tests require the least time to complete, averaging 15.2 min.

Discussion and conclusion

In compliance with URC 2021 guidelines, a time-efficient, multiple biomolecules (protein, carbohydrate and ammonium ions) based life detection protocol from soil sample analysis (MBLDP-R) is established through the research work described here. One of the significant outcomes of this research is using a validation methodology for developing multiple biomolecule-based life detection protocols and proposing a functional protocol for classifying the samples which shows consistency and rigidity in mock in situ tests. MBLDP-R proposes a multiple biomolecule-based detection framework to be embedded in a rover’s scientific subsystem for planetary exploration. A weighted test scoring algorithm is designed to find the most suitable test method for the given situation. The implemented test-scoring scoring method can also be applied in different situations by setting the weights accordingly. A robust scientific exploration rover subsystem is developed to evaluate the proposed protocol. The evaluation study shows that the suggested protocol successfully identifies the biosignatures in soil samples and categorizes them into the three desired classifications Extant, Extinct, and No Presence of Life (NPL). Furthermore, the study provides evidence for the suggested protocol’s usability and functionality in real-world situations by showing how well it performs in simulated test environments. The protocol scores an adequate AUC value (0.92 Micro and 0.97 Macro) and excellent recall (89.25), precision (88.38), and F1-Score (88.72) on the experimental tests of identifying signs of life in mock soil samples. Moreover, an average time of 17.6 min for completing a full detection ensures that the requirement of the protocol to be rapid is met. Table  4 provides a summary of the several MBLDP-R feature factors as well as other well-known life detection approaches that were evaluated for this study.

Additionally, in contrast to other established techniques based on a single biomolecule, several biomolecule integrations reduce the likelihood of false negatives. For instance: single biomolecule-based approach of Mora et al. 10 terms a sample as No Presence of Life (NPL) in the absence of amino acid in that sample. However, several other important biomolecules (lipid, carbohydrate, etc.) remain unidentified due to the architectural limitations of the protocol. Thus, many samples with the absence of amino acid but with the presence of other biomolecules are termed as NPL due to the lack of contribution of multiple biomolecules during the classification of the samples. A similar statement is also true for a handful of prevailing works 8 , 9 , 18 where only nucleic acid has been used to detect the signature of life.

On the contrary, MBLDP-R aims to maximize the likelihood of detecting at least one biomolecule in the sample, thereby reducing the risk of obtaining a false negative result, by establishing a threshold value. For example: MBLDP-R uses the presence of both carbohydrates and ammonium ions or solely carbohydrates to identify the sample as Extant , making it capable of detecting life even in the absence of amino acids. However, our framework does not completely eliminate the possibility of receiving a false positive due to the inherent limitations of the individual biomolecule tests, which should be further explored in the future.

The research compares the weightage of different biomolecules found in nature in life detection decisions, the complexity of testing them on mobile robotic platforms, and how resource and time-consuming these tests might be. Following this comparison, the research provides MBLDP-R, which covers both the theoretical and technical aspects of life detection. Finally, with the on-field evaluations, the research provides a clear image of how MBLDP-R will operate in a time and resource-constrained environment, which can assist in the development of more resilient and efficient future protocols. Additionally, the developed subsystem with the implemented protocol is resource and time efficient unlike some of the prior life detection protocols 8 , 10 , 18 . Finally, starting with the development of the protocol, choosing the best tests for biomolecule detection, development of a robotic subsystem capable of sample collection and onboard analysis, and lastly, evaluation with the real-time samples demonstrate the holistic approach of this research that can be helpful for other researchers to explore multiple biomolecules-based life detection methods.

Even with its efficiency and consistent performance, the work has some limitations. Firstly, the initial filtration of the biomolecules is based on the guidelines of URC 2021. If there is no time limit, the selected biomolecules could be different and thereby the principles for test scoring would be different. As was previously noted in sub-subsection Phase 1: potential List of Biomolecules , the addition of time-consuming procedures like nucleic acid and pigment detection will have a greater influence on life detection decisions. So, adding these time-consuming but precise test techniques can increase operating time while improving the framework’s resilience and decision-making capability, paving a path toward claiming distinct science fidelity. This can be a potential extension of the research in the future. Additionally, there is also a chance of false positives because the selected analytes can be created abiotically; this can be mitigated in future extensions of this study by taking into account the investigation of biomolecules that cannot be made abiotically. Furthermore, due to limited publicly available information regarding materials widely recognized by experts as analogues to Mars regolith or potential biosignatures, we face challenges in making detailed distinctions between our samples in this work and the Martian regoliths, which can be potentially added in the future works. Secondly, the protocol is evaluated using a moderately sized sample of 40. There are future scopes to evaluate this protocol with a large sample size along with an exploration of variations of this protocol. Another limitation is the sensitivity of the protocol. For the detection of the biomolecules, the protocol ensured that the samples to be detected as Extant, Extinct or NPL are exactly that. Therefore, the protocol had no scope to test how sensitive it is in determining the different biomolecules. Whether the protocol would detect trace amounts of the biomolecules is not apparent in this study, which can be an interesting future scope of this research. Additionally, because the rover is anticipated to operate in an extraterrestrial environment, the soil sample that it has collected is taken into account as a potential test subject as such by the protocol. While measuring the desired biomolecules, the soil inorganic particles are not separated from the sample. It would take time to complete the separating process and would compromise the rapidness of the developed procedure. However, one intriguing extension of the research can be to separate the organic particles before analyzing a soil sample utilizing the proposed MBLDP-R.

Data availability

The datasets used and/or analyzed during the current study available from the corresponding author on reasonable request.

Zorpette, G. Why go to Mars? Sci. Am. 282 , 40–43 (2000).

Article   CAS   PubMed   Google Scholar  

Zubrin, R. Why we earthlings should colonize mars! Theol. Sci. 17 , 305–316 (2019).

Article   Google Scholar  

Speight, J. G. Water systems. in Natural Water Remediation (ed Speight, J. G.) 1–51 (Elsevier, (2020).

Li, C. et al. China’s Mars exploration mission and science investigation. Space Sci. Rev. 217 , 57 (2021).

Article   ADS   Google Scholar  

Farley, K. A. et al. Mars 2020 mission overview. Space Sci. Rev. 216 , 1–41 (2020).

Aerts, J. W., Röling, W. F. M., Elsaesser, A. & Ehrenfreund, P. Biota and biomolecules in extreme environments on Earth: implications for life detection on Mars. Life . 4 , 535–565 (2014).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Abrahamsson, V. et al. Extraction and separation of chiral amino acids for life detection on Ocean Worlds without using organic solvents or derivatization. Astrobiology . 21 , 575–586 (2021).

Article   ADS   CAS   PubMed   Google Scholar  

Goordial, J. et al. In situ field sequencing and life detection in remote (79°26′N) Canadian high arctic permafrost ice wedge microbial communities. Front. Microbiol. 8 , (2017).

Kiflen, M., Shariff, O., Mahdi, H. & Balsara, F. Novel nucleic acid-based soil sample analysis system for planetary exploration. Preprint at (2020).

Mora, M. F., Kehl, F., da Costa, E., Bramall, N. & Willis, P. A. fully automated microchip electrophoresis analyzer for potential life detection missions. Anal. Chem. 92 , 12959–12966 (2020).

Wilson, J. NASA’s Orion flight test and the Journey to Mars. Preprint at (2017).

Wall, M. Astronauts will face many hazards on a journey to mars. Space Preprint at (2019).

Zaman, A. et al. Phoenix: towards Designing and developing a human Assistant Rover. IEEE Access. 10 , (2022).

Rećko, M., Tołstoj-Sienkiewicz, J. & Turycz, P. Versatile soil sampling system capable of collecting, transporting, storing and preliminary onboard analysis for Mars rover analogue. Solid State Phenom. 260 , 59–65 (2017).

Neveu, M., Hays, L. E., Voytek, M. A., New, M. H. & Schulte, M. D. The ladder of life detection. Astrobiology . 18 , 1375–1402 (2018).

Kite, E. S., Gaidos, E. & Onstott, T. C. Valuing life-detection missions. Astrobiology . 18 , 834–840 (2018).

Article   ADS   PubMed   Google Scholar  

Sharukh, M., Karim, B., Nv, S. & Thomas, J. Intelligent deployable mini rover from the mars rover for deep narrow scientific investigation. Int. J. Sci. Res. Comput. Sci. Eng. Inform. Technol. 174–184 (2019).

Mojarro, A., Ruvkun, G., Zuber, M. T. & Carr, C. E. Nucleic acid extraction from synthetic Mars analog soils for in situ life detection. Astrobiology . 17 , 747–760 (2017).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Tabor, A. ARADS project designs tools for finding signs of life. Preprint at (2019).

Gentry, D. et al. Field Exploration and Life Detection Sampling for Planetary Analogue Research (FELDSPAR): Variability and Correlation in Biomarker and Mineralogy Measurements from Icelandic Mars Analogues. in Lunar and Planetary Science Conference (2018).

Tabor, A. What is BASALT? - preparing for human deep space missions, on Earth. Preprint at (2019).

Colen, J. Drilling for data: Simulating the Search for Life on mars. Preprint at (2017).

MacKenzie, S. M. et al. The Enceladus Orbilander mission concept: balancing return and resources in the search for life. Planet. Sci. J. 2 , 77 (2021).

Hand, K. P. et al. Science goals and mission architecture of the Europa lander mission concept. Planet. Sci. J. 3 , 22 (2022).

Aerts, J., Röling, W., Elsaesser, A. & Ehrenfreund, P. Biota and biomolecules in extreme environments on earth: implications for life detection on mars. Life (Basel) . 4 (4), 535–565 (2014).

ADS   PubMed   Google Scholar  

Gómez-Elvira, J. et al. Biology for Life Detection (MoBiLD) on Mars Authors (ESA Publications Division, 2003).

Domagal-Goldman, Shawn & Wright, Katherine & Adamala, Katarzyna & Rubia, Leigh & Bond, Jade & Dartnell, Lewis & Goldman, Aaron & Lynch, Kennda & Naud, Marie-Eve & Paulino Lima, Ivan Glaucio & Singer, Kelsi & Walter-Antonio, Marina & Abrevaya, Ximena & Anderson, Rika & Arney, Giada & Atri, Dimitra & Azua, Armando & Bowman, Jeff & Brazelton, William & Wong, Teresa. (2016). The Astrobiology Primer v2.0. Astrobiology. 16. 561–653. 10.1089/ast.2015.1460.

Zisk, R. Searching for biomolecules on Mars. Payload. (2022)., September 15 https://payloadspace.com/searching-for-biomolecules-on-mars/

Hays, L. E. et al. Biosignature preservation and detection in Mars analog environments. Astrobiology . 17 , 363–400 (2017).

Poch, O. et al. Remote sensing of potential biosignatures from rocky, liquid, or icy (exo) planetary surfaces. Astrobiology . 17 , 231–252 (2017).

Harman, C. E. et al. Abiotic O2 levels on planets around F, G, K, and M stars: effects of lightning-produced catalysts in eliminating oxygen false positives. Astrophys. J. 866 , 56 (2018).

Nielsen-Marsh, C. et al. Cambridge University Press,. The chemical degradation of bone. in Human Osteology: In Archaeology and Forensic Science (eds. Cox, M. & Mays, S.) 548 (2000).

Nielsen-Marsh, C. Biomolecules in fossil remains: multidisciplinary approach to endurance. Biochem. (Lond) . 24 , 12–14 (2002).

Walworth, J. Nitrogen in soil and the environment. Arizona.edu Preprint at (2017).

Dwek, R. A. & Glycobiology Toward understanding the function of sugars. Chem. Rev. 96 , 683–720 (1996).

Rudd, P. M. & Dwek, R. A. Glycosylation: heterogeneity and the {3D} structure of proteins. Crit. Rev. Biochem. Mol. Biol. 32 , 1–100 (1997).

Cheshire, M. V. Origins and stability of soil polysaccharide. J. Soil. Sci. 28 , 1–10 (1977).

Article   CAS   Google Scholar  

Debosz, K., Vognsen, L. & Labouriau, R. Carbohydrates in hot water extracts of soil aggregates as influenced by long-term management. Commun. Soil. Sci. Plant. Anal. 33 , 623–634 (2002).

Singh, A., & Singh, R. DNA An Important Component of life. In Biochemistry, Biophysics, and Molecular Chemistry 195–207 (Apple Academic, 2020).

Seeman, N. C. & Sleiman, H. F. {DNA} nanotechnology. Nat. Rev. Mater. 3 , 17068 (2018).

Article   ADS   CAS   Google Scholar  

Zhao, Y., Chen, F., Li, Q., Wang, L. & Fan, C. Isothermal amplification of nucleic acids. Chem. Rev. 115 , 12491–12545 (2015).

Pääbo, S. Molecular cloning of ancient Egyptian mummy {DNA}. Nature . 314 , 644–645 (1985).

Evershed, R. P. Biomolecular archaeology and lipids. World Archaeol. 25 , 74–93 (1993).

Cross, B. E., Edinberry, M. N. & Turner, W. B. Pigments of Gnomonia erythrostoma. I. The structures of erythrostominone, deoxyerythrostominone, and deoxyerythrostominol. J. Chem. Soc. Perkin . 1 (3), 380–390 (1972).

Mizukami, H., Konoshima, M. & Tabata, M. Variation in pigment production in Lithospermum erythrorhizon callus cultures. Phytochemistry . 17 , 95–97 (1978).

Mumtaz, R., Bashir, S., Numan, M., Shinwari, Z. K. & Ali, M. Pigments from soil bacteria and their therapeutic properties: a mini review. Curr. Microbiol. 76 , 783–790 (2019).

Młodzińska, E. Survey of plant pigments: Molecular and environmental determinants of plant colors. Abcbot pl (2009).

de Marsac, N. Phycobiliproteins and phycobilisomes: the early observations. Photosynth Res. 76 , 193–205 (2003).

Jones, J. G. Studies on lipids of soil micro-organisms with particular reference to hydrocarbons. J. Gen. Microbiol. 59 , 145–152 (1969).

Stevenson, F. J. Lipids in soil. J. Am. Oil Chem. Soc. 43 , 203–210 (1966).

URC2020 Science Discussion. Youtube (2020). https://www.youtube.com/watch?v=jW21VtoHCag

Serra, J. A. Histochemical tests for proteins and amino acids; the characterization of basic proteins. Stain Technol. 21 , 5–18 (1946).

Lubran, M. M. The measurement of total serum proteins by the Biuret method. Ann. Clin. Lab. Sci. 8 , 106–110 (1978).

CAS   PubMed   Google Scholar  

Domon, B. & Aebersold, R. Mass spectrometry and protein analysis. Sci. (1979) . 312 , 212–217 (2006).

CAS   Google Scholar  

Potter, R. S. & Snyder, R. S. The determination of ammonia in soils. J. Ind. Eng. Chem. 7 , 221–226 (1915).

Kwak, D., Lei, Y. & Maric, R. Ammonia gas sensors: a comprehensive review. Talanta . 204 , 713–730 (2019).

Katoch, R. Carbohydrate Estimations. in Analytical Techniques in Biochemistry and Molecular Biology 67–76Springer New York, (2011).

Bakken, L. R. & Frostegård, Å. Springer Berlin Heidelberg,. Nucleic acid extraction from soil. in Soil Biology 49–73 (2006).

Jensen, A. et al. Quantitative paper chromatography of carotenoids. Acta Chem. Scand. 13 , 1863–1868 (1959).

Strain, H. H., Sherma, J., Benton, F. L. & Katz, J. J. One-way paper chromatography of the chloroplast pigments of leaves. Biochim. Biophys. acta . 109 , 1–15 (1965).

Brotas, V. & Plante-Cuny, M. R. The use of {HPLC} pigment analysis to study microphytobenthos communities. Acta Oecol. (Montrouge) . 24 , S109–S115 (2003).

Suzuki, Y. & Shioi, Y. Identification of chlorophylls and carotenoids in major teas by high-performance liquid chromatography with photodiode array detection. J. Agric. Food Chem. 51 , 5307–5314 (2003).

Yuan, J., Zhang, Y., Shi, X., Gong, X. & Chen, F. Simultaneous determination of carotenoids and chlorophylls in algae by high performance liquid chromatography. Se Pu . 15 , 133–135 (1997).

Portable high performance liquid chromatography. Axcendcorp.com Preprint at. (2020).

Mubarak, M., Shaija, A. & Suchithra, T. V. A review on the extraction of lipid from microalgae for biodiesel production. Algal Res. 7 , 117–123 (2015).

Patel, A. et al. Lipids detection and quantification in oleaginous microorganisms: an overview of the current state of the art. BMC Chem. Eng. 1 , (2019).

Oades, J. M. Soil organic matter and structural stability: mechanisms and implications for management. Plant. Soil. 76 , 319–337 (1984).

Swain, F. M. & Rogers, M. A. (2), (3),. Distribution of carbohydrate residues in some fossil specimens and associated sedimentary matrix and other geologic samples. J. Sediment. Res. 37, 12–24 (1967).

Palacas, J. G., Swain, F. M. & Smith, F. Presence of carbohydrates and other organic compounds in ancient sedimentary rocks. Nature . 185 , 234 (1960).

Briggs, D. E. G. & Summons, R. E. Ancient biomolecules: their origins, fossilization, and role in revealing the history of life: prospects \& overviews. Bioessays . 36 , 482–490 (2014).

Capone, D. G., Popa, R., Flood, B. & Nealson, K. H. Geochemistry. Follow the nitrogen. Sci. (1979) . 312 , 708–709 (2006).

MIST Mongol Barota - URC SAR 2021. Youtube (2021). https://www.youtube.com/watch?v=0ezyJEdleQM

Sokolova, M. & Lapalme, G. A systematic analysis of performance measures for classification tasks. Inf. Process. Manag . 45 , 427–437 (2009).

Narkhede, S. Understanding auc-roc curve. Towards Data Sci. 26 , 220–227 (2018).

Google Scholar  

Download references

This research is funded by the Institute of Advanced Research (Grant No. IAR-2023-Pub-006), United International University, Bangladesh.

Author information

Authors and affiliations.

Department of Computer Science and Engineering, United International University, United City, Vatara, 1200, Dhaka, Bangladesh

Department of Computer Science and Engineering, Military Institute of Science and Technology, Mirpur Cantonment, Dhaka, 1216, Bangladesh

Akib Zaman, Fardeen Ashraf, Sadia Nur Nazifa, Hafsah Mahzabin Chowdhury & Md. Mahbubur Rahman

Department of Biochemistry & Molecular Biology, Faculty of Biological Sciences, University of Dhaka, Curzon Hall Area College Road, Dhaka, 1000, Bangladesh

Haseena Khan & Faria Noshin Ahona

Department of Mechanical Engineering, Military Institute of Science and Technology, Mirpur Cantonment, Dhaka, 1216, Bangladesh

Oliullah Samir & Asif Mahmud Rayhan

You can also search for this author in PubMed   Google Scholar

Contributions

AZ: Methodology, Evaluation Study, Writing - Original Draft; FA: Investigation, Software, Writing - Original Draft; HK: Supervision, Writing- Original Draft, Experiment Design FNA: Resources, Investigation, Writing; OS: Design, Fabrication, Writing; AMR: Design, Fabrication; SNN: Investigation, Validation, Writing; HMC: Investigation, Validation; MR: Supervision, Writing- Reviewing and Editing;

Corresponding authors

Correspondence to Akib Zaman or Fardeen Ashraf .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Zaman, A., Ashraf, F., Khan, H. et al. A multiple biomolecules-based rapid life detection protocol embedded in a rover scientific subsystem for soil sample analysis. Sci Rep 14 , 26645 (2024). https://doi.org/10.1038/s41598-024-77808-6

Download citation

Received : 15 January 2022

Accepted : 25 October 2024

Published : 04 November 2024

DOI : https://doi.org/10.1038/s41598-024-77808-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Life detection
• Multiple biomolecule integration
• Qualitative test-scoring
• Life-detection rover subsystem
• Computer-aided analysis

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.