Is the empirical research we have the research we can trust? A review of distance education journal publications in 2021

Abstract

This study aims to investigate the trustworthiness of empirical research published in distance education (DE) journals, an area that has yet to be systematically explored. The review covers 238 empirical studies which were based on primary data and published in 2021 in eight DE journals listed in Social Science Citation Index (SSCI) or Emerging Sources Citation Index (ESCI). Findings suggest that they are fairly good in terms of clarity in context of study, purpose of study, and duration if a study involves an intervention or is otherwise length-sensitive, albeit the existence of small room for improvement. In contrast, a large proportion of the studies reviewed are less rigorous to different but considerable and even alarming degrees in terms of research approach and design, sampling strategies, source of data, researcher bias, ethical concerns, and limitations. Put specifically, less than one quarter of the sample studies adopted the qualitative approach while over 90% of the quantitative studies followed the survey and correlational designs, resulting in a staggering disproportion from the perspective of diversity in research approach and design. Over 60% of the sample studies did not spell out their sampling strategies and only about 20% of the specified sampling strategies were probabilistic in nature, limiting the generalizability of the findings. Over 90% employed questionnaires/scales/rubrics and/or interview protocols to collect data but over 70% of these two types of instruments were neither reviewed nor piloted before put to use with less than 50% available in full content, hence likely to undermine the value of the findings. Researcher bias, ethical concerns, and limitations were addressed in 10%, 50%, and 70% of the studies respectively. Implications for future research are also discussed in the light of these findings.