Measuring the impact of AI

What metrics are being used to illustrate that AI's impact on education is:

1. Evident,
2. Significant, and
3. Frequent

Many articles anecdotally give examples of how AI is impacting education, but having metrics with concrete examples would reveal the significance of what is being experienced. This might also give insight into an Impact Gaps Canvas (a method to help learn about a problem before trying to solve it) with AI in education.

Metrics may also assist in removing some of the negative cognitive perceptions, as what is being felt may not be of the same magnitude as what is being measured.

A researcher, Johannes Schrumpf, at the Institute of Cognitive Science, University of Osnabrück, Germany, noted in a research paper (2022) that the effectiveness of using AI in higher education can be measured by analyzing user engagement, feedback statistics, and the acceptance of resource recommendations. The results were mixed to negative, at best.

An ambitious recent article, Impact of artificial intelligence on assessment methods in primary and secondary education: Systematic literature review (2023), by a group of researchers from Spain, systematically reviewed findings and found some evidence that AI has the potential to improve education, in particular, student assessment, within both the primary and secondary grades.

The US Department of Education, Office of Educational Technology, also published a research paper, Artificial Intelligence and the Future of Teaching and Learning (2023) noting that AI has impacted students in a variety of ways, both positive and negative, namely in skill development, social implications, accessibility, etc. However, the data to support these observations is still forming and "will be continually adjusting as the goals for technology are consistently changing. We often don’t yet fully understand the context; and as we learn about the context, our goals must change."

What seems clear is that there is a sense of urgency because the pace of innovation is creating tension for teachers and students to some degree, to feel comfortable using AI intentionally, safely, and meaningfully in the classroom. Despite the growing interest and the initial findings presented, there is still a notable lack of metrics and concrete examples specifically tailored to the education sector. This gap highlights the critical need for developing and applying precise metrics to better understand and leverage AI's potential in education.