Antonius Rachmat Chrismanto, Willy Sudiarto Raharjo, Oscar Gilang Purnajati

Text Similarity Analysis for Evaluating Alignment Between Lesson Plans and Teaching Reports

Introduction

Text similarity analysis for evaluating alignment between lesson plans and teaching reports. Automate alignment evaluation between higher education lesson plans (RPS) and teaching reports (BPP) using text similarity methods like fuzzy, exact, and TF-IDF algorithms.

Abstract

RPS (Rencana Pembelajaran Semester, or called Lesson Plans) is a class activity planning document in the higher education learning process that includes learning outcomes, methods, learning strategy, and evaluation criteria. It is created by the lecturers in charge of the course and coordinated with the relevant department. This document needs to be monitored throughout the semester for its conformity with the implementation document (Borang Pelaksanaan Perkuliahan (BPP)). It was done manually through our eRPS system, but it requires a lot of effort and precision and is not time-efficient. This research focused on evaluating the effectiveness of several content-based text similarity methods to detect RPS conformity compared with the BPP, or called Teaching Reports document. The Boyer-Moore (B), Rabin-Karp (R), Jaccard (JC), Jaro-Winkler (JW), Smith-Waterman (SW), Knuth-Morris-Pratt (K), Levenehtein cosine similarity (C), Dice (D), Jaro (J), and Soundex (S) algorithms were evaluated in this paper. In the vector-based similarity method, TF-IDF was used. The evaluation of 11 string-matching algorithms across four scenarios demonstrated clear performance trends. Fuzzy algorithms (SW with accuracy 0,845–0,870, and JW with accuracy 0,840-0,850) achieved the highest accuracy in a single row of lecturer scenario, while exact/pattern-based algorithms (B, K, and S with accuracy 0,8625–0,8725) on a combination of all rows of lectures with minimal variance (≈0,005–0,015). Pre-processing benefits fuzzy algorithms (+2.5%) but is neutral for exact/pattern-based algorithms. The combined scenario improves the exact/phonetic algorithms (+6–7%) but reduces the fuzzy performance algorithm (−10–14%). The optimal thresholds were generally 40–50%, except for JW and J, which were 65%.

Review

This paper addresses a highly relevant and practical problem within higher education: the manual and time-consuming process of ensuring alignment between course lesson plans (RPS) and actual teaching reports (BPP). By proposing and evaluating automated text similarity methods, the authors aim to significantly improve the efficiency and accuracy of this critical monitoring task. The clear identification of the existing manual system's limitations, such as high effort and precision requirements, effectively sets the stage for the proposed technological solution and highlights the potential impact of this research. The methodology is comprehensive, evaluating a diverse array of 11 content-based text similarity algorithms, encompassing exact string matching (Boyer-Moore, Rabin-Karp, Knuth-Morris-Pratt), phonetic (Soundex), fuzzy string matching (Jaro, Jaro-Winkler, Levenshtein, Smith-Waterman), and token-based approaches (Jaccard, Dice, with TF-IDF for vector-based similarity). The evaluation across four distinct scenarios and the analysis of pre-processing benefits offer valuable insights into the algorithms' performance characteristics. Key findings indicate that fuzzy algorithms (Smith-Waterman, Jaro-Winkler) excel in single-row lecturer scenarios, while exact/pattern-based algorithms (Boyer-Moore, Knuth-Morris-Pratt, Soundex) show superior accuracy and minimal variance in combined-row scenarios. The differing impacts of pre-processing and scenario combination on algorithm performance are particularly noteworthy, providing a nuanced understanding of their applicability. Overall, this research presents a robust initial exploration into automating a vital educational quality assurance process. The thorough comparative analysis of numerous algorithms, coupled with specific performance metrics and insights into optimal thresholds, makes a significant contribution to the field. The findings provide clear guidance for practitioners and researchers seeking to implement text similarity solutions for document conformity assessment, particularly within an educational context. While the abstract offers a strong overview of the technical evaluation, the full paper would benefit from a deeper discussion on the qualitative aspects of "alignment" and how these algorithms practically capture it, further solidifying the practical implications of this well-executed study.

Full Text

Comments

You need to be logged in to post a comment.