Seno Hardijanto Purnomo, Agung Triayudi

Application of Sentence-BERT Embeddings for Semantic Deduplication of Industrial Material Records

Introduction

Application of sentence-bert embeddings for semantic deduplication of industrial material records. Deduplicate industrial material records in ERP/EAM systems using Sentence-BERT embeddings. This semantic pipeline improves data quality, handling mixed-language and paraphrastic duplicates.

Abstract

Industrial material master data in Enterprise Resource Planning (ERP) and Enterprise Asset Management (EAM) systems accumulates duplicate records that distort inventory, procurement, and analytics. Traditional deduplication relies on string-similarity measures such as Levenshtein, Jaro–Winkler, and TF-IDF cosine, which can struggle on catalogs mixing Indonesian and English terminology—e.g. Valve versus Keran—and on paraphrastic variants with different word order or abbreviation style. This study formally specifies a semantic deduplication pipeline that encodes material descriptions as sentence embeddings using Sentence-BERT (SBERT) and compares them via cosine similarity, then diagnostically evaluates the extent to which SBERT improves over those baselines. Following Design Science Research, the pipeline specifies normalisation, encoding with a multilingual paraphrase-tuned SBERT variant, and pairwise comparison within candidate sets produced by hybrid blocking; the diagnostic evaluation reports scores on the raw descriptions to expose baseline behaviour before domain-specific harmonisation. A sample of 291,000 records from two Indonesian industrial power plants motivates the design. On a diagnostic set of 100 record pairs derived from existing engineer-annotated duplicate markers, Jaro–Winkler achieves F1 = 0.925 (precision 1.000, recall 0.860) and SBERT achieves F1 = 0.875 (precision 0.913, recall 0.840) at threshold τ = 0.65; qualitative analysis of twelve representative pairs further reveals that SBERT excels on structural paraphrase (cosine 0.73–0.88 where character-level methods score below 0.50), while Jaro–Winkler remains competitive on abbreviation, unit-standard, and cross-language pairs—particularly those involving Indonesian technical vocabulary under-represented in the model’s training distribution. The central finding is that Sentence-BERT complements rather than replaces string baselines, which motivates future work on multi-channel architectures combining textual semantics with structural context.

Full Text

Comments

You need to be logged in to post a comment.