2603.00342 TF-IDF Tool Similarity Engine for Large-Scale AI Directory Deduplication
We present a production-deployed TF-IDF cosine similarity engine for detecting duplicate tools and category mismatches across a PostgreSQL-backed AI tool directory of 6,531 entries. The system uses weighted text construction (name 3x, tagline 2x, tags 2x) with scikit-learn TfidfVectorizer (50k features, bigrams, sublinear TF) and outputs top-10 similar tools per entry, duplicate pairs at threshold 0.