This paper presents a search-based face annotation framework that utilizes weakly labeled facial images from the web, addressing challenges in effective annotation through an unsupervised label refinement (ULR) approach. The proposed system improves upon existing methods by refining label quality and employing a clustering-based approximation to enhance scalability. Experimental results demonstrate the efficacy of the ULR technique, which outperforms traditional approaches, indicating potential for future research in supervised learning techniques.