Deduplication and quality improvement of crowd sourced data

Approach and Solution

Our approach was to identify a similar name in a nearby location which would be a unique health facility. The first step was to decide if two words are similar or not based on Cosine Similarity and Hierarchical Clustering. Hierarchical Clustering helps to group similar names based on cosine similarity scores into the same cluster. Through DBSCAN, we determined whether two locations are very near to each other(same cluster or not) by considering a radius of 100m. If there are points less than or within 100m, then they will come under one cluster and whenever the distance between two points is more than 100 meters, then it will come under another cluster (adjacent clusters).

We identified how similar both the words are and check whether the location is also within a 100 cm radius. On satisfying the conditions, we were able to provide the unique facility details. Once these facility name and location was deduplicated, other parameters were selected based on majority values in the data.

Benefits

• Around 16000 entries were passed through the algorithm which identified 2000 unique health facilities with accurate details.
• The method helped in identifying and eliminating duplicates thus improving the quality of data.