📝 Approach for a better clustering built in HackNLP
Our approach was to create an N-dimensional index representing the similarity of texts and create clusters from it.
- Take the matrix from Java
- Create N-dimensional index
- Create algorithm to create clusters
- Implement a 3D dimensional representation
We have taken the similarity score of all the possible pairs of vectors representation of several texts (requirements) given by ESSI University group project which is calculated using a Cosine distance. Then using that information as a matrix we have created an index using NMSLIB (source: https://github.com/nmslib/nmslib) and implemented a clusterization algorithm by thresholding and selecting a number of neighbours.
We did not have a lot of time to develop our ideas. The brainstorming was a little bit rush and we are not used to it. Moreover, the dataset and method to validate our model were a little difficult to deal with.
We've never used nmslib or neither done a clustering algorithm so we can say that almost everything of what we've done it was new to us.
Re-think about the way it is computed the accuracy for the model and experiment with several parameters to get the best result. We could try several ways to compute the distance and its similarity score instead of the Cosine distance.
python3 -m src.scripts.build_api_model data/input_buildModel_duplicates.json
python3 -m src.scripts.get_matrix data/input_computeClusters_duplicates.json data/score_matrix.json data/mapping.json
python3 -m src.build data
python3 -m src.find_clusters data 
python3 -m src.eval data/input_computeClusters_duplicates.json data/clusters.json && python3 -m src.eval data/input_computeClusters_duplicates.json data/clusters.json 
MIT © Similaripy