Take pictures using LERF embeddings; visualize grounding results in 3D mesh; parallelization

[jedyang97]

Try the newest demo at here!

Specifically, we have made improvement on:

• Use LERF embeddings + DBSCAN clustering to determine camera poses
• Take a picture for each object instance
• Use LLaVA-13B to caption each picture
• GPT-4 reads all captions and reason internally or ask user for clarification to ground object
• Display grounding results to user: object instances highlighted in a 3D mesh using bounding sphere
• Significantly speed up the pipeline with parallelization on rendering and LLaVA inference

"How many doors are there in this room?"

"find all the chairs"