Skip to content

Commit

Permalink
Added HoMe paper
Browse files Browse the repository at this point in the history
  • Loading branch information
shagunsodhani committed Oct 14, 2018
1 parent 777888d commit f659771
Show file tree
Hide file tree
Showing 3 changed files with 66 additions and 1 deletion.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ I am trying a new initiative - a-paper-a-week. This repository will hold all tho

* [Poincaré Embeddings for Learning Hierarchical Representations](https://shagunsodhani.in/papers-I-read/Poincare-Embeddings-for-Learning-Hierarchical-Representations)
* [When Recurrent Models Don’t Need To Be Recurrent](https://shagunsodhani.in/papers-I-read/When-Recurrent-Models-Don-t-Need-To-Be-Recurrent)
* [HoME - a Household Multimodal Environment](https://shagunsodhani.in/papers-I-read/HoME-a-Household-Multimodal-Environment)
* [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://shagunsodhani.in/papers-I-read/Emergence-of-Grounded-Compositional-Language-in-Multi-Agent-Populations)
* [A Semantic Loss Function for Deep Learning with Symbolic Knowledge](https://shagunsodhani.in/papers-I-read/A-Semantic-Loss-Function-for-Deep-Learning-with-Symbolic-Knowledge)
* [Hierarchical Graph Representation Learning with Differentiable Pooling](https://shagunsodhani.in/papers-I-read/Hierarchical-Graph-Representation-Learning-with-Differentiable-Pooling)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
layout: post
title: HoME - a Household Multimodal Environment
comments: True
excerpt:
tags: ['2017', 'Multi Modal', 'NIPS 2017', 'NIPS Workskop', 'Virtual Embodiment', AI, NIPS]
---

## Introduction

* Environment for learning using modalities like vision, audio, semantics, physics and interaction with objects and other agents.

* [Link to the paper](https://arxiv.org/abs/1711.11017)

## Motivation

* Humans learn by interacting with their surroundings (environment).

* Similarly training an agent in an interactive multi-model environment (virtual embodiment) could be useful for a learning agent.


## Characteristics

* Open-source and Open-AI gym compatible

* Built on top of 45000 3D house layouts from SUNCG dataset.

* Provides both 3D visual and audio recording.

* Semantic image segmentation and langauge description of objects.

## Components

* Rendering Engine

* Implemented using Panda 3D game engine.

* Renders RGB+depth scenes based on textures, multi-source lightings and shadows.

* Acoustic Engine

* Implemented using EVERT

* Supports multiple microphones, sound sources, sound absorption based on material, atmospheric conditions etc.

* Semantics Engine

* Provides a short textual description for each object, along with information like color, category, material size, location etc.

* Physics Engine

* Implemented using Bullet3 Engine

* Supports physical interaction, external forces like gravity and position and velocity information for multiple agents.

## Potential Applications

* Visual Question Answering

* Conversational Agents

* Training an agent to follow instructions

* Multi-agent communication
2 changes: 1 addition & 1 deletion site/_site
Submodule _site updated from d48f9c to 2d236a

0 comments on commit f659771

Please sign in to comment.