Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have 'ResourceExhaustedError' occured to you? #4

Open
illutheplanet opened this issue Nov 24, 2018 · 2 comments
Open

Have 'ResourceExhaustedError' occured to you? #4

illutheplanet opened this issue Nov 24, 2018 · 2 comments

Comments

@illutheplanet
Copy link

I just replaced input1.txt and input2.txt with my own dataset. When 'local variable 'final_features' referenced before assignment' occured, I deannotated 'final_features = feature_extractor('11_0.tif','16_0.tif')' in Train.py. Then I began to run Train.py, after a few minutes it showed like this:
Resource exhausted: OOM when allocating tensor with shape[1,64,224,224] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

@vbhavank
Copy link
Owner

@illutheplanet This could occur due to several reasons like:

  1. I would suggest just initializing final_features as np.empty([..,...]) with size equal to the number of samples you use.

  2. Your GPU runs out of memory due to large filter size and outputs that needs to be stored at intermediate steps. I would suggest storing variables to disk and continue after closing your sessions where ever possible. Please keep in mind you cannot clear sessions when traversing or computing your network graph.

@zyp19
Copy link

zyp19 commented Apr 3, 2020

I just replaced input1.txt and input2.txt with my own dataset. When 'local variable 'final_features' referenced before assignment' occured, I deannotated 'final_features = feature_extractor('11_0.tif','16_0.tif')' in Train.py. Then I began to run Train.py, after a few minutes it showed like this:
Resource exhausted: OOM when allocating tensor with shape[1,64,224,224] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Did you run it successfully?What files are "modelh6.h5" and "model.json"?

Thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants