out of memory issue

[KevinYuk]

./devel/lib/rangenet_lib/infer -p /home/2T/xxx/xxx/rangenet_lib/pre-trained_model/darknet53 -s /home/2T/xxx/xxx/rangenet_lib/catkin_ws/src/rangenet_lib/example/000000.bin --verbose

Success picking up ONNX model
Failure creating engine from ONNX model
Current trial size is 8589934592
Failure creating engine from ONNX model
Current trial size is 4294967296
Failure creating engine from ONNX model
Current trial size is 2147483648
Cuda error in file src/implicit_gemm.cu at line 648: out of memory

[KevinYuk]

wait for longer

[bin70]

@KevinYuk Hello! I was met the same problem with you, could you please tell me how you access it finally?

[bin70]

Just wait for the program automatically reduce the trial size.

[KevinYuk]

Just wait for the program automatically reduce the trial size.

Yes. Or reduce the batch size.

[LongruiDong]

Just wait for the program automatically reduce the trial size.

Yes. Or reduce the batch size.

Hi, I met similiar but a little different situation in #15

as you see,it terminated with "runtime_error"...

I have tried repeating run it, but still got the same error... , do you know how to pass it ?

Thanks in advance~

[LongruiDong]

Just wait for the program automatically reduce the trial size.

Yes. Or reduce the batch size.

where to reduce "batch size"?
what`s more, I set MAX_WORKSPACE_SIZE larger(16gb) and MIN_WORKSPACE_SIZE smaller (even to 1)

rangenet_lib/include/netTensorRT.hpp

Line 19 in f1114af

#define MAX_WORKSPACE_SIZE \

,
but still get run time error...

================================================================================
Setting verbosity to: false
Trying to open model
Trying to deserialize previously stored: src/rangenet_lib/model/darknet53//model.trt
Could not deserialize TensorRT engine. 
Generating from sratch... This may take a while...
Trying to generate trt engine from : src/rangenet_lib/model/darknet53//model.onnx
Platform HAS fp16 support.
No DLA selected.
----------------------------------------------------------------
Input filename:   src/rangenet_lib/model/darknet53//model.onnx
ONNX IR version:  0.0.4
Opset version:    9
Producer name:    pytorch
Producer version: 1.1
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
WARNING: ONNX model has a newer ir_version (0.0.4) than this parser was built against (0.0.3).
 ----- Parsing of ONNX model src/rangenet_lib/model/darknet53//model.onnx is Done ---- 
Success picking up ONNX model
Failure creating engine from ONNX model
Current trial size is 17179869184
Failure creating engine from ONNX model
Current trial size is 8589934592
Failure creating engine from ONNX model
Current trial size is 4294967296
Failure creating engine from ONNX model
Current trial size is 2147483648
Failure creating engine from ONNX model
Current trial size is 1073741824
Failure creating engine from ONNX model
Current trial size is 536870912
Failure creating engine from ONNX model
Current trial size is 268435456
Failure creating engine from ONNX model
Current trial size is 134217728
Failure creating engine from ONNX model
Current trial size is 67108864
Failure creating engine from ONNX model
Current trial size is 33554432
Failure creating engine from ONNX model
Current trial size is 16777216
Failure creating engine from ONNX model
Current trial size is 8388608
Failure creating engine from ONNX model
Current trial size is 4194304
Failure creating engine from ONNX model
Current trial size is 2097152
Failure creating engine from ONNX model
Current trial size is 1048576
Failure creating engine from ONNX model
Current trial size is 524288
Failure creating engine from ONNX model
Current trial size is 262144
Failure creating engine from ONNX model
Current trial size is 131072
Failure creating engine from ONNX model
Current trial size is 65536
Failure creating engine from ONNX model
Current trial size is 32768
Failure creating engine from ONNX model
Current trial size is 16384
Failure creating engine from ONNX model
Current trial size is 8192
Failure creating engine from ONNX model
Current trial size is 4096
Failure creating engine from ONNX model
Current trial size is 2048
Failure creating engine from ONNX model
Current trial size is 1024
Failure creating engine from ONNX model
Current trial size is 512
Failure creating engine from ONNX model
Current trial size is 256
Failure creating engine from ONNX model
Current trial size is 128
Failure creating engine from ONNX model
Current trial size is 64
Failure creating engine from ONNX model
Current trial size is 32
Failure creating engine from ONNX model
Current trial size is 16
Failure creating engine from ONNX model
Current trial size is 8
Failure creating engine from ONNX model
Current trial size is 4
Failure creating engine from ONNX model
Current trial size is 2
Failure creating engine from ONNX model
Current trial size is 1
terminate called after throwing an instance of 'std::runtime_error'
  what():  ERROR: could not create engine from ONNX.
Aborted (core dumped)