Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem with dope.launch #92

Closed
Dutzel opened this issue Dec 22, 2019 · 14 comments
Closed

Problem with dope.launch #92

Dutzel opened this issue Dec 22, 2019 · 14 comments

Comments

@Dutzel
Copy link

Dutzel commented Dec 22, 2019

Hello guy's,

after a while getting into this project everything worked fine until I have encountered an error. I have no idea how to fix it currently.

Running DOPE... (Listening to camera topic: '/dope/webcam/image_raw') Ctrl-C to stop [ERROR] [1577032753.118657]: bad callback: <bound method Subscriber.callback of <message_filters.Subscriber object at 0x7fb661218c50>> Traceback (most recent call last): File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback cb(msg) File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/__init__.py", line 75, in callback self.signalMessage(msg) File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/__init__.py", line 57, in signalMessage cb(*(msg + args)) File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/__init__.py", line 224, in add self.signalMessage(*msgs) File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/__init__.py", line 57, in signalMessage cb(*(msg + args)) File "/root/catkin_ws/src/dope/nodes/dope", line 248, in image_callback self.config_detect File "/root/catkin_ws/src/dope/src/dope/inference/detector.py", line 251, in detect_object_in_image out, seg = net_model(image_torch) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 491, in __call__ result = self.forward(*input, **kwargs) File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 112, in forward return self.module(*inputs[0], **kwargs[0]) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 491, in __call__ result = self.forward(*input, **kwargs) File "/root/catkin_ws/src/dope/src/dope/inference/detector.py", line 91, in forward out1 = self.vgg(x) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 491, in __call__ result = self.forward(*input, **kwargs) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/container.py", line 91, in forward input = module(input) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 491, in __call__ result = self.forward(*input, **kwargs) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py", line 301, in forward self.padding, self.dilation, self.groups) RuntimeError: CUDNN_STATUS_MAPPING_ERROR

Any ideas?
@TontonTremblay
Copy link
Collaborator

I am not sure, I have never seen this error before. How many models are you running at the same time? It seems you have something broken in your pytorch installation, rather pytorch itself, or cuda, or cudnn. I would try to use the docker image we provided.

@Dutzel
Copy link
Author

Dutzel commented Dec 24, 2019

Thanks for the quick answer. I am using docker, which is why I wonder and I made no changes to the drivers on my computer. Currently I try just to run one model, but this doesn't help too. It even takes about 2-3 minutes until the error comes up. Before it is just saying
"Loading DOPE parameters from '/home/dustin/catkin_ws/src/dope/config/config_pose.yaml'..."

@CthLeo
Copy link

CthLeo commented Dec 27, 2019

Maybe your pytorch and cuda versions do not match, you need to update pytorch.
It is best to go to the pytorch official website to check. @Dutzel

@okadahiroyuki
Copy link

I got the same error.
Did you get a solution?

@TontonTremblay
Copy link
Collaborator

Can you describe your hardware a little bit? Have you tried running DOPE in the docker?

@okadahiroyuki
Copy link

Can you describe your hardware a little bit? Have you tried running DOPE in the docker?
Yes

**HOST PC ***
Lenovo ThinkPad X1 Extreme(2019）
Memory:62.5GB
Intel Core i9-9880H [email protected]
GeForce GTX 1650/PCIe/SSE2
Ubuntu 18.04.4 LTS

rooworks@X1-Extreme-2nd:~$ nvidia-smi
Wed Feb 26 09:25:47 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1650 On | 00000000:01:00.0 Off | N/A |
| N/A 46C P8 3W / N/A | 381MiB / 3911MiB | 7% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1475 G /usr/lib/xorg/Xorg 159MiB |
| 0 1751 G /usr/bin/gnome-shell 134MiB |
| 0 17197 G ...AAAAAAAAAAAAAAgAAAAAAAAA --shared-files 82MiB |
| 0 21195 G gnome-control-center 1MiB |
+-----------------------------------------------------------------------------+

rooworks@X1-Extreme-2nd:~$ run_dope_docker.sh
root@X1-Extreme-2nd: roslaunch dope dope.launch

/usr/local/lib/python2.7/dist-packages/pkg_resources/py2_warn.py:22: UserWarning: Setuptools will stop working on Python 2

You are running Setuptools on Python 2, which is no longer
supported and

SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to setuptools<45
in your environment.
If you have done those things and are still encountering
this message, please comment in
pypa/setuptools#1458
about the steps that led to this unsupported combination.

sys.version_info < (3,) and warnings.warn(pre + "" * 60 + msg + "" * 60)
/usr/local/lib/python2.7/dist-packages/pkg_resources/py2_warn.py:22: UserWarning: Setuptools will stop working on Python 2

You are running Setuptools on Python 2, which is no longer
supported and

SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to setuptools<45
in your environment.
If you have done those things and are still encountering
this message, please comment in
pypa/setuptools#1458
about the steps that led to this unsupported combination.

sys.version_info < (3,) and warnings.warn(pre + "" * 60 + msg + "" * 60)
/usr/local/lib/python2.7/dist-packages/pkg_resources/py2_warn.py:22: UserWarning: Setuptools will stop working on Python 2

You are running Setuptools on Python 2, which is no longer
supported and

SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to setuptools<45
in your environment.
If you have done those things and are still encountering
this message, please comment in
pypa/setuptools#1458
about the steps that led to this unsupported combination.

sys.version_info < (3,) and warnings.warn(pre + "" * 60 + msg + "" * 60)
/usr/local/lib/python2.7/dist-packages/torch/cuda/init.py:114: UserWarning:
Found GPU0 GeForce GTX 1650 which requires CUDA_VERSION >= 9000 for
optimal performance and fast startup time, but your PyTorch was compiled
with CUDA_VERSION 8000. Please install the correct PyTorch binary
using instructions from http://pytorch.org

warnings.warn(incorrect_binary_warn % (d, name, 9000, CUDA_VERSION))
[ERROR] [1582677567.883066]: bad callback: <bound method Subscriber.callback of <message_filters.Subscriber object at 0x7f0f2c394110>>
Traceback (most recent call last):
File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
cb(msg)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/init.py", line 75, in callback
self.signalMessage(msg)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/init.py", line 57, in signalMessage
cb(*(msg + args))
File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/init.py", line 224, in add
self.signalMessage(msgs)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/init.py", line 57, in signalMessage
cb((msg + args))
File "/root/catkin_ws/src/dope/nodes/dope", line 248, in image_callback
self.config_detect
File "/root/catkin_ws/src/dope/src/dope/inference/detector.py", line 251, in detect_object_in_image
out, seg = net_model(image_torch)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 112, in forward
return self.module(*inputs[0], **kwargs[0])
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/root/catkin_ws/src/dope/src/dope/inference/detector.py", line 91, in forward
out1 = self.vgg(x)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_MAPPING_ERROR

@TontonTremblay
Copy link
Collaborator

I never tested on a 1650, I did on a 1660, I would think the 50 to be 'too weak' do you have an other GPU you could test on?

@saratrajput
Copy link

I wonder if 1650 is the problem. I had tested it on Jetson Xavier and it runs fine. Unless Xavier is somehow better than 1650.

@okadahiroyuki
Copy link

I tested another machine, GeForce RTX 2080 with Max-Q Design/PCIe/SSE2.
but Couldn't.

/usr/local/lib/python2.7/dist-packages/pkg_resources/py2_warn.py:22: UserWarning: Setuptools will stop working on Python 2

You are running Setuptools on Python 2, which is no longer
supported and

SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to setuptools<45
in your environment.
If you have done those things and are still encountering
this message, please comment in
pypa/setuptools#1458
about the steps that led to this unsupported combination.

sys.version_info < (3,) and warnings.warn(pre + "" * 60 + msg + "" * 60)
/usr/local/lib/python2.7/dist-packages/pkg_resources/py2_warn.py:22: UserWarning: Setuptools will stop working on Python 2

You are running Setuptools on Python 2, which is no longer
supported and

SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to setuptools<45
in your environment.
If you have done those things and are still encountering
this message, please comment in
pypa/setuptools#1458
about the steps that led to this unsupported combination.

sys.version_info < (3,) and warnings.warn(pre + "" * 60 + msg + "" * 60)
/usr/local/lib/python2.7/dist-packages/pkg_resources/py2_warn.py:22: UserWarning: Setuptools will stop working on Python 2

You are running Setuptools on Python 2, which is no longer
supported and

SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to setuptools<45
in your environment.
If you have done those things and are still encountering
this message, please comment in
pypa/setuptools#1458
about the steps that led to this unsupported combination.

sys.version_info < (3,) and warnings.warn(pre + "" * 60 + msg + "" * 60)
�[31m[dope-1] process has died [pid 15322, exit code -9, cmd /root/catkin_ws/src/dope/nodes/dope __name:=dope __log:=/root/.ros/log/bc7eceb4-5a8c-11ea-944c-d0abd507ad78/dope-1.log].
log file: /root/.ros/log/bc7eceb4-5a8c-11ea-944c-d0abd507ad78/dope-1*.log�[0m
... logging to /root/.ros/log/bc7eceb4-5a8c-11ea-944c-d0abd507ad78/roslaunch-m15-R2-15304.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
�]2;/root/catkin_ws/src/dope/launch/dope.launch�
�[1mstarted roslaunch server http://m15-R2:36371/�[0m

SUMMARY

CLEAR PARAMETERS

  • /dope/

PARAMETERS

  • /dope/class_ids/bleach: 7
  • /dope/class_ids/cracker: 1
  • /dope/class_ids/gelatin: 2
  • /dope/class_ids/meat: 3
  • /dope/class_ids/mustard: 4
  • /dope/class_ids/soup: 5
  • /dope/class_ids/sugar: 6
  • /dope/dimensions/bleach: [10.2677307128906...
  • /dope/dimensions/cracker: [16.4036006927490...
  • /dope/dimensions/gelatin: [8.91829967498779...
  • /dope/dimensions/meat: [10.1646738052368...
  • /dope/dimensions/mustard: [9.60241508483886...
  • /dope/dimensions/soup: [6.76593780517578...
  • /dope/dimensions/sugar: [9.26773071289062...
  • /dope/downscale_height: 500
  • /dope/draw_colors/bleach: [232, 222, 12]
  • /dope/draw_colors/cracker: [13, 255, 128]
  • /dope/draw_colors/gelatin: [255, 255, 255]
  • /dope/draw_colors/meat: [0, 104, 255]
  • /dope/draw_colors/mustard: [217, 12, 232]
  • /dope/draw_colors/soup: [255, 101, 0]
  • /dope/draw_colors/sugar: [232, 222, 12]
  • /dope/input_is_rectified: True
  • /dope/mesh_scales/bleach: 0.01
  • /dope/mesh_scales/cracker: 0.01
  • /dope/mesh_scales/gelatin: 0.01
  • /dope/mesh_scales/meat: 0.01
  • /dope/mesh_scales/mustard: 0.01
  • /dope/mesh_scales/soup: 0.01
  • /dope/mesh_scales/sugar: 0.01
  • /dope/sigma: 3
  • /dope/thresh_angle: 0.5
  • /dope/thresh_map: 0.01
  • /dope/thresh_points: 0.1
  • /dope/topic_camera: /dope/webcam/imag...
  • /dope/topic_camera_info: /dope/webcam/came...
  • /dope/topic_publishing: dope
  • /dope/weights/soup: package://dope/we...
  • /rosdistro: kinetic
  • /rosversion: 1.12.14

NODES
/
dope (dope/dope)

�[1mROS_MASTER_URI=http://localhost:11311�[0m
�]2;/root/catkin_ws/src/dope/launch/dope.launch http://localhost:11311�
�[1mprocess[dope-1]: started with pid [15322]�[0m
all processes on machine have died, roslaunch will exit
shutting down processing monitor...
... shutting down processing monitor complete
�[1mdone�[0m

@TontonTremblay
Copy link
Collaborator

the setuptool warning is normal since python 2 is depreciated. It looks to me like it is working? If not I would recommend to use anaconda to install your python instead of using python on the system.

@Dutzel
Copy link
Author

Dutzel commented Apr 16, 2020

@okadahiroyuki Did you solve the problem? Meanwhile i need u use my home computer again, why i am facing the same problem :( I made a clean reinstall of everything but i am still getting the same problem.

@Dutzel
Copy link
Author

Dutzel commented Apr 23, 2020

Simple solution solved this problem for me:

pip install torch==1.0.0 -f https://download.pytorch.org/whl/cu90/torch-1.0.0-cp27-cp27mu-linux_x86_64.whl

@Dutzel Dutzel closed this as completed Apr 23, 2020
@sivaeinfo
Copy link

Actually i have tried to use the dope package in https://github.com/yehengchen/DOPE-ROS-D435

I have installed require packages and dependencies.
In this, they have given two command.
1.roslaunch realsense2_camera rs_rgbd.launch
2. roslaunch dope dope.launch

This gives following error
IMG-20210518-WA0020.jpegIMG-20210518-WA0020.jpeg

Please help me to sort out this issues.

@youssefsamy65
Copy link

Actually i have tried to use the dope package in https://github.com/yehengchen/DOPE-ROS-D435

I have installed require packages and dependencies. In this, they have given two command. 1.roslaunch realsense2_camera rs_rgbd.launch 2. roslaunch dope dope.launch

This gives following error IMG-20210518-WA0020.jpegIMG-20210518-WA0020.jpeg

Please help me to sort out this issues.

this is the same error that is appening to me now did you fix it ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@okadahiroyuki @TontonTremblay @Dutzel @saratrajput @CthLeo @sivaeinfo @youssefsamy65