Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kernel not compiled with GPU support #55

Closed
Angzz opened this issue Oct 14, 2019 · 19 comments
Closed

Kernel not compiled with GPU support #55

Angzz opened this issue Oct 14, 2019 · 19 comments

Comments

@Angzz
Copy link

Angzz commented Oct 14, 2019

❓ Questions and Help

General questions about detectron2.

Hi, when train models with detectron2, the encountered error is shown below:

RuntimeError: Not compiled with GPU support (ROIAlign_forward at /mnt/lustre/liang1/project/detectron2/detectron2/layers/csrc/ROIAlign/ROIAlign.h:71)

I have install detectron2 with python setup.py build develop, will it compile only with CPU by default? thx.

@ppwwyyxx
Copy link
Contributor

It will compile with CPU only, only if pytorch cannot find gpus/cuda at the time you compile it.
It can be checked by

python -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)'

@Angzz
Copy link
Author

Angzz commented Oct 14, 2019

it' s very strange that:

print(torch.version.cuda)
print(torch.__version__)

and the outputs:

'9.2.148'
'1.3.0+cu92'

my cuda version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148

but when run commond above:

python -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)'

it still return false.

@ppwwyyxx
Copy link
Contributor

Since it is an pytorch/cuda installation issue, please ask in pytorch forum or issues instead. Thanks!

@zhudd-hub
Copy link

zhudd-hub commented Oct 15, 2019

I have the same issue when I run demo in jupyter notebook:
RuntimeError: Not compiled with GPU support (ROIAlign_forward at /home/workspace/zhudd/detectron2/detectron2/layers/csrc/ROIAlign/ROIAlign.h:71) frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7ff68cc04687 in /home/workspace/zhudd/anaconda3/envs/detectron/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: ROIAlign_forward(at::Tensor const&, at::Tensor const&, float, int, int, int, bool) + 0x149 (0x7ff67aaf62b9 in /home/workspace/zhudd/detectron2/detectron2/_C.cpython-37m-x86_64-linux-gnu.so) frame #2: <unknown function> + 0x2097f (0x7ff67ab0597f in /home/workspace/zhudd/detectron2/detectron2/_C.cpython-37m-x86_64-linux-gnu.so) frame #3: <unknown function> + 0x20a6e (0x7ff67ab05a6e in /home/workspace/zhudd/detectron2/detectron2/_C.cpython-37m-x86_64-linux-gnu.so) frame #4: <unknown function> + 0x1b358 (0x7ff67ab00358 in /home/workspace/zhudd/detectron2/detectron2/_C.cpython-37m-x86_64-linux-gnu.so) frame #5: _PyMethodDef_RawFastCallKeywords + 0x264 (0x562d6d8cd774 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #6: _PyCFunction_FastCallKeywords + 0x21 (0x562d6d8cd891 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #7: _PyEval_EvalFrameDefault + 0x4ede (0x562d6d93afce in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #8: _PyFunction_FastCallDict + 0x10b (0x562d6d87c92b in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #9: THPFunction_apply(_object*, _object*) + 0x8d6 (0x7ff6bf3ade96 in /home/workspace/zhudd/anaconda3/envs/detectron/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #10: _PyMethodDef_RawFastCallKeywords + 0x1e0 (0x562d6d8cd6f0 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #11: _PyCFunction_FastCallKeywords + 0x21 (0x562d6d8cd891 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #12: _PyEval_EvalFrameDefault + 0x47d4 (0x562d6d93a8c4 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #13: _PyFunction_FastCallDict + 0x10b (0x562d6d87c92b in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #14: _PyObject_Call_Prepend + 0x63 (0x562d6d89be23 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #15: PyObject_Call + 0x6e (0x562d6d88e51e in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #16: _PyEval_EvalFrameDefault + 0x1f4c (0x562d6d93803c in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #17: _PyEval_EvalCodeWithName + 0x2f9 (0x562d6d87b929 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #18: _PyFunction_FastCallDict + 0x1d5 (0x562d6d87c9f5 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #19: _PyObject_Call_Prepend + 0x63 (0x562d6d89be23 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #20: <unknown function> + 0x1702ea (0x562d6d8d52ea in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #21: _PyObject_FastCallKeywords + 0x3fb (0x562d6d8d616b in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #22: _PyEval_EvalFrameDefault + 0x4ac6 (0x562d6d93abb6 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #23: _PyFunction_FastCallDict + 0x10b (0x562d6d87c92b in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #24: _PyObject_Call_Prepend + 0x63 (0x562d6d89be23 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #25: PyObject_Call + 0x6e (0x562d6d88e51e in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #26: _PyEval_EvalFrameDefault + 0x1f4c (0x562d6d93803c in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #27: _PyEval_EvalCodeWithName + 0x2f9 (0x562d6d87b929 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #28: _PyFunction_FastCallDict + 0x1d5 (0x562d6d87c9f5 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #29: _PyObject_Call_Prepend + 0x63 (0x562d6d89be23 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #30: <unknown function> + 0x1702ea (0x562d6d8d52ea in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #31: _PyObject_FastCallKeywords + 0x3fb (0x562d6d8d616b in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #32: _PyEval_EvalFrameDefault + 0x53ae (0x562d6d93b49e in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #33: _PyFunction_FastCallKeywords + 0xfb (0x562d6d8cccfb in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #34: _PyEval_EvalFrameDefault + 0x4b69 (0x562d6d93ac59 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #35: _PyEval_EvalCodeWithName + 0xba9 (0x562d6d87c1d9 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #36: _PyFunction_FastCallDict + 0x1d5 (0x562d6d87c9f5 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #37: _PyObject_Call_Prepend + 0x63 (0x562d6d89be23 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #38: PyObject_Call + 0x6e (0x562d6d88e51e in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #39: _PyEval_EvalFrameDefault + 0x1f4c (0x562d6d93803c in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #40: _PyEval_EvalCodeWithName + 0x2f9 (0x562d6d87b929 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #41: _PyFunction_FastCallDict + 0x1d5 (0x562d6d87c9f5 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #42: _PyObject_Call_Prepend + 0x63 (0x562d6d89be23 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #43: <unknown function> + 0x1702ea (0x562d6d8d52ea in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #44: _PyObject_FastCallKeywords + 0x3fb (0x562d6d8d616b in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #45: _PyEval_EvalFrameDefault + 0x53ae (0x562d6d93b49e in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #46: _PyEval_EvalCodeWithName + 0xba9 (0x562d6d87c1d9 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #47: _PyFunction_FastCallDict + 0x1d5 (0x562d6d87c9f5 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #48: _PyObject_Call_Prepend + 0x63 (0x562d6d89be23 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #49: PyObject_Call + 0x6e (0x562d6d88e51e in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #50: _PyEval_EvalFrameDefault + 0x1f4c (0x562d6d93803c in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #51: _PyEval_EvalCodeWithName + 0x2f9 (0x562d6d87b929 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #52: _PyFunction_FastCallDict + 0x1d5 (0x562d6d87c9f5 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #53: _PyObject_Call_Prepend + 0x63 (0x562d6d89be23 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #54: <unknown function> + 0x1702ea (0x562d6d8d52ea in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #55: _PyObject_FastCallKeywords + 0x3fb (0x562d6d8d616b in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #56: _PyEval_EvalFrameDefault + 0x53ae (0x562d6d93b49e in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #57: _PyFunction_FastCallDict + 0x10b (0x562d6d87c92b in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #58: _PyEval_EvalFrameDefault + 0x1f4c (0x562d6d93803c in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #59: _PyEval_EvalCodeWithName + 0xba9 (0x562d6d87c1d9 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #60: _PyFunction_FastCallDict + 0x1d5 (0x562d6d87c9f5 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #61: _PyObject_Call_Prepend + 0x63 (0x562d6d89be23 in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #62: <unknown function> + 0x1702ea (0x562d6d8d52ea in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3) frame #63: _PyObject_FastCallKeywords + 0x3fb (0x562d6d8d616b in /home/workspace/zhudd/anaconda3/envs/detectron/bin/python3)
### whatsmore,when I run:
python -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)'
### it return :
True /usr/local/cuda
the version of cuda
`+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104 Driver Version: 410.104 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|=====================+=============+==============|
| 0 Tesla P100-PCIE... Off | 00000000:3B:00.0 Off | 0 |
| N/A 40C P0 29W / 250W | 4836MiB / 12198MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla P100-PCIE... Off | 00000000:AF:00.0 Off | 0 |
| N/A 37C P0 27W / 250W | 10MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla P100-PCIE... Off | 00000000:D8:00.0 Off | 0 |
| N/A 39C P0 28W / 250W | 10MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

my pytorch is 1.3
what should I do to solve the problem

@ppwwyyxx
Copy link
Contributor

You should uninstall and reinstall detectron2, when python -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)' is working

@SkeletonOne
Copy link
Contributor

@zhudd-hub I meet a similar problem as same as yours. Have you solved it?

@dale1202
Copy link

i run python3 -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)', it return True。
and i run detectron2, it still return "RuntimeError: Not compiled with GPU support", Have you solved it? @ppwwyyxx

@viven12138
Copy link
Contributor

i run python3 -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)', it return True。
and i run detectron2, it still return "RuntimeError: Not compiled with GPU support", Have you solved it? @ppwwyyxx

The same question! Have u solved it?

@dale1202
Copy link

dale1202 commented Oct 18, 2019

i have solved it, reinstall detectron2 : remove bulid file , execute "python setup.py build develop" again,make sure your cuda version is match with pytorch version @ppwwyyxx

@viven12138
Copy link
Contributor

viven12138 commented Oct 18, 2019

i have solved it, reinstall detectron2 : remove bulid file , execute "python setup.py build develop" again,make sure your cuda version is match with pytorch version @ppwwyyxx

@dale1202 I did it as what u said and ran into the same problem"RuntimeError: Not compiled with GPU support ". :(

@dale1202
Copy link

use cuda9.2,use "pip3 install torch==1.3.0+cu92 torchvision==0.4.1+cu92 -f https://download.pytorch.org/whl/torch_stable.html" to install pytorch。execute "nvcc --version" check cuda version

@viven12138
Copy link
Contributor

use cuda9.2,use "pip3 install torch==1.3.0+cu92 torchvision==0.4.1+cu92 -f https://download.pytorch.org/whl/torch_stable.html" to install pytorch。execute "nvcc --version" check cuda version
@dale1202 I install pytorch, cuda and cudnn in a conda environment. I can't use nvcc, cause I don't have root permission. What's more, the command python -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)' returns "True None". Is it related to the conda environment?
My environment info:

------------------------  --------------------------------------------------
sys.platform              linux
Python                    3.7.3 (default, Mar 27 2019, 22:11:17) [GCC 7.3.0]
Numpy                     1.17.2
Detectron2 Compiler       GCC 7.4
Detectron2 CUDA Compiler  not available
DETECTRON2_ENV_MODULE     <not set>
PyTorch                   1.3.0+cu92
PyTorch Debug Build       False
torchvision               0.4.1+cu92
CUDA available            True
GPU 0,1,2,3               GeForce RTX 2080 Ti
CUDA_HOME                 None
Pillow                    6.2.0
cv2                       4.1.1
------------------------  --------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.20.5 (Git Hash 0125f28c61c1f822fd48570b4c1066f96fcb9b2e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 9.2
  - NVCC architecture flags: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_50,code=compute_50
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=True, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

@ppwwyyxx
Copy link
Contributor

If you can't use nvcc, then you can't install detectron2.
nvcc never requires root permission.

@viven12138
Copy link
Contributor

If you can't use nvcc, then you can't install detectron2.
nvcc never requires root permission.

I can't use nvcc, it returns "apt install nvidia-cuda-toolkit", which needs root permission.
I actually see detectron2 in my conda list.

@viven12138
Copy link
Contributor

If you can't use nvcc, then you can't install detectron2.
nvcc never requires root permission.

I can't use nvcc, it returns "apt install nvidia-cuda-toolkit", which needs root permission.
I actually see detectron2 in my conda list.

I ask root to install cuda. And I met the issue "vcc fatal : Unsupported gpu architecture 'compute_75'", then i add "export TORCH_CUDA_ARCH_LIST="7.0"" to the environment to solve it!
reference

@marsggbo
Copy link

check two things:

  1. whether pytorch-gpu version matches your nvcc version
  2. whether your set CUDA_HOME correctly, e.g. export CUDA_HOME=/usr/local/cuda in ~/.bashrc

reference: https://zhuanlan.zhihu.com/p/93278639

@fatalfeel
Copy link

#if meet RuntimeError('Not compiled with GPU support#
(i have same problem and do it ok now)
#make sure /usr/local/cuda link to /usr/local/cuda-10.1#
ln -sf /usr/local/cuda-10.1 /usr/local/cuda
rm -R /root/projects/detectron2/build
python3 setup.py build develop

@KunalMGupta
Copy link

KunalMGupta commented Mar 26, 2020

In case someone is still wondering how to solve it. This is what I tried:

  1. Check the cuda version using: python -c 'import torch; print(torch.version.cuda)'
  2. remove detectron2
  3. reinstall with the correct cuda: python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cuABC/index.html

where ABC is the cuda version found in step 1) i.e. ABC = 101 (cuda 10.1) and 100 (for cuda 10.0)

Following this I was able to solve this error. Hope it helps!

@MarkAlive
Copy link

remember to remove 'build' dir before reinstall detectron2.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

10 participants