Python bindings do not compile with MSVC Visual Studio C++ compiler (cl.exe) work fine with LLVM's VS compatible compiler (clang-cl.exe)

[traversaro]

I know Windows + MSVC + Python is probably not used a lot, but I experienced a problem (that may be related to MSVC compiler limitation and I can't more spend in debugging it, so I will open an issue to report what I discovered).

It seems that compiling the python bindings on Windows works fine clang-cl.exe (the clang drop in replacement for VS's cl.exe) but fails with cl.exe .

To reproduce the problem, if you have vs2019 or vs2022 with C++ support installed in your system and pixi installed, run:

git clone -b winfailure https://github.com/traversaro/manif
# To run the build (and the failure) with VS2019
pixi run -e vs2019 build
# To run the build (and the failure) with VS2019
pixi run -e vs2022 build

The output on my system in vs2019:

D:\src\tst\manif>pixi run -e vs2019 build
✨ Pixi task (configure in vs2019): cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING:BOOL=OFF -DBUILD_PYTHON_BINDINGS:BOOL=ON -G Ninja -S . -B build2019
CMake Warning (dev) at .pixi/envs/vs2019/Library/share/cmake/pybind11/FindPythonLibsNew.cmake:101 (message):
  Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
  are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
  the cmake_policy command to set the policy and suppress this warning, or
  preferably upgrade to using FindPython, either by calling it explicitly
  before pybind11, or by setting PYBIND11_FINDPYTHON ON before pybind11.
Call Stack (most recent call first):
  .pixi/envs/vs2019/Library/share/cmake/pybind11/pybind11Tools.cmake:50 (find_package)
  .pixi/envs/vs2019/Library/share/cmake/pybind11/pybind11Common.cmake:228 (include)
  .pixi/envs/vs2019/Library/share/cmake/pybind11/pybind11Config.cmake:250 (include)
  CMakeLists.txt:73 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found pybind11: D:/src/tst/manif/.pixi/envs/vs2019/Library/include (found version "2.13.5")
-- Installing manifpy in D:\src\tst\manif\.pixi\envs\vs2019/manifpy
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: D:/src/tst/manif/build2019

✨ Pixi task (print_cl_version in vs2019): cl.exe
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30154 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

✨ Pixi task (build in vs2019): cmake --build build2019 --config Release  --parallel 1
[1/8] Building CXX object python\CMakeFiles\manifpy.dir\bindings_rn.cpp.obj
FAILED: python/CMakeFiles/manifpy.dir/bindings_rn.cpp.obj
C:\PROGRA~2\MICROS~4\2019\BUILDT~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\cl.exe  /nologo /TP -DEIGEN_DEFAULT_TO_ROW_MAJOR -Dmanifpy_EXPORTS -ID:\src\tst\manif\include -ID:\src\tst\manif\external\tl -external:ID:\src\tst\manif\.pixi\envs\vs2019\Library\include -external:ID:\src\tst\manif\.pixi\envs\vs2019\Include -external:ID:\src\tst\manif\.pixi\envs\vs2019\Library\include\eigen3 -external:W0 /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MD /O2 /Ob2 /DNDEBUG /GL /bigobj /MP /showIncludes /Fopython\CMakeFiles\manifpy.dir\bindings_rn.cpp.obj /Fdpython\CMakeFiles\manifpy.dir\ /FS -c D:\src\tst\manif\python\bindings_rn.cpp
D:\src\tst\manif\.pixi\envs\vs2019\Library\include\pybind11/pybind11.h(1669): error C2661: 'pybind11::cpp_function::cpp_function': no overloaded function takes 8 arguments
D:\src\tst\manif\python\bindings_lie_group_base.h(122): note: see reference to function template instantiation 'pybind11::class_<manif::R1d,manif::LieGroupBase<manif::R1d>> &pybind11::class_<manif::R1d,manif::LieGroupBase<manif::R1d>>::def<manif::Rn<double,1>(__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const,pybind11::arg,pybind11::arg_v,pybind11::arg_v,char[358]>(const char *,Func &&,const pybind11::arg &,const pybind11::arg_v &,const pybind11::arg_v &,const char (&)[358])' being compiled
        with
        [
            Func=manif::Rn<double,1> (__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const
        ]
D:\src\tst\manif\python\bindings_lie_group_base.h(104): note: see reference to function template instantiation 'pybind11::class_<manif::R1d,manif::LieGroupBase<manif::R1d>> &pybind11::class_<manif::R1d,manif::LieGroupBase<manif::R1d>>::def<manif::Rn<double,1>(__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const,pybind11::arg,pybind11::arg_v,pybind11::arg_v,char[358]>(const char *,Func &&,const pybind11::arg &,const pybind11::arg_v &,const pybind11::arg_v &,const char (&)[358])' being compiled
        with
        [
            Func=manif::Rn<double,1> (__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const
        ]
D:\src\tst\manif\python\bindings_rn.cpp(23): note: see reference to function template instantiation 'void wrap_lie_group_base<manif::R1d,manif::LieGroupBase<manif::R1d>>(pybind11::class_<manif::R1d,manif::LieGroupBase<manif::R1d>> &)' being compiled
ninja: build stopped: subcommand failed.

The output on my system in vs2022:

D:\src\tst\manif>pixi run -e vs2022 build
✨ Pixi task (configure in vs2022): cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING:BOOL=OFF -DBUILD_PYTHON_BINDINGS:BOOL=ON -G Ninja -S . -B build2022
CMake Warning (dev) at .pixi/envs/vs2022/Library/share/cmake/pybind11/FindPythonLibsNew.cmake:101 (message):
  Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
  are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
  the cmake_policy command to set the policy and suppress this warning, or
  preferably upgrade to using FindPython, either by calling it explicitly
  before pybind11, or by setting PYBIND11_FINDPYTHON ON before pybind11.
Call Stack (most recent call first):
  .pixi/envs/vs2022/Library/share/cmake/pybind11/pybind11Tools.cmake:50 (find_package)
  .pixi/envs/vs2022/Library/share/cmake/pybind11/pybind11Common.cmake:228 (include)
  .pixi/envs/vs2022/Library/share/cmake/pybind11/pybind11Config.cmake:250 (include)
  CMakeLists.txt:73 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found pybind11: D:/src/tst/manif/.pixi/envs/vs2022/Library/include (found version "2.13.5")
-- Installing manifpy in D:\src\tst\manif\.pixi\envs\vs2022/manifpy
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: D:/src/tst/manif/build2022

✨ Pixi task (print_cl_version in vs2022): cl.exe
Microsoft (R) C/C++ Optimizing Compiler Version 19.41.34120 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

✨ Pixi task (build in vs2022): cmake --build build2022 --config Release --parallel 1
[1/8] Building CXX object python\CMakeFiles\manifpy.dir\bindings_rn.cpp.obj
FAILED: python/CMakeFiles/manifpy.dir/bindings_rn.cpp.obj
C:\PROGRA~1\MIB055~1\2022\ENTERP~1\VC\Tools\MSVC\1441~1.341\bin\Hostx64\x64\cl.exe  /nologo /TP -DEIGEN_DEFAULT_TO_ROW_MAJOR -Dmanifpy_EXPORTS -ID:\src\tst\manif\include -ID:\src\tst\manif\external\tl -external:ID:\src\tst\manif\.pixi\envs\vs2022\Library\include -external:ID:\src\tst\manif\.pixi\envs\vs2022\Include -external:ID:\src\tst\manif\.pixi\envs\vs2022\Library\include\eigen3 -external:W0 /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MD /O2 /Ob2 /DNDEBUG /GL /bigobj /MP /showIncludes /Fopython\CMakeFiles\manifpy.dir\bindings_rn.cpp.obj /Fdpython\CMakeFiles\manifpy.dir\ /FS -c D:\src\tst\manif\python\bindings_rn.cpp
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): error C2661: 'pybind11::cpp_function::cpp_function': no overloaded function takes 8 arguments
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(186): note: could be 'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...) const &,const Extra ...)'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...) const &,const Extra ...)': template parameter 'Class' is ambiguous
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: could be 'manif::LieGroupBase<manif::R1d>'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: or       'manif::Rn<double,1>'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...) const &,const Extra ...)': could not deduce template argument for 'Return (__cdecl Class::* )(Arg...) const &' from '_Ty'
        with
        [
            _Ty=manif::Rn<double,1> (__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const
        ]
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(174): note: or       'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...) const,const Extra ...)'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...) const,const Extra ...)': template parameter 'Class' is ambiguous
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: could be 'manif::LieGroupBase<manif::R1d>'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: or       'manif::Rn<double,1>'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...) const,const Extra ...)': could not deduce template argument for 'Return (__cdecl Class::* )(Arg...) const' from '_Ty'
        with
        [
            _Ty=manif::Rn<double,1> (__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const
        ]
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(164): note: or       'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...) &,const Extra ...)'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...) &,const Extra ...)': template parameter 'Class' is ambiguous
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: could be 'const manif::LieGroupBase<manif::R1d>'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: or       'manif::Rn<double,1>'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...) &,const Extra ...)': could not deduce template argument for 'Return (__cdecl Class::* )(Arg...) &' from '_Ty'
        with
        [
            _Ty=manif::Rn<double,1> (__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const
        ]
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(152): note: or       'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...),const Extra ...)'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...),const Extra ...)': template parameter 'Class' is ambiguous
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: could be 'const manif::LieGroupBase<manif::R1d>'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: or       'manif::Rn<double,1>'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Return (__cdecl Class::* )(Arg...),const Extra ...)': could not deduce template argument for 'Return (__cdecl Class::* )(Arg...)' from '_Ty'
        with
        [
            _Ty=manif::Rn<double,1> (__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const
        ]
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(144): note: or       'pybind11::cpp_function::cpp_function(Func &&,const Extra ...)'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Func &&,const Extra ...)': could not deduce template argument for '<unnamed-symbol>'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(142): note: 'std::enable_if_t<false,void>' : Failed to specialize alias template
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(135): note: or       'pybind11::cpp_function::cpp_function(Return (__cdecl *)(Args...),const Extra ...)'
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: 'pybind11::cpp_function::cpp_function(Return (__cdecl *)(Args...),const Extra ...)': could not deduce template argument for 'Return (__cdecl *)(Args...)' from '_Ty'
        with
        [
            _Ty=manif::Rn<double,1> (__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const
        ]
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: while trying to match the argument list '(_Ty, pybind11::name, pybind11::is_method, pybind11::sibling, const pybind11::arg, const pybind11::arg_v, const pybind11::arg_v, const char [358])'
        with
        [
            _Ty=manif::Rn<double,1> (__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const
        ]
D:\src\tst\manif\.pixi\envs\vs2022\Library\include\pybind11/pybind11.h(1669): note: the template instantiation context (the oldest one first) is
D:\src\tst\manif\python\bindings_rn.cpp(23): note: see reference to function template instantiation 'void wrap_lie_group_base<manif::R1d,manif::LieGroupBase<manif::R1d>>(pybind11::class_<manif::R1d,manif::LieGroupBase<manif::R1d>> &)' being compiled
D:\src\tst\manif\python\bindings_lie_group_base.h(104): note: see reference to function template instantiation 'pybind11::class_<manif::R1d,manif::LieGroupBase<manif::R1d>> &pybind11::class_<manif::R1d,manif::LieGroupBase<manif::R1d>>::def<manif::Rn<double,1>(__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const,pybind11::arg,pybind11::arg_v,pybind11::arg_v,char[358]>(const char *,Func &&,const pybind11::arg &,const pybind11::arg_v &,const pybind11::arg_v &,const char (&)[358])' being compiled
        with
        [
            Func=manif::Rn<double,1> (__cdecl manif::Rn<double,1>::* )(const manif::LieGroupBase<manif::R1d> &,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>,tl::optional<Eigen::Ref<Eigen::Matrix<double,1,1,1,1,1>,0,Eigen::InnerStride<1>>>) const
        ]

[traversaro]

If you are experiencing this problem, using clang-cl.exe instead of cl.exe should fix the problem. However, I wanted to report the problem somewhere in case somebody was encountering it.