Body Tracking SDK (1.1.2) is not Working With ROS Humble Branch directly On Windows 10

[Mujadid-Ali]

I have some Azure Kinect Sensor SDK (v1.4.1) recordings. I need body tracking data to be published on a ROS2 (humble) topic. I cloned the Azure Kinect ROS Driver repo, checked out its humble branch and try to build the Azure Kinect ROS Driver repo. I encountered the following three erros (2 were buildtime, one is runtime)

1. Following was the first error. Even I have already installed and configure the both Azure Kinect Sensor SDK (v1.4.1) and Azure Kinect Body Tracking SDK (v1.1.2) and add its bin directory as environment variable.
   
   After investigation, I came to know that in new Azure Body Tracking SDK (v1.1.2) the dnn_model.onxx filename is changed to dnn_model_2_0_op11.onnx while in cmake file Findk4abt.cmake line 9 the file name is still dnn_model.onxx. Thats why when cmake file try to find and install this file it does not found and Body Tracking SDK were not configuring.

2. Following was the second error that occurred
   
   Again after investigation I came to know that in cpp file k4a_ros_device.cpp line 771 a double value is directly passing to the rclcpp::Duration() which don't have any contractor that takes double value. So I fixed this issue by calling rclcpp::Duration::from_second() function which takes double input value. As you can see here.
   After resolving the above issue, the Azure Kinect ROS Driver is build successfully.

3. Now when I run the Driver in reply mode with body tracking feature on by using the following command.
   ros2 launch azure_kinect_ros_driver driver.launch.py body_tracking_enabled:=true recording_loop_enabled:=true recording_file:="<ABSOLUTE_PATH_TO_MKV_RECORDING_FILE>"
   A runtime error has occurred as shown below
   
   After investigation, I came to know that it's due to the usage of wrong directml.dll file. Since this DLL file is not placed on the installation directory, so our program try to use the default system DLL file, and it's generating the above error. If you place the directml.dll file that is shipped with Azure Kinect Body Tracking SDK (v1.1.2) installation in program Install folder, then it will use the correct DLL file and Body tracking data can be published on ROS2 topic.

To Reproduce
Steps to reproduce the behavior:

1. Install Azure Kinect Sensor SDK v1.4.1 and add its bin directory to environment variable
2. Install Azure Body Tracking SDK v1.1.2 and add its bin directory to environment variable
3. Clone Azure Kinect ROS Driver source code and checkout humble branch
4. build repo using colcon build command. And you will encounter the first error.

Note: make sure you already have and setuped all Azure Kinect ROS Driver dependencies

Expected behavior
The branch should be build and run Body tracking feature without any problem

Logs

1. First issue logs

-- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19045.
-- Found ament_cmake: 1.3.4 (C:/dev/ros2_humble/share/ament_cmake/cmake)
-- Found rclcpp: 16.0.4 (C:/dev/ros2_humble/share/rclcpp/cmake)
-- Found rosidl_generator_c: 3.1.4 (C:/dev/ros2_humble/share/rosidl_generator_c/cmake)
-- Found rosidl_adapter: 3.1.4 (C:/dev/ros2_humble/share/rosidl_adapter/cmake)
-- Found rosidl_generator_cpp: 3.1.4 (C:/dev/ros2_humble/share/rosidl_generator_cpp/cmake)
-- Using all available rosidl_typesupport_c: rosidl_typesupport_fastrtps_c;rosidl_typesupport_introspection_c
-- Using all available rosidl_typesupport_cpp: rosidl_typesupport_fastrtps_cpp;rosidl_typesupport_introspection_cpp
-- Found rmw_implementation_cmake: 6.1.1 (C:/dev/ros2_humble/share/rmw_implementation_cmake/cmake)
-- Found rmw_fastrtps_cpp: 6.2.2 (C:/dev/ros2_humble/share/rmw_fastrtps_cpp/cmake)
-- Using RMW implementation 'rmw_fastrtps_cpp' as default
-- Found std_msgs: 4.2.3 (C:/dev/ros2_humble/share/std_msgs/cmake)
-- Found sensor_msgs: 4.2.3 (C:/dev/ros2_humble/share/sensor_msgs/cmake)
-- Found visualization_msgs: 4.2.3 (C:/dev/ros2_humble/share/visualization_msgs/cmake)
-- Found image_transport: 3.1.5 (C:/dev/ros2_humble/share/image_transport/cmake)
-- TinyXML2 was already found when tinyxml2_vendor was included, adding missing imported target tinyxml2::tinyxml2
-- Found tf2: 0.25.2 (C:/dev/ros2_humble/share/tf2/cmake)
-- Found tf2_ros: 0.25.2 (C:/dev/ros2_humble/share/tf2_ros/cmake)
-- Found tf2_geometry_msgs: 0.25.2 (C:/dev/ros2_humble/share/tf2_geometry_msgs/cmake)
-- Found eigen3_cmake_module: 0.1.1 (C:/dev/ros2_humble/share/eigen3_cmake_module/cmake)
-- Ensuring Eigen3 include directory is part of orocos-kdl CMake target
-- Found cv_bridge: 3.2.1 (D:/projects/SedeniusFreelaunc/vesc/ros2_ws2/install/cv_bridge/share/cv_bridge/cmake)
-- Found angles: 1.16.0 (D:/projects/SedeniusFreelaunc/vesc/ros2_ws2/install/angles/share/angles/cmake)
Finding K4A SDK binaries
-- Body Tracking SDK found: compiling support for Body Tracking
Body Tracking SDK found: compiling support for Body Tracking
K4A Libs: k4a::k4a;k4a::k4arecord;k4abt::k4abt

2. Second Issue logs

-- Configuring done (10.2s)
-- Generating done (1.4s)
-- Build files have been written to: 
Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

  Checking Build System
  Building Custom Rule D:/projects/SedeniusFreelaunc/vesc/ros2_ws/src/Bandit_ROS_Controller/Azure_Kinect_ROS_Driver/CMakeLists.txt
  k4a_ros_device.cpp
D:\projects\SedeniusFreelaunc\vesc\ros2_ws\src\Bandit_ROS_Controller\Azure_Kinect_ROS_Driver\src\k4a_ros_device.cpp(686,57): warning C4305: 'initializing': truncation from 'double' to 'const float' [D:\projects\SedeniusFreelaunc\vesc\ros2_ws\build\azure_kinect_ros_driver\azure_kinect_ros_driver_node.vcxproj]
D:\projects\SedeniusFreelaunc\vesc\ros2_ws\src\Bandit_ROS_Controller\Azure_Kinect_ROS_Driver\src\k4a_ros_device.cpp(730,57): warning C4305: 'initializing': truncation from 'double' to 'const float' [D:\projects\SedeniusFreelaunc\vesc\ros2_ws\build\azure_kinect_ros_driver\azure_kinect_ros_driver_node.vcxproj]
D:\projects\SedeniusFreelaunc\vesc\ros2_ws\src\Bandit_ROS_Controller\Azure_Kinect_ROS_Driver\src\k4a_ros_device.cpp(773,47): error C2440: '<function-style-cast>': cannot convert from 'double' to 'rclcpp::Duration' [D:\projects\SedeniusFreelaunc\vesc\ros2_ws\build\azure_kinect_ros_driver\azure_kinect_ros_driver_node.vcxproj]
D:\projects\SedeniusFreelaunc\vesc\ros2_ws\src\Bandit_ROS_Controller\Azure_Kinect_ROS_Driver\src\k4a_ros_device.cpp(773,47): message : No constructor could take the source type, or constructor overload resolution was ambiguous [D:\projects\SedeniusFreelaunc\vesc\ros2_ws\build\azure_kinect_ros_driver\azure_kinect_ros_driver_node.vcxproj]
D:\projects\SedeniusFreelaunc\vesc\ros2_ws\src\Bandit_ROS_Controller\Azure_Kinect_ROS_Driver\src\k4a_ros_device.cpp(1210,54): warning C4267: 'argument': conversion from 'size_t' to 'uint32_t', possible loss of data [D:\projects\SedeniusFreelaunc\vesc\ros2_ws\build\azure_kinect_ros_driver\azure_kinect_ros_driver_node.vcxproj]
D:\projects\SedeniusFreelaunc\vesc\ros2_ws\src\Bandit_ROS_Controller\Azure_Kinect_ROS_Driver\src\k4a_ros_device.cpp(1262,21): warning C4244: 'initializing': conversion from 'unsigned __int64' to 'float', possible loss of data [D:\projects\SedeniusFreelaunc\vesc\ros2_ws\build\azure_kinect_ros_driver\azure_kinect_ros_driver_node.vcxproj]

3. Third Issue logs

[node.EXE-1] [INFO] [1690912042.265085700] [k4a_calibration_transform_data]:             Extrinsics:
[node.EXE-1] [INFO] [1690912042.265150000] [k4a_calibration_transform_data]:                     Translation: -28.3245, -2.87884, -47.3492
[node.EXE-1] [INFO] [1690912042.265225600] [k4a_calibration_transform_data]:                     Rotation[0]: -0.00218384, -0.999955, -0.00927506
[node.EXE-1] [INFO] [1690912042.265278100] [k4a_calibration_transform_data]:                     Rotation[1]: 0.0203125, -0.00931753, 0.99975
[node.EXE-1] [INFO] [1690912042.265326700] [k4a_calibration_transform_data]:                     Rotation[2]: -0.999791, 0.0019949, 0.020332
[node.EXE-1] 2023-08-01 22:47:25.2389496 [E:onnxruntime:, inference_session.cc:1449 onnxruntime::InferenceSession::Initialize::<lambda_a7567fc66851077067532e04df6b55ab>::operator ()] Exception during initialization: C:\Users\vache\Desktop\Kinect\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\FusedGraphKernel.cpp(64)\onnxruntime.dll!00007FFC80C88B13: (caller: 00007FFC80C85FC9) Exception(1) tid(321c) 80004002 No such interface supported
[node.EXE-1]
[node.EXE-1] Exception during initialization: C:\Users\vache\Desktop\Kinect\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\FusedGraphKernel.cpp(64)\onnxruntime.dll!00007FFC80C88B13: (caller: 00007FFC80C85FC9) Exception(1) tid(321c) 80004002 No such interface supported
[node.EXE-1]
[node.EXE-1] [2023-08-01 22:47:25.313] [error] [t=12828] [K4ABT] ..\src\TrackerHost\TrackerHost.cpp (184): Create(). Exception during initialization: C:\Users\vache\Desktop\Kinect\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\FusedGraphKernel.cpp(64)\onnxruntime.dll!00007FFC80C88B13: (caller: 00007FFC80C85FC9) Exception(1) tid(321c) 80004002 No such interface supported
[node.EXE-1]
[node.EXE-1] [2023-08-01 22:47:25.318] [error] [t=12828] [K4ABT] ..\src\sdk\k4abt.cpp (39): tracker->Create(sensor_calibration, config) returned failure in k4abt_tracker_create()
[ERROR] [node.EXE-1]: process has died [pid 12536, exit code 3221226505, cmd 'D:\projects\SedeniusFreelaunc\vesc\ros2_ws2\install\azure_kinect_ros_driver\lib\azure_kinect_ros_driver\node.EXE --ros-args --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_z48dpz4f --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_416z75gi --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_2ehg6kbb --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_v07j83aa --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_anur5x4_ --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_2tyo71v6 --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_m4p2q2aq --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_v4e0ycxj --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_xm0u9cq9 --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_e66_g4ms --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_xicguwye --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_2lr2r244 --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_c3nhnaw_ --params-file C:\Users\syed_\AppData\Local\Temp\launch_params__cx2h0u_ --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_5hqk19qg --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_n_08bola --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_3l3_wq9w --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_6z8d1pg1 --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_2yqp8rtn --params-file C:\Users\syed_\AppData\Local\Temp\launch_params_3hl__jum'].
[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[INFO] [robot_state_publisher.EXE-2]: process has finished cleanly [pid 12480]
[INFO] [joint_state_publisher.EXE-3]: process has finished cleanly [pid 7708]

Desktop (please complete the following information):

• OS: Windows 10
• Version: 19041
• Azure Kinect Sensor SDK: 1.4.1
• Azure Body Tracking SDK: 1.1.2

Solutions

1. The solution to the first issue is simply chnage the file name from dnn_model.onnx to dnn_model_2_0_op11.onnx in file Findk4abt.cmake line 7
   

2. The solution to the second issue is simply add the ::from_second() function in file k4a_ros_device.cpp line 771 as
   

3. The solution to the third issue is place the code in cmake to install directml.dll files with other files too in the install directory so that node will always use the shipped DLL file,