UnicodeDecodeError decode error during linking

[NikolausDemmel]

I just got this error during linking the devel products (this is on pre-0.4.0-clean-pkgs):

Errors     << zeno_tracking_lib:symlink /home/den2pal/work/ros_ws/logs/zeno_tracking_lib/build.symlink.003.log                                                                                                                             
Stage `symlink` failed with arguments:
  package: {'package_format': 2, 'exports': [], 'buildtool_export_depends': [], 'exec_depends': [<catkin_pkg.package.Dependency object at 0x7f7a4522bb90>, <catkin_pkg.package.Dependency object at 0x7f7a4522bcb0>, <catkin_pkg.package.Dependency object at 0x7f7a4522bc50>, <catkin_pkg.package.Dependency object at 0x7f7a4522be90>, <catkin_pkg.package.Dependency object at 0x7f7a451eb050>, <catkin_pkg.package.Dependency object at 0x7f7a451eb0b0>, <catkin_pkg.package.Dependency object at 0x7f7a451eb110>, <catkin_pkg.package.Dependency object at 0x7f7a451eb170>, <catkin_pkg.package.Dependency object at 0x7f7a451eb1d0>, <catkin_pkg.package.Dependency object at 0x7f7a451eb230>, <catkin_pkg.package.Dependency object at 0x7f7a451eb290>], 'name': 'zeno_tracking_lib', 'filename': '/home/den2pal/work/ros_ws/src/pilot/TrackingLib/package.xml', 'replaces': [], 'version_abi': None, 'build_export_depends': [<catkin_pkg.package.Dependency object at 0x7f7a451eb2f0>, <catkin_pkg.package.Dependency object at 0x7f7a451eb350>, <catkin_pkg.package.Dependency object at 0x7f7a451eb3b0>, <catkin_pkg.package.Dependency object at 0x7f7a451eb410>, <catkin_pkg.package.Dependency object at 0x7f7a451eb470>, <catkin_pkg.package.Dependency object at 0x7f7a451eb4d0>, <catkin_pkg.package.Dependency object at 0x7f7a451eb530>, <catkin_pkg.package.Dependency object at 0x7f7a451eb590>, <catkin_pkg.package.Dependency object at 0x7f7a451eb5f0>, <catkin_pkg.package.Dependency object at 0x7f7a451eb650>, <catkin_pkg.package.Dependency object at 0x7f7a451eb6b0>], 'doc_depends': [], 'test_depends': [], 'maintainers': [<catkin_pkg.package.Person object at 0x7f7a45220150>], 'version': '0.0.0', 'build_depends': [<catkin_pkg.package.Dependency object at 0x7f7a451eb710>, <catkin_pkg.package.Dependency object at 0x7f7a451eb770>, <catkin_pkg.package.Dependency object at 0x7f7a451eb7d0>, <catkin_pkg.package.Dependency object at 0x7f7a451eb830>, <catkin_pkg.package.Dependency object at 0x7f7a451eb890>, <catkin_pkg.package.Dependency object at 0x7f7a451eb8f0>, <catkin_pkg.package.Dependency object at 0x7f7a451eb950>, <catkin_pkg.package.Dependency object at 0x7f7a451eb9b0>, <catkin_pkg.package.Dependency object at 0x7f7a451eba10>, <catkin_pkg.package.Dependency object at 0x7f7a451eba70>, <catkin_pkg.package.Dependency object at 0x7f7a451ebad0>], 'urls': [], 'authors': [<catkin_pkg.package.Person object at 0x7f7a45220490>], 'licenses': ['ZenoWay internal'], 'buildtool_depends': [<catkin_pkg.package.Dependency object at 0x7f7a451ebb30>], 'conflicts': [], 'description': u'Zeno Pilot TrackingLib'}
  metadata_path: /home/den2pal/work/ros_ws/.catkin_tools/profiles/default
  devel_manifest_path: /home/den2pal/work/ros_ws/.catkin_tools/profiles/default/packages/zeno_tracking_lib
  prebuild: False
  package_path: pilot/TrackingLib
  dest_devel_path: /home/den2pal/work/ros_ws/devel
  source_devel_path: /home/den2pal/work/ros_ws/devel/.private/zeno_tracking_lib
Traceback (most recent call last):
  File "/home/den2pal/work/py/catkin_tools/catkin_tools/execution/executor.py", line 148, in async_job
    event_queue))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 40: ordinal not in range(128)

It seems that I had a strange character in a library name. After deleting the .so file manually and retyping the library name in CMakeLists.txt the problem is gone and I cannot reproduce it.

[TDiazT]

I experienced a similar issue when building pr2_navigation in our PR2 robot.

We cloned the package from http://github.com/pr2/pr2_navigation
And then tried to build it doing catkin build pr2_navigation and we got these error messages that seems to be related to this issue:

.........................................................................................................................................................................
_________________________________________________________________________________________________________________________________________________________________________
Warnings   << pr2_navigation_slam:symlink /home/pr2nav/catkin_ws/logs/pr2_navigation_slam/build.symlink.000.log
Warning: Cannot symlink from /home/pr2nav/catkin_ws/devel/.private/pr2_navigation_slam/lib/pkgconfig/pr2_navigation_slam.pc to existing file /home/pr2nav/catkin_ws/devel/lib/pkgconfig/pr2_navigation_slam.pc
Warning: Source hash: aed78e57b1f60554bb6040c80e6c0b34
Warning: Dest hash: 9b14662db8cc4fcc94c7e260c3246d91
Warning: Cannot symlink from /home/pr2nav/catkin_ws/devel/.private/pr2_navigation_slam/share/pr2_navigation_slam/cmake/pr2_navigation_slamConfig.cmake to existing file /home/pr2nav/catkin_ws/devel/share/pr2_navigation_slam/cmake/pr2_navigation_slamConfig.cmake
Warning: Source hash: e3bda60c7b03dbc2c2a750d3371e8915
Warning: Dest hash: c46464a65249e4b50ef192388c61e182
.........................................................................................................................................................................
Finished  <<< pr2_navigation_slam                         [ 4.2 seconds ]
_________________________________________________________________________________________________________________________________________________________________________
Warnings   << pr2_navigation_local:symlink /home/pr2nav/catkin_ws/logs/pr2_navigation_local/build.symlink.000.log
Warning: Cannot symlink from /home/pr2nav/catkin_ws/devel/.private/pr2_navigation_local/lib/pkgconfig/pr2_navigation_local.pc to existing file /home/pr2nav/catkin_ws/devel/lib/pkgconfig/pr2_navigation_local.pc
Warning: Source hash: d56360549f03308d214b53966d556140
Warning: Dest hash: b23a9130b29180a1b52edf58a0b6a48a
Warning: Cannot symlink from /home/pr2nav/catkin_ws/devel/.private/pr2_navigation_local/share/pr2_navigation_local/cmake/pr2_navigation_localConfig.cmake to existing file /home/pr2nav/catkin_ws/devel/share/pr2_navigation_local/cmake/pr2_navigation_localConfig.cmake
Warning: Source hash: 6ba44902ad2b300b27fc7ebfa22cf5f5
Warning: Dest hash: 332390abfcf4af1ca054886d6c62efc5
.........................................................................................................................................................................
Finished  <<< pr2_navigation_local                        [ 3.1 seconds ]
_________________________________________________________________________________________________________________________________________________________________________
Errors     << laser_tilt_controller_filter:symlink /home/pr2nav/catkin_ws/logs/laser_tilt_controller_filter/build.symlink.000.log
Stage `symlink` failed with arguments:
  package: {'package_format': 1, 'exports': [<catkin_pkg.package.Export object at 0x342d050>, <catkin_pkg.package.Export object at 0x342d200>], 'buildtool_export_depends': [], 'exec_depends': [<catkin_pkg.package.Dependency object at 0x3444950>, <catkin_pkg.package.Dependency object at 0x34449b0>, <catkin_pkg.package.Dependency object at 0x3444a10>, <catkin_pkg.package.Dependency object at 0x3444a70>, <catkin_pkg.package.Dependency object at 0x3444ad0>], 'name': 'laser_tilt_controller_filter', 'filename': '/home/pr2nav/catkin_ws/src/pr2_navigation/laser_tilt_controller_filter/package.xml', 'replaces': [], 'version_abi': None, 'build_export_depends': [<catkin_pkg.package.Dependency object at 0x3444b30>, <catkin_pkg.package.Dependency object at 0x3444b90>, <catkin_pkg.package.Dependency object at 0x3444bf0>, <catkin_pkg.package.Dependency object at 0x3444c50>, <catkin_pkg.package.Dependency object at 0x3444cb0>], 'doc_depends': [], 'test_depends': [], 'maintainers': [<catkin_pkg.package.Person object at 0x3445d50>], 'version': '0.1.27', 'build_depends': [<catkin_pkg.package.Dependency object at 0x3444d10>, <catkin_pkg.package.Dependency object at 0x3444d70>, <catkin_pkg.package.Dependency object at 0x3444dd0>, <catkin_pkg.package.Dependency object at 0x3444e30>, <catkin_pkg.package.Dependency object at 0x3444e90>], 'urls': [<catkin_pkg.package.Url object at 0x3445f10>], 'authors': [<catkin_pkg.package.Person object at 0x3445f90>], 'licenses': ['BSD'], 'buildtool_depends': [<catkin_pkg.package.Dependency object at 0x3444ef0>], 'conflicts': [], 'description': u'laser_tilt_controller_filter'}
  metadata_path: /home/pr2nav/catkin_ws/.catkin_tools/profiles/default
  devel_manifest_path: /home/pr2nav/catkin_ws/.catkin_tools/profiles/default/packages/laser_tilt_controller_filter
  prebuild: False
  package_path: pr2_navigation/laser_tilt_controller_filter
  dest_devel_path: /home/pr2nav/catkin_ws/devel
  source_devel_path: /home/pr2nav/catkin_ws/devel/.private/laser_tilt_controller_filter
Traceback (most recent call last):
  File "/home/pr2nav/.local/lib/python2.7/site-packages/catkin_tools/execution/executor.py", line 146, in async_job
    event_queue))
  File "/home/pr2nav/.local/lib/python2.7/site-packages/catkin_tools/verbs/catkin_build/cli.py", line 393, in main
    parallel_jobs = int(opts.parallel_jobs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x81 in position 41: ordinal not in range(128)

.........................................................................................................................................................................
Failed     << laser_tilt_controller_filter:symlink        [ Exited with code 3 ]
Failed    <<< laser_tilt_controller_filter                [ 16.8 seconds ]
Abandoned <<< pr2_navigation_perception                   [ Unrelated job failed ]
_________________________________________________________________________________________________________________________________________________________________________
Errors     << semantic_point_annotator:symlink /home/pr2nav/catkin_ws/logs/semantic_point_annotator/build.symlink.000.log
Stage `symlink` failed with arguments:
  package: {'package_format': 1, 'exports': [], 'buildtool_export_depends': [], 'exec_depends': [<catkin_pkg.package.Dependency object at 0x344b530>, <catkin_pkg.package.Dependency object at 0x344b590>, <catkin_pkg.package.Dependency object at 0x344b5f0>, <catkin_pkg.package.Dependency object at 0x344b650>], 'name': 'semantic_point_annotator', 'filename': '/home/pr2nav/catkin_ws/src/pr2_navigation/semantic_point_annotator/package.xml', 'replaces': [], 'version_abi': None, 'build_export_depends': [<catkin_pkg.package.Dependency object at 0x344b6b0>, <catkin_pkg.package.Dependency object at 0x344b710>, <catkin_pkg.package.Dependency object at 0x344b770>, <catkin_pkg.package.Dependency object at 0x344b7d0>], 'doc_depends': [], 'test_depends': [], 'maintainers': [<catkin_pkg.package.Person object at 0x344a810>], 'version': '0.1.27', 'build_depends': [<catkin_pkg.package.Dependency object at 0x344b830>, <catkin_pkg.package.Dependency object at 0x344b890>, <catkin_pkg.package.Dependency object at 0x344b8f0>, <catkin_pkg.package.Dependency object at 0x344b950>], 'urls': [<catkin_pkg.package.Url object at 0x344a990>], 'authors': [<catkin_pkg.package.Person object at 0x344aa10>], 'licenses': ['BSD'], 'buildtool_depends': [<catkin_pkg.package.Dependency object at 0x344b9b0>], 'conflicts': [], 'description': u'A node which annotates 3D point cloud data with semantic labels.'}
  metadata_path: /home/pr2nav/catkin_ws/.catkin_tools/profiles/default
  devel_manifest_path: /home/pr2nav/catkin_ws/.catkin_tools/profiles/default/packages/semantic_point_annotator
  prebuild: False
  package_path: pr2_navigation/semantic_point_annotator
  dest_devel_path: /home/pr2nav/catkin_ws/devel
  source_devel_path: /home/pr2nav/catkin_ws/devel/.private/semantic_point_annotator
Traceback (most recent call last):
  File "/home/pr2nav/.local/lib/python2.7/site-packages/catkin_tools/execution/executor.py", line 146, in async_job
    event_queue))
  File "/home/pr2nav/.local/lib/python2.7/site-packages/catkin_tools/verbs/catkin_build/cli.py", line 393, in main
    parallel_jobs = int(opts.parallel_jobs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 24: ordinal not in range(128)

.........................................................................................................................................................................
Failed     << semantic_point_annotator:symlink            [ Exited with code 3 ]
Failed    <<< semantic_point_annotator                    [ 28.5 seconds ]
_________________________________________________________________________________________________________________________________________________________________________
Errors     << pr2_navigation_self_filter:symlink /home/pr2nav/catkin_ws/logs/pr2_navigation_self_filter/build.symlink.000.log
Stage `symlink` failed with arguments:
  package: {'package_format': 1, 'exports': [], 'buildtool_export_depends': [], 'exec_depends': [<catkin_pkg.package.Dependency object at 0x3442590>, <catkin_pkg.package.Dependency object at 0x34425f0>, <catkin_pkg.package.Dependency object at 0x3442650>, <catkin_pkg.package.Dependency object at 0x34426b0>, <catkin_pkg.package.Dependency object at 0x3442710>, <catkin_pkg.package.Dependency object at 0x3442770>, <catkin_pkg.package.Dependency object at 0x34427d0>, <catkin_pkg.package.Dependency object at 0x3442830>, <catkin_pkg.package.Dependency object at 0x3442890>, <catkin_pkg.package.Dependency object at 0x34428f0>], 'name': 'pr2_navigation_self_filter', 'filename': '/home/pr2nav/catkin_ws/src/pr2_navigation/pr2_navigation_self_filter/package.xml', 'replaces': [], 'version_abi': None, 'build_export_depends': [<catkin_pkg.package.Dependency object at 0x3442950>, <catkin_pkg.package.Dependency object at 0x34429b0>, <catkin_pkg.package.Dependency object at 0x3442a10>, <catkin_pkg.package.Dependency object at 0x3442a70>, <catkin_pkg.package.Dependency object at 0x3442ad0>, <catkin_pkg.package.Dependency object at 0x3442b30>, <catkin_pkg.package.Dependency object at 0x3442b90>, <catkin_pkg.package.Dependency object at 0x3442bf0>, <catkin_pkg.package.Dependency object at 0x3442c50>, <catkin_pkg.package.Dependency object at 0x3442cb0>], 'doc_depends': [], 'test_depends': [], 'maintainers': [<catkin_pkg.package.Person object at 0x34439d0>], 'version': '0.1.27', 'build_depends': [<catkin_pkg.package.Dependency object at 0x3442d10>, <catkin_pkg.package.Dependency object at 0x3442d70>, <catkin_pkg.package.Dependency object at 0x3442dd0>, <catkin_pkg.package.Dependency object at 0x3442e30>, <catkin_pkg.package.Dependency object at 0x3442e90>, <catkin_pkg.package.Dependency object at 0x3442ef0>, <catkin_pkg.package.Dependency object at 0x3442f50>, <catkin_pkg.package.Dependency object at 0x3442fb0>, <catkin_pkg.package.Dependency object at 0x3444050>, <catkin_pkg.package.Dependency object at 0x34440b0>], 'urls': [<catkin_pkg.package.Url object at 0x3443cd0>], 'authors': [<catkin_pkg.package.Person object at 0x3443d50>], 'licenses': ['BSD'], 'buildtool_depends': [<catkin_pkg.package.Dependency object at 0x3444110>], 'conflicts': [], 'description': u"Filters the robot's body out of point clouds."}
  metadata_path: /home/pr2nav/catkin_ws/.catkin_tools/profiles/default
  devel_manifest_path: /home/pr2nav/catkin_ws/.catkin_tools/profiles/default/packages/pr2_navigation_self_filter
  prebuild: False
  package_path: pr2_navigation/pr2_navigation_self_filter
  dest_devel_path: /home/pr2nav/catkin_ws/devel
  source_devel_path: /home/pr2nav/catkin_ws/devel/.private/pr2_navigation_self_filter
Traceback (most recent call last):
  File "/home/pr2nav/.local/lib/python2.7/site-packages/catkin_tools/execution/executor.py", line 146, in async_job
    event_queue))
  File "/home/pr2nav/.local/lib/python2.7/site-packages/catkin_tools/verbs/catkin_build/cli.py", line 393, in main
    parallel_jobs = int(opts.parallel_jobs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x90 in position 40: ordinal not in range(128)

.........................................................................................................................................................................
Failed     << pr2_navigation_self_filter:symlink          [ Exited with code 3 ]
Failed    <<< pr2_navigation_self_filter                  [ 38.0 seconds ]
[build] Summary: 6 of 10 packages succeeded.
[build]   Ignored:   2 packages were skipped or are blacklisted.
[build]   Warnings:  9 packages succeeded with warnings.
[build]   Abandoned: 1 packages were abandoned.
[build]   Failed:    3 packages failed.
[build] Runtime: 38.5 seconds total.
[build] Note: Workspace packages have changed, please re-source setup files to use them.

[jbohren]

@TDiazT Thanks for the additional report, it looks like you're also having the unicode error, but you're also getting numerous symlink errors. Are you re-using a develspace built with another tool?

[jbohren]

@TDiazT could you re-run your build with catkin_tools from this branch: https://github.com/catkin/catkin_tools/tree/better-tracebacks ?

[jbohren]

I've got this in the test harness with #368. It's failing on Python2 and succeeding on Python3.

[jbohren]

This should now be fixed in #368. It'd be great for you guys to verify that on your systems if you have a chance.

[TDiazT]

Hi @jbohren ,
We exercised using the better-tracebacks version doing this:

1. Removed older catkin_tools and installed better-tracebacks version

2. Created a new workspace and initialized using catkin init

3. Downloaded the tested packages pr2_navigation doing this:

   $ wstool init src/
   $ wstool set pr2_navigation --git https://github.com/pr2/pr2_navigation -v 0.1.27 -t src
   $ wstool up pr2_navigation -t src

4. Then we built the package catkin build and everything went fine:

   Finished  <<< pr2_navigation_perception                   [ 14.9 seconds ]                                                                                                         
   [build] Summary: All 10 packages succeeded!                                                                                                                                        
   [build]   Ignored:   1 packages were skipped or are blacklisted.                                                                                                                   
   [build]   Warnings:  10 packages succeeded with warnings.                                                                                                                          
   [build]   Abandoned: None.                                                                                                                                                         
   [build]   Failed:    None.                                                                                                                                                         
   [build] Runtime: 48.9 seconds total.                                                                                                                                               
   [build] Note: Workspace packages have changed, please re-source setup files to use them. 

We suspect that the previous error was due to using a develspace built with the regular catkin.

Thanks a lot!

[jbohren]

Thanks for the feedback, @TDiazT.

[MatthiasNieuwenhuisen]

Hi,

we experienced the same problem. Two packages install a library with the same name and generating the md5 sum of the libraries fails because of the encoding error. We have tried with catkin_tools 0.4.2, the master branch and the better-tracebacks version. If we remove the encode('utf-8') function from line 272-273 in catkin_tools/jobs/catkin.py the collisions are correctly identified and the build process finishes. The workspace was cleanly initialized with

source /opt/ros/indigo/setup.bash
catkin init
catkin build

We have tested it with Python 2.7.6.

> locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

A toy example to reproduce with the attached .so:

>>> import os
>>> from md5 import md5
>>> source_file = 'liblibelas.so'
>>> md5(open(os.path.realpath(source_file)).read()).hexdigest()
'fd72bce4e872072e940df0d08b5ded74'
>>> md5(open(os.path.realpath(source_file)).read().encode('utf-8')).hexdigest()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x90 in position 24: ordinal not in range(128)

liblibelas.zip

Best,
Matthias

[jack-oquin]

Two packages install a library with the same name

That sounds like a bug in those packages. Any generated binary packages would have to specify "Conflicts" for each other.

[MatthiasNieuwenhuisen]

I agree that this should not happen, but to my understanding the purpose of the failing part of catkin.py is to detect and report those collisions. Thus, it is never executed if no conflicting libraries are to be symlinked to the devel space and should work in that exact situation. Please let me know if I'm wrong.

[jack-oquin]

I see. You want it to handle and report the bug better.

[alecive]

I'm also having this error:

Stage `symlink` failed with arguments:
  package: {'package_format': 2, 'exports': [], 'buildtool_export_depends': [], 'exec_depends': [<catkin_pkg.package.Dependency object at 0x7f3b409c0830>], 'name': 'baxter_collaboration', 'filename': '/home/baxter/ros_devel_ws/src/baxter_collaboration/package.xml', 'replaces': [], 'version_abi': None, 'build_export_depends': [<catkin_pkg.package.Dependency object at 0x7f3b40246f50>], 'doc_depends': [], 'test_depends': [], 'maintainers': [<catkin_pkg.package.Person object at 0x7f3b4070eb10>], 'version': '0.1.0', 'build_depends': [<catkin_pkg.package.Dependency object at 0x7f3b40246fb0>], 'urls': [], 'authors': [<catkin_pkg.package.Person object at 0x7f3b40749b10>], 'licenses': ['MIT'], 'buildtool_depends': [<catkin_pkg.package.Dependency object at 0x7f3b40733050>], 'conflicts': [], 'description': u'The baxter_collaboration package'}
  metadata_path: /home/baxter/ros_devel_ws/.catkin_tools/profiles/default
  devel_manifest_path: /home/baxter/ros_devel_ws/.catkin_tools/profiles/default/packages/baxter_collaboration
  prebuild: False
  package_path: baxter_collaboration
  dest_devel_path: /home/baxter/ros_devel_ws/devel
  source_devel_path: /home/baxter/ros_devel_ws/devel/.private/baxter_collaboration
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/executor.py", line 146, in async_job
    event_queue))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 24: ordinal not in range(128)

My issue was similar to @MatthiasNieuwenhuisen : I had two libraries that tried to create the same library target, and for this reason there was a collision. But catkin_make correctly gave me the error below, whereas catkin build was not so kind to tell me the true reason of the problem.

CMake Error at baxter_collaboration/lib/CMakeLists.txt:14 (add_library):
  add_library cannot create target "arm_controller" because another target
  with the same name already exists.  The existing target is a shared library
  created in source directory
  "/home/baxter/ros_devel_ws/src/baxter_tictactoe/lib".  See documentation
  for policy CMP0002 for more details.