Python rules for Bazel

[joshfischer1108]

No description provided.

[joshfischer1108]

Steps to completion: Not exhaustive

• Add native python rules to workspace
• Update docker rules because of a deprecated pip_import dependency
• Remove pex rules from the project
• Add requirements.txt to support all of the python dependencies for the project. This is only a first pass to loading python deps. After the initial migration is done we can lazily load python deps for improved build times, if desired
• Remove tools/rules/newgenproto.bzl file as it seems to be a duplicate.
• Swap out pex_library with native py_libray rules
• and pex_binary with native py_binary rules
• Investigate the use of starlark to replace python build scripts
• Investigate the use for py_proto_libaries from google https://thethoughtfulkoala.com/posts/2020/05/08/py-protobuf-bazel.html
• pex_binary has a number of rules which do not translate over to py_binary: resources, zip_safe. We need to find an alternative to these.

[nicknezis]

If we are removing all use of pex, we will need to ensure we can still Launch HeronPy topologies. The Heron Executor code currently decides which binary type based on file extension. The logic in this file will need to be updated based on these changes.

incubator-heron/heron/executor/src/python/heron_executor.py

Lines 788 to 795 in fab089c

	if self.pkg_type == 'jar' or self.pkg_type == 'tar':
	retval.update(self._get_java_instance_cmd(instance_info))
	elif self.pkg_type == 'pex':
	retval.update(self._get_python_instance_cmd(instance_info))
	elif self.pkg_type == 'so':
	retval.update(self._get_cpp_instance_cmd(instance_info))
	elif self.pkg_type == 'dylib':
	retval.update(self._get_cpp_instance_cmd(instance_info))

[surahman]

Draft removal of most references to pex_binary and pex_library.

[nwangtw]

This would be really cool~

[surahman]

There is no zip_safe and resources in py_binary. I have opted to disable zip_safe and placed the change in its own commit so it can be dropped if needed.

resources in bazel_pex_rules appears to have switched to calling this data:

List of labels; Optional; Default is []

Files to include as resources in the final pex binary.

Putting other rules here will cause the outputs of those rules to be embedded in this one. Files will be included as-is. Paths in the archive will be relative to the workspace root.

I think this may align with Bazel's definition of data, but it uses those files at rule runtime:

List of labels; optional

Files needed by this rule at runtime. May list file or rule targets. Generally allows any target.

[surahman]

I just reverted the commits on pylint and cpplint.

Edit: I am still learning Bazel and understanding the build scripts. I can not trigger the infinite loop that CI is experiencing on my local machines:

(00:13:40) ERROR: Cycle detected but could not be properly displayed due to an internal problem. 
Please file an issue. Raw display: topLevelKey: ConfiguredTargetKey{label=//heron/api/src/cpp:cxx-api, config=BuildConfigurationValue.Key[a5ba794ad3f6c1fd9f8a7f4293435609fb1ef8a34f21b35e3d8a289fcb873f5e]}

The error seems to be caused by an infinite recursion/expansion of the build rule cpplint. I think the culprit is main being set to cpplint because when I set it to main.py there is no infinite loop anymore.

[surahman]

Could this be causing an infinite recursion/expansion of the build rule?

[joshfischer1108]

I guess it could be possible. @nicknezis would have a better idea. He's been our go to Bazel person,

[surahman]

I think the best way to approach this enormous task is to completely remove a single PEX build rule at a time and make sure everything compiles and all tests pass. Start with the simpler build rules and move to the more complex ones. That way we can verify in a concrete fashion that the changes we are making are correct. It also means the work we are completing can be incrementally merged back into master so that there is progress.

Conceivably a Kanban board that tracks the rules to remove? We can clump some together for fewer PR's.

[joshfischer1108]

I like the idea of removing a single build rule at a time. After jumping into a lot this code I think we've learned a lot (a good time for growth 😄 ) Are you thinking we should abandon this branch and start fresh?

[surahman]

Are you thinking we should abandon this branch and start fresh?

We are thinking alike 😁

[joshfischer1108]

Closing to account for smaller chunks of work.