[BUG]: Unclear error messages from Py4j tables crawler

[nfx]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

workflow is failing between these lines
https://github.com/databrickslabs/ucx/blob/main/src/databricks/labs/ucx/hive_metastore/tables.py#L526-L545
something related to shared state of spark session.

InternalError: failed-table-crawl: listing databases -> catalog : An error occurred while calling o400.sharedState. Trace: py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.internal.SharedState 
org.apache.spark.sql.SparkSession.sharedState() is not whitelisted on class class org.apache.spark.sql.SparkSession at py4j.security.WhitelistingPy4JSecurityManager.checkCall(WhitelistingPy4JSecurityManager.java:473) at py4j.Gateway.invoke(Gateway.java:305) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750)

Expected Behavior

clear error message that assessment workflow has to be run before fetching the list of tables. technically, we could also run the assessment job from within TablesCrawler.

Steps To Reproduce

No response

Cloud

AWS

Operating System

macOS

Version

latest via Databricks CLI

Relevant log output

No response