Feature/22-switch-to-jdbc-driver (#23)

* Update name of ODBC driver

* Update ODBC manifest

* Add JDBC connector

* Add JDBC capabilities

* Describe installation of jdbc driver

* Adapt capabilities for jdbc driver

* Add fingerprint

* Configure tdvt

* Add data source tests

* Add results workbook

* Update data source

* Update test result workbook

* Configure string trim for jdbc

* Add test with bad password

* Skip string.split test: function not supported by Exasol

* Rename column zzz2 to zzz

* Remove trailing spaces when extracting parts of a date

* Remove duplicate space

* Add rationale for skipping the 'split' function tests

* Change test user name to be equal to odbc tests

* Skip test for time data type

TIME not supported by Exasol

* Rename tdvt folder to odbc

* Adapt hostname for jdbc tests

* Adapt odbc tests to run

* Unify test configuration for jdbc & odbc

* Fix odbc issues with spaces

Remove trimming capabilities that hide issue with trailing spaces in TO_CHAR()

* Configure max number of threads to 1

* Remvove argument for number of threads

This is now configured in the ini file

* Add comment about failing tests

* Ignore Eclipse config files

* Update instructions for building the dockerfile

* Update dependencies

* Improve formatting of readme

* Add HtmlUnit driver

* Use Exasol 7.1.1, improve error handling

* Improve error handling in shell script

* Cleanup

* Adapt configuration to Eclipse 2021-09

* Validate & package jdbc connector

* Improve pom setup

* Cleanup tests: remove unused ports

* Read properties file in Java code to avoid having to set environment
variables when running tests from the IDE.

* Upgrade testcontainers to fix uploading files to the container

user id 'xxx' is too big ( > 2097151 ) when using commons-compress 1.21
with large user ID

testcontainers/testcontainers-java#4384

* Adapt UI-Tests to new Connector name / Tableau version

* Run tests in headless mode

* Document ui-test setup

* Build JDBC url with fingerprint and certificate validation arguments

* Update fingerprint in connection files

* Add developer guide

* Move test instructions to developer guide

* Add connection dialog screenshot

* Configure Eclipse formatter to never join already wrapped lines

* Extract base class for JDBC & ODBC connector tests

* Add UI tests for JDBC connector

* Fix failing StaplesDataTest

See tableau/connector-plugin-sdk#871 for details.

* Add changelog, improve documentation

* #22 Implement review findings by @AnastasiiaSergienko

* Apply suggestions from code review

Co-authored-by: Anastasiia Sergienko <[email protected]>

* Implement review findings by @AnastasiiaSergienko

Co-authored-by: Anastasiia Sergienko <[email protected]>

.gitignore

@@ -12,4 +12,4 @@
 **/tdvt_results_workbook.twbx
 .idea
 *.iml
-*.env
+*.env

README.md

@@ -4,7 +4,7 @@
 
 # Overview
 
-[Tableau](https://www.tableau.com/) is a business intelligence tool. You can use Tableau to visualize  data in [Exasol](https://www.exasol.com).
+[Tableau](https://www.tableau.com/) is a business intelligence tool. You can use Tableau to visualize data in [Exasol](https://www.exasol.com).
 
 The Exasol Tableau Connector is an adapter, that maps Exasol's analytical capabilities to Tableau's external interface, allowing Tableau to use Exasol-specific features.
 
@@ -13,6 +13,7 @@ The Exasol Tableau Connector is an adapter, that maps Exasol's analytical capabi
 * Exasol-specific connection dialog
 * Maps Tableau functions to Exasol functions
 
+
 # Table of Contents
 
 ## Information for Users
@@ -21,11 +22,15 @@ The Exasol Tableau Connector is an adapter, that maps Exasol's analytical capabi
 * [Tableau in the Exasol Online Documentation](https://docs.exasol.com/connect_exasol/bi_tools/tableau.htm)
 * [White paper on Exasol and Tableau](https://www.exasol.com/resource/tableau-exasol-better-together/)
 
+## Information for Developers
+
+* [Developer Guide](doc/developer_guide/developer_guide.md)
+
 # Dependencies
 
 ## Runtime Dependencies
 
-To use the Exasol Tableau Connector, you need Tableau Version 2020.3 or later either as desktop or server variant.
+To use the Exasol Tableau Connector, you need Tableau Version 2021.3 or later either as desktop or server variant.
 
 ## Test Dependencies
 

doc/changes/changelog.md

@@ -1,3 +1,4 @@
 # Changes
 
+* [0.2.0](changes_0.2.0.md)
 * [0.1.0](changes_0.1.0.md)

doc/changes/changes_0.2.0.md

@@ -0,0 +1,13 @@
+# Exasol Tableau Connector 0.2.0, released 2021-10-??
+
+Code name: JDBC Connector
+
+## Summary
+
+Version 0.2.0 of the Tableau Connector adds a JDBC connector that offers an option for certificate validation and allows entering a certificate fingerprint.
+
+The new release also fixes an issue with the ODBC connector that caused all strings being trimmed by removing capabilities `CAP_ODBC_TRIM_NO_EMPTY_STRING` and `CAP_ODBC_TRIM_VARCHAR_PADDING`. We also fixed some failing TDVT tests for the ODBC connector and improved the documentation for developers.
+
+## Features
+
+* #22: Add JDBC connector

doc/developer_guide/developer_guide.md

@@ -0,0 +1,186 @@
+# Developer Guide
+
+This developer guide describes how to
+
+* Test the connectors manually
+* Run Tableau's TDVT test suite
+* Run UI tests for Tableau Server
+
+## Manually Testing Connectors 
+
+To manually test the connectors in Tableau Desktop without packaging, add the following arguments when starting Tableau Desktop:
+
+```bat
+"C:\Program Files\Tableau\Tableau 2021.3\bin\tableau.exe" -DConnectPluginsPath=%USERPROFILE%\git\tableau-connector\src -DLogLevel=Debug
+```
+
+* `-DConnectPluginsPath=path\to\tableau-connector\src`: Path to the `src` directory of this cloned repository.
+* `-DLogLevel=Debug`: enable log output of `logging.Log()` in JavaScript files.
+
+After starting Tableau Desktop, click `More...` in the left bar under "To a Server", then click `Exasol JDBC by Exasol AG` or `Exasol ODBC by Exasol AG` to open the database connection dialog for JDBC resp. ODBC.
+
+Restart Tableau after modifying any connector file to reload changes.
+
+## Packaging the Connectors
+
+This requires `python3-venv` to be installed.
+
+To package the JDBC and ODBC connectors, execute
+
+```sh
+cd tableau-server-GUI-tests
+./set_up_scripts/package_connector.sh
+```
+
+This validates the connectors and creates the connectors at
+
+```
+tableau-server-GUI-tests/target/exasol_jdbc.taco
+tableau-server-GUI-tests/target/exasol_odbc.taco
+```
+
+To use the connectors, copy them to `C:\Program Files\Tableau\Connectors`.
+
+As the connectors are not signed, you need to start Tableau Desktop with argument `-DDisableVerifyConnectorPluginSignature`.
+
+## Running TDVT Tests
+
+You can run TDVT tests under Windows and macOS. This guide describes the setup for Windows. The setup for macOS is similar.
+
+### Initial Setup
+
+* Create a new Exasol database running on port `8563`.
+* Prepare database schema by running [tools/load_tvdt_test_data.sql](../../tools/load_tvdt_test_data.sql).
+* Configure hostname of the Exasol database: Add an entry to `c:\Windows\System32\Drivers\etc\hosts`. Adapt the IP to your database:
+
+    ```
+    10.0.0.2    exasol.test.lan
+    ```
+
+* Install TDVT as described in the [TDVT documentation](https://tableau.github.io/connector-plugin-sdk/docs/tdvt#set-up).
+* Update the Exasol certificate fingerprint in the four `*.tds` files:
+  * [tdvt_jdbc/tds/cast_calcs.exasol_jdbc.tds](../../tdvt_jdbc/tds/cast_calcs.exasol_jdbc.tds)
+  * [tdvt_jdbc/tds/Staples.exasol_jdbc.tds](../../tdvt_jdbc/tds/Staples.exasol_jdbc.tds)
+  * [tdvt_odbc/tds/Staples.exasol_odbc.tds](../../tdvt_odbc/tds/Staples.exasol_odbc.tds)
+  * [tdvt_odbc/tds/cast_calcs.exasol_odbc.tds](../../tdvt_odbc/tds/cast_calcs.exasol_odbc.tds)
+
+* Update the path to `tabquerytool.exe` (e.g. `C:\Program Files\Tableau\Tableau 2021.3\bin\tabquerytool.exe`) in
+  * [tdvt_jdbc/config/tdvt/tdvt_override.ini](../../tdvt_jdbc/config/tdvt/tdvt_override.ini)
+  * [tdvt_odbc/config/tdvt/tdvt_override.ini](../../tdvt_odbc/config/tdvt/tdvt_override.ini)
+
+### Configure Test Suites
+
+You can configure the tests suites to run in files
+* [tdvt_jdbc/config/exasol_jdbc.ini](../../tdvt_jdbc/config/exasol_jdbc.ini)
+* [tdvt_odbc/config/exasol_odbc.ini](../../tdvt_odbc/config/exasol_odbc.ini)
+
+After modifying these files you need to re-generate the test suite by adding the `--generate` argument to the `tdvt.tdvt run` command.
+
+See the [manual](https://tableau.github.io/connector-plugin-sdk/docs/tdvt#ini-file-structure) for details about the available tests.
+
+### Run Tests
+
+* JDBC Connector:
+
+    ```bash
+    cd tdvt_jdbc
+    python -m tdvt.tdvt run exasol_jdbc --generate
+    python -m tdvt.tdvt run exasol_jdbc
+    ```
+
+* ODBC Connector:
+
+    ```bash
+    cd tdvt_odbc
+    python -m tdvt.tdvt run exasol_odbc --generate
+    python -m tdvt.tdvt run exasol_odbc
+    ```
+
+See the [manual](https://tableau.github.io/connector-plugin-sdk/docs/tdvt#about-running-tdvt) for details.
+
+### Viewing Test Results
+
+TDVT writes the test result to `test_results_combined.csv`. You can view it by opening the `TDVT Results.twb` workbook with Tableau Desktop. After re-running the tests type `F5` to refresh the results. See [detailed instructions](https://tableau.github.io/connector-plugin-sdk/docs/tdvt#review-results).
+
+### Troubleshooting
+
+Log files of Tableau Desktop: `%USERPROFILE%\Documents\My Tableau Repository\Logs\`:
+* `log.txt`
+* `jprotocolserver.log`
+
+Also see the [FAQ and troubleshooting section of the manual](https://tableau.github.io/connector-plugin-sdk/docs/tdvt#frequently-found-issues-and-troubleshooting).
+
+## Tableau Server UI Tests
+
+### Setup
+
+1. Follow the [instructions](https://help.tableau.com/current/server-linux/en-us/server-in-container_setup-tool.htm) to create a container. This should result in an image `tableau_server_image:<version>`, e.g. `tableau_server_image:20213.21.0917.1006`.
+2. Update `tableau-server-GUI-tests/set_up_scripts/Dockerfile`:
+   * Use the same image created before in the `FROM` line, e.g. `tableau_server_image:20213.21.0917.1006`.
+   * Update the download urls to the latest Exasol release if available.
+3. Update constant `DEFAULT_DOCKER_DB_REFERENCE` in class `TableauServerGUITestIT` to the latest Exasol release.
+4. Build a new container including the Exasol ODBC and JDBC drivers by running
+
+    ```bash
+    cd tableau-server-GUI-tests/set_up_scripts
+    docker build . --tag tablau_server_with_exasol_drivers
+    ```
+
+### Run the Tests
+
+To run the tests you need to create the `tableau-server-GUI-tests/src/test/resources/credentials.properties` file with the following content (replace the placeholders for real values):
+
+```properties
+TABLEAU_USERNAME=<tableau-server-username>
+TABLEAU_PASSWORD=<tableau-server-password>
+TABLEAU_LICENSE_KEY=<tableau-server-license-key>
+```
+
+Start the tests with
+
+```sh
+cd tableau-server-GUI-tests
+mvn integration-test
+```
+
+### Run the Tests Remotely
+
+Startup of the Tableau Server takes a long time. We recommend using an AWS instance running "Amazon Linux 2" with 16 CPUs, e.g. `c5a.4xlarge`. With this instance type, startup will take about 15 to 20 minutes instead of 30 minutes.
+
+You can either run the tests completely on the remote machine or start them on your local machine.
+
+#### Start tests on remote machine
+
+This assumes you are using an AWS instance running Amazon Linux 2.
+
+1. Setup the instance with a UI as described [here](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-2-install-gui/).
+2. Install a Chrome browser as this will be used for running the UI tests.
+3. Install Java 11 (`yum install java-11-amazon-corretto`) and the latest Maven version.
+4. Start the VNC server on the instance with `vncserver -geometry 1920x1080`.
+5. Tunnel the VNC server port via `ssh <host> -L 5901:localhost:5901`.
+6. Start a local VNC client and connect to `localhost:5901`.
+7. In the VNC session make sure you can start the `chrome` browser.
+8. Run the tests via `mvn integration-test`.
+
+#### Start tests on local machine
+
+If you want to trigger the tests from your local IDE, you need to forward the required ports and configure your local environment:
+
+```sh
+ssh ec2-user@<remote machine address> -i <ssh-key> -L 2375:localhost:2375 -L 33445:localhost:33445 -L 33446:localhost:33446
+```
+
+Specify the host where the docker daemon is running and disable the Ryuk container:
+
+```sh
+export DOCKER_HOST=tcp://localhost:2375
+export TESTCONTAINERS_RYUK_DISABLED=true
+```
+
+### Troubleshooting
+
+#### Tableau Server Container Startup Fails
+
+When the Tableau server container stops after some minutes, you can start the container with environment variable `TSM_ONLY=1`, attach to the container and start the server with `"${DOCKER_CONFIG}"/config/tsm-commands`. See [detailed instructions](https://help.tableau.com/current/server-linux/en-gb/server-in-container_troubleshoot.htm).
+
+One possible root cause is an invalid license key. To check if the license is valid, run `tsm licenses list` in the container. You can try to get a trial license by running `tsm licenses activate --trial`.

doc/user_guide/user_guide.md

@@ -1,11 +1,24 @@
 # User Guide
 
 The Exasol Tableau Connector is distributed together with Tableau Desktop and Tableau Server applications. We recommend using the latest available version of Tableau products to access the connector.
+
+![JDBC connection dialog](../images/jdbc_connection_dialog.png "JDBC connection dialog")
+
 If you want to use the currently developed version of connector, you can follow the guide below and install the connector disabling sign verification.
 
 ## Testing In-Development Connector
 
-## With Tableau Desktop
+### Install the JDBC Driver
+
+* [Download the JDBC driver](https://www.exasol.com/portal/display/DOWNLOAD/)
+* Copy the driver to the Tableau installation directory:
+  * Windows: `C:\Program Files\Tableau\Drivers`
+  * Mac: `~/Library/Tableau/Drivers`
+  * Linux: `/opt/tableau/tableau_driver/jdbc`
+
+**Important note for Windows:** Make sure to download and install file `EXASOL_JDBC-<version>.msi`. The JDBC driver will be installed to `C:\Program Files\Exasol\EXASolution-7.1\JDBC\exajdbc.jar`. Only the JDBC driver for Windows supports Kerberos under Windows.
+
+### With Tableau Desktop
 
 * Clone the Tableau-Connector repository. You can execute the following command from [Git Bash](https://gitforwindows.org/) terminal:
 
@@ -16,16 +29,16 @@ If you want to use the currently developed version of connector, you can follow
 * Open Windows cmd and start Tableau Desktop providing the path to the connector directory and also disabling the signature check.
 
     ```shell
-    "<path to tableau.exe file>" -DConnectPluginsPath=<path to connector> -DDisableVerifyConnectorPluginSignature
+    "<path to tableau.exe file>" -DConnectPluginsPath=<path to connector>
     ```
 
     Example:
 
     ```shell
-    "C:\Program Files\Tableau\Tableau <version>\bin\tableau.exe" -DConnectPluginsPath=C:\Users\user\git\tableau-connector\src -DDisableVerifyConnectorPluginSignature
+    "C:\Program Files\Tableau\Tableau <version>\bin\tableau.exe" -DConnectPluginsPath=C:\Users\user\git\tableau-connector\src
     ```
 
-## With Tableau Server
+### With Tableau Server
 
 * Clone the Tableau-Connector repository:
 

src/exasol_jdbc/connectionBuilder.js

@@ -0,0 +1,27 @@
+(function dsbuilder(attr) {
+
+    const authentication = attr[connectionHelper.attributeAuthentication];
+    const fingerprint = attr["v-fingerprint"];
+    const validateServerCertificate = attr["v-validateservercertificate"];
+
+    logging.Log("connectionBuilder.js: input args: authentication='" + authentication
+        + "', fingerprint='" + fingerprint
+        + "', validateServerCertificate='"+ validateServerCertificate+"'");
+
+    var fingerprintArg = "";
+    if(fingerprint && fingerprint.trim().length > 0) {
+        fingerprintArg = "/" + fingerprint.trim();
+    }
+
+    // See https://docs.exasol.com/connect_exasol/drivers/jdbc.htm
+    const url = "jdbc:exa:"
+        + attr[connectionHelper.attributeServer]
+        + fingerprintArg
+        + ":"
+        + attr[connectionHelper.attributePort]
+        + ";validateservercertificate=" + validateServerCertificate
+        + ";feedbackinterval=1"
+        + ";clientname=TableauDesktop";
+
+    return [url];
+})

src/exasol_jdbc/connectionFields.xml

@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<connection-fields>
+  <field name="server" label="Server" value-type="string" optional="false" category="endpoint" />
+
+  <field name="port" label="Port" value-type="string" optional="false" category="endpoint" default-value="8563" />
+
+  <field name="authentication" label="Authentication" category="authentication" value-type="string" editable="false" default-value="auth-user-pass" />
+
+  <field name="username" label="Username" value-type="string" optional="false" category="authentication" />
+
+  <field name="password" label="Password" value-type="string" optional="false" category="authentication" secure="true" />
+
+  <field name="v-fingerprint" label="Server Certificate Fingerprint" value-type="string" optional="true" category="general" />
+
+  <field name="v-validateservercertificate" label="Validate Server Certificate" value-type="boolean" optional="false" category="general" default-value="1">
+    <boolean-options>
+      <false-value value="0" />
+      <true-value value="1" />
+    </boolean-options>
+  </field>
+</connection-fields>

src/exasol_jdbc/connectionMetadata.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<connection-metadata>
+  <database enabled='true' label='Database'>
+    <field optional='true' default-value='' />
+  </database>
+  <schema enabled='true' label="Schema" />
+  <table enabled='true' label="Table" />
+</connection-metadata>

src/exasol_jdbc/connectionProperties.js

@@ -0,0 +1,7 @@
+(function propertiesbuilder(attr) {
+    logging.Log("Running connectionProperties.js")
+    var props = {};
+    props["user"] = attr[connectionHelper.attributeUsername];
+    props["password"] = attr[connectionHelper.attributePassword];
+    return props;
+})

src/exasol_jdbc/connectionResolver.tdr

@@ -0,0 +1,27 @@
+<?xml version='1.0' encoding='utf-8' ?>
+<tdr class='exasol_jdbc'>
+	<connection-resolver>
+    <connection-builder>
+      <script file='connectionBuilder.js'/>
+    </connection-builder>
+    <connection-normalizer>
+      <required-attributes>
+        <setImpersonateAttributes/>
+        <attribute-list>
+          <attr>server</attr>
+          <attr>port</attr>
+          <attr>dbname</attr>
+          <attr>schema</attr>
+          <attr>authentication</attr>
+          <attr>username</attr>
+          <attr>password</attr>
+          <attr>v-fingerprint</attr>
+          <attr>v-validateservercertificate</attr>
+        </attribute-list>
+      </required-attributes>
+    </connection-normalizer>
+    <connection-properties>
+      <script file='connectionProperties.js'/>
+    </connection-properties>
+	</connection-resolver>
+</tdr>