[Python API] Reusing connection with islice() limit

[alex-mirkin]

When using python api to compare a list of tables, I'm making the connection outside the loop and then reusing it for all the tables I need to compare.
But after getting the first x isliceed rows from diffs, the differ continues to check the rest of the table and doesn't stop until the whole table is compared or the connection is closed.
This is problematic for huge tables or tables with many diffs.

Is this an expected behavior?
Should I create and close a connection for each table?

Simplified version of the code:

# create db connections
mysql_conn = connect(mysql_conn_details, thread_count=MAX_THREADS)
snowflake_conn = connect(snowflake_conn_details)

tables = ['foo.table1', 'bar.table2']

# iterate over tables to compare
for table in tables:
    # create a connection to the table
    table_seg_mysql = TableSegment(database, table_path, key_columns, extra_columns)
    table_seg_snowflake = TableSegment(database, table_path, key_columns, extra_columns)

    diffs = diff_tables(table_seg_mysql, table_seg_snowflake)

    # limit the number of diffs to check
    iterator = islice(diffs, 10)

    # check the diffs
    for sign, row in iterator:
        print(sign, row)

# close connections
mysql_conn.close()
snowflake_conn.close()

[erezsh]

I think the scheduling is supposed to be lazy with a lookahead. If it is eager, then maybe it's a bug. I will look into it.

[alex-mirkin]

I've tested and the diffs.close() works as expected.
Looks good!

Adding the with support will be a nice bonus.

[erezsh]

Added support for with in PR #39.

[alex-mirkin]

I'm seeing a strange behavior after the lazy commit in a specific scenario:
When comparing two tables in a loop and the first table is really huge one - after the diff finished all the tables, the program will not exit - because of stuck threads.

The number of stuck threads = max_threadpool_size

Anything I can check to see why it happens?

[erezsh]

Thanks for letting me know. Most likely it is caused by this commit: 0afbe65

I tried to fix it in this PR: #40

Could you give it a try and let me know if it solved the problem for you?

If not, I will revert the entire commit, and that should fix it for sure (if I'm not wrong).

[alex-mirkin]

I can confirm that PR: #40 fixes the issue for this test case.
Thanks!