Weekly Status Log: Query Interfaces

[domoritz]

This is for uwdata#483, uwdata#510, and uwdata#506. Code is in https://github.com/cmudig/mosaic/tree/prepare.

[dorianpan]

September 20, 2024

– should try a different way of extracting the params (visitor pattern as a ref) different from the one in toString()
– server implementation (python first)
– possibly try to refactor toString() into toSql() and have two versions for prepared and non-prepared

Goal next week:
– layout implementation plan for adding the prepare statement
– have an initial implementation based on the plan

[dorianpan]

September 25th, 2024

Discussion today
– rename toString() function
– safety issue with current implementation -- ensure the approach is robust
– use a helper function probably?
– better logic for adding “?”
https://docs.google.com/document/d/1RiO9qCwJS333HMGcDEAKb7xV80oZwGfYoFzIyARqeZg/edit

[dorianpan]

September 27th, 2024

Discussion today:
9.27 meeting
— front end sends the generic string & param list (still buggy but should be done soon)
— server almost the same as the old implementation (already passed the test)
— Dorian’s next step:

• Make sure the toSql() function works
• Test it with current backend implementation
  — Plan to be completed (doug)
• Node connector
• Update json format in data-server.js
• Add prepared config param
• arrowBuffer (DuckDBStatement): this.statement instead of this.con?
• Add tests
• cache: update cache to include prepared params, etc, update calls to cacheKey
• prepare(sql): add map to check for prepared
• What is bundle?
• Python connector
• What are handlers doing? (what is app?)

[domoritz]

What is bundle?

Bundles are prepared sets of queries that can be loaded into the cache. There is a demo app in https://github.com/uwdata/mosaic/blob/main/dev/bundle.html.

[dorianpan]

October 2, 2024

Douglas: updated the node server code to accommodate for the update on the sql builder; started python server update

-- todo: update python server code to prepare query in the sample script format

Dorian: the toSql function is able to return an object containing the param array and the query string with ? marks

-- todo: need to support both prepared and non-prepared versions by having an indicator variable as an input

Next step: test the implementation together & make changes if anything doesn't work

[dorianpan]

October 4, 2024

Pushed all changes to server and sql builder to remote branch prepare
Modified test cases for Python server so now able to run test locally

Pending: need to find a way to connect to the python server and run test with both sql builder and the python server; maybe write a script that make requests to localhost?

[dorianpan]

October 9, 2024

meeting updates
Douglas -- updated python and node server so now they are able to handle prepared statements

Dorian -- implemented the query generator to support generating both prepared and non-prepared query strings (buggy now needs a closer look)

goals next week:
-- figure out if prepared statement is per session
-- able to enable & disable prepared statement (mostly client side)
-- create unit tests and start to think about how to modify the interface so there will be an option to select whether the statement is prepared or not

[dorianpan]

October 23, 2024

meeting updates

Douglas:
-- added prepared statement support for bundles
-- added more test cases
-- updated query manager so that it's able to disable/enable prepared statement

Dorian:
-- rewrote the SQL builder to implement the visitor pattern logic to generate query strings
-- renamed the toString() method of the Query class to toSql() and modified it to accept visitors as inputs
-- added test cases for the updated Query class

goals next week:

Douglas:
-- figure out prepared logic: per connection / per query
-- look into rust server and make updates
-- start integration with the updated toSQL() function

Dorian:
-- add more test cases for the updated toSQL() function
-- change all other files that uses Query.toString() to Query.toSQL() (might include implicit toString calls)
-- refine the visitor logic if needed

[domoritz]

Relevant comment: uwdata#567 (comment)

[dorianpan]

November 6, 2024

Updates

Douglas:
looked into rust server, questions related to built-in cache for the rust server

merged current work with Dorian's branch, backend support for prepared statements ready for node.js server & python server

Dorian:
updated toSQL() function according to the comments from last meeting

updated code in core package related to the implicit use of toString; replaced them with non-prepared toSQL() function for now, passed all unit tests

merged code with Douglas' branch

Next step:

Douglas:
implement backend support for rust

Dorian:
implement prepared version of toSQL() with updated backend support

write unit tests for prepared statements

Both:
address any remaining merge conflicts or obsolete code caused by updated return type of toSQL()

[dorianpan]

November 12th, 2024

Updates: added several fixes and passed all unit testing with prepared toSQL function. Passed client test with nodeconnector.js

Next step:
Dorian:
test our implementation with the actual server using the browser interface next since we are only testing with the nodeconnector now which isn't used for production; update query consolidator to accommodate for the updated data structure

Douglas:
Implement the rust server and work on caching for the demo server