source-oracle-batch: new connector #2387
Conversation
mdibaiee
force-pushed
the
mahdi/oracle-batch-connector
branch
5 times, most recently
from
7f33f9f to
d32a10d
Compare
mdibaiee
force-pushed
the
mahdi/oracle-batch-connector
branch
from
d32a10d to
38d4f84
Compare
| Id string `json:"ID"` | ||
| } | ||
|
|
||
| func TestCaptureCheckpointSCN(t *testing.T) { |
There was a problem hiding this comment.
I tested this behavior by setting scnCursor=false and running the test and seeing it fail, while with scnCursor=true it works 👍🏽
There was a problem hiding this comment.
LGTM
The batch SQL captures are kind of a moving target at the moment, I'm in the process of doing some code health overhauls (see #2400 and #2406) and once those are complete I intend to add some new features like (optional) view discovery and the CDK-style full-refresh behavior with consistent keys and inferred deletions.
I think this implementation is good to merge as-is, just noting that there's a lot of stuff in the works that we'll need to apply to this connector as well.
|
|
||
| RUN go install -v ./go/... | ||
| RUN go install -v ./source-boilerplate/... | ||
| RUN go install -v github.com/jackc/pgx/v5 |
There was a problem hiding this comment.
Probably don't need PGX here?
| Connect func(ctx context.Context, cfg *Config) (*sql.DB, error) | ||
| TranslateValue func(val any, databaseTypeName string) (any, error) | ||
| GenerateResource func(resourceName, schemaName, tableName, tableType string) (*Resource, error) | ||
| DefaultQueryTemplate string |
There was a problem hiding this comment.
I've actually got a change out for review today (e931ede) which replaces this with a SelectQueryTemplate function.
The rationale here is that we've been talking about adding (probably optional) view discovery to the batch SQL captures, but of course views don't have any sort of XMIN / ROWSCN property for us to use so we'd probably want to make full-refresh and cursor-incremental capture behaviors work properly by default (that is, without forcing the user to input a query template).
Description:
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
(anything that might help someone review this PR)
This change is