source-hackernews #1656
source-hackernews #1656
Conversation
|
@danthelion is this something you're looking for a review on? |
danthelion
force-pushed
the
hn_capture
branch
from
14bc29f to
2641e60
Compare
danthelion
force-pushed
the
hn_capture
branch
from
2641e60 to
709f30a
Compare
|
@williamhbaker, a bit late but yes 😁 |
.idea/.gitignore
Outdated
| @@ -0,0 +1,8 @@ | |||
| # Default ignored files | |||
There was a problem hiding this comment.
Mind removing this .idea folder from the files to commit?
| async def spec(self, log: Logger, _: request.Spec) -> ConnectorSpec: | ||
| return ConnectorSpec( | ||
| configSchema=EndpointConfig.model_json_schema(), | ||
| documentationUrl="https://docs.estuary.dev", |
There was a problem hiding this comment.
FYI if you want to have actual docs for this connector, this is the URL to update for that.
source-hackernews/test.flow.yaml
Outdated
| import: | ||
| - acmeCo/flow.yaml | ||
| captures: | ||
| acmeCo/source-google-sheets: |
| @@ -0,0 +1,20 @@ | |||
| credentials: | |||
There was a problem hiding this comment.
I don't think this config file is for this capture, since it doesn't look like this capture requires any configuration at all.
There was a problem hiding this comment.
is the sops part required?
There was a problem hiding this comment.
If the configuration has fields that need to be encrypted, yeah. There's some instructions for how to do the encryption here. But if this capture doesn't require any kind of configuration, you could just delete the file.
|
|
||
| item = Item.model_validate_json(req) | ||
|
|
||
| # stop the backfill when we catch up |
There was a problem hiding this comment.
Since there's only a fetch_page defined, the connector will only backfill and will stop once item.time > log_cutoff, and then won't do anything else.
You might want to change this to be a fetch_changes function, that way it will continue to get records as new cursor values become available.
Also just noting that there are over 40MM total records and this is going to fetch them 1-by-1, so that is going to take an extremely long time to get to the present.
danthelion
force-pushed
the
hn_capture
branch
from
709f30a to
82edc8f
Compare
Description:
(Describe the high level scope of new or changed features)
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
(anything that might help someone review this PR)
This change is