Chore: Add ability to retry ingestion up to 3 times #1089
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
StrongMonkey
force-pushed
the
add-file-ingestion-retry
branch
from
35eb6fb to
5a229d1
Compare
Signed-off-by: Daishan Peng <[email protected]>
StrongMonkey
force-pushed
the
add-file-ingestion-retry
branch
from
5a229d1 to
657be9d
Compare
StrongMonkey
commented
Jan 2, 2025
| @@ -258,7 +258,7 @@ func (k *Handler) Sync(req router.Request, _ router.Response) error { | |||
|
|
|||
| invokeOpts := invoke.SystemTaskOptions{ | |||
| CredentialContextIDs: []string{credentialContextID}, | |||
| Timeout: 1 * time.Hour, | |||
| Timeout: 2 * time.Hour, | |||
There was a problem hiding this comment.
Increase the time since we want to increase the number of pages our scraper can scrape and sometimes it can take up more than 1 hour to finish.
StrongMonkey
commented
Jan 2, 2025
| @@ -61,7 +61,7 @@ type Config struct { | |||
| HelperModel string `usage:"The model used to generate names and descriptions" default:"gpt-4o-mini"` | |||
| AWSKMSKeyARN string `usage:"The ARN of the AWS KMS key to use for encrypting credential storage" env:"OBOT_AWS_KMS_KEY_ARN" name:"aws-kms-key-arn"` | |||
| EncryptionConfigFile string `usage:"The path to the encryption configuration file" default:"./encryption.yaml"` | |||
| KnowledgeSetIngestionLimit int `usage:"The maximum number of files to ingest into a knowledge set" default:"1000" env:"OBOT_KNOWLEDGESET_INGESTION_LIMIT" name:"knowledge-set-ingestion-limit"` | |||
| KnowledgeSetIngestionLimit int `usage:"The maximum number of files to ingest into a knowledge set" default:"3000" env:"OBOT_KNOWLEDGESET_INGESTION_LIMIT" name:"knowledge-set-ingestion-limit"` | |||
There was a problem hiding this comment.
Increasing the number of files one agent can ingest, since we should be able to ingest 3000 files per agent without any error.
Note: we haven't really systematically tested whether increasing the number of ingested file will deteriorate the retrival result. This is just the success rate of ingestion.
StrongMonkey
requested review from
thedadams,
njhale,
iwilltry42 and
g-linville
g-linville
approved these changes
Jan 3, 2025
iwilltry42
approved these changes
Jan 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add ability to re-enqueue the file ingestion process up to 3 times when it fails.
#1071