ES Search Query Collect All Response

[noah-paige]

Feature or Bugfix

• Bugfix

Detail

• For catalog_indexer_task ensure we collect all hits from query response for with_deletes option
  • Up the Query Size to 1000 results (default is 10)
  • Add logic to continue querying to collect all hits if there are more than the query size limit (i.e. > 1000)

Relates

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

• Does this PR introduce or modify any input fields or queries - this includes
  fetching data from storage outside the application (e.g. a database, an S3 bucket)?
  • Is the input sanitized?
  • What precautions are you taking before deserializing the data you consume?
  • Is injection prevented by parametrizing queries?
  • Have you ensured no eval or similar functions are used?
• Does this PR introduce any functionality or component that requires authorization?
  • How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  • Are you logging failed auth attempts?
• Are you using or adding any cryptographic features?
  • Do you use a standard proven implementations?
  • Are the used keys controlled by the customer? Where are they stored?
• Are you introducing any new policies/roles/users?
  • Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[noah-paige]

TESTING - Tested both locally and in AWS for the following:

• Test startReindexTask successfully runs catalog_indexer_task when withDeletes=False
• Test startReindexTask successfully runs catalog_indexer_task when withDeletes=True
• Test startReindexTask successfully runs catalog_indexer_task when withDeletes=True and # of objects to delete is > 10 and QUERY_SIZE is 10 (i.e. multiple search calls required to collect all responses)
• UI button works to invoke startReindexCatalog as Admin with either withDeletes Switch T/F

[dlpzx]

Looks good. I just want to confirm the new way we are dealing with the response of search.
response = {'hits':{'hits':[{'_id:1, ...}...]}}
Before:

• we extract docs.get('hits', {}).get('hits', []) in the catalog indexer task from search
• we extract hits-hits in the FE from search - with the Catalog DataSearch props...
  After:
• we directly get the hits in the catalog indexer task from search_all
• we extract hits-hits in the FE from search - we do not change anything to not mess up with the Catalog view

[noah-paige]

Looks good. I just want to confirm the new way we are dealing with the response of search. response = {'hits':{'hits':[{'_id:1, ...}...]}} Before:

* we extract docs.get('hits', {}).get('hits', []) in the catalog indexer task from search

* we extract hits-hits in the FE from search - with the Catalog DataSearch props...
  After:

* we directly get the hits in the catalog indexer task from search_all

* we extract hits-hits in the FE from search - we do not change anything to not mess up with the Catalog view

Correct! The FE Component we use automatically handles all the pagination for us