Use new Search query syntax when calling api #46481

Kicu · 2024-07-30T10:22:39Z

Details

Use the new query syntax when calling /search api endpoint
all the base values and filter values are being send as stringified json AST query format, and filters are now expected to work
Bugfix: I noticed that our sorting broke slightly after new query syntax ([Search v2] [App] Use new query syntax #45617) because when clicking on column name we rebuilt the query, but ignored the filters
since this PR only modifies fetching logic I provided only video for web; no UI or styling was affected

NOTE: there is bug on backend right now that only returns first page of results and ignores offset. @luacmartins is aware and PR fixing this is in progress. When reviewing please temporarily ignore that backend returns offset: 0.

Fixed Issues

$ #46684
PROPOSAL:

Tests

go to /search/filters and choose some filters, click ViewResults
verify that when /search request is sent, payload has an object with all the filter values inside
verify that search results are correctly returned and displayed
(temporary) ignore that lazy loading/pagination doesn't work as this is being fixed on backend
click on column headers and test that sorting works and correct params are sent inside query when calling api

Offline tests

QA Steps

go to /search/filters and choose some filters, click ViewResults
verify that when /search request is sent, payload has an object with all the filter values inside
verify that search results are correctly returned and displayed
verify that sorting works

PR Author Checklist

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

rec-web-query-api.mp4

MacOS: Desktop

Kicu · 2024-07-30T10:31:43Z

@luacmartins this is just a draft but there is 1 issue with offset that I need your help with.
Question: what is the proper way to send offset to api now, that we moved offset out of the parser.

Problems right now:

If I send offset wrapped inside the AST query then backend seems to ignore it - that is I always get first page.

BUT if I instead send it on top level backend call - I get an endless Search loop, because regardless of offset value api always returns hasMoreResults: true and app keeps refetching data.

rec-offset.mp4

Take a look at this on your end and tell us what is the preferred way to send offset.

luacmartins · 2024-07-30T15:12:18Z

@Kicu the way you described in 1 is the correct way to send it. I noticed that we have a bug in the backend that's not correctly parsing the new offset input. I'll work on a fix for that.

luacmartins · 2024-07-30T15:33:43Z

I have a PR up that fixes the bug. I'll let you know once that's deployed and fixed. For now, please continue with option 1.

melvin-bot · 2024-07-31T10:10:39Z

@rayane-djouah Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

Kicu · 2024-07-31T10:18:17Z

super simple and ready for review

rayane-djouah · 2024-07-31T10:48:18Z

Reviewer Checklist

I have verified the author checklist is complete (all boxes are checked off).
I verified the correct issue is linked in the ### Fixed Issues section above
I verified testing steps are clear and they cover the changes made in this PR
- I verified the steps for local testing are in the Tests section
- I verified the steps for Staging and/or Production testing are in the QA steps section
- I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
- I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
I checked that screenshots or videos are included for tests on all platforms
I included screenshots or videos for tests on all platforms
I verified tests pass on all platforms & I tested again on:
- Android: Native
- Android: mWeb Chrome
- iOS: Native
- iOS: mWeb Safari
- MacOS: Chrome / Safari
- MacOS: Desktop
If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
I verified proper code patterns were followed (see Reviewing the code)
- I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
- I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
- I verified that comments were added to code that is not self explanatory
- I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
- I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
- I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
- I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
- I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
- I verified the JSDocs style guidelines (in STYLE.md) were followed
If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
I verified that this PR follows the guidelines as stated in the Review Guidelines
I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
If a new component is created I verified that:
- A similar component doesn't exist in the codebase
- All props are defined accurately and each prop has a /** comment above it */
- The file is named correctly
- The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
- The only data being stored in the state is data necessary for rendering and nothing else
- For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
- Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
- All JSX used for rendering exists in the render method
- The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions
If any new file was added I verified that:
- The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory
If a new CSS style is added I verified that:
- A similar style doesn't already exist
- The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG)
If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
- I verified that all the inputs inside a form are aligned with each other.
- I added Design label and/or tagged @Expensify/design so the design team can review the changes.
If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

Screen.Recording.2024-08-01.at.12.16.07.PM.mov

Android: mWeb Chrome

Screen.Recording.2024-08-01.at.12.11.27.PM.mov

iOS: Native

Simulator.Screen.Recording.-.iPhone.15.Pro.Max.-.2024-08-01.at.12.01.44.mp4

iOS: mWeb Safari

Simulator.Screen.Recording.-.iPhone.15.Pro.Max.-.2024-08-01.at.11.55.33.mp4

MacOS: Chrome / Safari

Screen.Recording.2024-08-01.at.11.48.21.AM.mov

MacOS: Desktop

Screen.Recording.2024-08-01.at.11.55.45.AM.mov

rayane-djouah · 2024-07-31T11:41:29Z

Bug (Regression): Expenses are not grouped into IOU reports (in "Shared", "Drafts", and "Finished" tabs), and some expenses are missing (this is already known based on the above). There are differences between the staging and development versions. Please look at the screenshots below for both versions:

1- "Shared" Results:

Staging: View Screenshot
Development: View Screenshot

2- "Drafts" Results:

Staging: View Screenshot
Dev: View Screenshot

3- "Finished" Results:

Staging: View Screenshot
Dev: View Screenshot

luacmartins · 2024-07-31T12:36:20Z

Taking a look to see if my backend PR will solve the issue above

luacmartins · 2024-07-31T13:17:51Z

Ok, I see the issue. The backend is returning the wrong type. I put up a PR with a temporary fix until we convert the frontend to use expense type, instead of transaction and report

luacmartins · 2024-07-31T13:18:11Z

@rayane-djouah for now, we can continue review assuming the issue is resolved.

rayane-djouah · 2024-07-31T13:50:01Z

@Kicu - Do you know why the search API request runs twice (with the same payload) every time I open the page or filter the expenses?

luacmartins · 2024-07-31T14:02:49Z

I've noticed that too and it has been an issue on main. I'll create an issue to fix that

luacmartins · 2024-07-31T14:04:36Z

Created issue here for the duplicate requests

Kicu · 2024-07-31T14:11:01Z

isn't the double call simply because of StrictMode and it's expected? and it will never happen in prod? I think we can leave it be.
Usually double calls like this one could be a sign of some possible problems with React, but I think since its only initial call for us we can just ignore it. This is what StrictMode is supposed to be doing

rayane-djouah · 2024-07-31T14:16:57Z

isn't the double call simply because of StrictMode and it's expected? and it will never happen in prod? I think we can leave it be. Usually double calls like could be a sign of some possible problems with React, but I think since its only initial call for us we can just ignore it

Oh, that makes sense. I forgot about Strict Mode! 😄

I just tested it with Strict Mode disabled, and the duplicate requests is not happening 👍

Kicu · 2024-07-31T14:18:35Z

Oh and btw I've found a bug where our logic displaying sorting no longer makes sense and Im fixing it. Will push some new code in like ~15minutes, and then you can re-check.
I had to refactor a bunch of functions that build query strings but I think it makes for a much simpler code now

Kicu · 2024-07-31T14:40:15Z

@rayane-djouah @luacmartins some context for newest commit:

when we click on header we called the onSort function which now has to rebuild the whole query (after new syntax was introduced) and change the value of sortBy and/or sortOrder.
however we had no advanced filters when @adamgrzybowski was working on this, hence his function: buildSearchQueryString completely ignored filters, which means when sorting we were cutting out any filters from URL
I fixed this function and also aligned a few other functions so that we try to use the queryJSON object most of the time

Hope that makes it clear. All this parsing logic is still a bit complex 😅 but I think we will simplify it further, we're just learning which functions we really need in the code

luacmartins · 2024-07-31T14:41:35Z

Nice. Thanks for catching that! It seems like TS checks are failing though

Kicu · 2024-07-31T14:45:55Z

src/components/Search/index.tsx

@@ -170,15 +168,10 @@ function Search({queryJSON, policyIDs, isCustomQuery}: SearchProps) {
    const sortedData = SearchUtils.getSortedSections(type, data, sortBy, sortOrder);

    const onSortPress = (column: SearchColumnType, order: SortOrder) => {
-        const currentSearchParams = SearchUtils.getCurrentSearchParams();
-        const currentQueryJSON = SearchUtils.buildSearchQueryJSON(currentSearchParams.q, policyIDs);


Note: I removed this because it was super redundant. We already have the parsed queryJSON object in this component's props. Instead of using it we were getting the params and parsing them once again which didn't add anything to this function.

Kicu · 2024-07-31T14:51:19Z

TS fixed

rayane-djouah · 2024-07-31T15:03:05Z

@Kicu @luacmartins The filtering by policyIDs is currently broken. Should we leave it as is and address it in #46592, or should we implement a temporary fix here?

luacmartins · 2024-07-31T15:49:23Z

That's a regression, we could keep sending the policyID as a temporary solution. The real fix will be implemented as part of #46592

Kicu · 2024-08-01T10:13:47Z

@rayane-djouah @luacmartins I pushed a fix and policyIDs are now being sent the old way and the filtering seems to be working.

rec-web-policyids.mp4

rayane-djouah · 2024-08-01T10:29:00Z

@Kicu - Did you push your latest changes? I see that we're still not sending the policyIDs

Kicu · 2024-08-01T10:30:37Z

@rayane-djouah sorry 😅 Indeed forgot to push, its ok now

Kicu · 2024-08-01T10:31:56Z

src/components/Search/index.tsx

@@ -134,19 +137,6 @@ function Search({queryJSON, policyIDs, isCustomQuery}: SearchProps) {
        setDeleteExpensesConfirmModalVisible(true);
    };

-    useEffect(() => {


this was a duplicate of useEffect in line 96 - probably someone made a mistake when refactoring

rayane-djouah

LGTM and tests well apart from this bug

luacmartins

LGTM! The remaining bug will be fixed after the next API deploy today.

rayane-djouah · 2024-08-01T13:21:37Z

@luacmartins - Could you please create a payment issue for this PR? Thank you!
(the linked issue #45026 is for PR #45407)
Process: https://expensify.slack.com/archives/C02NK2DQWUX/p1721843458370249?thread_ts=1721843446.947239&cid=C02NK2DQWUX

OSBotify · 2024-08-01T13:31:16Z

✋ This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

luacmartins · 2024-08-01T19:16:30Z

Created here

OSBotify · 2024-08-02T01:42:41Z

🚀 Deployed to staging by https://github.com/luacmartins in version: 9.0.16-0 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

rayane-djouah · 2024-08-02T10:59:04Z

The bug is fixed after the API deploy 🚀

dangrous · 2024-08-02T15:10:22Z

oh hai! We've got this deploy blocker - #46720 - which seems very related (sort order doesn't actually change anything); I just tried the behavior on the production branch and it still is incorrect, so I think it's either a backend issue or a regression that wasn't caught before. Butttttt feels like you all might have some insight into a fix! @luacmartins do you think it might be backend? Did we merge something recently?

OSBotify · 2024-08-06T04:02:52Z

🚀 Deployed to production by https://github.com/marcaaron in version: 9.0.16-8 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

send JSON ast query to Search api

66834ca

This was referenced Jul 30, 2024

[Search v2.1][Auth] Deprecate old syntax code #45030

Closed

[PAID][Search v2.1][Web-E] Deprecate old Search params #45029

Closed

Kicu added 2 commits July 31, 2024 11:47

Correctly pass offset param when calling search api

9e2ac8f

Merge branch 'main' into kicu/45026-query-in-api

ead7ea9

Kicu marked this pull request as ready for review July 31, 2024 10:10

Kicu requested a review from a team as a code owner July 31, 2024 10:10

melvin-bot bot requested review from rayane-djouah and removed request for a team July 31, 2024 10:10

luacmartins self-requested a review July 31, 2024 12:33

rayane-djouah mentioned this pull request Jul 31, 2024

[$250] [Search v2.1] [POLISH] Prevent duplicate API request #46577

Closed

Fix missing filter query params when sorting search results

3ba594e

Fix TS errors in Search

b92ecb0

Kicu commented Jul 31, 2024

View reviewed changes

luacmartins mentioned this pull request Aug 1, 2024

advanced filters category #46197

Merged

50 tasks

Kicu added 2 commits August 1, 2024 10:46

Merge branch 'main' into kicu/45026-query-in-api

5dca4d7

Fix effects in Search after merge conflicts

f7b1cdd

Fix not sending policyIDs in search

3d23bb2

Kicu commented Aug 1, 2024

View reviewed changes

rayane-djouah approved these changes Aug 1, 2024

View reviewed changes

luacmartins approved these changes Aug 1, 2024

View reviewed changes

luacmartins merged commit 207365c into Expensify:main Aug 1, 2024
15 of 16 checks passed

github-actions bot mentioned this pull request Aug 2, 2024

Deploy Checklist: New Expensify 2024-08-02 #46687

Closed

50 tasks

rayane-djouah mentioned this pull request Aug 2, 2024

[PAID] [Search v2.1] - Sorting on Drafts and Finished does not work at all #46720

Closed

6 tasks

melvin-bot bot mentioned this pull request Aug 6, 2024

[HOLD for payment 2024-08-13] [Search v2.1] Use new query syntax in App #46684

Closed

Use new Search query syntax when calling api #46481

Use new Search query syntax when calling api #46481

Conversation

Kicu commented Jul 30, 2024 • edited by luacmartins Loading

Details

Fixed Issues

Tests

Offline tests

QA Steps

PR Author Checklist

Screenshots/Videos

Kicu commented Jul 30, 2024

luacmartins commented Jul 30, 2024

luacmartins commented Jul 30, 2024

melvin-bot bot commented Jul 31, 2024

Kicu commented Jul 31, 2024

rayane-djouah commented Jul 31, 2024 • edited Loading

Reviewer Checklist

Screenshots/Videos

rayane-djouah commented Jul 31, 2024

luacmartins commented Jul 31, 2024

luacmartins commented Jul 31, 2024

luacmartins commented Jul 31, 2024

rayane-djouah commented Jul 31, 2024 • edited Loading

luacmartins commented Jul 31, 2024

luacmartins commented Jul 31, 2024

Kicu commented Jul 31, 2024 • edited Loading

rayane-djouah commented Jul 31, 2024 • edited Loading

Kicu commented Jul 31, 2024

Kicu commented Jul 31, 2024

luacmartins commented Jul 31, 2024

Kicu Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

Kicu commented Jul 31, 2024

rayane-djouah commented Jul 31, 2024

luacmartins commented Jul 31, 2024

Kicu commented Aug 1, 2024

rayane-djouah commented Aug 1, 2024

Kicu commented Aug 1, 2024

Kicu Aug 1, 2024

Choose a reason for hiding this comment

rayane-djouah left a comment

Choose a reason for hiding this comment

luacmartins left a comment

Choose a reason for hiding this comment

rayane-djouah commented Aug 1, 2024 • edited Loading

OSBotify commented Aug 1, 2024

luacmartins commented Aug 1, 2024

OSBotify commented Aug 2, 2024

rayane-djouah commented Aug 2, 2024

dangrous commented Aug 2, 2024

OSBotify commented Aug 6, 2024

Kicu commented Jul 30, 2024 •

edited by luacmartins

Loading

rayane-djouah commented Jul 31, 2024 •

edited

Loading

rayane-djouah commented Jul 31, 2024 •

edited

Loading

Kicu commented Jul 31, 2024 •

edited

Loading

rayane-djouah commented Jul 31, 2024 •

edited

Loading

Kicu Jul 31, 2024 •

edited

Loading

rayane-djouah commented Aug 1, 2024 •

edited

Loading