Skip to content
This repository has been archived by the owner on Dec 14, 2021. It is now read-only.

initial autofilling #372

Merged
merged 16 commits into from
Jan 16, 2019
Merged

initial autofilling #372

merged 16 commits into from
Jan 16, 2019

Conversation

sashei
Copy link
Contributor

@sashei sashei commented Jan 11, 2019

Fixes #215

Testing and Review Notes

This will fill credentials, regardless of current DataStore lock status, into the appropriate views in an app as requested. This ticket currently does not differentiate between the app and a contained WebView, so browsers and apps using a WebView-based login (such as ours :p) will not receive the correct credentials to limit scope on this PR. I've filed a follow-up #371 specifically to handle the webview / geckoview view parsing.

There is also a follow-up issue #375 to address the limitations of the naive package name parsing (matching the app name with a credential website) that's being used here. You may see an error "unexpected package name format"

Apps I've tested with so far:

  • Twitter uses a WebView for login, so not supported yet (Autofill credentials in web pages #371) ⚠️
  • Facebook ✅
  • Reddit (only finds password field? can't figure out what's going on here) ⚠️
  • Instagram (fb login, Just Works) ✅
  • Walmart (doesn't kick off the autofillservice regardless of what I do, so ¯_(ツ)_/¯ ) ❌
  • Spotify (doesn't kick off the autofillservice regardless of what I do, so ¯_(ツ)_/¯ ) ❌
  • Snapchat ✅
  • Vanguard ✅
  • Netflix ✅
  • Slack -- autofill service only kicks off on the workspace URL page, not for login ⚠️
  • Venmo (doesn't kick off the autofillservice regardless of what I do, so ¯_(ツ)_/¯ ) ❌
  • Wayfair (only finds username in multi-step login) ⚠️
  • Letgo ✅
  • DoorDash ✅
  • GrubHub ✅
  • Duolingo ✅
  • Invision - WebView, supported in Autofill credentials in web pages #371 ⚠️
  • Geico - insanely complex view heirarchy (why does every textview have a linearlayout around it??) ❌
  • Skype (doesn't kick off the autofillservice regardless of what I do, so ¯_(ツ)_/¯ ) ❌

Helping test the AutofillService

I've only tested this against the top ~10 apps on the app store... if we can cast a wider net, that would be awesome! I've tried to add some useful error pop-ups from the AutofillService, and would really appreciate folks installing the build from this branch and trying to log in to several apps. Feel free to add any apps you test as comments (with the pop-up error if you catch it) or in the section above.

As we are only testing login screen parsing with this ticket, I have amended the package name parsing to only return "twitter". You will see your twitter credentials filled on every screen; I'm particularly interested in apps in which no login fields (username or password) can be found, or in which only one or the other is filled. I've pulled out the twitter login, but will be merging this with the rest of the popup errors. The domain one is particularly likely to pop up in this stage, and issues with that message can be moved to #375. If you'd like to test the view parsing functionality only and fill twitter credentials to everything, the app link below has that functionality.

✨✨✨✨✨✨APP LINK✨✨✨✨✨✨

To Do

  • add “WIP” to the PR title if pushing up but not complete nor ready for review
  • double check the original issue to confirm it is fully satisfied
  • add testing notes and screenshots in PR description to help guide reviewers
  • add unit tests
    • optional: consider adding instrumentation (integration/UI) tests
  • consider running this branch in a debug simulator and check for memory leak notifications or any warnings
  • request the "UX" team perform a design review (if/when applicable)
  • make sure CI builds are passing (e.g.: fix lint and other errors)
  • check on the accessibility of any added UI

Sorry, something went wrong.

@sashei sashei requested a review from a team as a code owner January 11, 2019 21:29
@ghost ghost assigned sashei Jan 11, 2019
@ghost ghost added the in progress label Jan 11, 2019
@sashei sashei force-pushed the 215-autofill-credentials branch 2 times, most recently from a80cdca to 8509f54 Compare January 14, 2019 19:59
Copy link
Contributor

@linuxwolf linuxwolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving as comment-only for now.

Overall it looks good. I have some concerns in the details, but it's shaping up nicely!

val usernamePresentation = RemoteViews(packageName, android.R.layout.simple_list_item_1)
val passwordPresentation = RemoteViews(packageName, android.R.layout.simple_list_item_1)
usernamePresentation.setTextViewText(android.R.id.text1, credential.username)
passwordPresentation.setTextViewText(android.R.id.text1, "Password for ${credential.username}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any localization concerns with that string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 good call-out. I actually wasn't sure what to put here; we could show the password in plaintext but that doesn't seem ideal. The designs don't cover the password field.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nickbrandt @changecourse ^ ?

For reference, currently when the user is focusing the password field, I'm sending back an autofill selection box that says Password for x where x is the username. What do y'all think?

Copy link
Contributor

@nickbrandt nickbrandt Jan 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sashei that makes sense to me 👍
Through my testing, I haven't encountered any apps where tapping on an autofill suggestion for the username field fills both username and password fields. Is that how this implementation will work, where the user will have to select the autofill suggestion for both the username and password fields?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nickbrandt in my testing, yes, we end up filling both fields when the user selects an account in one of them (can highlight username or password fields). You're welcome to grab the app from the link above and see if it works as you expect.

@sashei
Copy link
Contributor Author

sashei commented Jan 14, 2019

Autofill detection thoughts: if we have an EditText following a TextView and the TextView contains our keywords ("email", "username", "password" etc) then we can assume the EditText should have take the autofill ID for the appropriate credential.

@sashei sashei force-pushed the 215-autofill-credentials branch from ba6b6e4 to 1e2c7c3 Compare January 15, 2019 03:46
@sashei sashei requested a review from a team January 15, 2019 05:34
@nickbrandt
Copy link
Contributor

nickbrandt commented Jan 15, 2019

@sashei here is some of my testing results: (using build #2143, debug)
Device: Pixel 1 running Android 9

Apps tested:
Facebook: Only provides suggestion for Password ⚠️
Instagram: Not showing suggestions ❌
Reddit: Not showing suggestions ❌
Dropbox: Not showing suggestions ❌
Netflix: Only provides suggestion for Username ⚠️
Geico: Not showing suggestions ❌
Wayfair: 2 step sign-in, provides suggestion for step 1 (Username), not step 2 (password) ⚠️
Walmart: Not showing suggestions ❌

Perhaps I'm doing something wrong here. I'm confused why some of these are working for you/others, but not for me :/

I'm not seeing any errors either, except occasionally seeing "unexpected package name format" when on the entry list or an entry detail

@changecourse
Copy link
Contributor

changecourse commented Jan 15, 2019

I've loaded up the debug build of #2143 a few times on my Pixel 2 and have had issues just signing in. I was finally able to log in, but am experiencing some of the same issues as Nick with suggestions not showing up. I made sure to set up Lockbox as the AutoFill provider in settings once Lockbox was installed.

Apps tried:
USAA: Not showing any suggestions
Reddit: Not showing any suggestions
InVision: Not showing any suggestions

@sashei
Copy link
Contributor Author

sashei commented Jan 15, 2019

@nickbrandt @changecourse cool, thanks for testing y'all, I will look into those with some other device / version emulators!

@sashei
Copy link
Contributor Author

sashei commented Jan 15, 2019

FWIW I tried some apps with two-step sign in processes, and the autofill service frequently only gets triggered on the first page (e.g. Slack app), so that might be a system limitation :-/

@sashei sashei force-pushed the 215-autofill-credentials branch from 3ee92b8 to 6703db1 Compare January 16, 2019 02:14
@sashei sashei force-pushed the 215-autofill-credentials branch from 6703db1 to a7f2b81 Compare January 16, 2019 02:15
@sashei sashei force-pushed the 215-autofill-credentials branch from a7f2b81 to a38fcd3 Compare January 16, 2019 02:38
@sashei
Copy link
Contributor Author

sashei commented Jan 16, 2019

@changecourse @nickbrandt thanks again for testing this yesterday; I cleverly introduced a regression and did no further testing right before making this build so... yeah! Hopefully that is cleared up for y'all today; I've made a point to re-test against all the apps listed in the issue description!

@nickbrandt
Copy link
Contributor

@sashei it's working much better now, thanks! Below are my testing results:

Device: Pixel 1 running Android 9

Apps Tested
Netflix: Filled Both ✅
Walmart: Suggestions not showing ❌
Facebook: Filled Both ✅
Reddit: Suggestion for Password only ⚠️
Wayfair: Suggestion for Username only (2 step sign-in) ⚠️
Dropbox: Filled Both ✅
Instagram: Filled Both ✅

Errors:
InVision: "couldn't find a username or password field" (webview?)
Geico: "couldn't find a username or password field" (webview?)
Skype: "couldn't find a username or password field" (webview?)

@sashei sashei force-pushed the 215-autofill-credentials branch from f96b015 to 6e89ca7 Compare January 16, 2019 21:27
@sashei sashei changed the title [WIP] initial autofilling initial autofilling Jan 16, 2019
@sashei sashei requested a review from a team January 16, 2019 21:32
@sashei sashei force-pushed the 215-autofill-credentials branch from 6e89ca7 to eea524e Compare January 16, 2019 21:59
Copy link
Contributor

@linuxwolf linuxwolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking really awesome. Some very quick testing this is working pretty well, though i suspect we're entering an arms race w/r/t heuristics.

"Requesting Changes" for the discussion of a couple of points in-line; will happily change to "approved" once discussed.

View.AUTOFILL_HINT_EMAIL_ADDRESS,
"email",
"username",
"user name"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😆

@sashei
Copy link
Contributor Author

sashei commented Jan 16, 2019

A note on testing: The AutofillService is very hairy to unit test; even the google sample app doesn't do it. I've got a placeholder test file and some basic infrastructure, but couldn't get these working in a meaningful way. Most of the business logic is in the ParsedStructure (and presumably the domain parsing logic will be abstracted out as well), so I feel ok with leaving the LockboxAutofillService under-tested for now.

@sashei sashei merged commit c92c02e into master Jan 16, 2019
@ghost ghost removed the in progress label Jan 16, 2019
@sashei sashei deleted the 215-autofill-credentials branch January 16, 2019 23:16
devinreams pushed a commit that referenced this pull request Jan 26, 2019
initial autofilling
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide credentials via Autofill when available
4 participants