Java implementation of DevTools API

[shs96c]

This should allow users to take advantage of features offered by using the Chrome Debugging Protocol (and equivalents on other browsers), notably:

• Full page screenshots
• Streaming logs
• Waiting for mutation events on the page

[shs96c]

There's code for this in https://github.com/shs96c/selenium/tree/cdp

[joywave]

This would be really nice to have. I looked at this today and some previous attempts were #5262 #5159

[adiohana]

Hi,
We already implemented this feature in our selenium server. I will be happy to create a PR for contribution.
Working with DevTools API (especially the Network) allowing us to:

• filter URL's
• add custom headers to all requests
• manipulate desired requests and responses
• and more

[adiohana]

@shs96c
I see that this is part of Selenium 4. Is there a point for me to work on the PR now?
Chrome DevTools integration requires WebSocket functionality and some changed in RemoteWebDriver which is kind of a big change

[barancev]

Initial support has been implemented:
f280b8b
a99534e
5829863

[shs96c]

@adiohana Absolutely! We're super-keen on coming up with a nicer way of presenting this in the user-facing APIs, and we really need to find a way of generating the raw CDP classes rather than writing it all by hand. We'd really appreciate some help with this, and your changes sound great!

[adiohana]

@shs96c
Thanks for the confirmation, we will do it soon

[adiohana]

@shs96c
seems that generating the raw CDP is a complicated task and can be fragile because we are counting on Google developers not to change their CDP json schema as stated here: https://github.com/ChromeDevTools/devtools-protocol/blob/master/json/browser_protocol.json

I see that you already implemented (partially) the following domains:

• Target
• Runtime
• Console
• Log

I think that the 2 most critical domains (for the features I stated above and more) are:

• Network
• Performance

Do you think it worth implementing Network and Performance "manually" while we are working on the auto generation of the CDP classes?

BTW: creating all the CDP classes manually will probably take 1-2 days, we can finish that this week.

[shs96c]

@adiohana, writing those two manually makes sense as a stepping stone to getting everything running. If you've got the time, I'd be very happy to review a PR :) If you need a hand, I tend to lurk on the #selenium IRC and Slack channels (London work hours), and I can step you through anything you're finding tricky to deal with.

[adiohana]

Thanks @shs96c
I am working on this together with [email protected]
we will submit the PRs tomorrow probably.
We also joined the slack channel so we will reach out in case of any issues

[adiohana]

#7212

@shs96c
PR is ready. We have a good infrastructure for DevTools and Network and Performance implementations.

I will add the notes and gaps also to here:

• Need to add support for List as Command input.
• Add better error handling for DevTools operations
• Add more test scenarios
• Implement more DevTools methods according to https://chromedevtools.github.io/devtools-protocol/

let me know if you want to open tasks for the above notes. We will be happy to add more API's and more functionalities to DevTools in the near future.

[dratler]

Hi ,
I would like to implement "Profiler Domain" out of DevTools , is it fine ?

[shs96c]

@dratler Apologies for not responding sooner --- I'd missed the notification. Implementing any part of the CDP is fine. Putting forwards any diffs for Selenium is also fine. It's encouraged! Thank you for taking the time and effort to join in the fun! :)

[dratler]

Hi @shs96c ,
no problem just say when :).
P.S I have investigated more about auto generated and for our use case I think it will not do well.
For now I'm also working on Page CDP.|
also If you have other tasks that I could aid please make and I will do my best to tag in.

[SergeyPirogov]

Would be nice to have support for debugger and profiler

[dratler]

Hi @SergeyPirogov ,
profiler is implemented and for Debugger it's under construction

[barancev]

I'm working on the code generatior that will cover full CDP specification. Please be patient, it's going to be available soon.

[barancev]

What we really need help with is the current implementation of the Fetch domain. It does not pass its tests. If someone could investigate and fix it that would be very much appreciated.

[dratler]

I will fix it

[NoamGoren]

Hi @barancev ,
@dratler and I fixed this issue.
PR can be found on #7754

[diemol]

I'll close this since initial support has been added in the linked commits above, the PR discussion will be handled in the PR, and a more detailed discussion on how it will be supported can be followed at #8168