[🐛 Bug]: Cannot set read timeout value in Local Drivers

[TolstoyDotCom]

What happened?

Despite setting high timeouts ("{implicit: 10000000, pageLoad: 10000000, script: 10000000}"), calling executeAsyncScript will occasionally time out after exactly 3 minutes. This is using 4.5.0 of Selenium and Java 17 with Firefox 105 on Ubuntu 18. It's also using selenium-http-jdk-client.

How can we reproduce the issue?

Download https://github.com/TolstoyDotCom/more-speech and follow the instructions in the "2.1 Notes" section. You might need to use different handles and different numbers of pages to see the issue.

When you click 'Check timeline', the app gets some tweets from the user's profile page then visits each tweet's page.

On an individual tweet page, com.tolstoy.censorship.twitter.checker.app.webdriver.WebDriverFactoryNT first calls executeAsyncScript to load the tweets using the scripts in the stockscripts directory (jquery + suededenim + tweet_retriever).

Then, com.tolstoy.censorship.twitter.checker.app.browserproxy.BrowserProxyHAR calls executeAsyncScript using (jquery + suededenim + har_retriever).

Then, WebDriverFactoryNT calls executeAsyncScript using (jquery + suededenim + json_parser).

So, three calls to executeAsyncScript per page and if one of them takes longer than 3 minutes (likely if there are lots of tweets), it will probably time out.

Relevant log output

[INFO ] 2022-Oct-21 11:45:18 AM c.t.c.t.c.a.h.SearchRunTimelineBuilder - Loading reply page [...]
[...]
[ERROR] 2022-Oct-21 11:48:18 AM c.t.c.t.c.a.h.SearchRunTimelineBuilder - error logging in or building searchRun org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException
Build info: version: '4.5.0', revision: 'fe167b119a'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-125-generic', java.version: '17.0.4'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Command: [c57d3478-3d94-41ae-a991-f4792364f495, get {url=https://twitter.com/[...]}]
Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 105.0, moz:accessibilityChecks: false, moz:buildID: 20220915150737, moz:debuggerAddress: 127.0.0.1:36466, moz:geckodriverVersion: 0.32.0, moz:headless: false, moz:platformVersion: 5.4.0-125-generic, moz:processID: 6398, moz:profile: /tmp/rust_mozprofileAhZ2z2, moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, moz:windowless: false, pageLoadStrategy: normal, platformName: LINUX, proxy: Proxy(), se:cdp: ws://127.0.0.1:36466/devtoo..., se:cdpVersion: 85.0, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 10000000, pageLoad: 10000000, script: 10000000}, unhandledPromptBehavior: dismiss}
Session ID: c57d3478-3d94-41ae-a991-f4792364f495
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:65)
	at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
	at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:49)
	at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
	at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
	at org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:99)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:181)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:167)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:142)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:547)
	at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:310)
	at com.tolstoy.censorship.twitter.checker.app.helpers.SearchRunTimelineBuilder.getIndividualPage(SearchRunTimelineBuilder.java:353)
	at com.tolstoy.censorship.twitter.checker.app.helpers.SearchRunTimelineBuilder.getIndividualPages(SearchRunTimelineBuilder.java:323)
	at com.tolstoy.censorship.twitter.checker.app.helpers.SearchRunTimelineBuilder.buildSearchRunTimelineInternal(SearchRunTimelineBuilder.java:282)
	at com.tolstoy.censorship.twitter.checker.app.helpers.SearchRunTimelineBuilder.buildSearchRunTimeline(SearchRunTimelineBuilder.java:185)
	at com.tolstoy.censorship.twitter.checker.app.AppGUI$TimelineWorker.doInBackground(AppGUI.java:210)
	at com.tolstoy.censorship.twitter.checker.app.AppGUI$TimelineWorker.doInBackground(AppGUI.java:187)
	at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
	at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:206)
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:59)
	... 23 more

Operating System

Ubuntu 18

Selenium version

4.5.0 Java

What are the browser(s) and version(s) where you see this issue?

Firefox 105

What are the browser driver(s) and version(s) where you see this issue?

webdrivermanager 5.0.3

Are you using Selenium Grid?

no

[github-actions]

@TolstoyDotCom, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[diemol]

Can you provide a reduced test case where this can be reproduced?

[titusfortner]

The timeout you are getting has nothing to do with the WebDriver specified timeouts.

You need to update the Java http library read timeout (default is 3 minutes):

ClientConfig config = ClientConfig.defaultConfig().readTimeout(Duration.ofSeconds(300));
RemoteWebDriver driver = ChromeDriver.builder()
                .address(myHub)
                .config(config)
                .oneOf(options)
                .build();

[TolstoyDotCom]

This was happening when I tested with other versions of Selenium that don't use selenium-http-jdk-client, such as version 4.1.1 of Selenium.

I'm also loading the browser on the user's computer, I'm not using grid or some other remote setup. This is a desktop application that loads the user's browser and navigates to Twitter pages to compile a list of tweets. It's just as if the user opened their browser, navigated a few Twitter pages, and recorded which tweets appear where.

The application uses Javascript to scroll the Twitter pages, scrape info from the HTML, read the HAR logs, and merge data. That's where the stack trace happened: after the code called executeAsyncScript with the Javascript code that scrolls a Twitter page and scrapes the HTML. So, it seems like this is a script timeout, not, for instance, a timeout when asking Selenium to load a page and the connection takes a long time.

I tried the solution above. In WebDriverFactoryNT.makeWebDriver() I changed this:

WebDriver driver = new FirefoxDriver( firefoxOptions );

to this:

ClientConfig clientConfig = ClientConfig.defaultConfig().readTimeout( TIMEOUT ).connectionTimeout( TIMEOUT );
WebDriver driver = FirefoxDriver.builder().config( clientConfig ).oneOf( firefoxOptions ).build();

That results in RemoteWebDriverBuilder.getLocalDriver() throwing "ClientConfig instances do not work for Local Drivers".

This is how I'm currently setting timeouts (TIMEOUT is a Duration of over 2 hours):

firefoxOptions.setPageLoadTimeout( TIMEOUT );
firefoxOptions.setScriptTimeout( TIMEOUT );
firefoxOptions.setImplicitWaitTimeout​( TIMEOUT );
[...]
driver.manage().timeouts().scriptTimeout( TIMEOUT );

Is there some other way?

[titusfortner]

All http libraries have default read timeout values.

Unfortunately, we currently do not have a way to change them in local drivers in Java right now.

I think we need to add a ClientConfig parameter to driver constructors to allow this.

[TolstoyDotCom]

I pushed a new version that doesn't show the whole of my script in the stack trace.

I was able to get timeout errors twice using that version with these settings:
handle to check: alexborstein
skip login: [checked]
number of timeline replies to check: 1
how many pages on the timeline to scroll: 1
firefox profile name: twtr_censorship

I'd already created that Firefox profile, logged in to a testing Twitter account, and enabled the HAR extension.