Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[🐛 Bug]: Timeout exception thrown randomly with RemoteWebDriver #12558

Closed
Abhijith2893 opened this issue Aug 15, 2023 · 40 comments
Closed

[🐛 Bug]: Timeout exception thrown randomly with RemoteWebDriver #12558

Abhijith2893 opened this issue Aug 15, 2023 · 40 comments

Comments

@Abhijith2893
Copy link

What happened?

When executing tests at scale there are some random failures with read timeouts (seems to wait for 3 mins). We are observing this primarily when clicking an element but have also observed it during navigate to page. Seems to be a reoccurrence of #9528

We are using dockerized grid v4.10 with the hub/node model and we confirmed that the docker process starts with jdk-http-client

java -Dwebdriver.http.factory=jdk-http-client -jar /opt/selenium/selenium-server.jar --ext /opt/selenium/selenium-http-jdk-client.jar hub --session-request-timeout 3600 --session-retry-interval 15 --relax-checks true --bind-host false --config /opt/selenium/config.toml

How can we reproduce the issue?

Unfortunately similar to https://github.com/SeleniumHQ/selenium/issues/9528 we are unable to find a consistent repro for this. Seems flaky and rerunning the test passes but we are still seeing this happen daily.

Relevant log output

We use an internal framework to run selenium tests and the DEBUG logs indicate that a 500 was returned during the click event. We check to see if the element is enabled and clickable (which passes) but the subsequent click seems to fail.


DEBUG    2023-08-15 19:24:02.928  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-hub-nomad.service.eng.consul:4444/session/ab89759c17aac47ed60f6d6b9a8b6882/element {"using": "xpath", "value": "//*[@class='extension-name'][text()='JSON']"}
DEBUG    2023-08-15 19:24:02.943  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "POST /session/ab89759c17aac47ed60f6d6b9a8b6882/element HTTP/1.1" 200 95
DEBUG    2023-08-15 19:24:02.943  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=200 | data={"value":{"element-6066-11e4-a52e-4f735466cecf":"3AFE27AB794812F985C270183415B5A4_element_21"}} | headers=HTTPHeaderDict({'content-length': '95', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG    2023-08-15 19:24:02.944  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request
DEBUG    2023-08-15 19:24:02.944  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> GET http://selenium-four-hub-nomad.service.eng.consul:4444/session/ab89759c17aac47ed60f6d6b9a8b6882/element/3AFE27AB794812F985C270183415B5A4_element_21/enabled {"id": "3AFE27AB794812F985C270183415B5A4_element_21"}
DEBUG    2023-08-15 19:24:02.961  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "GET /session/ab89759c17aac47ed60f6d6b9a8b6882/element/3AFE27AB794812F985C270183415B5A4_element_21/enabled HTTP/1.1" 200 14
DEBUG    2023-08-15 19:24:02.962  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=200 | data={"value":true} | headers=HTTPHeaderDict({'content-length': '14', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG    2023-08-15 19:24:02.962  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request
DEBUG    2023-08-15 19:24:02.963  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-hub-nomad.service.eng.consul:4444/session/ab89759c17aac47ed60f6d6b9a8b6882/element {"using": "xpath", "value": "//*[@class='extension-name'][text()='JSON']"}
DEBUG    2023-08-15 19:24:02.978  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "POST /session/ab89759c17aac47ed60f6d6b9a8b6882/element HTTP/1.1" 200 95
DEBUG    2023-08-15 19:24:02.979  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=200 | data={"value":{"element-6066-11e4-a52e-4f735466cecf":"3AFE27AB794812F985C270183415B5A4_element_21"}} | headers=HTTPHeaderDict({'content-length': '95', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG    2023-08-15 19:24:02.979  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request
DEBUG    2023-08-15 19:24:02.980  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-hub-nomad.service.eng.consul:4444/session/ab89759c17aac47ed60f6d6b9a8b6882/element/3AFE27AB794812F985C270183415B5A4_element_21/click {"id": "3AFE27AB794812F985C270183415B5A4_element_21"}
DEBUG    2023-08-15 19:27:03.006  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "POST /session/ab89759c17aac47ed60f6d6b9a8b6882/element/3AFE27AB794812F985C270183415B5A4_element_21/click HTTP/1.1" 500 3517
DEBUG    2023-08-15 19:27:03.007  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=500 | data={
  "value": {
    "error": "script timeout",
    "message": "Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown",
    "stacktrace": "java.lang.RuntimeException: Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:160)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:384)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.util.concurrent.TimeoutException\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:377)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:54)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:91)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:125)\n\t... 26 more\n"
  }
} | headers=HTTPHeaderDict({'content-length': '3517', 'Cache-Control': 'none', 'Content-Type': 'application/json; charset=utf-8'})
DEBUG    2023-08-15 19:27:03.008  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request


### Operating System

Linux

### Selenium version

4.10

### What are the browser(s) and version(s) where you see this issue?

Chrome v.114.0.5735.133

### What are the browser driver(s) and version(s) where you see this issue?

v.114.0.5735.133

### Are you using Selenium Grid?

4.10
@github-actions
Copy link

@Abhijith2893, thank you for creating this issue. We will troubleshoot it as soon as we can.


Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

@titusfortner
Copy link
Member

Do you have the logs for the grid?

From my tests recently, Python does not seem to have a Read timeout as such (#12368). Java read timeout happens at 200 seconds.

Default script timeout is 30 seconds, have you changed that anywhere in your code?
This looks to me like the driver itself had a problem and has improperly reported it back to the grid as a script timeout error.

Would need to see logs.

@Abhijith2893
Copy link
Author

Including hub and node logs for an occurrence

Node logs

{"log":"22:23:09.342 INFO [LocalNode.newSession] - Session created by the Node. Id: d4045be089608482643a18631c05632e, Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 114.0.5735.133, chrome: {chromedriverVersion: 114.0.5735.90 (386bc09e8f4f..., userDataDir: /tmp/.com.google.Chrome.4J0f7U}, goog:chromeOptions: {debuggerAddress: localhost:42993}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://172.17.0.8:4444/sessio..., se:cdpVersion: 114.0.5735.133, se:vnc: ws://172.17.0.8:4444/sessio..., se:vncEnabled: true, se:vncLocalAddress: ws://172.17.0.8:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 300000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}"}

{"log":"22:23:19.278 WARN [SeleniumSpanExporter$1.lambda$export$3] - {\"traceId\": \"936d710c728d4da031330c73191e24fd\",\"eventTime\": 1692138199277271160,\"eventName\": \"HTTP request execution complete\",\"attributes\": {\"http.flavor\": 1,\"http.handler_class\": \"org.openqa.selenium.remote.http.Route$PredicatedRoute\",\"http.host\": \"10.240.9.228:22701\",\"http.method\": \"POST\",\"http.request_content_length\": \"42\",\"http.scheme\": \"HTTP\",\"http.status_code\": 500,\"http.target\": \"\\u002fsession\\u002fd4045be089608482643a18631c05632e\\u002fexecute\\u002fsync\",\"http.user_agent\": \"selenium\\u002f4.10.0 (python linux)\"}}"}

{"log":"22:33:57.595 WARN [SeleniumSpanExporter$1.lambda$export$3] - {\"traceId\": \"62ddd91fcc936eee13b5444feb276b01\",\"eventTime\": 1692138837593350679,\"eventName\": \"HTTP request execution complete\",\"attributes\": {\"http.flavor\": 1,\"http.handler_class\": \"org.openqa.selenium.remote.http.Route$PredicatedRoute\",\"http.host\": \"10.240.9.228:22701\",\"http.method\": \"POST\",\"http.request_content_length\": \"53\",\"http.scheme\": \"HTTP\",\"http.status_code\": 404,\"http.target\": \"\\u002fsession\\u002fd4045be089608482643a18631c05632e\\u002felement\\u002f3E2F9FC4413097A9ED3BAE7A95943A8D_element_21\\u002fclick\",\"http.user_agent\": \"selenium\\u002f4.10.0 (python linux)\"}}"}

{"log":"22:35:56.520 INFO [SessionSlot.stop] - Stopping session d4045be089608482643a18631c05632e"}

{"log":"22:35:56.521 WARN [LocalNode.lambda$new$5] - Removing Downloads folder associated with d4045be089608482643a18631c05632e"}

{"log":"22:35:56.522 WARN [LocalNode.lambda$new$6] - Removing Uploads folder associated with d4045be089608482643a18631c05632e"}

Hub logs

{"log":"22:23:09.351 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: d4045be089608482643a18631c05632e "}

{"log":"22:26:55.478 WARN [SeleniumSpanExporter$1.lambda$export$3] - {\"traceId\": \"62ddd91fcc936eee13b5444feb276b01\",\"eventTime\": 1692138415477352036,\"eventName\": \"exception\",\"attributes\": {\"exception.message\": \"Unable to execute request for an existing session: java.util.concurrent.TimeoutException\\nBuild info: version: '4.10.0', revision: 'c14d967899'\\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\\nDriver info: driver.version: unknown\",\"exception.stacktrace\": \"org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException\\nBuild info: version: '4.10.0', revision: 'c14d967899'\\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\\nDriver info: driver.version: unknown\\n\\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:394)\\n\\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:54)\\n\\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:91)\\n\\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:125)\\n\\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:384)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\\n\\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\\n\\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\\n\\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\\n\\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\\n\\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\\n\\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\\n\\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\\n\\tat java.base\\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\\n\\tat java.base\\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\\n\\tat java.base\\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\\n\\tat java.base\\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\\n\\tat java.base\\u002fjava.lang.Thread.run(Thread.java:829)\\nCaused by: java.util.concurrent.TimeoutException\\n\\tat java.base\\u002fjava.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)\\n\\tat java.base\\u002fjava.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)\\n\\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:377)\\n\\t... 29 more\\n\",\"exception.type\": \"org.openqa.selenium.TimeoutException\",\"http.flavor\": 1,\"http.handler_class\": \"org.openqa.selenium.grid.router.HandleSession\",\"http.host\": \"selenium-four-hub-nomad.service.eng.consul:4444\",\"http.method\": \"POST\",\"http.request_content_length\": \"53\",\"http.scheme\": \"HTTP\",\"http.target\": \"\\u002fsession\\u002fd4045be089608482643a18631c05632e\\u002felement\\u002f3E2F9FC4413097A9ED3BAE7A95943A8D_element_21\\u002fclick\",\"http.user_agent\": \"selenium\\u002f4.10.0 (python linux)\",\"session.id\": \"d4045be089608482643a18631c05632e\"}}"}

{"log":"22:35:56.099 WARN [SeleniumSpanExporter$1.lambda$export$3] - {\"traceId\": \"ae75ee023242815a7efc55eb5621159e\",\"eventTime\": 1692138956098356744,\"eventName\": \"exception\",\"attributes\": {\"exception.message\": \"Unable to execute request for an existing session: java.util.concurrent.TimeoutException\\nBuild info: version: '4.10.0', revision: 'c14d967899'\\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\\nDriver info: driver.version: unknown\",\"exception.stacktrace\": \"org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException\\nBuild info: version: '4.10.0', revision: 'c14d967899'\\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\\nDriver info: driver.version: unknown\\n\\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:394)\\n\\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:54)\\n\\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:91)\\n\\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:125)\\n\\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:384)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\\n\\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\\n\\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\\n\\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\\n\\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\\n\\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\\n\\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\\n\\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\\n\\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\\n\\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\\n\\tat java.base\\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\\n\\tat java.base\\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\\n\\tat java.base\\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\\n\\tat java.base\\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\\n\\tat java.base\\u002fjava.lang.Thread.run(Thread.java:829)\\nCaused by: java.util.concurrent.TimeoutException\\n\\tat java.base\\u002fjava.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)\\n\\tat java.base\\u002fjava.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)\\n\\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:377)\\n\\t... 29 more\\n\",\"exception.type\": \"org.openqa.selenium.TimeoutException\",\"http.flavor\": 1,\"http.handler_class\": \"org.openqa.selenium.grid.router.HandleSession\",\"http.host\": \"selenium-four-hub-nomad.service.eng.consul:4444\",\"http.method\": \"GET\",\"http.scheme\": \"HTTP\",\"http.target\": \"\\u002fsession\\u002fd4045be089608482643a18631c05632e\\u002fscreenshot\",\"http.user_agent\": \"selenium\\u002f4.10.0 (python linux)\",\"session.id\": \"d4045be089608482643a18631c05632e\"}}"}

{"log":"22:35:56.531 INFO [GridModel.release] - Releasing slot for session id d4045be089608482643a18631c05632e"}

{"log":"22:35:56.531 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: d4045be089608482643a18631c05632e"}

In our custom framework we capture a screenshot on failure. We observe that after the first timeout issue (on the click). All subsequent requests also timeout like the screenshot request.

We trying tweaking the scriptTimeout and set it to 5 mins but from the DEBUG logs it looks like the 500 is returned after exactly 3 mins.

@Abhijith2893
Copy link
Author

Adding some more logs from our custom framework showing the subsequent request failures, all after exactly 3mins

DEBUG    2023-08-15 22:23:55.474  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-hub-nomad.service.eng.consul:4444/session/d4045be089608482643a18631c05632e/element/3E2F9FC4413097A9ED3BAE7A95943A8D_element_21/click {"id": "3E2F9FC4413097A9ED3BAE7A95943A8D_element_21"}
DEBUG    2023-08-15 22:26:55.480  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "POST /session/d4045be089608482643a18631c05632e/element/3E2F9FC4413097A9ED3BAE7A95943A8D_element_21/click HTTP/1.1" 500 3517
DEBUG    2023-08-15 22:26:55.482  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=500 | data={
  "value": {
    "error": "script timeout",
    "message": "Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown",
    "stacktrace": "java.lang.RuntimeException: Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:160)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:384)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.util.concurrent.TimeoutException\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:377)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:54)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:91)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:125)\n\t... 26 more\n"
  }
} | headers=HTTPHeaderDict({'content-length': '3517', 'Cache-Control': 'none', 'Content-Type': 'application/json; charset=utf-8'})
DEBUG    2023-08-15 22:26:55.482  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request
KIT_DEBUG 2023-08-15 22:26:55.483  KIT:iframe_switcher.py:110 -> Switching to parent frame.
DEBUG    2023-08-15 22:26:55.484  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-hub-nomad.service.eng.consul:4444/session/d4045be089608482643a18631c05632e/frame/parent {}
DEBUG    2023-08-15 22:29:55.494  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "POST /session/d4045be089608482643a18631c05632e/frame/parent HTTP/1.1" 500 3517
DEBUG    2023-08-15 22:29:55.495  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=500 | data={
  "value": {
    "error": "script timeout",
    "message": "Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown",
    "stacktrace": "java.lang.RuntimeException: Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:160)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:384)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.util.concurrent.TimeoutException\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:377)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:54)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:91)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:125)\n\t... 26 more\n"
  }
} | headers=HTTPHeaderDict({'content-length': '3517', 'Cache-Control': 'none', 'Content-Type': 'application/json; charset=utf-8'})
DEBUG    2023-08-15 22:29:55.496  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request
KIT_DEBUG 2023-08-15 22:29:55.496  KIT:iframe_switcher.py:102 -> Switching to iframe: //div[contains(@class,'load-xm-events-modal_iframeWrapper')]/iframe
DEBUG    2023-08-15 22:29:55.497  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-hub-nomad.service.eng.consul:4444/session/d4045be089608482643a18631c05632e/element {"using": "xpath", "value": "//div[contains(@class,'load-xm-events-modal_iframeWrapper')]/iframe"}
DEBUG    2023-08-15 22:32:55.506  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "POST /session/d4045be089608482643a18631c05632e/element HTTP/1.1" 500 3517
DEBUG    2023-08-15 22:32:55.507  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=500 | data={
  "value": {
    "error": "script timeout",
    "message": "Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown",
    "stacktrace": "java.lang.RuntimeException: Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:160)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:384)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.util.concurrent.TimeoutException\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:377)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:54)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:91)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:125)\n\t... 26 more\n"
  }
} | headers=HTTPHeaderDict({'content-length': '3517', 'Cache-Control': 'none', 'Content-Type': 'application/json; charset=utf-8'})
DEBUG    2023-08-15 22:32:55.507  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request
DEBUG    2023-08-15 22:32:56.092  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> GET http://selenium-four-hub-nomad.service.eng.consul:4444/session/d4045be089608482643a18631c05632e/screenshot {}
DEBUG    2023-08-15 22:35:56.101  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "GET /session/d4045be089608482643a18631c05632e/screenshot HTTP/1.1" 500 3517
DEBUG    2023-08-15 22:35:56.101  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=500 | data={
  "value": {
    "error": "script timeout",
    "message": "Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown",
    "stacktrace": "java.lang.RuntimeException: Unable to execute request for an existing session: java.util.concurrent.TimeoutException\nBuild info: version: '4.10.0', revision: 'c14d967899'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-477.13.1.el8_8.x86_64', java.version: '11.0.19'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:160)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:384)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:347)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.util.concurrent.TimeoutException\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)\n\tat java.base\u002fjava.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:377)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:54)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:91)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:125)\n\t... 26 more\n"
  }
} | headers=HTTPHeaderDict({'content-length': '3517', 'Cache-Control': 'none', 'Content-Type': 'application/json; charset=utf-8'})
DEBUG    2023-08-15 22:35:56.102  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request

@titusfortner
Copy link
Member

Can you set grid to log with level FINE?

@titusfortner
Copy link
Member

Aha. The error is throwing us off - #12484
It won't fix the underlying issue, but grid is doing the wrong thing and we'll get more insight after the next release.

@Abhijith2893
Copy link
Author

Thank you for the update. Due to the traffic we run on the grid I'll try capture FINE level logs during the weekend and attach them here.

@Abhijith2893
Copy link
Author

Attaching grid FINE level logs. I tried extracting the relevant portion. Searching for element_21/click in the logs is where we observe the script timeout error.
grid-debug-log.txt

@diemol
Copy link
Member

diemol commented Aug 24, 2023

I cannot figure out much from the logs. Is there a way to reproduce this?

@Abhijith2893
Copy link
Author

Unfortunately we are unable to find a consistent repro for this similar to #9528

@diemol
Copy link
Member

diemol commented Sep 4, 2023

Any updates on how to reproduce this?

@joerg1985
Copy link
Member

This might be related to closing the httpclient to fast, it might still be waiting for a response. It should only be closed when the session is gone, not on a time base in my mind.

this.httpClients =
CacheBuilder.newBuilder()
.expireAfterAccess(Duration.ofMinutes(1))
.removalListener(
(RemovalListener<URL, HttpClient>) removal -> removal.getValue().close())
.build();

@Abhijith2893 is it possible the navigation (until the page is loaded) takes more than 60 seconds?

@Abhijith2893
Copy link
Author

Yes, that could be a possibility. We haven't been able to get a consistent repro but during debugging we noticed that adding a sleep.wait before the click command where we encounter this issue seems to reduce the failures. Interestingly, as observed in the logs, we check if the button is enabled and clickable (both checks pass) before the actual click action which return the 500.

@joerg1985
Copy link
Member

@diemol I did increase the timeout to 4 minutes in 5f3f7d0, we should not close potentially waiting HttpClients in any case.

@Abhijith2893 can you use the nightly build to check if this fixes this issue?

@titusfortner
Copy link
Member

If we change defaults for the languages, #12368, we should keep this value in mind as well.

@joerg1985
Copy link
Member

Yes, thanks for the hint. I have a change prepared to completly remove this timeout. We can check every minute and if it has not been used for some time, if not check the session is alive or not. If the session is alive let the client alive, if not close it.

But lets see if this issue is resolved with the increased timeout first.

@Abhijith2893
Copy link
Author

I've been running some tests with the nightly build but have not faced the issue yet. The increased timeout looks promising.

@Abhijith2893
Copy link
Author

After updating the hub to 4.12.1 we observe a different error message for the same flaky issue from this fix #12484

But unfortunately it does not give any more context.

BUG    2023-09-21 20:08:31.974  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-hub-nomad.service.eng.consul:4444/session/23cf22e138f012c6d1a51c62d9cd3fee/element {"using": "xpath", "value": "//*[@class='extension-name'][text()='JSON']"}
DEBUG    2023-09-21 20:08:31.998  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "POST /session/23cf22e138f012c6d1a51c62d9cd3fee/element HTTP/1.1" 200 95
DEBUG    2023-09-21 20:08:31.998  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=200 | data={"value":{"element-6066-11e4-a52e-4f735466cecf":"73F6FFD324B8AC09F44D2FA95B95DA84_element_22"}} | headers=HTTPHeaderDict({'content-length': '95', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG    2023-09-21 20:08:31.999  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request
DEBUG    2023-09-21 20:08:31.999  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> GET http://selenium-four-hub-nomad.service.eng.consul:4444/session/23cf22e138f012c6d1a51c62d9cd3fee/element/73F6FFD324B8AC09F44D2FA95B95DA84_element_22/enabled {"id": "73F6FFD324B8AC09F44D2FA95B95DA84_element_22"}
DEBUG    2023-09-21 20:08:32.024  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "GET /session/23cf22e138f012c6d1a51c62d9cd3fee/element/73F6FFD324B8AC09F44D2FA95B95DA84_element_22/enabled HTTP/1.1" 200 14
DEBUG    2023-09-21 20:08:32.025  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=200 | data={"value":true} | headers=HTTPHeaderDict({'content-length': '14', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG    2023-09-21 20:08:32.025  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request
DEBUG    2023-09-21 20:08:32.026  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-hub-nomad.service.eng.consul:4444/session/23cf22e138f012c6d1a51c62d9cd3fee/element {"using": "xpath", "value": "//*[@class='extension-name'][text()='JSON']"}
DEBUG    2023-09-21 20:08:32.049  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "POST /session/23cf22e138f012c6d1a51c62d9cd3fee/element HTTP/1.1" 200 95
DEBUG    2023-09-21 20:08:32.049  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=200 | data={"value":{"element-6066-11e4-a52e-4f735466cecf":"73F6FFD324B8AC09F44D2FA95B95DA84_element_22"}} | headers=HTTPHeaderDict({'content-length': '95', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG    2023-09-21 20:08:32.050  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request
DEBUG    2023-09-21 20:08:32.050  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-hub-nomad.service.eng.consul:4444/session/23cf22e138f012c6d1a51c62d9cd3fee/element/73F6FFD324B8AC09F44D2FA95B95DA84_element_22/click {"id": "73F6FFD324B8AC09F44D2FA95B95DA84_element_22"}
DEBUG    2023-09-21 20:11:32.074  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-hub-nomad.service.eng.consul:4444 "POST /session/23cf22e138f012c6d1a51c62d9cd3fee/element/73F6FFD324B8AC09F44D2FA95B95DA84_element_22/click HTTP/1.1" 500 2606
DEBUG    2023-09-21 20:11:32.076  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=500 | data={
  "value": {
    "error": "unknown error",
    "message": "\u003cno message present in throwable>",
    "stacktrace": "java.lang.NullPointerException\n\tat org.openqa.selenium.remote.tracing.Tags.lambda$static$2(Tags.java:60)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:89)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:125)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:382)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\n"
  }
} | headers=HTTPHeaderDict({'content-length': '2606', 'Cache-Control': 'none', 'Content-Type': 'application/json; charset=utf-8'})
DEBUG    2023-09-21 20:11:32.076  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request

@joerg1985
Copy link
Member

Okay this NullPointer is verry interesting, i have reviewed the code and ended up in the RetryRequest class. I think this class is broken, as all concurrent requests use the same static final field to save the fallback value, this has the potential to return the Response of a different Request when two threads hit the max retries concurrently.

I could not find the actual root cause of the NPE from reviewing the code. The Failsafe mechanism used here makes it hard for me ro read the code and follow the execution. Using a loop might simplify the code drastically and might make the dependency obsulete?

@joerg1985
Copy link
Member

@pujagani i think the RetryRequest has the following issue. When a unhandled exception is thrown the fallback value is returend, this might be null or the value last time set by a different request reaching the max retries.

@pujagani
Copy link
Contributor

@joerg1985 I think your observation regarding the two threads hitting max retries concurrently might be right.
I am afraid I don't fully understand the "When a unhandled exception is thrown the fallback value is returend" bit.
The retry kicks in only when it sees a TimeoutException in this case. If the retries are exceeded then it returns the fallback response.

@pujagani
Copy link
Contributor

Actually, the Fallback is threadsafe https://failsafe.dev/javadoc/core/dev/failsafe/Fallback.html. I am not sure how to reproduce the NPE. It would be nice if we could reproduce it.

@pujagani
Copy link
Contributor

I also don't see RetryRequest in the stack trace shared in #12558 (comment). Am I missing something?

@joerg1985
Copy link
Member

joerg1985 commented Sep 27, 2023

@pujagani here is some code to provoke this, the fallback (aka null) is taken as the operation itself failing.

  @Test
  void unexpectedExceptionsFallthrough() {
    HttpHandler handler = new RetryRequest().andFinally((HttpRequest request) -> {
      throw new UnsupportedOperationException();
    });

    Assertions.assertThrows(UnsupportedOperationException.class, () -> handler.execute(new HttpRequest(GET, "/")));
  }

@pujagani
Copy link
Contributor

Thank you @joerg1985 for having this handly. I am going to take a look.

@joerg1985
Copy link
Member

joerg1985 commented Sep 27, 2023

Actually, the Fallback is threadsafe https://failsafe.dev/javadoc/core/dev/failsafe/Fallback.html. I am not sure how to reproduce the NPE. It would be nice if we could reproduce it.

The library itself is fine, i think it's our custom code that makes this issue.

  @Test
  void thisIsNotARealTest() {
    HttpHandler handler1 = new RetryRequest().andFinally((HttpRequest request) -> {
      throw new TimeoutException();
    });

    // will write the value to RetryRequest.fallBackResponse
    Assertions.assertEquals(HTTP_GATEWAY_TIMEOUT, handler1.execute(new HttpRequest(GET, "/")).getStatus());

    // new RetryRequest() here for the second handler
    HttpHandler handler2 = new RetryRequest().andFinally((HttpRequest request) -> {
      throw new UnsupportedOperationException();
    });

    // will read the value of RetryRequest.fallBackResponse set by handler1
    Assertions.assertNull(handler2.execute(new HttpRequest(GET, "/")));
  }

PS: in the real world handler1 and handler2 are created in different threads.

@pujagani
Copy link
Contributor

@pujagani here is some code to provoke this, the fallback (aka null) is taken as the operation itself failing.

  @Test
  void unexpectedExceptionsFallthrough() {
    HttpHandler handler = new RetryRequest().andFinally((HttpRequest request) -> {
      throw new UnsupportedOperationException();
    });

    Assertions.assertThrows(UnsupportedOperationException.class, () -> handler.execute(new HttpRequest(GET, "/")));
  }

The null error here is due to the final result not available to the RetryPolicy to create a fallback. https://github.com/SeleniumHQ/selenium/blob/trunk/java/src/org/openqa/selenium/remote/http/RetryRequest.java#L100 The value at that line is null, hence the response object is not created.

@joerg1985
Copy link
Member

@pujagani here is some code to provoke this, the fallback (aka null) is taken as the operation itself failing.

  @Test
  void unexpectedExceptionsFallthrough() {
    HttpHandler handler = new RetryRequest().andFinally((HttpRequest request) -> {
      throw new UnsupportedOperationException();
    });

    Assertions.assertThrows(UnsupportedOperationException.class, () -> handler.execute(new HttpRequest(GET, "/")));
  }

The null error here is due to the final result not available to the RetryPolicy to create a fallback. https://github.com/SeleniumHQ/selenium/blob/trunk/java/src/org/openqa/selenium/remote/http/RetryRequest.java#L100 The value at that line is null, hence the response object is not created.

So the original exception is silently swallowed and the code will continue execution with a null response and failing with the NPE we see above.

@joerg1985
Copy link
Member

@titusfortner i just pushed the timeout less version in e8c77b8. So #12368 must not care about this timeout any more.

@pujagani
Copy link
Contributor

pujagani commented Sep 27, 2023

Yes, I think so. So now that it is narrowed down, I am going to try to fix our code tomorrow.

I also observed that if the ErrorFilter is added then the response contains the exception details. So the error is not swallowed.

HttpHandler handler =
        new RetryRequest()
            .andThen(new ErrorFilter())
            .andFinally(
                (HttpRequest request) -> {
                  throw new UnsupportedOperationException();
                });
    HttpResponse response = handler.execute(new HttpRequest(GET, "/"));

@joerg1985
Copy link
Member

I also observed that if the ErrorFilter is added then the response contains the exception details. So the error is not swallowed.

HttpHandler handler =
        new RetryRequest()
            .andThen(new ErrorFilter())
            .andFinally(
                (HttpRequest request) -> {
                  throw new UnsupportedOperationException();
                });
    HttpResponse response = handler.execute(new HttpRequest(GET, "/"));

The ErrorFilter should handle the exception and transform into a response, the RetryRequest should not retry it then. But i might be wrong :D

@pujagani
Copy link
Contributor

pujagani commented Sep 28, 2023

Yes! You are right. That is exactly what happens.

@titusfortner
Copy link
Member

closing based on 5f3f7d0, e8c77b8 & #12838

@Abhijith2893
Copy link
Author

Hey, updated to the latest 14.14.1 version for the hub but are still seeing the original issue. From this comment it seemed like the fix for the original issue is still being looked at ?

Attached is the DEBUG logs from version 4.14.1

DEBUG    2023-10-16 22:24:10.437  selenium.webdriver.remote.remote_connection:remote_connection.py:303 -> POST http://selenium-four-test-hub.service.eng.consul:4444/session/43a6ab97f39b0a6b8490c1191e90d65c/element/DB09BC8A8D795348EE8AAE387DC087C2_element_22/click {"id": "DB09BC8A8D795348EE8AAE387DC087C2_element_22"}
DEBUG    2023-10-16 22:30:10.460  urllib3.connectionpool:connectionpool.py:473 -> http://selenium-four-test-hub.service.eng.consul:4444 "POST /session/43a6ab97f39b0a6b8490c1191e90d65c/element/DB09BC8A8D795348EE8AAE387DC087C2_element_22/click HTTP/1.1" 500 3776
DEBUG    2023-10-16 22:30:10.461  selenium.webdriver.remote.remote_connection:remote_connection.py:319 -> Remote response: status=500 | data={
  "value": {
    "error": "unknown error",
    "message": "\u003cno message present in throwable>",
    "stacktrace": "java.lang.NullPointerException\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute0(JdkHttpClient.java:383)\n\tat org.openqa.selenium.remote.http.RetryRequest.lambda$apply$6(RetryRequest.java:93)\n\tat dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)\n\tat dev.failsafe.Functions.lambda$get$0(Functions.java:46)\n\tat dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)\n\tat dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)\n\tat dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)\n\tat dev.failsafe.internal.FallbackExecutor.lambda$apply$0(FallbackExecutor.java:51)\n\tat dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)\n\tat dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)\n\tat dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)\n\tat org.openqa.selenium.remote.http.RetryRequest.lambda$apply$7(RetryRequest.java:93)\n\tat org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:55)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:359)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:54)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:89)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:382)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\n"
  }
} | headers=HTTPHeaderDict({'content-length': '3776', 'Cache-Control': 'none', 'Content-Type': 'application/json; charset=utf-8'})
DEBUG    2023-10-16 22:30:10.461  selenium.webdriver.remote.remote_connection:remote_connection.py:346 -> Finished Request

We also noticed that in this version the subsequent requests were not timing out. Looking at the failure screenshot generated, it seems the click action was successful but the response being returned is 500 leading to test failure.

@pujagani
Copy link
Contributor

Thank you for trying the latest version. I am a little confused, are you still seeing timeouts? Based on the logs you are still seeing NPE. If only there was a way to reproduce this.

@joerg1985
Copy link
Member

I have one action point left in this, to not close the client based on a timeout.
But this might hide the root issue here.

I think a TimeoutException occurs here and the retry machanism is retring.
The client is closed after 4 minutes, even when the retry machanism is still running.
This makes the code fail with the NPE shown here, in the past this was not happening with the netty client.

@pujagani
Why is there a retry for the TimeoutException at all, i can understand to retry on a connection timeout.
But retring on a TimeoutException might send the command multiple times to the driver.
Or do i miss the way how this is prevented by the retry machanism?
Another point is the client has the same read timeout than the server.
So even if the retry is working the client has already reached the read timeout and will not read the response?

I think there is general topic with timeouts (pageLoadTimeout > default read timeout of the http client) too, but this another story...

@pujagani
Copy link
Contributor

I think a TimeoutException occurs here and the retry machanism is retring.
@Abhijith2893 Can you share your debug logs? If it is retrying then we should be able to see logs. It will help us narrow down the problem. Thank you!

@joerg1985
My understanding is that retry for TimeoutException is because there could be a chance that it took too long for the client (in this case Selenium) to get a response from the server(WebDriver), this can be due to serval reasons. We retry with the intent that the server can handle the request the next time we retry. We also cap the number of times we retry. We retry only if WebDriver cannot fulfil the request. So even if the request is sent multiple times, but not completed, then the command is not run multiple times.

Your second point makes sense, but if the client has reached the timeout, it retries again if the maximum retries are not exceeded then it will be able to get the response.

It is just a safety net to ensure that we don't respond with an error immediately and give the server a few chances to respond.

@joerg1985
Copy link
Member

@pujagani i think i have some code to provoke this, i will create a new issue for the click retry on timeout.

aguspe pushed a commit to aguspe/selenium that referenced this issue Oct 22, 2023
aguspe pushed a commit to aguspe/selenium that referenced this issue Oct 22, 2023
diemol added a commit that referenced this issue Nov 13, 2023
* fixed merge conflict

* [grid] use a URI as key to the map

* [java] make the json parsing exception text more helpful

* [java] fixed the get session url endpoint

* [java] use sessions.getUri when only the session uri is needed

* Revert "[java] make the json parsing exception text more helpful"

This reverts commit ce7cfc8.

* [java] format script

---------

Co-authored-by: Diego Molina <diemol@users.noreply.github.com>
Copy link

github-actions bot commented Dec 2, 2023

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked and limited conversation to collaborators Dec 2, 2023
@pujagani
Copy link
Contributor

pujagani commented Dec 6, 2023

I think this should be resolved via b5a2e11

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants