Skip to content

Improve stability of Selenium tests #5367

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 8, 2025
Merged

Conversation

Avery-Dunn
Copy link
Contributor

@Avery-Dunn Avery-Dunn commented Jun 27, 2025

Attempts to resolve some of the flaky tests described in #5357

One of the most common failure occurs during interactive tests, occasionally there's a timeout when trying to create the web driver for Selenium:

Test method Microsoft.Identity.Test.Integration.SeleniumTests.InteractiveFlowTests.Interactive_AADAsync threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:52547/session timed out after 120 seconds.

Stack trace
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at OpenQA.Selenium.Remote.HttpCommandExecutor.<MakeHttpRequest>d__36.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

// ...


at OpenQA.Selenium.Chrome.ChromeDriver..ctor(String chromeDriverDirectory, ChromeOptions options)
[at Microsoft.Identity.Test.Integration.Infrastructure.SeleniumExtensions.CreateDefaultWebDriver() in /_/tests/Microsoft.Identity.Test.Integration.NetCore/Infrastructure/SeleniumExtensions.cs:line 41](https://identitydivision.visualstudio.com/IDDP/_git/9a0dc2f2-2cc8-42ef-a2af-2b659b9c4530?path=%2F_%2Ftests%2FMicrosoft.Identity.Test.Integration.NetCore%2FInfrastructure%2FSeleniumExtensions.cs&version=GBmain&_a=contents&line=41&lineEnd=42&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Microsoft.Identity.Test.Integration.Infrastructure.SeleniumWebUI.InitDriverAndGoToUrl(String url) in /_/tests/Microsoft.Identity.Test.Integration.NetCore/Infrastructure/SeleniumWebUI.cs:line 95](https://identitydivision.visualstudio.com/IDDP/_git/9a0dc2f2-2cc8-42ef-a2af-2b659b9c4530?path=%2F_%2Ftests%2FMicrosoft.Identity.Test.Integration.NetCore%2FInfrastructure%2FSeleniumWebUI.cs&version=GBmain&_a=contents&line=95&lineEnd=96&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Microsoft.Identity.Test.Integration.Infrastructure.SeleniumWebUI.<SeleniumAcquireAuthAsync>d__10.MoveNext() in /_/tests/Microsoft.Identity.Test.Integration.NetCore/Infrastructure/SeleniumWebUI.cs:line 104](https://identitydivision.visualstudio.com/IDDP/_git/9a0dc2f2-2cc8-42ef-a2af-2b659b9c4530?path=%2F_%2Ftests%2FMicrosoft.Identity.Test.Integration.NetCore%2FInfrastructure%2FSeleniumWebUI.cs&version=GBmain&_a=contents&line=104&lineEnd=105&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)

This PR attempts to improve the stability of the webdriver creation by implementing some simple retry behavior, and giving it a much shorter timeout period before it gives up and tries again.

This should reduce the number of failed pipeline runs, and overall make the interactive tests run a bit quicker.

@Avery-Dunn Avery-Dunn requested a review from a team as a code owner June 27, 2025 23:04
@Avery-Dunn Avery-Dunn requested review from gladjohn and trwalke July 7, 2025 15:07
Copy link
Contributor

@neha-bhargava neha-bhargava left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tried to run this PR through one branch build? If not, I would recommend running it before merging this PR.

@Avery-Dunn
Copy link
Contributor Author

Avery-Dunn commented Jul 8, 2025

Have you tried to run this PR through one branch build? If not, I would recommend running it before merging this PR.

Yeah, if you look at the run history around June 25-26 you'll see a bunch of manual runs by me. Here's an example, and after about 10 runs I didn't see the flaky tests fail: https://identitydivision.visualstudio.com/IDDP/_build/results?buildId=1498688&view=results

@Avery-Dunn Avery-Dunn merged commit 39c0eb0 into main Jul 8, 2025
10 checks passed
@Avery-Dunn Avery-Dunn deleted the avdunn/flaky-tests-fix branch July 8, 2025 17:18
@gladjohn gladjohn added this to the 4.74.0 milestone Jul 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants