I have a HtmlUnit "script" that has been running for months, but started to fail about a week ago. Some changes were most probably made on the backend of the web site targeted by my script, but I don't have all the details unfortunately.

Basically the script throws an exception on the first attempt to access the main page. The code looks like this:

try (final WebClient webClient = new WebClient())
{
    webClient.getOptions().setRedirectEnabled(true);
    webClient.getCookieManager().setCookiesEnabled(true);
    HtmlPage loginPage = webClient.getPage("https://example.com/");
}

The exception thrown:
com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException: 421 Misdirected Request for https://example.com/

I am able to open that same page with curl using this command:

curl -L https://example.com

Same thing when I try with OKHttp client, it works and causes no error.

Since I could not find a solution with HtmlUnit, I have tried generate the redirected url using HttpClient:

private static Optional<String> getRedirectUrl() throws Exception
{
    final HttpClient client = HttpClient.newBuilder()
        .version(HttpClient.Version.HTTP_2)
        .followRedirects(HttpClient.Redirect.NEVER)
        .build();

    final HttpResponse<String> response = client.send(HttpRequest.newBuilder(new URI(EXAMPLE_SITE)).GET().build(), HttpResponse.BodyHandlers.ofString());

    return response.headers().firstValue("location");
}

But then I get a 403 Forbidden error when I attempt to access this redirected url using HtmlUnit (it works if I open the redirected URL in a standard browser).