based on JavaScript execution exeption question on Stackoverflow
HtmlUnit Version: 2.50.0
During getPage call of webpage flashscore.com, i got following exceptions
2021-07-07 08:46:05.408 WARN 4828 --- [nio-8080-exec-1] c.g.htmlunit.IncorrectnessListenerImpl : Obsolete content type encountered: 'text/javascript'.
2021-07-07 08:46:05.564 ERROR 4828 --- [nio-8080-exec-1] c.g.h.j.DefaultJavaScriptErrorListener : Error during JavaScript execution
com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot find function entries in object function Object() { [native code] }. (script in https://www.flashscore.com/unsupported/ from (31, 9) to (53, 10)#35)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:949) ~[htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:598) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:487) ~[htmlunit-core-js-2.50.0.jar:na]
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.callSecured(HtmlUnitContextFactory.java:353) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:829) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:805) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:796) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:942) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.ScriptElementSupport.executeInlineScriptIfNeeded(ScriptElementSupport.java:378) ~[htmlunit-2.50.0.jar:2.50.0]
I've tried with two different classes and problem still occur.
@PostMapping("/startScraping")
public ResponseEntity<FlashScraper> startScraping(@NonNull @RequestBody FlashScraper flashScraper) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
logger.info("startScraping request incomming");
logger.info("Call URL: " + flashScraper.getScrapeUrl());
String url = "https://flashScore.com";
try (final WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED)) {
HtmlPage page = webClient.getPage(url);
webClient.waitForBackgroundJavaScript(3_000);
System.out.println();
System.out.println();
System.out.println("----------------");
System.out.println(page.asNormalizedText());
System.out.println("----------------");
}
return new ResponseEntity(flashScraper, HttpStatus.OK);
}
@PostMapping("/startScraping")
public ResponseEntity<FlashScraper> startScraping(@NonNull @RequestBody FlashScraper flashScraper) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
logger.info("startScraping request incomming");
logger.info("Call URL: " + flashScraper.getScrapeUrl());
final WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.waitForBackgroundJavaScriptStartingBefore(1000);
HtmlPage scrapePage = webClient.getPage(flashScraper.getScrapeUrl());
webClient.waitForBackgroundJavaScript(3000);
System.out.println(scrapePage.getByXPath("//*[@id=\"g_25_rwPxTVj1\"]"));
return new ResponseEntity(flashScraper, HttpStatus.OK);
}
Pay now to fund the work behind this issue.
Get updates on progress being made.
Maintainer is rewarded once the issue is completed.
You're funding impactful open source efforts
You want to contribute to this effort
You want to get funding like this too