Bom dia. Preciso criar um webcrawler que acesse uma página que exige uma autenticação por certificado digital. Para tal, foi sugerido o Selenium com PhantomJS. O problema é que a autenticação do certificado digital simplesmente não funciona.
Segue abaixo o código utilizado:
DesiredCapabilities capabilities = DesiredCapabilities.phantomjs();
capabilities.setJavascriptEnabled(true);
capabilities.setCapability("takesScreenshot", true);
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
ImmutableMap<String, String> commandLineArguments = ImmutableMap.<String, String>builder()
.put("web-security", "false")
.put("ssl-protocol", "any")
.put("ignore-ssl-errors", "true")
.put("webdriver-loglevel", "DEBUG")
.put("ssl-client-certificate-file", "C:\\temp\\cert.cer")
.put("ssl-client-key-file", "C:\\temp\\cert-key.key")
.put("ssl-client-key-passphrase", "1234")
.build();
String[] params = commandLineArguments.entrySet().stream()
.map(e -> String.format("--%s=%s", e.getKey(), e.getValue())).collect(Collectors.toList())
.toArray(new String[0]);
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, params);
WebDriver driver = new PhantomJSDriver(capabilities);
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
driver.navigate().to("https://www1c.siscomex.receita.fazenda.gov.br/siscomexImpweb-7/inicio.html");
System.out.println(driver.getTitle());
driver.quit();
Eis as informações exibidas no console:
dez 11, 2017 8:34:49 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFORMAÇÕES: executable: C:\Java\workspace-testes\poc-crawler\phantomjs.exe
dez 11, 2017 8:34:49 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFORMAÇÕES: port: 9648
dez 11, 2017 8:34:49 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFORMAÇÕES: arguments: [--web-security=false, --ssl-protocol=any, --ignore-ssl-errors=true, --webdriver-loglevel=DEBUG, --ssl-client-certificate-file=C:\temp\cert.cer, --ssl-client-key-file=C:\temp\cert-key.key, --ssl-client-key-passphrase=1234, --webdriver=9648, --webdriver-logfile=C:\Java\workspace-testes\poc-crawler\phantomjsdriver.log]
dez 11, 2017 8:34:49 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFORMAÇÕES: environment: {}
[DEBUG - 2017-12-11T10:34:54.104Z] Config - config.init - {"ip":"127.0.0.1","port":"9648","hub":null,"proxy":"org.openqa.grid.selenium.proxy.DefaultRemoteProxy","version":"","logFile":null,"logLevel":"DEBUG","logColor":false}
[INFO - 2017-12-11T10:34:54.113Z] GhostDriver - Main - running on port 9648
[DEBUG - 2017-12-11T10:34:54.535Z] RouterReqHand - _handle - {"headers":{"Accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","Connection":"keep-alive","Host":"localhost:9648","User-Agent":"Java/1.8.0_151"},"httpVersion":"1.1","method":"GET","url":"/status","urlParsed":{"anchor":"","query":"","file":"status","directory":"/","path":"/status","relative":"/status","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/status","queryKey":{},"chunks":["status"]}}
[DEBUG - 2017-12-11T10:34:54.637Z] RouterReqHand - _handle - {"headers":{"Accept-Encoding":"gzip,deflate","Connection":"Keep-Alive","Content-Length":"1293","Content-Type":"application/json; charset=utf-8","Host":"localhost:9648","User-Agent":"Apache-HttpClient/4.5.3 (Java/1.8.0_151)"},"httpVersion":"1.1","method":"POST","post":"{\n \"desiredCapabilities\": {\n \"acceptSslCerts\": true,\n \"browserName\": \"phantomjs\",\n \"takesScreenshot\": true,\n \"javascriptEnabled\": true,\n \"phantomjs.cli.args\": [\n \"--web-security\\u003dfalse\",\n \"--ssl-protocol\\u003dany\",\n \"--ignore-ssl-errors\\u003dtrue\",\n \"--webdriver-loglevel\\u003dDEBUG\",\n \"--ssl-client-certificate-file\\u003dC:\\\\temp\\\\cert.cer\",\n \"--ssl-client-key-file\\u003dC:\\\\temp\\\\cert-key.key\",\n \"--ssl-client-key-passphrase\\u003d1234\"\n ],\n \"version\": \"\",\n \"platform\": \"ANY\"\n },\n \"requiredCapabilities\": {},\n \"capabilities\": {\n \"desiredCapabilities\": {\n \"acceptSslCerts\": true,\n \"browserName\": \"phantomjs\",\n \"takesScreenshot\": true,\n \"javascriptEnabled\": true,\n \"phantomjs.cli.args\": [\n \"--web-security\\u003dfalse\",\n \"--ssl-protocol\\u003dany\",\n \"--ignore-ssl-errors\\u003dtrue\",\n \"--webdriver-loglevel\\u003dDEBUG\",\n \"--ssl-client-certificate-file\\u003dC:\\\\temp\\\\cert.cer\",\n \"--ssl-client-key-file\\u003dC:\\\\temp\\\\cert-key.key\",\n \"--ssl-client-key-passphrase\\u003d1234\"\n ],\n \"version\": \"\",\n \"platform\": \"ANY\"\n },\n \"requiredCapabilities\": {},\n \"alwaysMatch\": {\n \"browserName\": \"phantomjs\"\n },\n \"firstMatch\": []\n }\n}","url":"/session","urlParsed":{"anchor":"","query":"","file":"session","directory":"/","path":"/session","relative":"/session","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/session","queryKey":{},"chunks":["session"]}}
[INFO - 2017-12-11T10:34:54.647Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - page.settings - {"XSSAuditingEnabled":false,"javascriptCanCloseWindows":true,"javascriptCanOpenWindows":true,"javascriptEnabled":true,"loadImages":true,"localToRemoteUrlAccessEnabled":false,"userAgent":"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1","webSecurityEnabled":false}
[INFO - 2017-12-11T10:34:54.647Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - page.customHeaders: - {}
[DEBUG - 2017-12-11T10:34:54.647Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - Session.desiredCapabilities - {"acceptSslCerts":true,"browserName":"phantomjs","takesScreenshot":true,"javascriptEnabled":true,"phantomjs.cli.args":["--web-security=false","--ssl-protocol=any","--ignore-ssl-errors=true","--webdriver-loglevel=DEBUG","--ssl-client-certificate-file=C:\\temp\\cert.cer","--ssl-client-key-file=C:\\temp\\cert-key.key","--ssl-client-key-passphrase=1234"],"version":"","platform":"ANY"}
[INFO - 2017-12-11T10:34:54.647Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - Session.negotiatedCapabilities - {"browserName":"phantomjs","version":"2.1.1","driverName":"ghostdriver","driverVersion":"1.2.0","platform":"windows-10-32bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
[INFO - 2017-12-11T10:34:54.647Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: ed3a4300-de5e-11e7-81a9-f74adf80e087
dez 11, 2017 8:34:54 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFORMAÇÕES: Detected dialect: OSS
[DEBUG - 2017-12-11T10:34:54.685Z] RouterReqHand - _handle - {"headers":{"Accept-Encoding":"gzip,deflate","Connection":"Keep-Alive","Content-Length":"30","Content-Type":"application/json; charset=utf-8","Host":"localhost:9648","User-Agent":"Apache-HttpClient/4.5.3 (Java/1.8.0_151)"},"httpVersion":"1.1","method":"POST","post":"{\"type\":\"implicit\",\"ms\":30000}","url":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/timeouts","urlParsed":{"anchor":"","query":"","file":"timeouts","directory":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/","path":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/timeouts","relative":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/timeouts","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/timeouts","queryKey":{},"chunks":["session","ed3a4300-de5e-11e7-81a9-f74adf80e087","timeouts"]}}
[DEBUG - 2017-12-11T10:34:54.691Z] SessionReqHand - _postTimeout - {"type":"implicit","ms":30000}
[DEBUG - 2017-12-11T10:34:54.701Z] RouterReqHand - _handle - {"headers":{"Accept-Encoding":"gzip,deflate","Connection":"Keep-Alive","Content-Length":"84","Content-Type":"application/json; charset=utf-8","Host":"localhost:9648","User-Agent":"Apache-HttpClient/4.5.3 (Java/1.8.0_151)"},"httpVersion":"1.1","method":"POST","post":"{\"url\":\"https://www1c.siscomex.receita.fazenda.gov.br/siscomexImpweb-7/inicio.html\"}","url":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/url","urlParsed":{"anchor":"","query":"","file":"url","directory":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/","path":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/url","relative":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/url","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/url","queryKey":{},"chunks":["session","ed3a4300-de5e-11e7-81a9-f74adf80e087","url"]}}
[DEBUG - 2017-12-11T10:34:54.701Z] SessionReqHand - _postUrlCommand - Session 'ed3a4300-de5e-11e7-81a9-f74adf80e087' is about to load URL: https://www1c.siscomex.receita.fazenda.gov.br/siscomexImpweb-7/inicio.html
[DEBUG - 2017-12-11T10:34:54.702Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - page.onResourceRequested - {"headers":[{"name":"Accept","value":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1"}],"id":1,"method":"GET","time":"2017-12-11T10:34:54.699Z","url":"https://www1c.siscomex.receita.fazenda.gov.br/siscomexImpweb-7/inicio.html"}
[DEBUG - 2017-12-11T10:34:54.702Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _oneShotCallback - onLoadStarted
[DEBUG - 2017-12-11T10:34:54.712Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _execFuncAndWaitForLoadDecorator - Page Loading in Session: true
[DEBUG - 2017-12-11T10:34:54.816Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _execFuncAndWaitForLoadDecorator - Page Loading in Session: true
[DEBUG - 2017-12-11T10:34:54.916Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _execFuncAndWaitForLoadDecorator - Page Loading in Session: true
[DEBUG - 2017-12-11T10:34:55.027Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _execFuncAndWaitForLoadDecorator - Page Loading in Session: true
[DEBUG - 2017-12-11T10:34:55.153Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _execFuncAndWaitForLoadDecorator - Page Loading in Session: true
[DEBUG - 2017-12-11T10:34:55.187Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - page.onResourceError - {"errorCode":2,"errorString":"Connection closed","id":1,"status":null,"statusText":null,"url":"https://www1c.siscomex.receita.fazenda.gov.br/siscomexImpweb-7/inicio.html"}
[DEBUG - 2017-12-11T10:34:55.187Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - page.onResourceReceived - {"contentType":null,"headers":[],"id":1,"redirectURL":null,"stage":"end","status":null,"statusText":null,"time":"2017-12-11T10:34:55.186Z","url":"https://www1c.siscomex.receita.fazenda.gov.br/siscomexImpweb-7/inicio.html"}
[DEBUG - 2017-12-11T10:34:55.188Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _oneShotCallback - onLoadFinished
[DEBUG - 2017-12-11T10:34:55.188Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _execFuncAndWaitForLoadDecorator - onLoadFinished: fail
[DEBUG - 2017-12-11T10:34:55.269Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _execFuncAndWaitForLoadDecorator - Page Loading in Session: false
[DEBUG - 2017-12-11T10:34:55.269Z] SessionReqHand - _SuccessHandler - status: fail
[DEBUG - 2017-12-11T10:34:55.285Z] RouterReqHand - _handle - {"headers":{"Accept-Encoding":"gzip,deflate","Cache-Control":"no-cache","Connection":"Keep-Alive","Host":"localhost:9648","User-Agent":"Apache-HttpClient/4.5.3 (Java/1.8.0_151)"},"httpVersion":"1.1","method":"GET","url":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/title","urlParsed":{"anchor":"","query":"","file":"title","directory":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/","path":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/title","relative":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/title","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087/title","queryKey":{},"chunks":["session","ed3a4300-de5e-11e7-81a9-f74adf80e087","title"]}}
[DEBUG - 2017-12-11T10:34:55.288Z] RouterReqHand - _handle - {"headers":{"Accept-Encoding":"gzip,deflate","Connection":"Keep-Alive","Host":"localhost:9648","User-Agent":"Apache-HttpClient/4.5.3 (Java/1.8.0_151)"},"httpVersion":"1.1","method":"DELETE","url":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087","urlParsed":{"anchor":"","query":"","file":"ed3a4300-de5e-11e7-81a9-f74adf80e087","directory":"/session/","path":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087","relative":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/session/ed3a4300-de5e-11e7-81a9-f74adf80e087","queryKey":{},"chunks":["session","ed3a4300-de5e-11e7-81a9-f74adf80e087"]}}
[DEBUG - 2017-12-11T10:34:55.288Z] Session [ed3a4300-de5e-11e7-81a9-f74adf80e087] - _deleteClosingPage
[DEBUG - 2017-12-11T10:34:55.291Z] RouterReqHand - _handle - {"headers":{"Accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","Connection":"keep-alive","Host":"localhost:9648","User-Agent":"Java/1.8.0_151"},"httpVersion":"1.1","method":"GET","url":"/shutdown","urlParsed":{"anchor":"","query":"","file":"shutdown","directory":"/","path":"/shutdown","relative":"/shutdown","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/shutdown","queryKey":{},"chunks":["shutdown"]}}
[INFO - 2017-12-11T10:34:55.291Z] ShutdownReqHand - _handle - About to shutdown
Alguém pode dar uma força. Eu agradeço.