Troubleshooting WebTests

Issue Tracker

When in trouble, you may want to visit the WebTest GitHub Issue Tracker.

Out of Memory

ANT may eat up a lot of memory when started for the use with Canoo WebTest. You can give it more space to live in by specifying a higher "maximum heap size" through the environment variable ANT_OPTS, e.g.:

ANT_OPTS=-Xmx128m

See the ANT documentation for more details.

In order to save memory avoid calling the taskdef more than once. When your test suite grows beyond some thousand steps consider using more than one file for collecting the reports.

java.lang.VerifyError

There may be non-compliant ANT version that interferes with the ant.jar provided in <WEBTESTHOME>/lib.

Consider to copy <WEBTESTHOME>/lib/ant.jar to <ANT_HOME>/lib.

While installTest.xml reports this error on some occasions, it cannot catch all possible cases.

Parsing issues

If it happens for instance that Canoo WebTest shows an error indicating that it did not find a button that was referenced in a test although the button is clearly visible in the browser (and works as well), one of the most common reasons is incorrectly formatted HTML that isn't understood by the parser.

"Normal" browsers are able to cope with really messy HTML code. WebTest too.. but not to the same extend.

An HTML Validator will be able to tell you if your HTML code is valid or not and indicate how to fix it.

To "see" the current state of a page (the DOM) in memory, you can dump it to the console with a small step:

JavaScript Errors

There are two types of JavaScript errors: "real" JavaScript errors occurring as well in a "real" browser and JavaScript errors occurring only in WebTest due to insufficient/incorrect reproduction of the simulated browser's behavior.

If your test fails due to a "real" JavaScript error, it is wise to fix your application. If you can't do it, you can choose to ignore JavaScript errors.

Following steps may help to understand the cause of a JavaScript error reported by WebTest:

Verify in the "real" browser

The quality of the browser simulation including JavaScript execution is already very good and it often happens that WebTest reports exactly the same JavaScript errors than the simulated browser (sometimes even with a better precision!). To detect if you have encountered a real JavaScript error, you need to do exactly the same actions in your browser than the ones done in your webtest. This requires high concentration! Be sure to really click the same links and buttons as your WebTest and not any other than may appear in the page (even with same label). In Firefox you can see JavaScript errors for instance in the JavaScript console. Other browsers have similar ways to acces to JavaScript errors.

Ugrade to latest WebTest and HtmlUnit version

If no error occurs in the real browser that you want to simulate with your webtest, then it probably means that WebTest's behavior is incorrect. Be sure to use the latest WebTest build (replacing as well the HtmlUnit version contained in WebTest by the latest HtmlUnit snapshot when possible): it's possible that the problem you encounter has already been fixed.

Report the problem

If the problem still occurs with the most recent versions of WebTest (and HtmlUnit) then it means that you've hint a new problem. In this case, please report it. Problems are sometimes fixed within a very short time. A look at HtmlUnit's Submitting JavaScript bugs documentation may help to find the best way to report a bug.

Button found more than once

For clicking on a button or a link, Canoo WebTest needs to locate that button or that link on the page. The above message occurs if the reference is not unique, which generally indicates an error in the test.

Consider using the name attribute or a locator element to create a unique reference. Refer to the syntax description and the samples on how to do this.

If the message still shows up, although the reference should be unique, it is likely that the page's HTML source is invalid. A common error is improper nesting of form elements with table rows and cells, they cannot be mixed. "Parsing issues" may be helpful.

Inspect HTTP traffic

It is sometimes interesting to inspect the HTTP traffic to compare the requests sent by WebTest and the responses received from the server(s) when WebTest is running with the ones of the "real" browser being simulated. Caution: remember that WebTest will do less requests than a normal browser, for instance it won't download images and stylesheets unless it detects that they play a role in the JavaScript execution.

There are different solutions to view the HTTP traffic, for instance these ones:

Use DebuggingWebConnection in WebTest

Following code instruments the web connection of the underlying WebClient. This will generate a report of the HTTP traffic in the sub-folder named mytest of your temporary directory for the rest of your webtest:

Wrap HTMLUnit web connection to track HTTP traffic

<groovy>
import com.gargoylesoftware.htmlunit.util.DebuggingWebConnection as DWC

def w = step.context.webClient
w.webConnection = new DWC(w.webConnection, "mytest")
</groovy>

Use Firebug in Firefox

The Network monitoring features of the Firebug extension for Firefox allow to track all HTTP requests.

Use a proxy both for WebTest and your "normal" browser

An other solution to track HTTP traffic both for WebTest and for a "normal" browser is to use an HTTP proxy that can show you the request and responses. Different proxies are available for this purpose. For Java developers Paros may be interesting as it is Open Source and written in Java.

Getting more information

A test fails, but you don't know why? If that is because of the ANT structure of the tests, unresolved properties or unexpected directory handling, you should consider running your ANT tests with the -v option. This gives additional verbose information about target sequence, unresolved properties and attempts to override property values.

If it is a problem within the HTML, try to use the reporting as described in "Out of Memory". It is often useful to inspect your page with an HTML Validator.

Debugging WebTest

The last ditch attempt to troubleshoot Canoo WebTest is to go for Java debugging on the source code level. You are likely to need not only the Canoo WebTest sources but also the source of the packages that Canoo WebTest is built on. That is ANT, and HtmlUnit. You may need even more, e.g. when stepping into the HTML parsing. But if you are so experienced, you probably know how to do this.

You need a Java debugger, a class to start the debugger on, the command line parameters and the proper classpath and sourcepath. We cannot help you with the debugger itself. You probably have one within your IDE. The JDK contains the jdb, but that one is not very comfortable. The remaining parameters can be determined from your ant starter script that resides under <ANT_HOME>/bin. Put all jars on the CLASSPATH that reside under <ANT_HOME>/lib and <WEBTESTHOME>/lib. Put the sources on the -sourcepath. Provide the option -Dant.home="<ANT_HOME>" and start the debugger on the class org.apache.tools.ant.Main. Provide the command line parameters -buildfile <file> and <target> as necessary.

When debugging, a good thing to look for is implicit tags that the parser inserts into incomplete HTML.

Tests get messy

If your tests get messy, this is mostly because they contain duplication. Avoid this at any costs.

Put all the references that occur more than once into an ANT property. Consider putting shared properties into a properties file. Use the ant and antcall tasks to extract reusable logic into subbuild files and subtargets. Use XML entities for textual includes. See the samples on how to do this. Also consider using Macros - see the ANT documentation for more details.

Keep your targets short and avoid putting too many webtests into one target.

Make use of the description attribute for main targets and use the description attribute for test steps.

Weird Errors running your script

If you get weird errors when running your script inside an IDE it could be that your IDE is using a custom ant runner. For some reason this seems to sometimes cause a conflict. If you find yourself in this situation, try to configure your IDE to use a standard ANT runner.

Possible technical explanation: Canoo WebTest relies on HtmlUnit which parses HTML using NekoHTML. NekoHTML is set up as a custom ContentHandler for the Xerces XML Parser. Your custom ANT runner may also have set itself up as a custom ContentHandler. For some reason this seems to sometimes cause a conflict.

As an example, in IntelliJ, if you run a script which uses the standard embedded ant then if you have a step which parses HTML you might see the following exception thrown: java.lang.AbstractMethodError: org.cyberneko.html.HTMLScanner.getCharacterOffset()I. This will occur if you for instance use an invoke step. The workaround is simply to configure in your ant build settings to use an external ant rather than the embedded one.