Skip to content
July 18, 2013 / selenium34

URL Testing

I’m excited for Java 8 having closures. It certainly reduces the amount of code one needs to write, callbacks are as easy as can be, and it’s nice to have other ways to solve a problem. We wished to bring some tests over from our previous project which was using TestNG and data providers. I certainly could have just moved it over, and called it a day, however I felt it was time to look a little bit closer at utilizing closures within groovy.

The first thing we did was create a URLRunner class. It’s goal is to open a connection to the URL input, and then verify that it was able to retrieve the data successfully (for us, this is any code under 300).

class URLRunner implements Runnable {
  def url
  def valid
  public void run() {
    try {
      url.toURL().openConnection().with {
        connectTimeout = TimeUnit.SECONDS.toMillis(10)
        valid = responseCode < 300
    } catch(e) {
      valid = false

From there we needed a simple way to test these URLs. Again, I could have stuck with what we had, but it was good to tinker. We created a URLExecutorService to handle the threading.

class ULRExecutorService {
  def pool = Executors.newFixedThreadPool(30)
  def throwException = true

  def testUrlCollection(collection) {
    /* A lot happens in these lines
       1) we cast the collection to a set to remove duplicates
       2) we strip urls we located which don't begin with http or are blank
       3) we iterate over that collection, creating a new URLRunner for each url we located
       4) we submit the url to the pool
       5) we add the URLRunner to our list (we injected an empty list as our starting value)
    def results = stripNonURLData(collection as Set).inject([]) { list, url ->
      def u = new URLRunner(url:url)
      list << u
    def timeout = 10 * results.size()
    pool.awaitTermination(timeout, TimeUnit.SECONDS)
    def failed = results.findAll { !it.valid }
    def sb = new StringBuilder("Failed URLS\n")
    failed.each { sb.append(failed.url).append("\n") }
    if (throwException) {
      assert failed.empty

  def stripNonURLData(collection) {
    collection.findAll { value -> 
      StringUtils.isNotBlank(value) && StringUtils.startsWith(StringUtils.strip(value), "http")

You’re probably wondering why we have the option to throw exceptions in the Executor.  In certain cases we don’t mind if a link is dead — editors are people, they can make mistakes. We don’t want to fail a build if they have a dead link, so when we test all the links on a page, we let those things slide, and simply emit the warning message.

If you’re interested, this is what the test for checking the links on a page would look like:

class LinkCheckingTest {

  def executorService = new URLExecutorService(throwException:false)  

  def "test all links on the page"() {
    def doc = Jsoup.connect(baseUrl + "path/to/page.html").get() //We get the baseUrl from the geb configuration, this test simply doesn't run in a browser
    def elements = doc.getElementsByTag("a")
    def urls = []
    elements.each { urls << it.attr("abs:href") } //This should alleviate the check above with links not starting with http

We have other tests to verify if javascript / css are on the page, where we will fail the build if those files are not located.


There you have it, in under 50 lines of code we created a Thread pool, located all the links on a page, opened a straight http connection to the resource(s), and verified they loaded correctly. Granted, it is nearly the same number of lines we had with our data provider / TestNG version, but we do get a huge win. We can easily modify this to use a dynamic ThreadPool size based on environment — we don’t have that luxury with TestNG, it’s an annotation value, which needs to be set at compile time. I’m guessing we aren’t the only people whose lower environments hate to be pounded with that many threads…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: