Browsermob Programmatic Proxy for Automated Testing

The web applications we build and test these days are far from a simple collection of HTML pages designed and coded from scratch by a small team of people. Some of the functionalities we see on sites are not even developed by the engineers coding the web page, but actually were created by different third parties. Such third party services are thereafter embedded wherever they are needed.

You can have third parties for adverts, login, comments, video content, statistics and so on. These are usually embedded in a website by calling a script from the third party vendor. For example, the following code embeds YouTube API functionality onto your webpage:

<script src="https://www.youtube.com/iframe_api"></script>

One common problem a website encounters when embedding many third parties is slow loading speed, due to the third parties either not responding or responding very slowly. This is a problem when running automation tests that load the web page, because it can result in long running times or even false negative results.

To solve this problem, we can use a programmatic proxy to block the calls to and from the third parties we select. The following code examples will show how we can use the Browsermob Proxy with Capybara and Ruby to filter out unwanted HTTP calls.

Installing the Browsermob Proxy Pack

From the command line, install the Browsermob Proxy Ruby gem

gem install browsermob-proxy

Download the Browsermob Proxy tool, available here, and unpack it into your automation project folder like so:

Starting the Proxy from the Code and Defining a Blacklist

First, we have to reference the Browsermob Proxy library we’ve installed with gem.

require 'browsermob/proxy'

Then we kick off the Browsermob by creating a new proxy object.

server = BrowserMob::Proxy::Server.new('./resources/browsermob-proxy-2.0.0/bin/browsermob-proxy', :port => 8181)  # port number for ruby to communicate with the browsermob tool
server.start
proxy = server.create_proxy(5454)    # port number for online proxy

After this, we need to create a blacklist in which we introduce all of the URLs that we want to block, with each of them in a regular expression. We also need to define the response code that the proxy should send when a request comes for a blacklisted site – in this case we’ll use 404.

proxy.clear_blacklist
proxy.blacklist(/^.+youtube.com\/.+$/, 404)

Once we define the full blacklist, we will need to specify a filename for the Browsermob Proxy to log all of the traffic.

proxy.new_har "my_har"

Now we can save the proxy object in a Selenium profile that will be passed to the browser running the test.

profile = Selenium::WebDriver::Firefox::Profile.new
profile.proxy= proxy.selenium_proxy
Capybara::Selenium::Driver.new(app, :browser => driver.browser, 
:http_client => client, :profile => profile)

After defining all of the actions that our test will perform, we will need to save the HAR file and close the proxy while when cleaning up the test environment.

proxy.har.save_to "./resources/hars/my_har.har"
proxy.close

Filtering out third parties that are not needed by a test can be very beneficial: we can better test the application in isolation and increase the speed and reliability of the test. The Browsermob Proxy is a very powerful tool when attempting this.