WebDriver lacks HTTP response header and status code methods

[lukeis]

Originally reported on Google Code with ID 141

It would be useful to have methods to read the HTTP status code and headers from an
HTTP 
response.  HtmlUnit already has this functionality, but it's not been brought through
to WebDriver.

Reported by stephendv on 2009-02-01 16:05:44

[lukeis]

Issue 152 has been merged into this issue.

Reported by simon.m.stewart on 2009-09-14 17:42:11

[lukeis]

good feature

Reported by risticnemanja on 2009-10-16 13:27:22

[lukeis]

good feature

Reported by risticnemanja on 2009-10-16 13:27:31

[lukeis]

We will not be adding this feature to the WebDriver API as it falls outside of our current

scope (emulating user actions).

Reported by jmleyba on 2009-12-17 17:49:33

• Status changed: WontFix

[lukeis]

This would make WebDriver unusable for testing.
Selenium is supporting this feature, so I guess I am stuck with selenium forever :)

Reported by ivanjanic on 2009-12-17 18:23:32

[lukeis]

Adding access for status code and response headers would be a really good addition.


If these features will not be added, then WebDriver cannot be regarded as a full web
testing tool, but merely a wrapper for various HTTP/S interaction APIs.

Reported by timo.westkamper on 2010-02-09 10:12:25

[lukeis]

"Emulating user actions" might be useful for a generic browser driver, but for a 
TESTING tool it will be necessary to read the status code, headers and cookie values.

This is needed for performing non-functional automated tests, e.g.:
- Does the logon form always result in a 302 redirect?
- Is the session cookie set before logon, or after (to identify session fixation attacks)?
- Is the page with my bank details cached or not?

Until these features are added, I too cannot use webdriver for testing.

Reported by stephendv on 2010-02-09 10:21:41

[lukeis]

I appreciate having a clean API, but this is honestly something that is very
important in a lot of my test cases so I hope it gets fixed.

Reported by ponzao on 2010-02-10 08:46:24

[lukeis]

yes agree with above commnets. For a web testing tool its is imperative to be able to
read HTTP headers.

Reported by pnakhat on 2010-03-05 09:16:04

[lukeis]

Issue 404 has been merged into this issue.

Reported by dawagner on 2010-03-05 11:41:07

[lukeis]

For the time being we are standing by our decision not to add this feature.  If we attempt
to make WebDriver the silver bullet of web testing, then it will suffer in overall
quality.

A few points to consider:

1. driver.get(url) blocks until the browser has loaded the page.  In the case of a
login redirect, what status/headers would you want returned?  By the semantics of the
API, we would 
return the response for the final loaded page.  You would end up with a 200 instead
of the 302 you're looking for.

2. WebDriver does support checking cookies, just not at the HTTP protocol level.  Take
a look at the WebDriver.Options interface:
http://selenium.googlecode.com/svn/trunk/docs/api/java/org/openqa/selenium/WebDriver.Options.html

3. If you want to test HTTP request/response headers, then write an appropriately sized
test and test those directly.  All you need is a simple HTTP client library (and as
you mentioned, 
HtmlUnit already offers some of these capabilities).  If you really want to combine
the two levels of testing, setup a proxy for your browsers so you can capture net traffic
and verify 
headers there.

Reported by ashley.leyba on 2010-03-10 12:41:45

[lukeis]

This means that the only reason is "it falls outside of our current scope (emulating
user actions)".

I find it a bit strange. Why doesn't this apply to managing cookies?

Additionally if in the "GettingStarted" Wiki page you write:
"WebDriver is a tool for automating testing web applications, and in particular to
verify that they work as expected". The possibility to access response code is very
useful for this purpose.

Reported by mguillemot@yahoo.fr on 2010-03-10 12:53:43

[lukeis]

In my case I'm trying to test that the last step in a flow, and that is supposed to
generate a PDF. I won't try to validate the contents, but I do need to check that the
page isn't an error. The PDF generation I can test at a different level.

Now, I could implement this with HtmlUnit or something, but that basically means that
my nicely reusable Page objects (which all use WebDriver) are worhthless. I'd be
re-implementing a lot of the steps that WebDriver already fixed for me in such a nice
way.

Hope you'll reconsider.

Reported by anders.r.sveen on 2010-03-17 12:59:22

[lukeis]

This feature is highly needed. When I call get() on a URL, I would like to know if the
correct page was loaded or if it was a 404. Currently I can only get page source and
look for the 404 string but it is highly dependent of underlying server (because I
don't have a custom 404 page in my webapp).

Reported by glaurung.aubrane on 2010-08-25 20:53:34

[lukeis]

Please reconsider this issue.  
@jmleyba/ashley :-)

In case #1 it is reasonable for Webdriver to not be very verbose with redirects, etc.
 I would expect to be able to see the HTTP headers / status code of the rendered page
that was returned by driver.get(url)

Testers frequently use tools like Firebug, HttpFox, Web Developer Toolbar (Firefox),
iehttpheaders (IE), the built in developer tools in Chrome, and Safari Web Inspector
to inspect headers from the browser.

I don't think that looking at headers of the rendered page is outside the scope of
a browser driver used as a testing tool.

Reported by colin.harrington on 2010-08-26 14:22:14

[lukeis]

I also find it crucial that WebDriver exposes the HTTP status code of the rendered page.
The same way you can get the raw, HTML by calling getPageSource() on the driver, or
getCurrentUrl(), you should be able to call getResponseCode(). How are the first two
methods so different to the last one here? 

If we apply your logic for not adding a method to get the HTTP status code, then you
also shouldn't add getPageSource(), since any data on the page should be extracted
from the DOM via the driver.

Reported by npellow on 2010-09-07 05:14:46

[lukeis]

This is a crucial feature amiss in the WebDriver API today and doing it with a light-weight
HTTP-only library doesn't cut it as what WebDriver does through its browser and HtmlUnit
integration is so much more than what happens at the HTTP level and thus impossible
(or at least, very, very hard) to replicate in a "lightweight" HTTP library.

For instance, when you submit a form, you would like to know if the resulting page
returns HTTP Status Code 400 because of bad content within the posted content or 200
if everything went okay. Testing this when the form itself relies on JavaScript that
might alter the form values before they are posted makes it virtually impossible to
test this in an HTTP-only library.

The reason we use WebDriver is be able to test the functionality as close to the user
experience as possible, and whether you want to admit it or not, HTTP status codes
are so close to the user experience that it makes no sense to hide them in the API.

Most users know what "404 Not Found" means and this is the response they get when browsing
to a page that doesn't exist in most cases. While the user doesn't look at the actual
HTTP status code, but an HTML representation of it, he is exposed to the status code
while WebDriver itself doesn't expose it to the test that is supposed to carry out
the same action as a regular user.

If you don't want to do anything complicated here, then please at least just expose
the Request and Reponse as strings so we can parse and make sense of it ourselves.
Hiding this information is limiting the ways WebDriver can be used, especially when
it's testing through HtmlUnit where everything happens programmatically anyway.

Reported by asbjornu on 2010-10-20 11:07:02

[lukeis]

This feature isn't going to happen. The recommended approach is to either extend the
HtmlUnitDriver to access the information you require or to make use of an external
proxy that exposes this information such as the BrowserMob Proxy (http://proxy.browsermob.com)

Reported by simon.m.stewart on 2010-10-20 11:37:49

[lukeis]

It's impossible to extend the HtmlUnitDriver when you're writing your tests in C#, since
there is no HtmlUnitDriver in C#. Or do you expect me to write a new "selenium-server-standalone-2.0a6.jar"
that exposes HTTP information and then write a new version of WebDriver that exposes
HTTP information to be able to get access to it?

If you enable people to get the source of the web page, you should enable people to
dig even further down into the protocol to detect whether the web application behaves
as it should or not. Just asserting on "visual" artifacts of a web page is simply just
not enough to get a full view of how your web application is behaving. It only tells
a part of the story and there's really no reason WebDriver can't tell the full story
(even though limited to how it would be seen through a web browser).

Reported by asbjornu on 2010-10-22 06:30:50

[lukeis]

Issue 919 has been merged into this issue.

Reported by dawagner on 2010-10-22 13:32:49

[lukeis]

WebDriver is concerned with driving the browser. Where more sophisticated information
is required it is designed to integrate effectively with other tools which can provide
this deeper level of information. In the case of inspecting HTTP traffic, the only
way to do this in a cross-browser way is via a proxy. WebDriver exposes APIs to make
it easy to set the proxy to use for a browser instance, so you can hook in whichever
is the most suitable tool for your project.

As a starting point, we recommend looking at the BrowserMob Proxy (http://proxy.browsermob.com
)

Reported by simonstewart@google.com on 2010-10-25 11:21:16

[lukeis]

Issue 950 has been merged into this issue.

Reported by dawagner on 2010-11-05 16:53:24

[lukeis]

How can I control browser locale without this? 

Reported by serge.shikov on 2010-12-16 12:43:17

[lukeis]

for testing mobile site is absolutely necessary for us to be able to pass headers with
user-agent and phone num.
I appreciate you don't want to pollute the api, but we need so we're going to write
it.
Is there a way to make it available to others as well as plugin or some more nice method
than a diff? 

Reported by uberto.gama on 2011-02-02 12:25:59

[lukeis]

The best way to contribute would be to contact Patrick Lightbody and see what can be
added to the BrowserMob Proxy. That's a project based on the proxy used by selenium,
but the focus there is on adding the functionality that projects, such as yours, require.

Reported by simon.m.stewart on 2011-02-02 13:30:08

[lukeis]

Not found the required reading.I also want this feature for this software.Thanks!

Reported by luoxuwy on 2013-11-04 03:05:39

[lukeis]

I can understand the "out of scope" thing. But, i need that feature too. I am attending
to test our users's users. And so it is a NxN diagram to test. They can change the
theme. But they can not change the request headers. And that is the only way for me
to test that issue. For this reason, it is not out of scope, i think.

I am not capable of building my own solution, except using a ProxyServer.  Make him
log. Read logs then solve the issue.

Reported by macitemre on 2013-11-19 16:46:16

[lukeis]

This feature would be very useful to me.  I understand the desire to keep a clean API,
but ultimately the end-user of the API are the developers and if many developers want
a feature than a patch should be accepted at the minimum.

Reported by dan.sandberg on 2013-12-07 07:16:53

[lukeis]

Issue 2047 has been merged into this issue.

Reported by nirvdrum on 2013-12-16 14:13:00

[lukeis]

For our website mobile testing, we'd like to be able to set the user-agent.

(This echoes comment #103 from the Wikipedia team above).

It seems a bit strange that you can't do something as simple as setting the user-agent
in Selenium.

Is that also out of scope, like getting back the HTTP Status codes?

Reported by victorhooi on 2014-01-28 01:17:00

[lukeis]

Issue 2047 has been merged into this issue.

Reported by nirvdrum on 2014-01-28 04:26:32

[lukeis]

#123 - Please see the discussion on issue 2047 for the matter of request headers.

Reported by nirvdrum on 2014-01-28 04:28:17

[lukeis]

Came upon this discussion because I am dealing with an issue right now where having
access to the underlying http headers would be extremely convenient. As much as I would
like to weigh in on the side of adding it, I understand the team's position. It's a
simple fact that http protocol information is per-request, and the output of webdriver
is a fully-rendered page which will have made a number of requests since the initial
action to request the resource. Should they really try to populate an array with the
results of every HTTP request made on the page? Doesn't make sense to me. So, bottom
line: it's a one-to-many problem, and I can understand them ruling it out of scope.

Reported by betz.mark on 2014-04-10 21:12:42

[lukeis]

While I can definitely understand the one to many aspect of it, at the very least I
would see it being a useful thing to add for the context of the page you loaded directly
(eg the URL you navigated to). Sure, there may have been many requests as part of that,
but I think you would cover a good chunk of what people want (80/20 rule) with a fairly
minor (arbitrary guess) amount of overall effort.

My 2c

Reported by glenn@ensquared.net on 2014-07-28 01:16:17

[lukeis]

When I'm testing or debugging a web application, the very first thing I do is open up
the web developer's console. Regardless of which browser I'm using, I'll see timing
information, headers, cookies, post data, and RESPONSE CODES.

Maybe back in 2009 it wasn't easy to see all this data in a browser, but 5 years later,
response codes are routinely examined in a browser when you're looking at a 302 redirection
chain.

Browsers have gotten better since this issue was first closed, why shouldn't WebDriver
keep up?

Reported by ironchefpython on 2014-09-15 19:00:04

[lukeis]

Up ! It should be really useful...

Reported by lucas@cherifi.info on 2014-09-29 16:04:40

[lukeis]

I was kind of shocked when I realized response codes were not accessible via WebDriver.
 I'm integrating HTMLUnitDriver and WebDriver with JMeter, and it would be super-useful
to be able to check the response codes, rather than having to parse the response data.

Reported by mark.bonassera on 2014-10-02 18:00:36

[lukeis]

i do not understand why you do not offer the HTTP error codes via the webdriver. i am
testing a webpage and occassionaly get something else than HTTP 200. should i now parse
the html and define every possible HTTP error that might come? What if somebody changes
the webserver or webserver version? are you expecting me to rewrite the html parsing
then again? Wouldn't it be easier just to check for the error code in the browser?

Reported by erik.dobak on 2014-10-06 08:22:50

[lukeis]

If people want to implement their own workaround, see my solution: https://github.com/MachinePublishers/ScreenSlicer/blob/master/core/firefox-addon/lib/main.js

It's a firefox addon that calls 127.0.0.1:8888/httpstatus/{statuscode}. You'd need
a webservice listening at that extension. With some work you might be able to write
that status into the html page itself. You could write something similar for if you're
using other browser drivers. The executable addon that you can drop into the extensions
dir of your firefox profile is: https://github.com/MachinePublishers/ScreenSlicer/raw/master/core/firefox-addon/build/jid1-QcfAcjQBdRo0lA%40jetpack.xpi

Reported by dan2k7 on 2014-10-17 16:40:16

[lukeis]

I was shocked when I learned that it could not implement already for 4 years, even if
only for readonly mode.

Reported by june.universum on 2014-10-21 10:30:18

[lukeis]

You can just ignore another users request, aka "please make headers accessible", but
you just have another user less. This is the open source version of https://www.youtube.com/watch?v=6wS5xOZ7Rq8
 I will neither use selenium+webdriver nor advertise it somewhere (because I wont use
it). I guess  you guys dont mind, but I now regret to have taken the 2 hours to get
everything set up with it and blogged it. There are alternatives and I am gonna use
them... for any other reference see #108/#109 

Creating fans the selenium way ... ;)

Reported by sebs@2xs.org on 2014-10-23 13:49:55

[lukeis]

I want to be able to set HTTP headers before clicking buttons in the interface.

Reported by asx34rtg.klagagagagag656 on 2014-10-26 12:37:55

[lukeis]

Another vote to expose the response code

Reported by Einevea on 2014-10-27 13:32:20

[lukeis]

I need to set headers to access our staging area, through firewalls, when using cloud
browsers.

Also our "users", when testing from outside firewall change their headers accordingly.
So as WebDriver should mimic users, I'd like it to mimic our test users.

Reported by mikael.x.wallin on 2014-10-28 13:48:01

[lukeis]

+1 for response codes etc, things have change dramatically since 2009, websites are
now webapps with a ton of functionality and browsers have changed dramatically why
not keep up with the changing times?

Reported by casey@lucidchart.com on 2014-10-29 15:04:36

[lukeis]

vote to have the response code

Reported by salustsai on 2014-10-30 18:21:32

[lukeis]

One more vote to have http status and headers introspection.

Rationale:
1. Browsers can check that data via js code, so I do not see why the webdriver should
be more limited / not expose it
2. in todays restful world, http internals are much more important than they were 5
years ago
2. having to set up a tool such as a proxy or http client is quite a chore, whereas
selenium is today integrated transparently into testing suites such as behat, which
make writing tests much nicer and faster
3. it seems that user have been consistently requesting this feature for 5 years, with
no-one opposing it except core devs...

Reported by ggiunta@kaliop.com on 2014-11-18 11:42:01

[lukeis]

Reported by luke.semerau on 2014-11-18 19:27:09

• Labels added: Restrict-AddIssueComment-Commit