Microsoft has posted the results for 192 tests grouped in 8 categories for HTML5, SVG 1.1, CSS3, and DOM Level 2&3 showing that IE9 Preview passes all of them with flying colors while Chrome, Firefox, Opera, and Safari have mixed results varying from 0% to 100% depending on the category. The conclusion, that IE9 is the most compliant with W3C standards, is contested by Google, Mozilla, Opera.
Microsoft has created 192 “test pages that we [Microsoft] developed in conjunction with the World Wide Web Consortium (W3C) working groups” and the results of those tests for the major browsers are:
The results summary is followed by detailed results with links to web pages containing the respective tests, so anyone could perform the respective tests. The conclusion is: IE 9 Preview is 100% compliant with W3C standards covered by tests while the other browsers lag behind. The best result for the other browsers is for Firefox with 100% for DOM 2 tests, but most of the browsers show results in yellow and red for many categories. InfoQ has talked to Google, Mozilla, and Opera to find out what is their opinion on tests, results and conclusion.
Google has pointed to Google I/O Keynote Day 1 where they covered the issue of HTML5 in browsers and their expectancy for the end of 2010 is for all browsers except IE to fully implement the HTML5 standard and some other specs (9min:35secs in the keynote):
Mozilla considers Microsoft’s tests as having limited coverage, being inaccurate and even misleading:
While we are glad that Microsoft is keen on using test suites to measure standards compliance, these particular tests on MSDN only cover a very limited spectrum of capabilities, and statements about browser compliance made here are frankly misleading.
The SVG-in-HTML tests cited by MSDN are known to be broken, do not conform to the HTML5 parsing rules or the DOM specification, and in fact make use of an IE bug. Microsoft's engineers know about this bug *and* the fact that the test itself is wrong [1][2], so it is surprising that this test is touted as an important one in furthering interoperability between browsers. Firefox's beta builds, on the other hand, actually do have an HTML5 parser in place [3], and nightly builds [4] will pass legitimate tests for SVG-in-HTML parsing.
Additionally, some of the CSS tests listed on MSDN are inaccurate, and browsers that actually pass these tests are not compliant with the standard. There is an official CSS3-selectors test suite where we pass all the relevant pieces [5] and we think that for web developers, sites like caniuse.com [6] are much more useful when discussing HTML5, CSS3, and SVG support in browsers. Mozilla will continue to contribute to open and fair test suites that adhere to the specifications in question [7], and encourage other browser vendors to do so as well.
Update: Mozilla has updated their statement with the following short but meaningful piece of information:
We've heard back from Microsoft and are now working with them to fix the tests.
Opera is aware of Microsoft’s tests and results, but they are using a different test suite having different results:
We haven't run these specific tests internally, but we do run a broad gamut of tests all the time to ensure that our contribution to the HTML5 standard does get implemented in our browser products. Opera is a big contributor to the HTML5 standard, it was after all Opera who proposed the Video tag that got the ball rolling on the standard. We constantly strive to improve our products and make sure the follow the standards as best possible.
This test is very selective in what elements it chooses to test for, and the results seem to favor one over the others. When evaluating HTML5 tests, the criteria should be the same as for all other kinds of tests. The test authority should be independent and the tests should reflect the nature of the specification. It's easy to point to other HTML5-relates tests where Opera do much better than IE like this one: http://www.codedread.com/svg-support.php. …
We run tests all the time to see how compatible our browser is, but HTML5 is a work in progress. So regarding the web pages of Microsoft, they seem to test a very selective part of HTML5 while leaving out large parts like the Canvas element.
Interestingly, Opera pointed to test results showing that the Opera 10.53 browser implements 94.89% of the SVG 1.1 test, while IE9 Preview 2 implements only 30.55% of the same test. Opera 10.52 does about the same in Microsoft’s test, but IE9 has 100% in that test which is far away from 30.55%.
Jeff Schiller, the software developer who ran the tests mentioned by Opera, commented on Microsoft’s test results:
While I am very excited about the direction IE9 is taking and have been impressed with the level of participation in the SVG Working Group, I have a problem with the “IE Testing Center” as I feel it is very misleading. Even Microsoft admits that they are only including tests that they wrote (for which they claim IE9 to pass) not the entire test suite.
For instance: the SVG Test Suite has 275 tests. Microsoft only talks about 31 tests on that page. As an example, IE9 does not even support gradients yet.
As another example, their HTML5 tests that talk about inline SVG markup are actually incorrect. They only pass in IE9 due to a browser bug dealing with childNodes. If I correct those tests they pass in Firefox nightlies and fail in IE9.
Apple did not reply to our inquiry, but they published an HTML5 Showcase which sparked a debate of its own.