Monday, October 28, 2013

Windows 8.1 Exhibits Gaming Performance Deficit

Microsoft just released Windows 8.1, and with that I subjected the latest version of the operating system to some gaming performance testing. The results illustrate a small decline in the performance of Windows when moving from 8.0 to 8.1.



System as Tested


  • Asus z87-Pro Motherboard
  • Intel i7-4770k @ 4.7Ghz/4.4Ghz Uncore
  • Corsair Dominator Pro 16GB (8x2) @ 2133, 10-11-11-27
  • Nvidia 670 2GB @  995/3204
    • ForceWare 320.49 on both 8.0 and 8.1
  • Sound Blaster ZxR Rendering in 5.1
  • "Upgrade Install" of 8.1 from 8.0. 8.0 was installed fresh. 
  • All drivers the same 
  • App/Service footprint nearly identical

Testing Methodology


For the last 13 years or so, I've used a benchmarking script to test gaming performance. This script performs automated testing and logs results from each game to a central file. The script runs each game n (i.e. as many as I would like) times and then generates a median (rather than average). I have added games to this periodically since its inception.

I say all this because you'll notice there are a few old games listed. I have found including these results useful to judge CPU/Memory performance and identify bottlenecks specific to different APIs.

For these tests, each game was run at least three times (most more) for each set of settings. There are 4 tested settings that apply to most games: 4xAA/16xAnisotropic filtering, 0x/0x, and all combinations thereof. Normally these results would be split, but for the purposes of this platform comparison they have been combined. Bioshock Infinite does not support any AA, so that is not affected by the AA setting. All tests are performed at 1600x1200 which maintains compatibility with the older titles and while being a less than ten percent deviation from the 1080p effective pixel count.

For the first set of results there were over 250 test runs. These runs have been compared in total between Windows 8.0 and Windows 8.1 on a per game basis to generate a % performance change.

Games tested as follows: (in order of age, oldest->newest)
  • Quake 3 (1999, OpenGL)
  • Comanche 4 (2001, D3D8)
  • Doom 3 (2004 OpenGL)
  • Serious Sam 2 (2005 OpenGL)
  • Company of Heroes (2006 DX10)
  • Crysis (2007 DX10 Very High Settings)
  • Dirt 2 Ultra (2009 DX11Ultra Settings)
  • Alien vs. Predator (2010 DX11)
  • Bioshock Infinite Ultra (2013 DX11 Ultra Settings)
  • Unigine Valley (2013 DX11)

Results 


As illustrated in the bar chart below, the results indicate that Windows 8.1 lags in performance when compared to Windows 8.0 original release.



Game; All Scores Median of Multiple Runs 8.0 4xAA 16xAni 8.1 4xAA 16xAni % Diff 8.0 0xAA 0xAni 8.1 0xAA 0xAni % Diff 8.0 4xAA 0xAni 8.1 4xAA 0xAni % Diff 8.0 0xAA 16xAni 8.1 0xAA 16xAni % Diff
Quake3 904.4 904.73 0.04 912.37 910.97 -0.15 903.43 903.8 0.04 911.1 908.6 -0.27
Comanche4 198.47 198.5 0.02 198.53 198.97 0.22 198.33 199.87 0.78 199.5 198.93 -0.29
Serious Sam 2 486.7 469.6 -3.51 549.3 536.2 -2.38 521.5 507.5 -2.68 547.9 538.7 -1.68
Doom3 338 334.7 -0.98 473.2 472.1 -0.23 339.8 337.7 -0.62 468.5 472.7 0.90
Company of Heroes_DX10 261.55 246.5 -5.75 290.25 268.05 -7.65 267.3 248.7 -6.96 285.15 264.05 -7.40
Crysis_Island_DX10 63.06 62.12 -1.49 81.16 79.59 -1.93 66.03 64.73 -1.97 77.82 76.32 -1.93
Dirt 2 DX11_Ultra 129.46 127.62 -1.42 167.47 165.05 -1.45 142.65 139.97 -1.88 155.99 154.13 -1.19
Alien vs. Predator DX11 63 62.2 -1.27 110.3 108.6 -1.54 65.4 64.6 -1.22 104.6 103.3 -1.24
Bioshock Infinite Ultra_DX11 100.4 99.29 -1.11 99.73 99.11 -0.62 99.53 97.93 -1.61 100.2 98.48 -1.72
Unigine_Valley 58.63 58.28 -0.60 79.75 79.67 -0.10 60.13 59.08 -1.75 78.25 77.6 -0.83


While most percentages are relatively low, the trend appears consistent and the high sample size lends to the reliability of the data. I find this a bit troubling since there are no obvious reasons performance degradation should be introduced when looking at the 8.1 changes. Note there is still work to do; I'd like to get additional opinions including ATI results.

Now let's take a look at Futuremark 3dMark Advanced Edition. These tests were run through a couple times each and averaged. They were performed on default settings. (1080p/720p/720p)





3DMark Test win 8.0 win 8.1 % Diff
Fire Strike 6560 6502 -0.88
Cloud Gate 25138 24331 -3.21
Ice Storm 179351 176476 -1.60


Again, consistent downturns in performance.

This is where I was going to publish this article, but as my finger hovered over the "Publish" button I had one lingering concern: What if I'm wrong? What if I missed a setting or something in the upgrade that might have this impact?

So Then We Did it Over Again


I wouldn't feel right publishing this article without doing a clean install and absolute parity of all operating systems involved. For the sake of being complete, I included a Windows 7 install as well. What I found stunned me... 8.1 isn't the start of a downturn, it's a continuation. On a smaller subset of tests I performed the same amount of runs. (average of 14 runs per title) The operating system/drivers were as follows:

  • Windows 7 SP1 fully patched, clean install
  • Windows 8 fully patched, clean install
  • Windows 8.1 fully patched, clean install
  • Nvidia driver version 327.23 on all platforms
 Here are the results using Windows 7 as baseline:



As you can see, Windows 8.1 lags behind Windows 8 and Windows 8 lags behind Windows 7. This is a troubling trend. Here's 3dMark (at least 3 runs per test):



The "Fire Strike" test gives us the only real hope of progress in Windows 8, but Windows 8.1 gives that gain right back. Overall, more troubling results.

This, coupled with the lingering mouse and input issues make for a bit of a shaky upgrade to 8.1 from a gaming perspective. As I outlined in my previous article, I believe it's time for Microsoft to step up their game (hah!) in light of new market pressures. This trend is unfortunate since there are a lot of minor changes in 8.1 that are a step in the right direction. (UI, etc.) Let's hope they iron out this performance issue quickly.

Update (11/4/2013): It looks as if Microsoft has acknowledged the mouse issue which I expect means a fix would be coming. Let's hope performance and other issues are addressed as well.

Update (11/9/2013): Microsoft has released a patch to deal with mouse issues. 

4 comments:

Nathan Co said...
This comment has been removed by a blog administrator.
Nathan Co said...

I don't always check my google email but more on my yahoo.com email. Please help.

Thanks again.

Toby Meyer said...

Thanks Nathan! I've removed your first comment because I want to make sure your e-mail is protected should any spammers comb the site. :) I appreciate your kind words. Here is a link on how to follow Blogger blogs. As for blocking a gaming site on the router the easiest way would probably be to redirect the desired URL to a bad address with dnsmasq, but that's a discussion for a different post. :)

Alex Cicala said...

I reckon it comes down to hardware and settings and what you add or use with the game in terms of thirdparty antialiasing and mods

For me there is no noticeable difference on my system and I run dxtory 24/7 so im aware of how games run on my comp...

In the end Windows 8/8.1 is so similar to Windows 7, the general public would disagree, but if I got a Windows 7 user to use my comp they would not be able to tell the difference unless they checked System Properties.. basically killed metro and charms and all that "crap" they wanted to add to try and make Windows Accessible.