Windows 8.1 Exhibits Gaming Performance Deficit

Microsoft just released Windows 8.1, and with that I subjected the latest version of the operating system to some gaming performance testing. The results illustrate a small decline in the performance of Windows when moving from 8.0 to 8.1.

System as Tested

• Asus z87-Pro Motherboard
• Intel i7-4770k @ 4.7Ghz/4.4Ghz Uncore
• Corsair Dominator Pro 16GB (8x2) @ 2133, 10-11-11-27
• Nvidia 670 2GB @  995/3204
• ForceWare 320.49 on both 8.0 and 8.1
• Sound Blaster ZxR Rendering in 5.1
• "Upgrade Install" of 8.1 from 8.0. 8.0 was installed fresh. 
• All drivers the same 
• App/Service footprint nearly identical

Testing Methodology

For the last 13 years or so, I've used a benchmarking script to test gaming performance. This script performs automated testing and logs results from each game to a central file. The script runs each game n (i.e. as many as I would like) times and then generates a median (rather than average). I have added games to this periodically since its inception.

I say all this because you'll notice there are a few old games listed. I have found including these results useful to judge CPU/Memory performance and identify bottlenecks specific to different APIs.

For these tests, each game was run at least three times (most more) for each set of settings. There are 4 tested settings that apply to most games: 4xAA/16xAnisotropic filtering, 0x/0x, and all combinations thereof. Normally these results would be split, but for the purposes of this platform comparison they have been combined. Bioshock Infinite does not support any AA, so that is not affected by the AA setting. All tests are performed at 1600x1200 which maintains compatibility with the older titles and while being a less than ten percent deviation from the 1080p effective pixel count.

For the first set of results there were over 250 test runs. These runs have been compared in total between Windows 8.0 and Windows 8.1 on a per game basis to generate a % performance change.

Games tested as follows: (in order of age, oldest->newest)

• Quake 3 (1999, OpenGL)
• Comanche 4 (2001, D3D8)
• Doom 3 (2004 OpenGL)
• Serious Sam 2 (2005 OpenGL)
• Company of Heroes (2006 DX10)
• Crysis (2007 DX10 Very High Settings)
• Dirt 2 Ultra (2009 DX11Ultra Settings)
• Alien vs. Predator (2010 DX11)
• Bioshock Infinite Ultra (2013 DX11 Ultra Settings)
• Unigine Valley (2013 DX11)

Results 

As illustrated in the bar chart below, the results indicate that Windows 8.1 lags in performance when compared to Windows 8.0 original release.

Game; All Scores Median of Multiple Runs	8.0 4xAA 16xAni	8.1 4xAA 16xAni	% Diff	8.0 0xAA 0xAni	8.1 0xAA 0xAni	% Diff	8.0 4xAA 0xAni	8.1 4xAA 0xAni	% Diff	8.0 0xAA 16xAni	8.1 0xAA 16xAni	% Diff
Quake3	904.4	904.73	0.04	912.37	910.97	-0.15	903.43	903.8	0.04	911.1	908.6	-0.27
Comanche4	198.47	198.5	0.02	198.53	198.97	0.22	198.33	199.87	0.78	199.5	198.93	-0.29
Serious Sam 2	486.7	469.6	-3.51	549.3	536.2	-2.38	521.5	507.5	-2.68	547.9	538.7	-1.68
Doom3	338	334.7	-0.98	473.2	472.1	-0.23	339.8	337.7	-0.62	468.5	472.7	0.90
Company of Heroes_DX10	261.55	246.5	-5.75	290.25	268.05	-7.65	267.3	248.7	-6.96	285.15	264.05	-7.40
Crysis_Island_DX10	63.06	62.12	-1.49	81.16	79.59	-1.93	66.03	64.73	-1.97	77.82	76.32	-1.93
Dirt 2 DX11_Ultra	129.46	127.62	-1.42	167.47	165.05	-1.45	142.65	139.97	-1.88	155.99	154.13	-1.19
Alien vs. Predator DX11	63	62.2	-1.27	110.3	108.6	-1.54	65.4	64.6	-1.22	104.6	103.3	-1.24
Bioshock Infinite Ultra_DX11	100.4	99.29	-1.11	99.73	99.11	-0.62	99.53	97.93	-1.61	100.2	98.48	-1.72
Unigine_Valley	58.63	58.28	-0.60	79.75	79.67	-0.10	60.13	59.08	-1.75	78.25	77.6	-0.83

While most percentages are relatively low, the trend appears consistent and the high sample size lends to the reliability of the data. I find this a bit troubling since there are no obvious reasons performance degradation should be introduced when looking at the 8.1 changes. Note there is still work to do; I'd like to get additional opinions including ATI results.

Now let's take a look at Futuremark 3dMark Advanced Edition. These tests were run through a couple times each and averaged. They were performed on default settings. (1080p/720p/720p)

3DMark Test	win 8.0	win 8.1	% Diff
Fire Strike	6560	6502	-0.88
Cloud Gate	25138	24331	-3.21
Ice Storm	179351	176476	-1.60

Again, consistent downturns in performance.

This is where I was going to publish this article, but as my finger hovered over the "Publish" button I had one lingering concern: What if I'm wrong? What if I missed a setting or something in the upgrade that might have this impact?

So Then We Did it Over Again

I wouldn't feel right publishing this article without doing a clean install and absolute parity of all operating systems involved. For the sake of being complete, I included a Windows 7 install as well. What I found stunned me... 8.1 isn't the start of a downturn, it's a continuation. On a smaller subset of tests I performed the same amount of runs. (average of 14 runs per title) The operating system/drivers were as follows:

• Windows 7 SP1 fully patched, clean install
• Windows 8 fully patched, clean install
• Windows 8.1 fully patched, clean install
• Nvidia driver version 327.23 on all platforms

 Here are the results using Windows 7 as baseline:

As you can see, Windows 8.1 lags behind Windows 8 and Windows 8 lags behind Windows 7. This is a troubling trend. Here's 3dMark (at least 3 runs per test):

The "Fire Strike" test gives us the only real hope of progress in Windows 8, but Windows 8.1 gives that gain right back. Overall, more troubling results.

This, coupled with the lingering mouse and input issues make for a bit of a shaky upgrade to 8.1 from a gaming perspective. As I outlined in my previous article, I believe it's time for Microsoft to step up their game (hah!) in light of new market pressures. This trend is unfortunate since there are a lot of minor changes in 8.1 that are a step in the right direction. (UI, etc.) Let's hope they iron out this performance issue quickly.

Update (11/4/2013): It looks as if Microsoft has acknowledged the mouse issue which I expect means a fix would be coming. Let's hope performance and other issues are addressed as well.

Update (11/9/2013): Microsoft has released a patch to deal with mouse issues.