Monday, October 28, 2013

Windows 8.1 Exhibits Gaming Performance Deficit

Microsoft just released Windows 8.1, and with that I subjected the latest version of the operating system to some gaming performance testing. The results illustrate a small decline in the performance of Windows when moving from 8.0 to 8.1.



System as Tested


  • Asus z87-Pro Motherboard
  • Intel i7-4770k @ 4.7Ghz/4.4Ghz Uncore
  • Corsair Dominator Pro 16GB (8x2) @ 2133, 10-11-11-27
  • Nvidia 670 2GB @  995/3204
    • ForceWare 320.49 on both 8.0 and 8.1
  • Sound Blaster ZxR Rendering in 5.1
  • "Upgrade Install" of 8.1 from 8.0. 8.0 was installed fresh. 
  • All drivers the same 
  • App/Service footprint nearly identical

Testing Methodology


For the last 13 years or so, I've used a benchmarking script to test gaming performance. This script performs automated testing and logs results from each game to a central file. The script runs each game n (i.e. as many as I would like) times and then generates a median (rather than average). I have added games to this periodically since its inception.

I say all this because you'll notice there are a few old games listed. I have found including these results useful to judge CPU/Memory performance and identify bottlenecks specific to different APIs.

For these tests, each game was run at least three times (most more) for each set of settings. There are 4 tested settings that apply to most games: 4xAA/16xAnisotropic filtering, 0x/0x, and all combinations thereof. Normally these results would be split, but for the purposes of this platform comparison they have been combined. Bioshock Infinite does not support any AA, so that is not affected by the AA setting. All tests are performed at 1600x1200 which maintains compatibility with the older titles and while being a less than ten percent deviation from the 1080p effective pixel count.

For the first set of results there were over 250 test runs. These runs have been compared in total between Windows 8.0 and Windows 8.1 on a per game basis to generate a % performance change.

Games tested as follows: (in order of age, oldest->newest)
  • Quake 3 (1999, OpenGL)
  • Comanche 4 (2001, D3D8)
  • Doom 3 (2004 OpenGL)
  • Serious Sam 2 (2005 OpenGL)
  • Company of Heroes (2006 DX10)
  • Crysis (2007 DX10 Very High Settings)
  • Dirt 2 Ultra (2009 DX11Ultra Settings)
  • Alien vs. Predator (2010 DX11)
  • Bioshock Infinite Ultra (2013 DX11 Ultra Settings)
  • Unigine Valley (2013 DX11)

Results 


As illustrated in the bar chart below, the results indicate that Windows 8.1 lags in performance when compared to Windows 8.0 original release.



Game; All Scores Median of Multiple Runs 8.0 4xAA 16xAni 8.1 4xAA 16xAni % Diff 8.0 0xAA 0xAni 8.1 0xAA 0xAni % Diff 8.0 4xAA 0xAni 8.1 4xAA 0xAni % Diff 8.0 0xAA 16xAni 8.1 0xAA 16xAni % Diff
Quake3 904.4 904.73 0.04 912.37 910.97 -0.15 903.43 903.8 0.04 911.1 908.6 -0.27
Comanche4 198.47 198.5 0.02 198.53 198.97 0.22 198.33 199.87 0.78 199.5 198.93 -0.29
Serious Sam 2 486.7 469.6 -3.51 549.3 536.2 -2.38 521.5 507.5 -2.68 547.9 538.7 -1.68
Doom3 338 334.7 -0.98 473.2 472.1 -0.23 339.8 337.7 -0.62 468.5 472.7 0.90
Company of Heroes_DX10 261.55 246.5 -5.75 290.25 268.05 -7.65 267.3 248.7 -6.96 285.15 264.05 -7.40
Crysis_Island_DX10 63.06 62.12 -1.49 81.16 79.59 -1.93 66.03 64.73 -1.97 77.82 76.32 -1.93
Dirt 2 DX11_Ultra 129.46 127.62 -1.42 167.47 165.05 -1.45 142.65 139.97 -1.88 155.99 154.13 -1.19
Alien vs. Predator DX11 63 62.2 -1.27 110.3 108.6 -1.54 65.4 64.6 -1.22 104.6 103.3 -1.24
Bioshock Infinite Ultra_DX11 100.4 99.29 -1.11 99.73 99.11 -0.62 99.53 97.93 -1.61 100.2 98.48 -1.72
Unigine_Valley 58.63 58.28 -0.60 79.75 79.67 -0.10 60.13 59.08 -1.75 78.25 77.6 -0.83


While most percentages are relatively low, the trend appears consistent and the high sample size lends to the reliability of the data. I find this a bit troubling since there are no obvious reasons performance degradation should be introduced when looking at the 8.1 changes. Note there is still work to do; I'd like to get additional opinions including ATI results.

Now let's take a look at Futuremark 3dMark Advanced Edition. These tests were run through a couple times each and averaged. They were performed on default settings. (1080p/720p/720p)





3DMark Test win 8.0 win 8.1 % Diff
Fire Strike 6560 6502 -0.88
Cloud Gate 25138 24331 -3.21
Ice Storm 179351 176476 -1.60


Again, consistent downturns in performance.

This is where I was going to publish this article, but as my finger hovered over the "Publish" button I had one lingering concern: What if I'm wrong? What if I missed a setting or something in the upgrade that might have this impact?

So Then We Did it Over Again


I wouldn't feel right publishing this article without doing a clean install and absolute parity of all operating systems involved. For the sake of being complete, I included a Windows 7 install as well. What I found stunned me... 8.1 isn't the start of a downturn, it's a continuation. On a smaller subset of tests I performed the same amount of runs. (average of 14 runs per title) The operating system/drivers were as follows:

  • Windows 7 SP1 fully patched, clean install
  • Windows 8 fully patched, clean install
  • Windows 8.1 fully patched, clean install
  • Nvidia driver version 327.23 on all platforms
 Here are the results using Windows 7 as baseline:



As you can see, Windows 8.1 lags behind Windows 8 and Windows 8 lags behind Windows 7. This is a troubling trend. Here's 3dMark (at least 3 runs per test):



The "Fire Strike" test gives us the only real hope of progress in Windows 8, but Windows 8.1 gives that gain right back. Overall, more troubling results.

This, coupled with the lingering mouse and input issues make for a bit of a shaky upgrade to 8.1 from a gaming perspective. As I outlined in my previous article, I believe it's time for Microsoft to step up their game (hah!) in light of new market pressures. This trend is unfortunate since there are a lot of minor changes in 8.1 that are a step in the right direction. (UI, etc.) Let's hope they iron out this performance issue quickly.

Update (11/4/2013): It looks as if Microsoft has acknowledged the mouse issue which I expect means a fix would be coming. Let's hope performance and other issues are addressed as well.

Update (11/9/2013): Microsoft has released a patch to deal with mouse issues. 

Thursday, October 3, 2013

The Gathering Storm in the PC Gaming Industry

As a result of recent events in the computing world, I think we are entering into a very special time in PC gaming. This industry has been close to my heart since I loaded up Police Quest 2 on my first turbo-button equipped and ultra-beige PC clone. Since then, there have been many evolutions of the gaming platform but no revolutions. That may finally change in the near future. Despite the decline in PC sales, the PC gaming industry continues to grow, and the PC as we know it is morphing into device types that don't fit the molds to which we're accustomed.



With this article I aim to illustrate why we're about to go through a particularly intriguing time in gaming by covering how we got here, where we are now, what has recently changed, and finally why the future is so exciting.

With that said, let's start with the foundation of the modern gaming platform; we'll cover a bit of history and the pace of innovation. 

Hardware

My new PC Gets 35 MPG. 

  • CPU Performance has increased steadily and closely followed Moore's law since the adoption of the Intel 8080 in the late 1970s. Intel has largely dominated this space from then until now save a few year interruption in the mid 2000s when AMD introduced the K8.
  • Gaming relevant display adapters started with "Windows Accelerator" boards (Late 1980s) and since the introduction of the first 3d accelerators (Mid 1990s), the progress has been steady with minor interruptions by things like programmable shaders.
  • Audio progression was initially rapid and has stagnated significantly in the last ten years. Things got kicked off in 1988 with the Roland LAPC-I, then the Adlib and then on to the Creative Labs cards which is where we remain today. We nearly had a major evolution in gaming sound in the late 90s with the introduction of A3d by Aureal, but unfortunately they were forced out of business by the legal maneuvering of Creative. 

Operating Systems

Everything Old is New Again.




For the popular gaming platform this one is easy. After the C64 and other early platforms faded away it's been Microsoft, Microsoft, Microsoft. DOS to Windows. DOS was new then it was old. Windows was new then it was old. NT was new now it's aging, and the commitment to the OS as a gaming platform has waned for years.

Input and Output Devices

Because the IBM Model M Still goes for Sixty Bucks on Ebay.

Little has changed with the mouse and keyboard in the last 30 years. As with everything else, performance and ability to customize has increased bit by bit. At least most gamepad supporting games these days have standardized on the Xbox360 controller layout rather than forcing you to map buttons to your gamepad du jour. 

The monitor hasn't progressed much since VGA was introduced in 1987. Slow and steady improvements, with a dip in performance when LCDs became the popular solution. As it pertains to pure gaming performance (input lag/pixel persistence/response time), LCDs are just now catching up to CRTs.

Evolution Summary

Putting it together.

As you can see, not much has really changed given we're talking about nearly 30 years. We're still playing similar games on similar platforms using similar interfaces. That's not a complaint mind you, but relative to the change we have seen in other sectors in recent years this space seems long in the tooth. Now let's shift to the present...




Setting the Stage for Change

It takes a nation of millions a few companies to hold us back.

First, a few facts: Depending on who/where/how you ask, the average gamer ranges from 30 to 37 years old. Nearly half are women, and over half of Americans play video games.



The gaming industry is an odd thing when you look at it as a whole; while we take the status quo for granted it really has not fully evolved. My generation (Gen X) is the first generation that has grown up with this entertainment medium, and while we've aged & settled down the market has not. There still exists a perception among the baby boomer generation that games are "kids stuff", and with boomers still in control of many large companies it's no surprise that these issues exist. A single viewing of the "Spike Video Game Awards" is all you need to illustrate the damage of this ill effect. This is improving with time, but like all new mediums before it, the gaming industry still needs time to realize its potential. We're still waiting for anything the likes of 2001: A Space Odyssey or The Godfather.

We're Getting Close Though

Couple this with extreme confusion on the part of the gaming system providers and the conflict of interest that it creates. With Microsoft being the predominant PC gaming platform one would think they would be eager to lead the space. I think their own conflict of interest and lack of a credible competitor has caused them to become complacent. Time and time again Microsoft has claimed to commit to the platform, but since the release of the original Xbox in 2001 they back down in favor of that lone consumer-image win. The most important and perfect example of this is DirectX, which started out strong in 1995 with ten major releases in the first ten years, but since 2006 has seen only one major release (DX11). To help put this in perspective, Internet Explorer 7 was released the same year as DirectX 10. Many developers in the industry have been complaining for years that DirectX is holding the industry back and it even appears at times that Microsoft uses relatively minor updates in an attempt to push gamers to the newest version of their operating system. (See Windows 8.1 and in a more defensible example, Windows Vista) Update 10/17/2013: Microsoft has published an article in defense of the Direct3d progress.

Additionally, Microsoft doesn't want you to hook up a PC to your TV. While home theater computing has never been a major market, it was inevitable that Microsoft would be a major player because of industry partnerships and the hardware support picture. Now they want to cut costs in that area and push the sale of the Xbox One instead, and that leaves space in this, albeit small, market segment.

And this is just the start. The desire to move away from the traditional PC platform has been an undercurrent for years, but without a viable competitor there is no reason for hardware manufacturers to spend money developing for an alternative. That's not to say others haven't tried though, and their frustration can be best summarized by this (NSFW, Language/Text) statement from Linus Torvalds.

Market Trends and Impact on Hardware Support

"Close the watertight doors." -Capt. EJ Smith 


Mobile Computing is Gaining in Popularity

With the monumental rise of portable computing (Smartphones, etc.) there is an effect here that is easily overlooked. Android (Linux) and iOS are absolutely dominating the mobile space, and with that comes the demand for and expertise in coding for other operating systems. This sort of experience doesn't directly undermine the current PC gaming picture, but it helps to build an environment more suitable for change.

Microsoft seems primed to slowly abandon the traditional PC space in favor of pursuing the built-for-purpose device model of Apple coupled with the software-as-a-service model of Google. It isn't that they want to abandon gaming, they just want you to use a device for that. Over the course of the last year and a half they have taken definitive steps to reorganize and re-purpose themselves as a devices and services company.

The problem with this strategy is they're trying to sell a platform that, when compared to the traditional PC or newer solutions (below), limits functionality. PC gaming thrives in part due to the community's ability to customize, add-on, and change the experience. As of now, nearly anything one may want to contribute, be it modding, culture, or even whole games is open more often than not. In a future Xbox (or PS4) gaming space like the one Microsoft has portrayed those things become more difficult or impossible to do for the average lone and unassociated contributor.

Enter the Contender

Is that the protagonist from Mike Tyson's Punchout?


As I'm sure you're aware, Valve recently announced the SteamMachines, Steam Controller, and most importantly (to my story at least),  SteamOS. This article isn't about an in-depth analysis of these announcements, but for the sake of context I'll introduce each:

SteamMachines/Controller

The SteamMachines aren't a platform but rather a series of similar platforms. It's more of a hardware spec for a living room focused PC. While Valve will be making (and offering in a limited 300 person beta) their own "SteamBox", the playing field is wide open for any manufacturer to create and ship a SteamMachine. Boy if Bissell ever wants to do a market sector transition this one would be easy. The controller is a stab at providing a traditional controller footprint that will more easily map to keyboard/mouse focused games. Initial impressions on the haptic feedback mechanism and other details have been relatively positive. That said, the controller is really an attempt to bridge the "couch gap" created by targeting this platform to the living-room/entertainment space. Should you choose you can still use an Xbox360 controller with the appropriate receiver or a traditional keyboard/mouse.

SteamOS

This piece is what I find the most interesting and has the largest potential for impact. The software that will be run on any SteamMachine, SteamOS, has the following characteristics:

  • An entire operating system distribution based on the open-source juggernaut Linux
  • Will run and be supported (to an extent) on nearly any standard PC platform
  • Capable of streaming games from your existing Windows gaming PC, which not only leverages any existing heavy duty hardware but also helps to bridge the Linux compatibility gap. 
Valve CEO Gabe Newell and His Heavy Friend
Since this platform is based on Linux it creates quite an opportunity to leverage community improvements to its core. Say you want an API to have a certain feature for an upcoming piece of hardware... the vendor, the community, and Valve could all work in tandem to provide the solution quickly and efficiently for all. If exercised correctly, the partnership between private sector and community development can be very powerful. All that's needed to facilitate that partnership is financial incentive mixed with a motivated, intelligent user base. Both those elements stand ready to contribute should the pieces fall into place.

Contender to Whom?

I want you to be nice until it's time to not be nice.

So this is where I'm supposed to make grand statements about head-to-head match-ups with consoles and/or PCs, but I won't. I don't think those match-ups will really happen. This platform isn't taking on any of the established players directly.... at least not yet.

The beauty of this idea is that it is potentially more sustainable than any other model. While the purely private entities need to fight over exclusives and drop lots of cash developing their platforms, the SteamOS initiative can leverage an existing and growing library of top-quality gaming experiences and could rely on the slow, steady pace of innovation of the underlying platform to gradually drain market share from its competitors.

Wildcards

Nobody gave the 2007 Giants a chance.

This is the particularly interesting part of this equation. Seeing as how the PC platform is arguably (in theory) the most adaptable gaming platform that is or will be available, it is worth tracking several new technologies that may contribute to the success or failure of the SteamOS or traditional Windows based gaming PC platform:

Oculus Rift and other Output Devices


Jimmy Fallon Models the Oculus Rift and Looks... Comfortable?

The Oculus Rift is the first VR headset that stands to be both cost effective and performant enough to catch on in the consumer space. This device is a potential slam dunk for the living room space (space to move around + reliance on controller). Whoever has the most effective support out of the gate could dominate a small market of enthusiasts and genre devotees (flight sims, racing sims) right off the bat.

New Input Controls



Not to be left behind, the input space also has some new VR initiatives. There are quite a few out there, but the most interesting I've seen are the relatively portable treadmill solution Omni, the Stem VR controller system, and the Tactical Haptics controller feedback system. Again, these are longshot devices that will only drive a small market initially, but the concept of a PC in the living room is a much better fit for any of these than a PC in the office or a console with limited support.

Rebirth of the Home Theater PC

By targeting the living room space the SteamMachine/SteamOS combo also taps into another potential market: the aforementioned HTPC crowd and additionally, if the price and functionality are right, the growing "cord cutters" crowd.

Should Valve's media playback efforts prove not only sufficient, but best in class, it would open up the market for low-spec/low-cost devices from hardware partners targeted explicitly at that space. The prospect of a roku-like device that can also play casual games and stream more power hungry ones could be a very interesting alternative to the bevy of devices found in many living rooms now, especially if a solution was provided to play games on low-end hardware. Speaking of transitions:

Network Streaming


"I'm not really here, I'm in a datacenter."

Valve has already announced that you'll be able to stream games from another PC on your local network, but what if they were to introduce a "cloud" gaming service down the line? The technology is feasible; Onlive has been doing it for some time already. This would be an interesting alternative for people that don't want to spend a lot of money up front on a gaming caliber box, but still be able to play "AAA" titles provided they aren't bothered by the I/O latency. This space would be a natural transition for Valve's gaming model, and create all sorts of new opportunities for charging models such as rental/pay by the hour without leaving your living room.

Note too that while Valve has stated they are working closely with both AMD and Nvidia, the first announced Steam Machine platform directly from Valve includes Nvidia hardware.  Keep in mind Nvidia already offers cloud gaming solutions and has the local streaming thing pretty much figured out (running on Linux no less). It would seem all the pieces may already be there. Microsoft and Sony are obviously in the race on this one as well.

The Performance Problem/Possibilities


Currently, the performance picture on Linux is somewhat spotty. There have been strides made in the last few months but the performance still generally trails its Windows counterpart. These performance differences are due almost entirely to drivers from hardware manufacturers. Of those, video card drivers have the most impact. For further reading on the current state of things, see: Nvidia, AMD, Intel. As for the future, there have already been announcements pledging better support.

Most games on the Windows platform utilize the API Direct3d, which acts as a unifying layer between the game and the video card. This allows for the same code to run on hardware from different vendors (in theory) without issue. This API is created by Microsoft, so it is unavailable on the Linux (SteamOS, Linux, MacOSX) platform. The standard API utilized by those platforms for 3d acceleration is and has been OpenGL for quite some time, which also runs on Windows. This would be the obvious choice for games going forward should they want to target both major PC platforms.

Oddly though, that isn't what a lot of developers would want to target anyhow. The more potentially lucrative pursuit would be to target consoles and the PC space, and anything coded to OpenGL and, to a lesser extent, Direct3d, would need work to be ported to consoles.

And now there's Mantle.

Mantle is the product of AMD's "next generation" console wins. The AMD Jaguar platform powers both the PS4 and Xbox One. To facilitate a unified API level and expose all the power possible on the platform, AMD created the "Mantle" API. Now AMD plans to bring this API to the PC. (Windows first, but it could work on any OS) It has already been adopted by the folks at Dice and they reported substantial performance gains. We'll have to wait for December to find out, but it will be very interesting to see if these performance promises pan out.

The potential here, however, is the possibility that this could further upset the entrenched standards in the industry. I don't think a hardware proprietary API is a good solution for the gaming community (remember Glide?), but if it catches on in any way the competition will be forced to respond in kind. AMD claims they'll open the platform eventually but I'd wager that will end up like the Nvidia "offer" to share PhysX. If the performance promises do pan out, one has to expect that Gabe (Valve CEO) would love to nudge its adoption if only to further undermine Direct3d.


Unintended Consequences


With all of the pressures introduced by the SteamOS introduction, Windows 8 lukewarm reception, the rise of the mobile market, and everything else mentioned above there will undoubtedly be changes across the board in an attempt to gain new or retain old customers. All these factors combined lead us to...

The Point.

So why do we care about this?


The pace of PC innovation hasn't kept up with other similar markets and it is now at its weakest point since its inception. Microsoft has done a poor job of keeping focus on the Windows based gaming space. Valve has introduced not only a hardware spec but a potential sans-Microsoft OS solution. Input, output, and consumption models are on the verge of being flipped on their head, and hardware manufacturers are moving to take more control of the API space. The probability of change to this platform stands higher than ever.

We don't know what will happen. For the first time in many years we don't know what will happen. The opportunities are many and the space is an incredibly exciting one to witness and participate in.

And this is the point. I'm intrigued by what this market could become for the first time in a long, long time. Perhaps Microsoft will re-invigorate the DirectX platform, or maybe SteamOS will finally bring the "year of the Linux desktop". Maybe the Oculus Rift combined with a Kinnect will launch the Xbox One to dominance, or maybe Nvidia releases a competing API that obsoletes everything prior.

Or maybe there's room for all of this. The market is growing and we're finally getting some competition in the field that is long overdue.

In times of rapid change the smallest actions can have large, lasting impacts. Even your product selection will impact the direction of video gaming in a small way, and any contributions you make to gaming communities even more-so. Now is our time to choose the direction, so go and contribute what you can. I know we'll all have fun along the way.