Computer Performance Metrics

ebergerly

After many years of poking around various computer tech forums on the internet, I’ve noticed that there is a continuing confusion regarding computer performance metrics. This is sometimes magnified by vendors’ intentionally vague descriptions which are often intended to create excitement rather than convey useful fact.

So I thought I’d give my simple description of the real issues to get folks thinking about the topic. This is hopefully going to be just a bunch of facts, with little or no personal opinion, so I won’t be engaging in any sort of debate. If I make an error feel free to correct, but most of this should be pretty straightforward.

Let’s start with a simple example:

I’m driving my car at 30 miles per hour. That is my speed, as shown on the speedometer. It is a clear and unambiguous number.

Now, let’s say I press harder on the accelerator until my speed increases to 60 miles per hour. My speed is now twice (2 times, in math terms) my initial speed. Am I going “fast”? Heck, I don’ t know, nor do I care. It depends on what you mean by “fast”, and ultimately it’s a fairly useless term as long as you know the exact speed. 

Am I going “faster”? Well, yeah, but only if you define “faster” as “speed in miles per hour relative to your initial speed”. Because again, “faster” is a fairly vague and ambiguous term, and as we’ll see in a bit it can have many definitions.

So the accurate and unambiguous description is “I’m now going twice (2x) my initial speed “.

So let’s say you strap a jet engine to your car and you’re now travelling at 300 miles per hour. The correct description is “I’m now going at ten times (10x) my initial speed”. Can you go 100x your inital speed, or even 500x? Well, yeah, if you get a big enough rocket. There’s pretty much no limit (well, there is but...).

So now  you have your jet car, and your buddy says “Hey, I just heard there’s a Led Zeppelin reunion concert in ½ an hour!! Can we get there in time in your super awesome jet car?”

And you respond, “Dude, my car is now 50x faster !!!”.

“Ummm, okay, but can we get there in time?”

“But its performance is awesome!! It’s a beast!!”

“Yeah, but can we get there in time?”

Hmmm...now it’s getting a lot more complicated. Now you need to calculate a completely different, and possibly far more complicated number (aka “metric”). You need to figure distance to the concert, and what your speed will be (considering real-world factors like stop lights, speed limits, etc.), and a bunch of other real world stuff to determine “how much time will it take to complete the task?”. In some ways, your rated speed may be somewhat irrelevant, due to a bunch of real-world considerations. And keep in mind it may be almost impossible to figure out what your final travel time will be due to all the variables. Maybe you just have to wait until you get there and then figure out how long it actually took. 

It’s very important to recognize the difference between “speed” and “time required to complete a task”. These are two totally different concepts, with totally different metrics.

So if you can travel at, say, an average of 60 mph the entire way, and the concert is 60 miles away, it will take an hour. If you can somehow manage to go at 120 mph average the entire way, it will take half that time, or 30 minutes. If you can somehow go at 600 mph the entire way, it will take 1/10 that time, or 6 minutes.

But no matter how fast you go, you’ll never get there instantaneously. It’s physically impossible.

So your speed is increasing astronomically, but your time to complete the task is getting smaller, because it’s the inverse. If you go 10x speed, travel time is 0.1 times the initial travel time. If you go 20x speed, travel time is 0.05x.

But keep in mind, and this is very important, in the real world there is NOT a direct or clear or easy-to-figure out relationship between speed and travel time. Because in a real trip, there are a lot of factors which affect speed during the trip. It depends on a lot of variables. Traffic jams, traffic lights, construction sites, etc. You have to look at the specifics of the ACTUAL journey and factor all the things that will affect speed during the trip, and come up with a final value. And this can be VERY difficult to determine. This is very much like computers, where it depends on the specific task, the software and driver capabilities, and so on. Your computer can have amazing specs, and can do a zillion calculations per second under certain conditions, but on your specific task, under specific conditions, it might be slower than molasses. You can’t say “Oh, this computer has a 10x performance increase therefore my renders will be 10x faster”. Partly because it’s totally ambiguous and meaningless, and partly because it’s far more complicated “under the hood”. Just like your trip to see Led Zeppelin, it may be impossible to predict render times.

But vendors prefer to talk in terms of speed increases, because it’s a big number and sounds better and more exciting. It’s a whole lot better for them to say “It has 10x speed/performance/awesomeness!!”, rather than “if you’re rendering a certain benchmark scene using the latest hardware and drivers and software (NGC, Optix, Physx, CUDA, etc.), and you’ve enabled Optix and de-noising, and you’re rendering using Iray version xxx.xx and in Studio version xx.xx, and using this hardware on a computer that fully supports it, then our results show a 50% decrease in render times compared to the same scene on the same computer using previous versions of drivers which are more efficient with the older technology....” and so on). It’s very complicated.  

What we care about is how long it takes to complete a specific task (ie, a render). Speed can be somewhat irrelevant.

What we care about is how a new technology will improve render times. And clearly what that means is “how much decrease in render times will this new technology produce with a benchmark scene compared to the previous technology?”, so that we can do a meaningful comparison with the old technology.

So if our old technology renders the scene in 10 minutes, and the new technology does it in 5 minutes, is it “faster”? Well yeah, but in this case “faster” implies “the time to complete a task is less than before”, not “speed in miles per hour or calculations per second”. So “faster” can mean many different things. So it’s best to be specific, and discuss actual render times, since we now know that speed and render time can be two totally different things.

One way to measure that, and one that is arguably the most useful, is to determine the “percent improvement (aka, decrease) in render times from old technology to new technology”.  It gives you a very clear idea of how much direct improvement you might expect in render times. What it means in math terms is that you find the difference in render times between the two technologies (10 – 5 = 5) and convert that to a percent. And you want to make it a percent of the initial (old tech) render time, so that the final number reflects the “percent improvement (aka, decrease) in render times from old technology to new technology”, for that particular scene under those particular conditions. In this case, the new technology has decreased render times by 5 minutes, which is 50% of the initial render time, which means it has improved/decreased render times by 50%.

The nice thing about this metric is you can look at an old technology render time and fairly quickly calculate an expected render time with the new technology. A 50% improvement means it cuts render time in half. A 33% improvement means it cuts render times by 33% (or cuts 3.3 minutes off a 10 minute render), and so on.

Now, this number, just like the time it takes to travel on your journey in a rocket car, can never be 100%. It will always take SOME time to complete the task. Hypothetically it can get real close to 100%, but in the real world it won’t be 100%. Because a 100% decrease in render times means you go from 10 minutes to zero. Speed can go to a zillion %, but “percent decrease in render times” can never quite get to 100%. Even if you can somehow maintain an overall render speed that’s 1,000 times your initial speed, your render time might turn out to be 1/1,000, or 0.001 of the initial render time, and that’s like a 99.99% decrease in render times.   

Now you can certainly take the two render times and manipulate them to come up with various ways to describe the impacts of the new technology to meet your needs. For example, if you go from a 10 minute render with old technology to a 5 minute render with new technology, you might describe this as “50% faster”. But since “faster” is more ambiguous, you’d need to further describe this is “faster = percent decrease in render times”.

Now one of the most popular industry/marketing descriptions of new technology is something like “8x faster”. But it is clear this type of marketing term is likely referring to a speed increase, which can be unrelated to our main interest of render times.

In any case, the bottom line of all of this is that what’s really important to people doing renders is to determine, for your particular application, how long it will take to render a commonly accepted benchmark scene, under specific conditions (software/driver versions, system parameters, etc.). What is of little use are terms like “faster”, or “10X performance”, or any of the other highly ambiguous phrases, UNLESS there is a clear definition of what those terms mean, what’s actually being measured, and under what specific conditions. And if you hear “performance” numbers that go above 100%, be skeptical, because, while true under specific and unspecified conditions, they may be totally irrelevant to your situation.

nicstt

Good explanation.

I've been watching the discussion and argument of the percentages and figured I didn't need the hastle that joining in would involve.

I'm fine with faster and slower, and it tells me what direction I can expect the changes to occur; like you I like numbers, which can still be manipulated - especially when coupled with meaningless new terms (Gigarays). No doubt they will become useful eventually -or so I hope.