Unity Performance: Editor vs. Standalone
Unity apps are usually developed within the Unity Editor and then published to standalone apps for iOS, Android, Windows, Mac, and so forth. We’d therefore like to think that what we’re seeing in the Editor is very close to what we’ll see once we publish the app. In many cases, this is true. However, when it comes to performance this is sometimes a bad assumption to make.
In my introductory article to performance testing I mentioned this about performance testing in the Editor:
The final issue is that code running in the Unity Editor is running in something of a debug or development mode. It’s roughly equivalent to the performance you can expect from a distributed game, but sometimes slower. For micro-benchmarks like these, it’s important to actually make a distribution build of the game.
Today I’ll show one case where performance in the Editor is slower than in a published app. As it happens, this is an important case because it’s triggered directly by the auxiliary MonoBehaviour
used to capture and forward events in a “pure code” app.
The test sets up 1000 game objects and attaches one of two MonoBehaviour
scripts to them. The first is a simple Updater
script that has an empty Update
on it:
using UnityEngine; public class Updater : MonoBehaviour { void Update() { } }
The second is the full EventForwarder
script that handles every Unity event type as of Unity 4.6. It’s a bit long so I won’t repeat it here, but you can find its source in the original article.
To tie these together into a test, here’s a simple script to attach to your main camera in an otherwise empty scene:
using System; using UnityEngine; public class TestScript : MonoBehaviour { private const int NumGameObjects = 1000; private const float UpdateInterval = 1; private Rect drawRect; private bool showModeScreen; private float totalTime; private int numFrames; private float timeleft; private float fps; void Start() { drawRect = new Rect(0, 0, Screen.width, Screen.height); showModeScreen = true; } void OnGUI() { if (showModeScreen) { if (GUI.Button(new Rect(0, 0, 100, 100), "All Events")) { StartTest<EventForwarder>(); } else if (GUI.Button(new Rect(100, 0, 100, 100), "Just Update")) { StartTest<Updater>(); } } else { timeleft -= Time.deltaTime; totalTime += Time.timeScale / Time.deltaTime; numFrames++; if (timeleft <= 0) { fps = totalTime / numFrames; timeleft = UpdateInterval; totalTime = 0; numFrames = 0; } GUI.Label(drawRect, "FPS: " + fps); } } private void StartTest<TMonoBehaviour>() where TMonoBehaviour : MonoBehaviour { for (var i = 0; i < NumGameObjects; ++i) { var otherGameObject = new GameObject("GameObject #" + i); otherGameObject.AddComponent<TMonoBehaviour>(); } showModeScreen = false; timeleft = updateInterval; } }
I ran the tests on this computer:
- 2.3 Ghz Intel Core i7-3615QM
- Mac OS X 10.10.2
- Unity 4.6.2
The first test was in the Unity Editor to establish a baseline. Here are the results:
Test | FPS |
---|---|
All Events | 32 |
Just Update | 60 |
There is definitely a performance difference here. The “Just Update” test that only listens for Update
events seems to be pegged at 60 FPS. Meanwhile, the “All Events” test is running about twice as slow at only 32 FPS. This seems like a huge gap until we run the test as a standalone app.
Here are the test results run as a Mac OS X standalone app where the build settings were set to x86_64, non-development and the run settings were set to 640×480, Fastest, and Windowed.
Test | FPS |
---|---|
All Events | 57 |
Just Update | 668 |
The 2x performance gap has widened to almost 12x! The 60 FPS cap in the Editor was hiding the full impact of the extra event handling. This leads us to ask if the only issue is the FPS cap in the Editor. So let’s adjust the test to 15,000 game objects instead of only 1,000.
Here are the updated results in the editor:
Test | FPS |
---|---|
All Events | 3 |
Just Update | 51 |
The performance gap has now widened even further to 17x! Let’s see what happens if we run as a standalone app:
Test | FPS |
---|---|
All Events | 3 |
Just Update | 246 |
Now the performance gap is 82x! We’ve gone from 2x to 12x to 17x to 82x by only changing the way we run the app and the number of test game objects.
So what is the “true” performance difference between “All Events” and “Just Update”. Well, end users will never run the app in the Unity Editor so we can throw out those numbers. We’re left with 12x for 1,000 game objects and 82x for 15,000. Since these gaps are in the relative performance—not the absolute—we can conclude that the performance difference widens as you add more and more game objects. The reason for this is unclear as we can’t profile at this level within the Unity engine.
The main takeaway here isn’t the performance difference between “All Events” and “Just Update”. Instead, it’s to highlight the differences in testing environment. It is key to understand each environment’s performance characteristics, such as the FPS cap in the Unity Editor and that lack of a cap—event at monitor refresh rate—in standalone apps. It’s also important to not assume linear performance scaling, as we’ve seen with the varying performance gaps.
In the end you should always test in your target environment, be it a physical iPad or a designated “minimum specification” computer. Testing frequently will keep you in touch with the actual performance your end users will see. Varying your testing parameters will give you insight into the performance impacts of potential decisions, such as scaling up the number of entities in your game scene. Do not simply rely on the performance you see in the Unity Editor!
If you know of any other performance differences the Unity Editor imposes, post a comment about it!
#1 by Aleks on February 23rd, 2015 ·
Some time ago I wanted to translate unity’s magic functions to an event bus based on standard C# events and it turned out to be very slow so I abandoned that idea. I think this could have an effect on your test.
#2 by jackson on February 23rd, 2015 ·
The test is definitely showing that more of the “magic functions” are slower than fewer, but not any details about which ones are slower than others. It also shows the performance for just one (Update) and it seems fine, at least in isolation. A general test of C# events performance is probably in order. If they turn out to be prohibitively slow, you can always use simple callbacks instead. Stay tuned, that performance test will come up soon.
#3 by Aleks on February 24th, 2015 ·
Yes, but I think that empty Update is not equal to Update() { UpdateEvent(); }
So it would be faster even without calling the rest of the magic functions.
#4 by jackson on February 24th, 2015 ·
It seems like
Update() { UpdateEvent(); }
would be slower thanUpdate() { }
by the cost of dispatching one event. I’ll be measuring that event dispatch cost versus a normal function (i.e. callback) call to see if theEventForwarder
can be optimized for simpler cases.#5 by MDR on February 26th, 2017 ·
Unity Time.time, and Time.RealtimeSinceStartup are not perfectly in tune across devices and platforms. In the editor you get one scale, and that scale is skewed slightly on every platform and device. These tests show there are huge differences, but these numbers cannot be exact. If you try this same test on any two pairs of PC/Android/iOS/whatever devices – those Time.whatever numbers will be different. Even on two of the same model devices, with matching specs, with same system version, with same image…. you get what I mean :)
#6 by jackson on February 26th, 2017 ·
That’s a great point! Always test on your target devices with your actual game rather than assuming that performance (or even measurement tools) on one platform will be the same everywhere.