Cloud Gaming: stress testing cloud gaming services using poor internet connection

A detailed comparison of five of the most popular cloud gaming Services

This is a translation of the article, which was initially published by user Gaddictive on a Russian resource Habr. We sincerely and wholeheartedly thank Gaddictive for such enormous technological research and for the permission to publish a translation of the original work.

The video used in the original article is also created by Gaddictive — here is the link to the original video in Russian. We also thank Gaddictive for providing us with both the original video footage and the permission to translate the video into English.

About a year ago I published an article “Cloud gaming: a hands-on review of the low-end PC gaming services”. I analyzed every cloud gaming service with its pros and cons. I did so by running games on the servers and describing my user experience.

The opinions in the comment section on that and other similar articles were always mixed. Contradiction opinions were a common sight. Certain users reported flawless and smooth gameplay, whereas others were experiencing freezes and delays. And this gave me an idea of testing the same cloud services but in different conditions. The point is that users cannot always have an ideal internet connection, so testing the services in different network conditions from nearly flawless to the worst possible is something that is worth proper research.

The main issue

As stated previously, the biggest issue is the connection quality. To be more precise, packet loss is the main culprit. The more data gets lost, the less a gamer is satisfied with the quality of the service. And let’s be fair, most of us can’t boast having a fiber connection that goes directly to their machine. In fact, most ISPs share the channel between all the people who live in the same housing/area.

Here is how it works: at 25 Mbps connection speed sending a single frame requires 40/50 packets. Lower connection quality means more packets are lost and occasional freezes and lags become more and more noticeable. It can get to the point when the game simply becomes unplayable.

Naturally, cloud gaming services have no control over the end users’ connection quality and cannot influence it. However, there are always ways to compensate for occasional data loss. Which service is the best when it comes to working with poor connections? Let’s find out.

What is being compared?

Regular PC (Intel i3–8100, GTX 1060 6 GB, 8GB RAM), GeForce Now (its Russian partner GFN with the servers located in Moscow), Loudplay, Vortex, Playkey, Stadia. On all services, except for Google Stadia, we run The Witcher. As Google has no such game in the library, we run a similar in terms of requirements and picture quality game Assassin’s Creed Odyssey.

What are the conditions for testing?

The location is Moscow. ISP — MGTS, with the 500 Mbps connection speed, direct cable connection, no Wi-Fi. Default graphic settings at Full HD resolution.

We are using Clumsy to imitate possible connection issues, such as packet loss of different kinds and volumes.

• Evenly distributed packet loss. Losses are distributed more or less evenly, and there are occasional single packet drops. Hence, in the case of a 10% evenly distributed loss for 100 packets each 10th packet is lost. It is important to remember that it will always be just a single packet. This could happen during various distortions on the channel from the server to the end-user.

We will be testing evenly distributed packet loss at 5%, 10%, and 25%.

• Mass packet drops, losing approximately 40–70 packets at once. This could happen when there are issues with the hardware (routers, cables and such) both on user’s or ISP’s end. This could also be the result of buffer overflow on the channel between the server and the user. Wi-Fi signals could also get distorted in a similar manner when they go through thick walls. Wireless network overload might also cause similar issues, it is especially common in apartments and crowded offices with many devices connected to one network or several networks being present in a small area creating interference.

We will test with occasional mass packet drops at 0,01%, 0,1%, 0,5%.

Below you can find my analysis for all of the above-mentioned cases along with the video examples. And at the very end, there will be a link to row uncut footage from all services in all possible use cases — where you can take a closer look at the artifacts as well as the detailed tech information (tech stats are recorded for all services except Stadia, as I could not find a way to access that information).

Let’s go!

Use case #1. Ideal conditions. Zero data loss.

Cloud gaming the way it was intended. No drops or connection issues, your internet access is flawless. Practically all cloud gaming services are doing great.

PC

In each of the use cases, the PC is taken for reference. The games are run locally, so network interference does not influence picture quality in any way. The footage is here mostly for reference as well as to answer the question if there are any noticeable downgrades when it comes to cloud gaming services. In the ideal conditions, there are no noticeable differences and cloud gaming services provide experience identical to a game run locally. I will not refer to the PC here anymore, but it will be in the video comparison.

GeForce Now

Looks great. No lags or freezes, the picture is clear. A smooth experience.

Vortex

Vortex seems to perform the worst in our comparison. The service has the worst picture quality and there were noticeable stutters. It did worse than any other service in all of the comparisons. Perhaps, the reason for that is that the servers are quite far from Moscow and the hardware is relatively weak, so it has some problems with Full HD content. If you had a different experience with Vortex please write in the comments below and let us know where and in what conditions you were using the service.

Playkey

Works great, as if the game was run locally. No freezes or lags.

Loudplay

Great picture quality, no visible issues.

Stadia

Although Google has no official support for Stadia in Russia everything runs just fine. No issues here. Although we had to take Odyssey instead of The Witcher, as the latter was not available on Stadia. Still, a fair replacement choice. Both are quite demanding games, same genre, the same concept of a guy cutting and slashing whatever gets on his way.

Use case #2. Evenly distributed 5% data loss

In this test out of every 100 packets approximately every 20th is lost. Just as a reminder, drawing a single frame requires about 40 or 50 packets.

GeForce Now

Nvidia does a really good job here. Although Playkey provides a slightly clearer picture everything runs great and The Witcher is still very playable.

Vortex

It is getting even worse. Looks like the service implies little to no redundant data sent. In this case, redundancy is encoding data in such a way that it can still be read even if there are connection issues along the way. (We are referring to FEC — Forward Error Correction). There are many ways to implement such a technology and based on our results Vortex failed to make use of this technology. On top of that, with evenly spread relatively insignificant losses the service struggles. In any of the following tests, the service simply refused to work.

Playkey

Everything runs just fine. Perhaps, having servers in Moscow helps with the data transfer, or the above mentioned FEC is implemented in a better way.

Loudplay

The service becomes unplayable, although in the previous testing it had no issues. What could be the reason? Well, I might suggest that the cause of this is in the TCP protocol utilized in the service. No new data is being sent unless there is a confirmation (an acknowledgment) that a previous packet has been received. Hence, when there is no acknowledgment for the data received, the entire thing freezes.

Said acknowledgment would not be necessary if the service switched to UDP protocol. As far as I can tell all the other services in comparison are, in fact, using UDP. Please, correct me in the comments if I’m wrong.

Stadia

Stadia is very playable. There are occasional pixelation in frames here and there. Insignificant input lag appears. It is possible that the error correction algorithm is not perfect, but things are very playable nonetheless.

Use case #3. Evenly distributed 10% loss

Losing every 10th packet out of 100. This is quite a challenge for cloud gaming services as this requires redundant data being sent. This means either sending packets more than once or recovering data.

GeForce Now

GeForce shows a visible decrease in video quality. Looks like GFN does detect network issues and is trying to make up for that. The service does so by lowering the bitrate. The idea is that the service considerably decreases load on the network to increase connection stability. It does keep the connection more stable, although this hurts picture quality. We see a very noticeable pixelation in the picture, as an attempt to stabilize the network. And since the testing implies constant data loss lowering the bitrate does not help the situation.

In a real-life scenario, this would mean that the video quality would fluctuate from higher to lower bitrate and then back. Basically, the service lowers picture quality when it detects connection issues, which does impact the gameplay experience negatively.

Playkey

No noticeable problems here. My guess is that the algorithm detects packet loss and focuses on data redundancy instead of lowering the bitrate. So, the user is very unlikely to notice any picture quality drops.

Loudplay

Unplayable. The service simply refused to work here and in further testing. Looks like the server simply does not work with connection issues. Perhaps, the TCP is to blame. In any case, this would be a very unpleasant news for real-life scenarios.

Vortex

Unplayable, but it does work. The picture is still here, the character does move but the frames are missing, there are freezes and the picture quality is low. Yet again, this might be caused by poorly optimized data redundancy, so the client app just has no idea what to display for a large portion of time. And as the data is lost and there is no means of recovering it, poor picture quality and freezes make the game unplayable.

Stadia

Things are not looking great for the service. As the data stream is interrupted, the picture stutters. The cause of this is likely in the lack of FEC. I asked a few people who follow tech news and are familiar with the service. Apparently, Stadia waits for the whole frame to load. So, unlike GFN there is no bitrate decrease, but the trade-off is that we get occasional freezes.

Stadia seems to be the only service that waits for each frame to load, as other services just use previous frames to fill in the missing data. This solution seems to work, as it is extremely difficult for a user to notice the difference. After all, the services load and display over 30 frames each second.

Use case #4. Evenly distributed 25% data loss

This is a pretty bad interesting case, as every fourth data packet is dropped. To be fair, these conditions mean that practically any cloud gaming experience is very difficult to achieve. And yet, some of the participants are coping with such conditions, although the issues are visible.

GFN

The picture is far from being clear, there is noticeable pixelation. This is still playable, although much different from what was promised by GFN. This is simply not the way it’s meant to be played.

Playkey

The gaming experience is quite smooth. The picture quality does take a hit, but it is still decent. If we take a look at the numbers (check console in the video), the service manages to recover 96% of all data packets.

Loudplay

Would not run.

Vortex

Unplayable. Freezes, dropped frames, the picture quality is extremely low.

Stadia

Practically unplayable. As mentioned above, the service waits for each frame to load, especially with minimal data redundancy. Stadia struggles with this challenge.

Use case #5. Mass packet drop 0,01%.

For each 10 000 packets about 40–70 packets are dropped at once. Which means losing approximately 1 out of 200 frames. This happens when there is a buffer overflow and the hardware simply drops the data it is unable to take.

GFN

There is a very slight decrease in video quality, however, things are still very playable.

Playkey

Looks great, smooth gaming experience.

Loudplay

The picture was crystal clear, the service could have won this test. Unfortunately, the very first packet loss ended the connection. The TCP protocol is not helping this service at all.

Vortex

The usual stuff, freezes, skipped frames. The experience is what we would call hardly playable.

Stadia

Playable. Occasional bitrate drops and pixelation, but is still very playable.

Use case #6. Mass packet drop 0,1%.

For each 10 000 packets, there are about 10 drops of 40–70 packets each time. We are losing about 10 frames out of 200.

Most services did experience issues here. The worst part is that data redundancy would not help much here. It would slightly impact the experience, but the benefits are very insignificant.

The thing is that the data stream has to be uninterrupted, otherwise, there are dropped frames and missed control inputs from the player, so the video stream gets choppy and there are freezes/drops.

GFN

The quality is much lower, so is the bitrate.

Playkey

It seems to be coping better than GFN, the FEC and bitrate algorithm are well optimized which helps the service to keep video quality reasonable.

Loudplay

Would not run.

Vortex

The service did launch, although this does not make the situation any better. Picture quality is far from decent, constant freezes and frame drops.

Stadia

The picture is stuttering, this is a solid proof that error correction is not well optimized. The picture freezes, then different frames appear. So, it is sort of playable if you are absolutely desperate or exploring new methods of self-torture.

Use case #7. Mass packet drops 0,5%

For 10 000 there are 50 drops with about 40–70 packets each time. So, we lose about 50 frames out of 200.

That is the worst-case scenario, there is smoke coming out of your router and it looks like someone has attempted to cut your cable connection several times. And yet, you are still willing to give cloud gaming a try. So, which service is the best at handling data in these situations?

GFN

This is sort of playable, although the picture quality is extremely low. The service is lowering the bitrate as much as it is possible. This makes me question, whether there are any ways of recovering lost data or any algorithms of working with data redundancy.

Playkey

Frames are visibly distorted, the experience is not quite smooth. It is very noticeable that the service is using previous frames to recover the lost ones. Overall, it is still playable, although, in intense moments, the frame drops and lag will make it an unpleasant experience.

Loudplay

Would not run.

Vortex

It launched the game, although the result makes you wish it did not. Unplayable.

Stadia

Unplayable. The service waits for frames to load and does not make use of data redundancy.

And the winner is…

The chart and comparison are extremely subjective. Feel free to disagree in the comments. Evidently, cloud gaming services are very sensitive to the quality of your connection, so in the real world, a gaming PC is still the best way to play games. However, if you do not have one, see the results below.

1. Local PC
2. Playkey
3. GeForce Now
4. Google Stadia
5. Vortex
6. Loudplay

So, as a conclusion let’s outline the factors that determine the quality of your cloud gaming experience.

1. The type of protocol used by the service. UDP seems to work the best, although I believe Loudplay opted for TCP for some reason. I might be wrong here, but the results speak for themselves.
2. FEC — Forward Error Correction a.k.a. Data redundancy. Being able to recover lost packets or make up for occasional drops means a lot for a cloud gaming service.
3. Bitrate adaptation. Lowering bitrate could be a great way to keep the gameplay smooth. However, it may impact the picture quality too much. Balancing bitrate and redundant data is the best way to achieve smooth cloud gaming.
4. Post-processing settings. The way a client app handles missed or dropped packets. The decision between waiting for a whole frame to load or just using the previous frames to fill in the lost parts.
5. The closeness of the servers also has a great impact on the quality of user experience. This is also true even for ideal conditions when there is no data loss. High ping makes it very difficult to play anything, no matter how reliable a connection is.

And just as I promised, here is a link to the raw footage from all services: https://www.youtube.com/channel/UCggfBhPjs9JdlTbBFMvBeIg/