I feel like some HFT/fintech concepts are ~ideal MMO architectures.
With a 2 tier setup, you could theoretically have a central server processing batches of events that are aggregated across multiple players/regions/etc. A single thread can service upwards of half a billion events per second, and if each of those events covers multiple potential players, then I'd argue we are in a pretty good position.
For me, having truly 1 consistent global universe is the only thing that would get me to consider an MMO in 2023+. The technical limitations forcing sharded worlds were excusable when WoW was released, but I don't have patience for those arguments anymore. Not if you want me to expressly burn my time and get paid for it on a recurring basis.
That's what I was hoping for from the "metaverse" crowd - really big, seamless worlds. Didn't happen. For the amount of money spent, the results are very disappointing. Facebook/Meta spent how many tens of billions of dollars on this? The crypto crowd produced zero functioning high-quality big-world systems.
I had some hopes for Improbable. Improbable has a big seamless world metaverse back end system. It involves a lot of remote procedure calls between multiple servers. The result seems to be excessive server costs. Five indy games, some quite good ("Worlds Adrift" was one) tried it, went live, and went broke. Improbable, a VC-funded company with about $400 million, has been thrashing around ever since. They tried setting up their own game studio, and produced "Scavengers". That was a game where a few thousand players all charge the same goal. More of a crowd tech demo than a game. After that flop, Improbable pivoted to making simulators for the British military, which apparently worked, the military not being too concerned about a few dollars per hour server cost during war games. That job done, they tried hooking up with Yuga Labs, the Bored Ape / Otherside people. Did another zerg rush demo for them. But cost per user per hour was so high they only ran that twice, for a few hours each time. Now they're pivoting again, to servicing baseball ("MLB", as the baseball industry calls itself) so that fans can watch games in VR, or something like that.
So what went wrong? The business problem was that ad-supported metaverses don't work. There is no role for "brands" in a highly immersive world. Quite a few companies have now figured this out the hard way. Ignoring that, though, what are the technical problems with scaling?
Having spent too much time inside Second Life client code, and written my own client, I can answer that. First, the user needs a "gamer PC" and serious network bandwidth to deal with a highly detailed dynamic world. With user created content, a key metaverse feature, there's far less instancing, and you need about 3x the GPU memory of a curated game. So you need roughly an NVidia 1060 and a few hundred Mb/s of network connection. The average Steam user has that, (see Steamcharts) but the average Facebook user does not. If you want to support WalMart $99 PCs and phones, there's a big problem.
"Cloud gaming", where the GPU is in a data center, lets anything that can play NetFlix play AAA title games. The problem is cost. Most of the cloud gaming hosting services gave up. Even Google gave up. NVidia GeForce Now remains, but they've raised their prices several times. Each user is using a dedicated PC-class system with a good GPU, so this isn't cheap. So trying to solve the user cost problem with cloud gaming doesn't work.
Server side is actually less of a problem. The Second Life server architecture isn't bad. Second Life
was supposed to be the "next Internet" when it was designed over 20 years ago, and as a result, it's overdesigned compared to most MMOs. User-developed clients are not only encouraged, most users use a third party viewer, rather than the open source Linden Lab viewer.
The networking architecture to the client is not too unusual. The time-critical stuff is on UDP, and the bulk data transfers are on HTTP. The UDP system supports out of order delivery to eliminate head of line blocking. (Reliable delivery, in-order delivery, no head of line blocking - pick two.) The trouble with allowing out of order delivery is that higher level operations with state can get into trouble. Hence discussions like this.[1]
The most unusual thing is that each client talks directly to multiple region simulators in the same area. This is what produces the seamless world illusion. The simulators also talk to each other, but mostly about objects crossing the boundaries between regions. Inter-simulator traffic is thus manageable. It's more of a state locking problem than a bandwidth problem. The design predates the theory of eventually consistent systems and conflict-free replicated data types, leading to immersion-breaking out-of-sync problems, including teleport failures and getting stuck crossing the boundary between regions.
With this architecture, there's no major limit to the size of the world. The internal traffic within the data center per region simulator does not grow with the size of the the world. Nor does the per-client traffic. Traffic to the asset servers does grow, but that's cached (AWS + Akamai). Shared services (login, billing, etc.) have multiple servers with load balancers.
So that's the successful path to a big, seamless world.
There's much trouble with seemingly random sluggishness, but that turns out to be due to various specific problems, some of which are being fixed.
Yes! 1 consistent global universe would get me to jack into whatever matrix they were selling.
It could even be multiple games connected with sufficient break in context. Eve for space and another game for ground assault. But they actually interact live.
Then throw in next gen VR and you basically have ready player one.
CCPs previous FPS DUST514 also linked with EVE. All the resources were in the EVE economy and ships could perform orbital bombardments. Although it mostly demonstrated how horrible it is to try to intertwine two very different games together!
I worked with Oddur and Ivar there quite a few years ago now so am looking forward to SEED!
I originally bought a PS3 to play Dust 514, and only got to play for a few months before it got shut down. I really like the concept of different games sharing state and universe. I hope they eventually crack this nuts.
> I feel like some HFT/fintech concepts are ~ideal MMO architectures.
Can you explain/link to some?
I suspect MMO servers, particularly when server authoritative end up having to do a lot more processing per event and tick. Distributing world state updates is also an extremely thorny issue with lots of inter-dependencies.
Whatever is good for low-latency trading systems (i.e. where you are paying contractually for microsecond-level guarantees), is also maybe good for gaming.
The whole concept with these patterns is that there is only ever a single writer at any given moment. Single writer principle w/ batching is what gives you enormous uplift in throughput, even when you are necessarily constrained to fully serialized business semantics. I'd argue that an MMO does not need to be as strictly serialized as whatever CBOE, et. al. are doing.
Interesting, thanks! Skimming the Disruptor paper its very similar to how you keep audio context fed using a ring buffer to write ahead as the audio context chews through behind you so the entire thing can remain lock-free and low latency. Typically in a separate thread as the latency and timing requirements are very strict.
Aeron on the face of it looks very similar to the architecture diagrams you'd get popping up designing a networking layer in a game already. I suspect the cross-pollination in both directions could be interesting.
Neither seem to particularly help with the intensive/interesting bit of an MMO which is running the actual simulation. For example big fleet fights in EVE as far as I remember have never been IO bound rather single-core performance bound. It's common for the bigger Corporations to contact CCP ahead of big fights so the node it's likely to happen on can be moved to a bigger machine. It's actually quite interesting because a lot of it is written in Stackless Python so not the most performant of languages and trapped in a single-threaded context so there is (nominally at least) quite a lot of headroom.
For big seamless worlds how you distribute the simulation is the tough bit. Hence companies like Improbable trying to solve that generally.
With a 2 tier setup, you could theoretically have a central server processing batches of events that are aggregated across multiple players/regions/etc. A single thread can service upwards of half a billion events per second, and if each of those events covers multiple potential players, then I'd argue we are in a pretty good position.
For me, having truly 1 consistent global universe is the only thing that would get me to consider an MMO in 2023+. The technical limitations forcing sharded worlds were excusable when WoW was released, but I don't have patience for those arguments anymore. Not if you want me to expressly burn my time and get paid for it on a recurring basis.