Game panels have been a touchy topic for server owners for a few years, some considering them a must-have, either for ease of use or other features the panels provided, while others consider they are useless bulk for the underlying game-servers. In this article, I'll discuss the benefits of using pterodactyl, a few less known features of it, along with my personal experience with the panel.

What is Pterodactyl Panel

Pterodactyl Panel is an open-source game panel that has become extremely popular with the Minecraft community recently. Unlike panels like multicraft, pterodactyl uses docker to secure each server in its own container. This improves security, performance and adds configurability for the server owner.

Docker, what?

Docker is a tool that provides OS-level virtualization to deliver software using containers. Containers are isolated from each other, and work as simpler and lightweight virtual machines. This makes the process of hosting game servers more secure, as clients are unable to access containers, not in their possession. This is extremely useful on Minecraft servers, as the large surface of attack (tens of plugins, bugs surfacing each day) make it harder to secure your server. With docker, if an attacker gains access to one of your containers, he will be mostly isolated there. While docker doesn't replace proper security policies, it definitely strengthens them and gives you more time to mitigate a potential security breach than not using it at all.

Optimizing Pterodactyl

One important note for pterodactyl users is the necessity to disable userland-proxy. While userland proxy may be useful for legacy software, it is often bloat for networking intensive Minecraft servers. You can disable userland-proxy by creating a file named /etc/docker/daemon.json and inputting the following code:

{
    "userland-proxy": false
}

Better Management

Unlike running Minecraft directly in Linux, pterodactyl means you can give people privileges based on their usage requirements. Your admins may need to run console commands, but not access files. This is now possible with pterodactyl panel. This means faster development times for you, and less worry about potential abuse.

Schedules

Schedules mean you can automatically do backups remotely, restart at night, and do more without too much hassle. Even the basic fact that backups are so easy to do is a huge preventive measure about potential information loss.

Disadvantages

Pterodactyl is often a slower way of managing your servers. You need a different SFTP session for each server, which is bothersome and often takes a lot of time that could be spent elsewhere. For many experienced sysadmins, the slowdown is enough to drive them away. Additionally, pterodactyl takes time to install, manage and maintain. It is just another thing that can go wrong. There is also a minor, yet noticeable performance loss when using pterodactyl. By disabling userland-proxy, you can minimize it, but it still exists.

Personal Opinion I personally prefer using pterodactyl for ease of use and the ability for me to invite others to contribute to the project, without too many risks on my end. Additionally, it is a lifesaver to less technically able users that may be brought onboard to help with management, etc.

07/09/2021 UPDATE I'm currently embracing pterodactyl with ReallyWorld, my largest client as of yet. Since implementing Pterodactyl as our method of managing the network, we have seen the following impacts:

1. Reduced time waste scrolling through 10+ dedicated servers.
2. Improved security significantly as attackers are now not able to get access to more than 1 server (or application) at a time.
3. Stability grew exponentially. Disabling userland-proxy has completely disabled any performance impact of pterodactyl, and the panel has helped us handle peaks of load way better than before.

This has resulted in over 2000-3000 more players and a revenue boost of more than 30%. We believe this is a true demonstration of Pterodactyl's use, and how it can perfect your server.

With the average Minecraft server having under 1 player online at peak time, very few servers have difficulties in building their infrastructures. Even so, the ones that do often have to go through the mud until they manage to build something stable. This blog post will explain some of the infrastructure designs used in the industry, with examples for minigame networks, game mode networks, and hybrids; all with the pros and cons of each of the solutions.

The Hosting Choice

Since hosting is the root of your entire infrastructure, taking the right choice here can mean the difference between being able to sustain tens of thousands of players vs barely reaching 1000. There are a few kinds of hosting methods I want to touch upon, along with my personal picks in terms of providers.

• Cloud Hosting: With cloud hosting being more prevalent each day, many minigame networks have decided to move to it to reduce costs and quickly scale with size. Since many networks are based on one timezone (Gamster included), this choice is able to reduce about 30-40% of hosting costs vs choosing dedicated. Also, this means that minigame networks no longer need to purchase expensive hardware even when they expect a large inflow of players, with the automatic cloud deployment system taking care of all the work. Even though this may be a dream solution for minigame networks, it is not so viable for servers mainly focusing on game modes like Survival, Factions, etc; where servers have difficulty in converting the game modes to work on the cloud. Favorite providers include: OVH, Hetzner, DigitalOcean.

• Dedicated Hosting: Dedicated is the second solution used by networks worldwide. With providers like OVH (SYS Subsidiary) offering effective dedicated solutions for low costs, including DDoS protection and effective support (If you call them); it's the easiest solution as it doesn't require developing a cloud deployment system for your network. Dedicated server hosting is most ideal for servers with stable player counts or network hosting game modes like Survival, Factions, etc. A good point is that poor DDoS protection providers often have the best pricing for the provided hardware! Favorite providers include OVH Subsidiary SYS, Hetzner, MYLOC subsidiary WebTropia.

Solving inherent hosting difficulties:

• Low cloud RAM: OpenJ9 is a JVM with lower memory usage (~30-40%) at the cost of CPU power (10-15%). It has been optimized for running in the cloud. It can also be used outside of cloud environments, with Gamster using a hybrid OJ9, Hotspot approach. More info available in TUX's blog post .
• Unreliable DDoS Protection: With most hosting providers having difficulties when you are under attack, the best option is to route your traffic through something you can trust, such as OVH. Providers like TCPShield can help you with that, or you can develop your own in-house solution. I've explained a bit about the former in a previous blog post, and I'll provide some images to explain some of the proposed infrastructure schemes. Avoid using reverse DNS or pointing DNS to your unprotected machines to avoid leaking your infrastructure info and being susceptible to denial of service. For OVH, buy a secondary IP and have a haproxy instance running on your main IP pointing to a closed internal bungee on the second IP. OVH will drop TCP connections to backend servers hosted on other providers when DDoSed. When using 2 IPs, with 1 closed and unavailable to the attacker, the second IP will never disconnect from your backend servers versus directly proxying being susceptible to DDoS, even though OVH Game!!!

Multiple Bungeecord Instances

Once you pass a point (~500 players), it's a good idea to start to use multiple bungeecord instances to ensure your server won't flop when your playerbases spikes. Good solutions include using RedisBungee (performance issues at huge player base spikes) or developing your own in-house solution to sync player counts via ping packets, and implementing between bungee sync whenever required. Because using multiple A records will mean bungeecord instances WILL NOT be properly load balanced, a good idea would be to have a haproxy cluster running behind to ensure almost perfect player spread on your bungee servers. Please note this will consume MORE bandwidth.

Infrastructure Examples

The following infrastructure is designed to allow the lowest pricing possible by allowing low-cost hosting providers. It uses HaProxy instances running on OVH, with a backend of cheap yet DDoS susceptible machines. Cloud can also be added into the mix if the user so desires.

PROS

• Easy to implement
• Cheap in comparison to other alternatives

CONS

• Consumes a lot of bandwidth. Almost double the bandwidth vs players directly connecting to the bungee, since the traffic is proxied 2 times (PLAYER <-> HA <-> BUNGEE vs PLAYER <-BUNGEE)
• Doesn't scale so well when you have tens of thousands of players.

The following infrastructure is designed for larger networks (5000+ players) that can save a lot by removing HaProxy from the mix. It is currently untested by me but is what Hypixel is most probably using. DNS service providers with smart queries are generally on the more expensive end, but once you rack tens of thousands, it's will reduce networking costs significantly, as instead of having player <-> HA <-> bungee you only have player<->bungee PROS

• Cheapest solution. Cloudflare with a watcher service on your end can be used for virtually 0$/month A balancing.
• Highly effective at huge player counts (5000+)
• Almost no extra bandwidth used.

CONS

• Your bungees need to be protected against DDoS vs your HaProxies
• Removes some of the proxying features that haproxy can offer.
• Rough load balancing due to not all resolvers following TTL.

Ever since Minecraft has become a global phenomenon, companies have popped everywhere to benefit from the growing market. From children buying Minecraft, building teams helping companies exploit the openness and ease of use that Minecraft offers, to community servers finding an extremely expansive market in hosting games for players. With the market still in growth and never truly dying, people have found ways to optimize their business models and formed multi-million dollar companies.

Revenues, in order of niche:

1. Community Servers: Since community servers benefit from the largest market in the Minecraft community (players), their revenues are the highest. Players purchase in-game benefits, either cosmetic or with real in-game value for large sums of money. The amount of server earns depends highly on the culture of the community they focus on, with the largest earners being focused on US players (for which, the average earn is over 30$/h), and the smallest earners focusing on eastern countries (Turkey, Russia, Romania: where earn is only 2$/player). Because of that, the market is highly differentiated based on the country they focus on. Players have different expectations, expenditures, and behavior.
2. Building Teams: Building teams also benefit from the large market of players, but since their work often provides less entertainment than live interaction with other players, their earn is also reduced. With the release of the Minecraft bedrock marketplace, builders are now able to monetize their work at way better margins than before. This encouraged innovation in building, which in turn grew demand, and along with it, prices. The largest downside with build teams is their requirement to work with Mojang to use their marketplace. This means fewer percentages left to the builders and a big dependence on Mojang's policies.
3. Developers: With these two main markets emerging and thriving, second-layer providers such as Developers now have a market to provide services to. While this niche is the worst paid, it also benefits from reasonable profits that can benefit sellers from third world countries and empower them to grow their skills. In my opinion, the limit that this market imposes is too small for any large company to develop, but this doesn't mean people haven't tried with moderate success.
4. Youtubers: Youtubers have become a large reason for Minecraft's booming success. The large community and simplicity of the game make it a perfect public presentation tool. This gave rise to diverse youtube niches, all stemming from the game. Their revenues depend on the country of origin and the contracts they have, but it is safe to assume it's a good and sustainable business model.

Monetisation Platforms With gaming servers being harder to monetize than other venues, several dedicated community platforms were created:

1. BuyCraft (now Tebex): BuyCraft was the first player in town, with its first plugin release on Aug 29, 2011. They quickly gathered most of the market and built a strong reputation for being reliable, efficient, and easy to use. Even so, their large costs meant both small servers (which couldn't afford the monthly cost) and enterprise networks (which wanted to avoid the 1% payment tax) started to look for alternatives
2. Enjin + DonationStore : Enjin was the second player on the market and quickly started to be used by several large networks (eg: Mineplex). They cost less, came with a decent amount of features (forum, web hosting, etc) and supported a wide array of payment providers. Even so, the old look that enjin had along with decreasing support for Minecraft (as they moved their focus to crypto) again created gaps in the market.
3. MinecraftMarket (now defunct): Minecraft market was the third player in the monetization platform business. With low costs and a "lifetime" plan costing only 15£, it quickly gathered many smaller income servers to use it. Unlike the previous two players, minecraftmarket came with more customization capability and was focused on the small server market. It suffered a security breach on 6/21/18, which along with a poor business model meaning always-decreasing profit margins made it die on June 21, 2018. This came with a large leak of client information containing hashed passwords, email, home addresses, etc.
4. CraftingStore (Recommended by us): With minecraft-market defunct, low-income servers started looking for low-cost alternatives. CraftingStore's effective support and reasonable feature set made it a good choice for server owners to move to. With many large networks (eg: Gamster) also using their solution due to the large feature set and 0% income tax. It is now the biggest contender with Tebex. Since CraftingStore is a smaller player and has a less mature solution, it also means less stability and security. Luckily, they have been quick in patching security issues.
5. DonationStore : Donation store is currently the only contender in the self-hosted marketplace market. With its business model being catered to a small number of networks, it is in my opinion the least favorable of choices. Being self-hosted means you take care of handling, updating and securing your own store, which drives focus from running your server to doing maintenance on things you can pay 15$ a month to get somebody to take care of it for you. The fact that the platform is self-hosted also means it is more prone to vulnerabilities, as the platform can't force updates on its customer base, and potential attackers can analyze the code to find vulnerabilities and exploit them. Again though, their team was quick to solve security issues, which along the self-hosted nature brings some value to the table.

The EULA controversy

With such a large market forming and relying on their game, it was reasonable to assume Mojang would take a stance. With their updated EULA, servers were now discouraged from exploiting the market (mostly comprised of children). This meant lower revenue margins for many servers, which now had to either comply or risk being blacklisted by Mojang.

Community drama Even though Mojang seemingly stepped out of the EULA battle, the expectations that players had changed. Many now viewed EULA compliance as a must and a sign of good server management. On top of that, several YouTubers jumped on the train, pointing to EULA non-compliance being a reason to cheat and attack servers. %[https://youtu.be/_gF67eaiLIk] With the video being a large success and garnering over a million views, several other YouTubers have also jumped on the train to exploit the trend. This restricted some communities to only following the EULA or face backlash from their playerbases.

Coming up with solutions Even though the EULA came with harsh restrictions, server owners have found ways to go around it and still keep their profit margins high. A few includes:

• Bypassing the EULA: As time passed, people found a way to bypass Mojang's EULA by using SRV records. Many large-scale networks decided to ignore the EULA and benefit from this bypass to keep their margins the same. While this technique is relatively safe for medium to large servers (under 3000 PB), it is impossible to implement by the large players (Hypixel, GommeHD, CubeCraft) without being noticed by Mojang.

• In-Game trades with real money: Players are encouraged to sell EULA friendly ranks to each other for real benefits. This allows players to pay to receive money or OP items from other members of the community. Since the server is not selling anything infringing on the EULA, it makes it both allowed and a way to reduce inflation due to injecting generated items. To incentivize this, server owners have to create rules and systems to ease this kind of transaction and reduce scamming. A good solution we have found is selling a virtual currency that can be used to buy store coupons. For example, if you sell 100 vcash for 1$, and the vcash can be transacted in-game either via in-game items (vcash vouchers) or commands (/vcash pay) and then cashed for coupons, this means players can virtually transact any item for virtual money.

• Giving away global rewards: XP Boosters, Spawner boosters can be a good way for a server to still provide rewards for spending money in-game without infringing the EULA. Additional systems such as extra boosts, if the server is low on players, can make sure as few players receive the perks while still rewarding the purchasers.

Our Solution Our solution to incentivize payments by allowing players to transact with real money. This has proven to be the most effective method of increasing growth at a smaller scale (~75 PB) but has faced issues with larger, less close together communities. To offset that, we have created a safe to use coupon plugin that prevents scamming and handles potential dupes. It is built on top of CraftingStore's recently added coupon API and allows owners to grant virtual cash to players. Get ERCoupons Now

For years, denial of service attacks have been a risk all Minecraft server owners had to take into account. With DDoS mitigation still being in its infancy by the time Minecraft gained popularity, many networks had to orient and build their servers around providers that allowed them to grow unaffected.

Introducing OVH

OVH has quickly risen out as one of the most reliable hosting providers which included denial of service mitigation. Their low pricing, along with the always improving infrastructure meant they were in the reach of most server owners while not hurting profit margins. With many enterprise networks currently embracing OVH as the best solution out there, and hosting providers reselling their services to the casual server market at a low price, the issue seemed fixed.

Minecraft-specific denial of service attacks

Just as any application that interacts with the internet, Minecraft needs a protocol (eg: a language) to allow the client and server to communicate. The popularity of Minecraft meant the protocol specification was reverse-engineered and publicized. With the protocol in hand and most of the server software open-sourced, malicious developers began developing software to crash servers. Plugin developers started popping up and providing solutions to denial of service attacks, most of them expensive and having ties with known denial of service perpetrators. While this has reduced the risk for a while, it only meant there was more on the line.

My initial solution

While working with the Gamster team, their decision not to pursue a long-term solution and instead resort to paying monthly "blacklisting" fees made me look into alternatives that would be able to handle large amounts of bots without large performance hits to the server. My solution was to combine ipsets with iptables to reduce the number of bots that would reach the server to a graspable amount. This was enough to prevent the pre-existent antibot plugins from being overloaded and crashing the server. The solution was presented to gamster management and was quickly implemented and further open-sourced for everybody to use. My post on spigotmc was moderately successful, with many antibot providers starting to implement the solution in an attempt to improve their plugins' ability to deal with large amounts of bots. Sadly, most of these implementations missed the idea behind the solution, and didn't result in major improvements to mitigating speed:

• ipsets were used to blacklist bots instead of to whitelist real players.
• There was no limit to the number of connections the server would have to handle, which meant large attacks were still able to cripple networks and shut down networks. While we further developed our solution and catered it to Gamster's infrastructure, the community was yet to adopt a widely effective strategy for mitigating botting.

Bungeecord Forks

A few Russian servers quickly developed their own in-house solution to handle botting, using drop checks along with captchas to detect and block bots from joining the server. While this was initially not widely adopted by the Minecraft community, several paid bungeecord forks such as Aegis have made using this system easy to grasp. This was enough to handle most attacks under 5000 connections per second but had trouble when attacks started overwhelming the system as a whole. Enterprise networks were still not covered.

TCPShield

TCPShield is a new solution that quickly came to popularity. Since their proxy services are currently free of charge, many servers have hopped on the train and finally received reasonable and accessible mitigation from Minecraft denial of service attacks. Even so, there were situations where people were denied from using tcpshield, still leaving a gap to be closed.

Our updated solution

With many networks contacting us to help mitigate their own botting issue, we have found a combination that has good results and is completely free of charge and can be used in-house:

• This Shell Script limits the maximum amount of connections your server will need to mitigate, making it easier for plugins to mitigate large attacks.
• The 2LS Antibot is a free and barely intrusive method to mitigate the bots that manage to pass through and smart bots at the same time.
• Depending on your infrastructure, a tool to automatically whitelist player IPs when they are validated to be true players (10+ minutes in-game, joined a server outside of your lobby servers, or when they finish a challenge) is recommended. For example, at gamster, we use a REST API to automatically update the ipsets on all of our haproxy servers every 5 minutes and whitelist our trusted players from being tested by the system.