|
Post by SchaumG550 on Oct 2, 2017 3:05:44 GMT
The server setup wasn't sloppy, we have one of the best servers possible with single threaded performance. The specific type I have on our host won't allow any type of RAID, the most I could do now was get drives attached via USB, I thought the server I picked was a great choice. The upcoming change I plan on making will implement drives in a RAID so it's easier to manage. 500GB isn't that much data but It's millions of individual files and trying to backup onto USB or with ftp isn't ideal at all. The last backups I have are too far back and wouldn't make sense to use them.
|
|
|
Post by Kazuniro on Oct 2, 2017 3:15:49 GMT
The server setup wasn't sloppy, we have one of the best servers possible with single threaded performance. The specific type I have on our host won't allow any type of RAID, the most I could do now was get drives attached via USB, I thought the server I picked was a great choice. The upcoming change I plan on making will implement drives in a RAID so it's easier to manage. 500GB isn't that much data but It's millions of individual files and trying to backup onto USB or with ftp isn't ideal at all. The last backups I have are too far back and wouldn't make sense to use them. Well, what's important is that you tried. And We all know you did. Just keep being a Supreme Walrus Lord. #HailThyWalrus
|
|
|
Post by Shadow on Oct 2, 2017 3:20:22 GMT
Alright, so. I've been keeping quiet for the most part and just tending to the FAQ with whatever information I'm given. Note that the following is not on behalf of staff, and purely from my individual perspective. This has not been the outcome that was expected when the server initially went down. Schaum did try. All was going well until it just wasn't anymore. It's disappointing and was overall a bit of a shock. There were ways it could have been prevented, yes. This is a fact. Was this known at the time? No. Is there anything we can do about the lost files themselves? No. This is also a fact. I understand the upset completely. A planned reset would have panned out much differently and could have been handled accordingly. However, it has always been a thing of "we will only reset if absolutely necessary," even when it's come up in staff discussions. That's part of Safe Survival's charm - a completely safe experience. Minimal hostile mobs, plenty of sethomes, warps, PvP only if you want to, and the promise of being able to build without staff plotting to reset everything. That's part of what brought me back here after my hiatus, it was the ideal experience for a more casual player. It's also what kept me voting and what got me to eventually donate. I adore this server and its community. It pains me that our hands are tied so tight here and I know everyone is upset with us. But there's nothing I can say that'll make it all better, honestly. We've been discussing the issues that have been coming up in depth in hopes of finding some form of a compromise. While Schaum has been working on getting the server set back up, we've been collaborating as a team (him included) to try to figure out what we're going to do. Donors and voters have been taken into consideration. Most things are very much "up in the air," if you will. We're still figuring it out. Note that we are still people and players that are under a lot of pressure right now. Not every staff member can even be present so our eighteen staff total is limited to a select group of people having to decide the fate of hundreds of players. Don't get me wrong - we did sign up for this job. We understood that it was not always going to be easy (give this person a /nick, ban this person for x-ray, warn this person for that, etc). We're doing this because we care about the server and are trying to make it the best it can be. We won't always be able to please everyone or even come to a conclusion that we're all happy with. As I stated before, it's a matter of compromise. It saddens me the amount of players that are choosing to leave or are, in short, furious that it happened. I wish there was more that we could do at this point. I am completely open to suggestions and I'm sure my teammates are as well. All I ask is that you all try to exercise patience with us. Please. I understand that the lack of a backup has caused a lot of players to be quite angry, especially for long-time players and donors. I, myself, am very much the create a backup-of-the-backup type but Schaum has explained above regarding the backup situation. That is what was meant by the "infeasible to backup 500gb" response. I understand that mistakes happen. Sometimes small, sometimes on a grander scale. Thing is, once it's happened, it's happened. Repeatedly attacking the source of it does not solve the problem. If anything, it escalates it and contributes to the toxic environment that I've heard players point out quite frequently over the past few months, especially. So, I've had my rant of sorts. I'd like to pass this off to you all. What do you suggest we do moving forward? Please refer to the following threads to see if your question has already been answered. [Server Reset FAQ] [Rulebook Update] [Redstone Rules Thread] <- Add redstone rule suggestions there, by the way.
|
|
|
Post by Ikith on Oct 2, 2017 3:23:26 GMT
The server setup wasn't sloppy, we have one of the best servers possible with single threaded performance. The specific type I have on our host won't allow any type of RAID, the most I could do now was get drives attached via USB, I thought the server I picked was a great choice. The upcoming change I plan on making will implement drives in a RAID so it's easier to manage. 500GB isn't that much data but It's millions of individual files and trying to backup onto USB or with ftp isn't ideal at all. The last backups I have are too far back and wouldn't make sense to use them. Schaum, I really hate to beef on you man but this is poor planning on your part weather you want to admit it straight up or not. Enterprise and small business servers do backups pretty regularly, and the files are a mixed bag when it comes to size, Word documents, Excel spreadsheet, Access files, ISOs for installations, Hard drive backup images from bare metal devices that have been removed from deployment and so on, all of these files get backed up for numerous reasons especially the instance you described in your original post which is hard drive failure. Enterprises and small businesses will take the steps necessary to be sure of data integrity on their servers. One thing that I must bring up is you never had to do this over FTP, there are plugins for Minecraft that will make a backup of folders specified then you can run a script on the Dedicated server to move that back up over to the USB drive! It's really that simple and should be your obligation as a server owner to make sure that backups run and that they are recoverable, you can even add to the script mentioned previously to prune old backups so your external hard drive doesn't fill up with old useless backups. I hope in the future you make your best effort to prevent this from happening again, do the external hard drive thing, this is your obligation not just to yourself but to your players and donators. You need to make an effort to preserve the data so this does not happen again. I highly suggest using Google to do this, look up cron jobs, look up file backup plugins etc, yes they will have an impact on performance while the backup is running, but really is it worth trading that performance hit and not backing up only to later down the line have to do this again and upset players and donators alike when you lose all of their data because you didn't want to put the work in to it? Alright, so. I've been keeping quiet for the most part and just tending to the FAQ with whatever information I'm given. Note that the following is not on behalf of staff, and purely from my individual perspective. This has not been the outcome that was expected when the server initially went down. Schaum did try. All was going well until it just wasn't anymore. It's disappointing and was overall a bit of a shock. There were ways it could have been prevented, yes. This is a fact. Was this known at the time? No. Is there anything we can do about the lost files themselves? No. This is also a fact. I understand the upset completely. A planned reset would have panned out much differently and could have been handled accordingly. However, it has always been a thing of "we will only reset if absolutely necessary," even when it's come up in staff discussions. That's part of Safe Survival's charm - a completely safe experience. Minimal hostile mobs, plenty of sethomes, warps, PvP only if you want to, and the promise of being able to build without staff plotting to reset everything. That's part of what brought me back here after my hiatus, it was the ideal experience for a more casual player. It's also what kept me voting and what got me to eventually donate. I adore this server and its community. It pains me that our hands are tied so tight here and I know everyone is upset with us. But there's nothing I can say that'll make it all better, honestly. We've been discussing the issues that have been coming up in depth in hopes of finding some form of a compromise. While Schaum has been working on getting the server set back up, we've been collaborating as a team (him included) to try to figure out what we're going to do. Donors and voters have been taken into consideration. Most things are very much "up in the air," if you will. We're still figuring it out. Note that we are still people and players that are under a lot of pressure right now. Not every staff member can even be present so our eighteen staff total is limited to a select group of people having to decide the fate of hundreds of players. Don't get me wrong - we did sign up for this job. We understood that it was not always going to be easy (give this person a /nick, ban this person for x-ray, warn this person for that, etc). We're doing this because we care about the server and are trying to make it the best it can be. We won't always be able to please everyone or even come to a conclusion that we're all happy with. As I stated before, it's a matter of compromise. It saddens me the amount of players that are choosing to leave or are, in short, furious that it happened. I wish there was more that we could do at this point. I am completely open to suggestions and I'm sure my teammates are as well. All I ask is that you all try to exercise patience with us. Please. I understand that the lack of a backup has caused a lot of players to be quite angry, especially for long-time players and donors. I, myself, am very much the create a backup-of-the-backup type but Schaum has explained above regarding the backup situation. That is what was meant by the "infeasible to backup 500gb" response. I understand that mistakes happen. Sometimes small, sometimes on a grander scale. Thing is, once it's happened, it's happened. Repeatedly attacking the source of it does not solve the problem. If anything, it escalates it and contributes to the toxic environment that I've heard players point out quite frequently over the past few months, especially. So, I've had my rant of sorts. I'd like to pass this off to you all. What do you suggest we do moving forward? Please refer to the following threads to see if your question has already been answered. [Server Reset FAQ] [Rulebook Update] [Redstone Rules Thread] <- Add redstone rule suggestions there, by the way. While this occurrence was not expected there were PLENTY of things that could have been done before this ever came up to preserve data and allow recovery of the server, maintaining servers is always a "be proactive not reactive" mentality and making backups as infeasible as it may seem even with the hurdles that Schaum described is one of those "proactive" things that should have been done as a server owner. As I discussed above, what Schaum described is still not an issue, you have a plugin which is allowed by semi-vanilla rules (as it falls under server administration plugins) that takes a backup every -insert date here-, then you have a cronjob (automated task that runs every set date, time or whatever combination of the two (theres more it can do but just as an example)) that runs AFTER that backup is done (if you want to be safe run your backups on a Monday then have the script move your backups over to the USB external on a Sunday or something like that) have that same script run a command that prunes all backups older than a certain date to prevent the external from filling. This would allow for data recovery in case of hardware failure.
|
|
|
Post by zot on Oct 2, 2017 5:01:22 GMT
Further more, even if Minecraft is written in java, all of the data is stored separately outside of the minecraft JARs in data files, all of these data files could have been backed up at the very least! This includes, the map, votes, claimblocks, claims, regions, coreprotect (stored in SQL but you can make a script that backs up SQL in fact you can probably find one on google and adjust it to your liking) and so on! All of this is stored separate from the minecraft JARs. To be clear, I to expected that they had backups and was disappointed that they didn't. But I understand why they are difficult and suggested some options to make them more likely in the future. But I want to be clear, when you are backing up hundreds of files, changes (updates) during the backup can create inconsistencies. And we all know that Minecraft really isn't very robust to begin with. So, to do backups, the server must be idle. Using techinternets.com/copy_calc?do for mydata, I looked at the time costs of transferring 500GB to backup media. These are very much the best possible numbers. Reality would be much worse. This is time that the server must be idle: USB 2: 2 hours 36 minutes USB 3: 23 minutes 1000BaseT (gigabit ethernet): 1 hour 15 minutes 10 GB ethernet (Note that disk drives don't reach this speed): 7 minutes SATA 3: 12 minutes There are certainly hardware solutions that can transfer 500GB in less than a second, but they are far beyond what you are going to get on a donation-based minecraft server. They require Fortune 500 (or NSA) IT budgets. I pretty much agree with all your other points.
|
|
|
Post by Ikith on Oct 2, 2017 5:12:39 GMT
Further more, even if Minecraft is written in java, all of the data is stored separately outside of the minecraft JARs in data files, all of these data files could have been backed up at the very least! This includes, the map, votes, claimblocks, claims, regions, coreprotect (stored in SQL but you can make a script that backs up SQL in fact you can probably find one on google and adjust it to your liking) and so on! All of this is stored separate from the minecraft JARs. To be clear, I to expected that they had backups and was disappointed that they didn't. But I understand why they are difficult and suggested some options to make them more likely in the future. But I want to be clear, when you are backing up hundreds of files, changes (updates) during the backup can create inconsistencies. And we all know that Minecraft really isn't very robust to begin with. So, to do backups, the server must be idle. Using techinternets.com/copy_calc?do for mydata, I looked at the time costs of transferring 500GB to backup media. These are very much the best possible numbers. Reality would be much worse. This is time that the server must be idle: USB 2: 2 hours 36 minutes USB 3: 23 minutes 1000BaseT (gigabit ethernet): 1 hour 15 minutes 10 GB ethernet (Note that disk drives don't reach this speed): 7 minutes SATA 3: 12 minutes There are certainly hardware solutions that can transfer 500GB in less than a second, but they are far beyond what you are going to get on a donation-based minecraft server. They require Fortune 500 (or NSA) IT budgets. I pretty much agree with all your other points. The server wouldn't really need to be idle, maybe a warning that the server might be slow while it runs backups, I can't imagine it would take too long with low compression or differential backups. Same with copying files I don't think the server would take a major hit when transferring files VIA usb directly instead of via FTP :\
|
|