Well, it’s 7:13 PM CST at the moment, and I’m shacked up in DFW due to some server upgrades for Goo at the moment. Unfortunately, we’re also recovering from our worst loss yet, a full wipe and deletion of all files stored on Goo’s main file server.
So, what happened? Well, based on what I can tell, when I loaded the the OST into the main MDS for Lustre, it caused a race which ended up blanking out our MDS, requiring a full wipe and reload of both OST’s, and the MDS itself. For those not familiar with the terms, OST is the object storage, or file storage, and MDS is the metadata server. Files are two parts, metadata, and the actual physical file. Unfortunately, when you lose the metadata, you lose all reference to the location on the drive where the physical file is stored. Due to this, the loss was pretty much catastrophic, wiping out every developer’s /home/ drive.
Good news though!
Web01
Distro03
Distro04
Storage02
lgswitch
Have all gone into production (Well, not the switch yet, more on that later.) These boxes will be doing as their names suggest, handling the web frontend, distribution of files, and additional storage space (10+tb now, for anyone asking….) This will take load off the current servers, and help to ensure a good experience overall.
Now, for the switch.
Lgswitch is sexy, and fun, and a blast. But unfortunately, we didn’t receive the switchmodule needed to plug our SFP’s into for the 10gbps fiber drop seen in the pictures below (It’s the yellow loop, if you’re curious.) So we’re stuck on 1gbit for the next two weeks, as I’ll be on call this coming week, and won’t be able to drive to Dallas, however, the following week, I plan on driving up and installing the module. Tuesday if I can wangle the day off, if not sooner, because I WANT MOAR SPEED!
Pictures!
Start:
In-Progress:
Done with current servers:
Done with all installs:
So basically the whole Goo file repository was sacrificed in the name of speed? That doesn’t sound very smart.
Not deliberately, unfortunately, as oxygen crashed during the server addition, it killed the metadata repository, which is unrecoverable. Believe me, I have no interest in killing the storage of our servers, as it’s what makes us so valuable to the community.
As a dev, I can always reupload. I’m thankful for the upgrade.
sucks about the storage wipe, but its a great way to get rid of dead files anyways, and love irssi running on that last pic! Great news on the server/bandwidth upgade tho! keep up the awesome work guys!
Irssi is all I use. At home, on the laptop, and at the DC.
Bummer. I thought the issue was initially on my Touchpad running CM9 until a Google search turned up this page. Just goes to show you don’t miss your GooManager until the server runs dry. Best to all running recovery.
Any idea how long till our home dirs are put back so we can reupload our files?
They should all be back at this time, if you’re still seeing issues, let me know, and I’ll check into it. Snipa on Freenode, or contact@snipanet.com
Thanks for doing all this, i wanted to clean up anyway ;D
[...] and users re-linked to /dev/ folders on Goo.im at this time. Full details can be found at: Lustre, Pain, And Goo. We're requesting that files be uploaded at your convenience, we're also beginning to work on a [...]
Who are (Snipa), By the way?
Would it be possible to rebuild some of the metadata from cached copies of your site?
For example, pull the MD5s and filenames off of http://webcache.googleusercontent.com/search?q=cache:U_WHb95l_NAJ:goo.im/cm/crespo4g/nightly&cd=2&hl=en&ct=clnk&gl=us (or just googling MD5 hashes of the files you still have) and rebuild (at least partially) the metadata?
You didn’t make a nandroid first???!!!
Can you spell?
It seems that you are using some form of Americanised English. Just for your records its MORE and not MOAR. Even the retarded IT technicians I know have some grasp of spell check.
Of course I can, I’m fully aware it’s more, rather than moar, however, I deliberately chose that spelling, mostly because I was annoyed at the time.