Lustre, Pain, And Goo.

Well, it’s 7:13 PM CST at the moment, and I’m shacked up in DFW due to some server upgrades for Goo at the moment. Unfortunately, we’re also recovering from our worst loss yet, a full wipe and deletion of all files stored on Goo’s main file server.

So, what happened? Well, based on what I can tell, when I loaded the the OST into the main MDS for Lustre, it caused a race which ended up blanking out our MDS, requiring a full wipe and reload of both OST’s, and the MDS itself. For those not familiar with the terms, OST is the object storage, or file storage, and MDS is the metadata server. Files are two parts, metadata, and the actual physical file. Unfortunately, when you lose the metadata, you lose all reference to the location on the drive where the physical file is stored. Due to this, the loss was pretty much catastrophic, wiping out every developer’s /home/ drive.

Good news though!
Web01
Distro03
Distro04
Storage02
lgswitch

Have all gone into production (Well, not the switch yet, more on that later.) These boxes will be doing as their names suggest, handling the web frontend, distribution of files, and additional storage space (10+tb now, for anyone asking….) This will take load off the current servers, and help to ensure a good experience overall.

Now, for the switch.
Lgswitch is sexy, and fun, and a blast. But unfortunately, we didn’t receive the switchmodule needed to plug our SFP’s into for the 10gbps fiber drop seen in the pictures below (It’s the yellow loop, if you’re curious.) So we’re stuck on 1gbit for the next two weeks, as I’ll be on call this coming week, and won’t be able to drive to Dallas, however, the following week, I plan on driving up and installing the module. Tuesday if I can wangle the day off, if not sooner, because I WANT MOAR SPEED!

Pictures!

Start:

In-Progress:

Done with current servers:

Done with all installs:

Batch 4

Type: Beer
Style: Hefenweizen
Base: Wheat Beer — Coopers
Batch Size: 5 Gal.

Additions:

  1. Additional Wheat Malt – Approx 2lb

Hops:

  1. 1oz German Hallertau – 30 min boil, 4.1% Alpha Acids
  2. 1oz Cascade – 5 min boil, 8.9% Alpha Acids
  3. 1oz Centennial – Added during cooling, approx 10 minutes, 10.3% Alpha Acids

Yeast:

  1. White Labs WLP300
  2. 1 pkt Bru Vigor

Data:

  • SG at Pri Ferment: 1.047
  • SG at Sec Ferment: 1.010
  • SG at Bottling:
  • Primary Ferment: 08/04/2012
  • Secondary Ferment: 08/10/2012
  • Bottled:

Notes:
Wort cooled using copper immersion chiller, inserted at 5 minutes prior to flame out, at the time of the Cascade Addition. Managed to avoid any boil overs. Yeast prepared as part of a starter in 1.040 gravity LME wort, 24h in advance, allowed to chill 3 hours to settle, spare LME wort poured off to clarify the yeast. Pitched at 78 deg F, yeast went right to work. Sitting in fridge at 65 deg f, good solid krausen with a heavy, stable foam. Bubbling approximately 1 bubble every 1-2 seconds. Bit of a sulphur scent, but that’s not unusual according to what I’ve read. No fermaid-K or other additives mixed in, does not appear to be required for something this low of gravity.

Batch 3

Type: Infusions

General:

  • Bottling Date: 07/25/2012
  • Clean Date: 08/08/2012

Infusion 1:
Alcohol: Vodka, White Gold
Additives:

  • Vanilla

Status:
Settling and clearing

Infusion 2:
Alcohol: Vodka, White Gold
Additives:

  • Saffron

Status:
Settling and clearing

Infusion 3:
Alcohol: Rum, Ron Barcelo Blanco
Additives:

  • Vanilla
  • Oak Cubes

Status:
Settling and clearing

Infusion 4:
Alcohol: Rum, Bacardi Gold
Additives:

  • Vanilla

Status:
Settling and clearing

Infusion 5:
Alcohol: Bourbon, Makers Mark
Additives:

  • Vanilla

Status:
Settling and clearing