The McDonalds of IDC’s?

I mean that in a good way.  I can see why they offer such a compelling IDC choice for many companies.  From being carrier neutral, easy cross connects, standardised security and operational procedures – heck even the free wifi is based on the same docket printing and login system.

While I give kudos to EQUINIX Sydney for providing a couch area with TV, ping pong, food and coffee.  I must say that charging people for a cup of machine coffee is a little on the tight side.  The showers are a nice touch – but LG DATACOM in Seoul still takes the cake as the best IDC that “Gets it” when it comes to poor nocturnal IT staff.  They provide a mini hotel/dormitory area where you can also catch some sleep in private while that massive rsync finishes or your team works in shifts to complete work within the downtime window.

I always try to get the best prices within the scope of common sense for clients. Mainly because as a integrated stack vendor, the pricing on one component can blow out the total price of the whole stack. And hardware is the first place to start (given an understanding of the management systems and risk credentials of the project at hand already).

One thorn in my side is mandated “Installation and Configuration” charges. Which are basically extortion attempts.

On the one hand no tech from SUN or IBM from their mainframe/mini computer division is going to know or be able to setup all variables in our system as we need – and as we will do over a long period of time – IE: the process of “Configuration” is Iterative and not finite to one small window of time. :-\

And going back to that whole vertical stack business. Why should a client pay a hardware vendor for support and then pay the integrator for support? These built in service charges make it harder for us to win business and sell anything as we have to protect ourselves as we will be doing the support – and that usually means that the client pays twice!

As much as these hardware firms like to chase the integrators and system builders as a great “Channel” to their products buyers in the market place. Mandatory setup and Configuration fees (Yes they apparently are mutually exclusive in charging if not in the vernacular of the Queen’s English!) are a great way of say “Up Yours” to said channel partners/integrators.

It is not yet clear who will get the Gold Medal – but the race is very competitive!

:-\

Select from database ID where name = RICHARD;

Returns and ID of 55 for example.

Then go and do a query like this:

Select * from some_other_table where ID = 55;

Returns, “Sorry does not exist, time to die…..”

Well apparently indexes when corrupt – which is NOT SUPPOSED TO HAPPEN – can cause PostgreSQL to go all stupid and not do a table lookup for real. This happened to me. So I found this:

PlanetPostgresql

Turns out that a reindex and a full vacuum can do wonders – even though a full vacuum is not needed with autovacuum and indexes can’t get corrupted…..or so they say.

I have now added a system wide reindex maintenance plan for PostgreSQL every night. I know that MS-SQL server has an option for this with their maintenance jobs inside enterprise manager. Maybe someone should make an enterprise manager for PostgreSQL too?

In any case, the short of it is, that we were juggling PERC4 cards around servers (PCI-X here, PCIe there..) and also complete raid 1 and raid 10 arrays too. The cards are supposed to “detect” the correct array type from the drives if the firmware was missing. Anyway, through a comedy of errors, it worked exactly 1/3 times. The other times we had to remember the exact settings of our arrays (stripe, etc) and how it was structured. So we could clear PERC cards and then recreate the arrays – taking special care to not initalise the new arrays.

So in the end, you can move arrays and channels about. And with LVM, even designations like /sda /sdb reording is also not an issue. However you should rely on good old fashioned hand held way of doing things. Before you start write down all the salient details of your arrays first.

DRAC RESET

What is a mixed bag in my mind is the new ESX3i. It will be first released in the new Dell VORSO servers, originally due to ship this month and as confirmed by my DELL Sales manager, is slipping into the new year. These servers will not need hard disks (So what about core dumps, logs and swap?) but will have embedded flash with the hypervisor installed.

On the surface it looks like the Hypervisor is now part of the system board. It ain’t. So I don’t expect ESX3i to give any speed increase. What I do expect is that it will give a massive boost to deployment times. If you are deploying new servers at the rate of one or more a week, this is cool. Right now at CANDIS with blades, we have all the wiring already in place due to the chassis. But we still need to go in through the management card to setup the OS. Yes we can open the box and slide in the blade and have a new server fully setup physically. But not the software. From the looks of ESX3i, it goes a step further. Power on the system, it finds a VC server, configures itself, boots up and presents it’s resources to the farm, ready to be dynamically allocated by the VC server.

Oh, they also have smart power management that can power down systems when their computing resources are not needed. This is cool. Since we pay about .2 RMB per kilowatt of power, it does add up. The new low voltage XEON’s are great power savers, as are the new high efficiency power supplies. However add in those dynamic server on/off controls and you will be able to make a serious dent in your power bill. Oh, and keep the “Eco-Terrorists” happy too and thus maybe avoid a politically correct jihad aimed at you, unlike poor Apple Inc.

VMWARE

CPU

System load was DOWN. This is due to much better disk IO (reduced wait times) and as such, user processes get to breath and send stuff back to clients in a speedy fashion.

LOAD

Load being a matrix balanced scorecard measurement over various metrics is also down considerably. The trend to take note of here is the 15min average more than the 1 min average. While spikes and load increases over short periods may upset some users experience. If the delays compound and extend over a longer time frame, you will be tying up more people in the slowdown. Again, this is leading a bit to the parallel vs serial arguments. And in the case of a database, sometimes fewer ‘in and out’ transactions give a better user experience over time because more gets done in a given time period, relative to many threads fighting at once. Think orderly lines at the bank in a developed country compared to the Beijing Subway at Jianguomen or Fuxingmen!

TEMPERATURE

A good 5 degrees lower in temp too. That is not insignificant. It all adds up to longer system life, better availability and lower costs.

POWER

Well out APC metered units aren’t that precise! While you can’t see any change from last week to this one – probably due to the load balancing effect of our redundant power supplies and redundant PDU’s AND UPS’ in between PDU’s and Servers (in addition to the IDC’s UPS’s and 4,000 Car Batts). You CAN see the increase in Saturday and Sunday while the massive disk arrays churn flat chat to reinitialise….that will need more power and also make more heat, needing more fan speed.

I used this excellent resource before.

Even the very rudimentary tweaks I did, and with a RAID array that is in a 90% rebuild rate, background initialisation, this 8.1 version is FAST! I have no idea why people use MySQL, it really is such a piece of crud.

Let me expand on this;

This type of backup file is advertised as “more convenient” and offers more options for restore time selective data restores, data re-ording, index tricks and the like. However no matter WHAT I did, it reported and sent a copy of the current pg_restore process and all the data being restored to standard output too!! This means that basically, I was going to have the same full text of 20GB worth of database data shoved down my SSH session!

Yes – this makes the whole affair much slower!

Luckily, being the lateral thinking kind of dude that I am, I never put all my eggs in one basket. That is just a recipe for data omelette.

I also had some plain text file dumps made from pg_dump. So I went into the the PostgreSQL template1 sessions and then used the “\i” command to import my big .SQL file. Perfect! Only important updates sent to std output and not the whole damn enchilada.

Disk restore is still going on. At a max through put of 20MByte a second with a 1000BaseT network, that is at best 1.2G a minute, 72GB an hour, so for 250GB approximately 3.5 hours. However I was still doing a raid array background initialisation, so even after setting the rebuild rate to 10%, the system still needs a lot of time to move the files back – I didn’t bother to do the maths, because 3.5 hours or 10 hours it would all breach my maintenance window.

Because my advertised downtime window to clients was rapidly approaching, I had no choice but to continue to allow the copy to proceed, but redirect the main cluster to access the data store via NFS over the network and bring services back online. I will then be able to do a RSYNC later in less than 1 hour to bring the haphazardly copied set on the main cluster in line with the now used and modified data store on my hot standby server.

First I noticed that the usual night time “Vacuum’s” that are needed to keep the PostgreSQL planar at it’s most efficient and indexes clean, was running right into the day time! It usually needed less than an hour for the 20GB database we currently have.

So after many failed attempts to get an online vacuum done. I stayed up really late, took the cluster down and did FULL vacuum. Full vacuum’s are slow, and you can’t run anything while they happen because they do full table locks, where as online vacuum’s do quick tuple/row level locks.

Anyway, database seemed speedier, but system was still sluggish. I have all data separated. Database files are on large RAID10 arrays with U320 SCSI drives spinning at 15K – split over TWO SCSI buses! Yeah tis fast. Big disks are used because it means relative to the size of the disk, more data is on the outer edge of the platters, that spin faster than the centre of the platters. I also keep PostgreSQL’s transaction log on a separate RAID1 array with 73GB 15K U320 drives as well with a 256MB battery backed cache.

Now database based operations were zippy again….the system while set by me to not rate a disk search at too high a cost due to the super speed disk IO that I have, still should get SOME data out of the cache and not run to the comparatively sloooow disks straight away. After doing this all was cool again database wise.

However operations that needed the data store to be accessed were still piss slow. So email ingestion, file usage, etc crawled. The data store which compliments the database data weighs in at about 250GB now. And this is on a single RAID1 array with 300GB U320 10K drives. I also remembered that over a 1000BaseT network to backup to a robotic tape library, the data store array (RAID1) maxes out at about 750MB a min. While the RAID10 with the database does it at 2700+MB a min.

So I thought, “lets add some more disks to the data store array”. I get more space, so relatively speaking more of the data is on the outside edges of all the platters (another topic for what one can do with LVM), the array has double the heads and spindles too – and some RAID0 goodness inside that RAID10 nested set.

Now Dell’s storage white paper from 2005 does state that a RAID1 to a RAID10 migration/raid level reconstruction is supported. RAID10 being two RAID1 arrays striped together in RAID0 (with the mirror part straddling 2 SCSI buses for speed and channel redundancy). However when I went into Open Manage after putting the drives in the chassis, expecting to be able to “reconstruct/migrate levels” to a RAID10, then pop into LVM, create some new data devices, add them to my Volume group and then expand my partition and then finally my file system – all while still being online……..I was greeted with only the option to reconstruct/migrate to RAID5 (Get ^*^*&^) or RAID0 (Uh, yeah, OK..).

So feeling quite annoyed now. I then went and made the two new drives into a new RAID1 set, thinking that I would then be able to add this to a final “nested” array with the current RAID1 set and make a RAID10 out of them. Well I thought it was working, but it wasn’t. All I managed to do was end up with my existing array being made into a RAID0 (heart attack!!) and the new array sitting there untouched. Also the Open Manage array management setup gave me no choice of stripe size. It defaulted to 64K. I prefer 128K to ensure the best chance of the records/tuples fitting into one whole stripe to maximise concurrency of head operations on different data records – and also because the PERC4 family of RAID cards does not suffer any penalty if a stripe size is too big for the data used – so why not eh?

So what am I doing now at 5:32 AM?

Rsyncing all data again to the hot standby server, will then power off the cluster, log in with the DRAC card, go into the BIOS of the RAID card and redo my friggin array from scratch, as RAID10 and 128K stripe and then boot up and copy my damn data back.

Not happy pappy. Not happy. I think it is SAN time…unless SAN’s can be this anal as well?

So what does this have to do with PostgreSQL upgrade? Well since the cluster is down anyway, I might as well go from 7.4 to 8.1. Get some of that auto vacuum goodness and lap up some of the apparent massive speed boosts in the 5 years of development between the two versions.

This confused him. “If I have it, why do I have to pay for it?”. Well without getting into the nitty gritty of supply chain analysis, economies of scale and mass production relative to IP and licencing obligations. I tried to make him feel better by referring to some of the IBM pSeries servers that come with multiple CPU’s but only one can be used unless you buy an activation key from IBM to enable subsequent processors.

At the end of the day I can see his point and also Dell’s.