In any case, the short of it is, that we were juggling PERC4 cards around servers (PCI-X here, PCIe there..) and also complete raid 1 and raid 10 arrays too. The cards are supposed to “detect” the correct array type from the drives if the firmware was missing. Anyway, through a comedy of errors, it worked exactly 1/3 times. The other times we had to remember the exact settings of our arrays (stripe, etc) and how it was structured. So we could clear PERC cards and then recreate the arrays – taking special care to not initalise the new arrays.

So in the end, you can move arrays and channels about. And with LVM, even designations like /sda /sdb reording is also not an issue. However you should rely on good old fashioned hand held way of doing things. Before you start write down all the salient details of your arrays first.

It is golden week here and all are away on break. So instead of forcing a staff member to come back, I thought I would take care of some stuff myself.

My problems started when a client who has a large advertising cluster, was running their main statistics database (for click fraud detection) on a Dell 1950 with only 1 SAS 15K drive.

I had suggested that this node, not being redundant like the tomcat servers be individually redundant, so DRAC card, redundant power and RAID.

Anyway, some new blades, Dell 1955′s arrived for the cluster and I thought, well, lets save the client some money, image the old 1950 DB server and load it onto a new 1955 server?

I thought this would be simple with Acronis.

No it wasn’t.

It turns out that a 3.5 Inch 73GB SAS drive is not the same size as a 2.5 Inch 73GB SAS drive. So I could not write my system image to the blades raid 1 array of 2 x 15K 73GB SAS drives.

Shite! If I reinstall the DB server it is not worth my time. Cheaper to buy the upgrade parts for the 1950.

Then I thought, well, I have LVM, I should be able to do this, after all I have used LVM before many times on large storage arrays.

So my goal was this, I needed a system image that was a couple hundred megs smaller than it was now, so it will go into the 1955′s ok.

This is where I added in VMWARE to the mix and made this an easy task. The steps are below:

1. Image the 1950 server to some ACRONIS TIB files somewhere. I used FTP
2. Image the TIB file to a new VPS made with a 73GB virtual disk
3. Create and attach a new 65GB virtual disk to the virtual machine
4. Image the MBR and /boot partitions using acronis onto the new 65GB virtual disk
5. Boot virtual machine with a rescue/live CD
6. Load FDISK for /dev/sdb and create a new LVM (Type 8e) partition in the remaining space on the 65GB virtual drive
7. Enter LVM with the “lvm” command
8. Activate all Volume Groups with the command “vgchange -a y”
9. EXIT out of LVM and then run this command to resize the EXT3 file system, “resize2fs /dev/VolGroup/LogVol00 40G”, you may have to run “e2fschk -f /dev/VolGroup00/LogVol00″ first too
10. Enter LVM again with the “lvm” command
11. Now we can reduce the Logical Volume that had the recently shrunk EXT3 file system on it with this command, “lvreduce LogVol00 -L 45G”
12. Because we are making a new custom boot image and we have already imaged over the /boot partition and MBR, we now want our old 73GB virtual drive to not have any of the same markings as our embryonic new 65GB virtual drive. To do this we need to change the Volume Group and Logical Volume names to something new:
13. lvrename VolGroup00 LogVol00 LogVol10
14. lvrename VolGroup00 LogVol01 LogVol11
15. lvchange LogVol10 -a n
16. lvchange LogVol11 -a n
17. vgchange VolGroup00 VolGroup10
18. Now we can create the new Logical Volumes and Volume Groups on the 65GB virtual disk in preparation for cloning our now 40GB EXT3 file system
19. Make a new Physical Volume first wit, “pvcreate /dev/sdb2″
20. pvscan
21. vgcreate VolGroup00 /dev/sdb2
22. vgscan
23. lvcreate VolGroup00 /dev/sdb2 -n LogVol01 -L 2G
24. lvcreate VolGroup00 /dev/sdb2 -n LogVol00 -L 50G
25. lvscan
26. Now we need to make our new LVM’s online and visible to the system so we run the command vgchange again, “vgchange -a y”
27. EXIT
28. Back at the command prompt we need to now setup our new SWAP partition, so we issue the command, “mkswap /dev/VolGroup00/LogVol01″
29. Now we can clone our old 40GB EXT3 partion that we shrunk to our new LVM which is larger than 40GB, but smaller than 65GB, so it will image onto the PE1955 2.5 SAS drive array
30. We use an old favourite for this, “dd if=/dev/VolGroup10/LogVol10 of=/dev/VolGroup00/LogVol00″
31. Once done, shutdown the VPS, boot up in Acronis and image the new 65GB virtual drive and then load it onto the PE1955 server. DONE!

I did all of this in single user mode so as to minimise the need for KUDZU to rescan and change all hardware. I can do this because of the hardware commonality in the 9th generation Dell servers.

Also once done, the new LVM will be smaller than the full capacity of the 73GB SAS 2.5 SAS drive. This is easily fixed while online. by making a new partition /dev/sda3 of LVM type (8e), making it into a Physical Volume, adding it to VolGroup00 and then extending the LogVol00 logical volume with the newly added extents. Once that is done, go back to the command prompt and use the ext2online command to finally expand your EXT3 partition to use all the space on the LVM.

I used this excellent resource before.

Even the very rudimentary tweaks I did, and with a RAID array that is in a 90% rebuild rate, background initialisation, this 8.1 version is FAST! I have no idea why people use MySQL, it really is such a piece of crud.

Let me expand on this;

This type of backup file is advertised as “more convenient” and offers more options for restore time selective data restores, data re-ording, index tricks and the like. However no matter WHAT I did, it reported and sent a copy of the current pg_restore process and all the data being restored to standard output too!! This means that basically, I was going to have the same full text of 20GB worth of database data shoved down my SSH session!

Yes – this makes the whole affair much slower!

Luckily, being the lateral thinking kind of dude that I am, I never put all my eggs in one basket. That is just a recipe for data omelette.

I also had some plain text file dumps made from pg_dump. So I went into the the PostgreSQL template1 sessions and then used the “\i” command to import my big .SQL file. Perfect! Only important updates sent to std output and not the whole damn enchilada.

Disk restore is still going on. At a max through put of 20MByte a second with a 1000BaseT network, that is at best 1.2G a minute, 72GB an hour, so for 250GB approximately 3.5 hours. However I was still doing a raid array background initialisation, so even after setting the rebuild rate to 10%, the system still needs a lot of time to move the files back – I didn’t bother to do the maths, because 3.5 hours or 10 hours it would all breach my maintenance window.

Because my advertised downtime window to clients was rapidly approaching, I had no choice but to continue to allow the copy to proceed, but redirect the main cluster to access the data store via NFS over the network and bring services back online. I will then be able to do a RSYNC later in less than 1 hour to bring the haphazardly copied set on the main cluster in line with the now used and modified data store on my hot standby server.

First I noticed that the usual night time “Vacuum’s” that are needed to keep the PostgreSQL planar at it’s most efficient and indexes clean, was running right into the day time! It usually needed less than an hour for the 20GB database we currently have.

So after many failed attempts to get an online vacuum done. I stayed up really late, took the cluster down and did FULL vacuum. Full vacuum’s are slow, and you can’t run anything while they happen because they do full table locks, where as online vacuum’s do quick tuple/row level locks.

Anyway, database seemed speedier, but system was still sluggish. I have all data separated. Database files are on large RAID10 arrays with U320 SCSI drives spinning at 15K – split over TWO SCSI buses! Yeah tis fast. Big disks are used because it means relative to the size of the disk, more data is on the outer edge of the platters, that spin faster than the centre of the platters. I also keep PostgreSQL’s transaction log on a separate RAID1 array with 73GB 15K U320 drives as well with a 256MB battery backed cache.

Now database based operations were zippy again….the system while set by me to not rate a disk search at too high a cost due to the super speed disk IO that I have, still should get SOME data out of the cache and not run to the comparatively sloooow disks straight away. After doing this all was cool again database wise.

However operations that needed the data store to be accessed were still piss slow. So email ingestion, file usage, etc crawled. The data store which compliments the database data weighs in at about 250GB now. And this is on a single RAID1 array with 300GB U320 10K drives. I also remembered that over a 1000BaseT network to backup to a robotic tape library, the data store array (RAID1) maxes out at about 750MB a min. While the RAID10 with the database does it at 2700+MB a min.

So I thought, “lets add some more disks to the data store array”. I get more space, so relatively speaking more of the data is on the outside edges of all the platters (another topic for what one can do with LVM), the array has double the heads and spindles too – and some RAID0 goodness inside that RAID10 nested set.

Now Dell’s storage white paper from 2005 does state that a RAID1 to a RAID10 migration/raid level reconstruction is supported. RAID10 being two RAID1 arrays striped together in RAID0 (with the mirror part straddling 2 SCSI buses for speed and channel redundancy). However when I went into Open Manage after putting the drives in the chassis, expecting to be able to “reconstruct/migrate levels” to a RAID10, then pop into LVM, create some new data devices, add them to my Volume group and then expand my partition and then finally my file system – all while still being online……..I was greeted with only the option to reconstruct/migrate to RAID5 (Get ^*^*&^) or RAID0 (Uh, yeah, OK..).

So feeling quite annoyed now. I then went and made the two new drives into a new RAID1 set, thinking that I would then be able to add this to a final “nested” array with the current RAID1 set and make a RAID10 out of them. Well I thought it was working, but it wasn’t. All I managed to do was end up with my existing array being made into a RAID0 (heart attack!!) and the new array sitting there untouched. Also the Open Manage array management setup gave me no choice of stripe size. It defaulted to 64K. I prefer 128K to ensure the best chance of the records/tuples fitting into one whole stripe to maximise concurrency of head operations on different data records – and also because the PERC4 family of RAID cards does not suffer any penalty if a stripe size is too big for the data used – so why not eh?

So what am I doing now at 5:32 AM?

Rsyncing all data again to the hot standby server, will then power off the cluster, log in with the DRAC card, go into the BIOS of the RAID card and redo my friggin array from scratch, as RAID10 and 128K stripe and then boot up and copy my damn data back.

Not happy pappy. Not happy. I think it is SAN time…unless SAN’s can be this anal as well?

So what does this have to do with PostgreSQL upgrade? Well since the cluster is down anyway, I might as well go from 7.4 to 8.1. Get some of that auto vacuum goodness and lap up some of the apparent massive speed boosts in the 5 years of development between the two versions.