I like their ratio of engineers to users… in China it is about the opposite.  We had 6 people come to change a light bulb last week.  Labour is a cheap commodity – but smart use of technology will be even cheaper!

FaceBook Infrastructure

Well my recipe was good. I just have to find it again. While no one forgets to ride a bike – my brain can’t keep all the tech stuff that is life as an IT pro current and in the level one cache of my cerebral cortex always. I need to push stuff out to swap now and then.

But to close up this lame virtual memory metaphor for my brain….. I have a serious page fault right now.

http://drupal.org/node/357715

Go read www.beijinglug.org for more.

Now if only IDG could put on a half decent Linux World instead of the token crap they excrete out like a constipated hippo.

Sorry if I talked too much.  Sorry if the other Aussies did like wise.  It is very Australian to just tell it how it is and not filter for PC or sensitivities.

Anyway, here is an excellent read from Cory Doctorow that I recently read.  No wonder none of his books are on audible!

Corey Doctorow’s speach at Microsoft

Select from database ID where name = RICHARD;

Returns and ID of 55 for example.

Then go and do a query like this:

Select * from some_other_table where ID = 55;

Returns, “Sorry does not exist, time to die…..”

Well apparently indexes when corrupt – which is NOT SUPPOSED TO HAPPEN – can cause PostgreSQL to go all stupid and not do a table lookup for real. This happened to me. So I found this:

PlanetPostgresql

Turns out that a reindex and a full vacuum can do wonders – even though a full vacuum is not needed with autovacuum and indexes can’t get corrupted…..or so they say.

I have now added a system wide reindex maintenance plan for PostgreSQL every night. I know that MS-SQL server has an option for this with their maintenance jobs inside enterprise manager. Maybe someone should make an enterprise manager for PostgreSQL too?

The “gam_server” process. Set to identify when any file in the system is changed. A useful action that has benefits. But not when it does it 3-5 times per second and the sever is serving NFS and PostgreSQL!

To fix it, just ensure that somewhere in /etc/ (RedHat Base) or /etc/gamin/ (Debian Base) has a file called:

“gaminrc”

Add in one line for each file system type that you have. For example mine is now:

#

fsset ext3 poll 10

fsset nfs poll 10

fsset xfs poll 10

#

The daemon will now scan every 10 seconds. Of course for it to do that, make sure to invoke a “kill -s 9″ first!

It is golden week here and all are away on break. So instead of forcing a staff member to come back, I thought I would take care of some stuff myself.

My problems started when a client who has a large advertising cluster, was running their main statistics database (for click fraud detection) on a Dell 1950 with only 1 SAS 15K drive.

I had suggested that this node, not being redundant like the tomcat servers be individually redundant, so DRAC card, redundant power and RAID.

Anyway, some new blades, Dell 1955′s arrived for the cluster and I thought, well, lets save the client some money, image the old 1950 DB server and load it onto a new 1955 server?

I thought this would be simple with Acronis.

No it wasn’t.

It turns out that a 3.5 Inch 73GB SAS drive is not the same size as a 2.5 Inch 73GB SAS drive. So I could not write my system image to the blades raid 1 array of 2 x 15K 73GB SAS drives.

Shite! If I reinstall the DB server it is not worth my time. Cheaper to buy the upgrade parts for the 1950.

Then I thought, well, I have LVM, I should be able to do this, after all I have used LVM before many times on large storage arrays.

So my goal was this, I needed a system image that was a couple hundred megs smaller than it was now, so it will go into the 1955′s ok.

This is where I added in VMWARE to the mix and made this an easy task. The steps are below:

1. Image the 1950 server to some ACRONIS TIB files somewhere. I used FTP
2. Image the TIB file to a new VPS made with a 73GB virtual disk
3. Create and attach a new 65GB virtual disk to the virtual machine
4. Image the MBR and /boot partitions using acronis onto the new 65GB virtual disk
5. Boot virtual machine with a rescue/live CD
6. Load FDISK for /dev/sdb and create a new LVM (Type 8e) partition in the remaining space on the 65GB virtual drive
7. Enter LVM with the “lvm” command
8. Activate all Volume Groups with the command “vgchange -a y”
9. EXIT out of LVM and then run this command to resize the EXT3 file system, “resize2fs /dev/VolGroup/LogVol00 40G”, you may have to run “e2fschk -f /dev/VolGroup00/LogVol00″ first too
10. Enter LVM again with the “lvm” command
11. Now we can reduce the Logical Volume that had the recently shrunk EXT3 file system on it with this command, “lvreduce LogVol00 -L 45G”
12. Because we are making a new custom boot image and we have already imaged over the /boot partition and MBR, we now want our old 73GB virtual drive to not have any of the same markings as our embryonic new 65GB virtual drive. To do this we need to change the Volume Group and Logical Volume names to something new:
13. lvrename VolGroup00 LogVol00 LogVol10
14. lvrename VolGroup00 LogVol01 LogVol11
15. lvchange LogVol10 -a n
16. lvchange LogVol11 -a n
17. vgchange VolGroup00 VolGroup10
18. Now we can create the new Logical Volumes and Volume Groups on the 65GB virtual disk in preparation for cloning our now 40GB EXT3 file system
19. Make a new Physical Volume first wit, “pvcreate /dev/sdb2″
20. pvscan
21. vgcreate VolGroup00 /dev/sdb2
22. vgscan
23. lvcreate VolGroup00 /dev/sdb2 -n LogVol01 -L 2G
24. lvcreate VolGroup00 /dev/sdb2 -n LogVol00 -L 50G
25. lvscan
26. Now we need to make our new LVM’s online and visible to the system so we run the command vgchange again, “vgchange -a y”
27. EXIT
28. Back at the command prompt we need to now setup our new SWAP partition, so we issue the command, “mkswap /dev/VolGroup00/LogVol01″
29. Now we can clone our old 40GB EXT3 partion that we shrunk to our new LVM which is larger than 40GB, but smaller than 65GB, so it will image onto the PE1955 2.5 SAS drive array
30. We use an old favourite for this, “dd if=/dev/VolGroup10/LogVol10 of=/dev/VolGroup00/LogVol00″
31. Once done, shutdown the VPS, boot up in Acronis and image the new 65GB virtual drive and then load it onto the PE1955 server. DONE!

I did all of this in single user mode so as to minimise the need for KUDZU to rescan and change all hardware. I can do this because of the hardware commonality in the 9th generation Dell servers.

Also once done, the new LVM will be smaller than the full capacity of the 73GB SAS 2.5 SAS drive. This is easily fixed while online. by making a new partition /dev/sda3 of LVM type (8e), making it into a Physical Volume, adding it to VolGroup00 and then extending the LogVol00 logical volume with the newly added extents. Once that is done, go back to the command prompt and use the ext2online command to finally expand your EXT3 partition to use all the space on the LVM.