Anyway – this problem gets worse with the distributed IDC/Cloud.  So just add this little line to the advnced properties of any (read all) hosts.

keyboard.typematicMinDelay = 2000000

While one normally uses SSH or RDP to get to remote machines – when boot strapping, or post cloning and need to change a mac address binding or any other of small thing, this keyboard bug will drive you nuts.  Especially when a 30 second downtime blows out.  And all you want to do is edit 2 or 3 characters.

Breath…. please take this all and be happy as I am now. 

http://drupal.org/node/357715

- Grid Computing

- China wide redundant “Storage Fabric”

- China Optimised International and Local Routing

- Geographic IP and IDC Failover

- More thin client goodness

BTW – never use Windows Server 2008. It sucks. It has “VISTA” written all over it….literally in all the help files and what not. It is a true dog of an OS…but the Terminal Services support is WAAAAAY better. Not that W2K3 was bad – this is just even more fluid – especially under virtualisation.

Will update soon.

I was going about expanding virtual disks and reallocating resources for one client. Now, I have done this MANY times, so I thought that “the 2 day old backup is sufficient” and did not wait 3-4 hours for a new backup, right before what will be a 10 min task.

I went to expand the virtual disks from the COS and noticed that there were some “Virtual-Disk-000001-delta.vmdk” and “Virtual-Disk-000001.vmdk” files present.

“Oh, a snapshot is here for some reason..?”, I pondered. I then went into the VI3 management console, drilled down to said VPS and went to the snapshot manager, expecting to find a snapshot and then simply commit it to the main disk so I could get back to expanding.

What I found was “No Snapshots for this Virtual Server”.

Hmmm….. “maybe they are old snapshot files that should have been deleted, but weren’t”, I further mused. And =IF= it is a snapshot, surely vmkfstools will not let me run a dangerous or incompatible command. So off I went to expand this virtual disk by another 100GB.

- Expand disk:

“vmware-cmd –X 220GB Virtual-Disk.vmdk”

Expansion done. All looks good. Fire up VPS…..

“Sorry VPS can’t be started because one of the base files that a given snapshot is based on has been modified and thus can’t be mounted”.

” *^&#*^&@(*&!(@&(!*&(&! “

Ok, no harm no foul. The actual disk is not changed. Doing an expand with vmkfstools just adds a marker for more size… surely I can just remove the extra addition, ‘rollback the expansion’ so to speak and all will be spiffy?

Nup. Even though I knew in the back of my head that shrinking a VMDK was NOT POSSIBLE in ESX3 as it was in ESX2.5, I still went searching in the faint hope that I had overlooked some trick during past information gathering exercises when I was not under so much pressure and panic as I was this time.

No dice. What I knew was confirmed. I can’t shrink it. I can’t even load up ghost and mirror, because the main problem is that this Virtual-Disk-000001-delta.vmdk file should be appended to the end. And seeing as it was 25GB in size, for what is a 100GB virtual disk and the data stamp was some 3 months prior – there is A LOT of data and changes that are at risk now.

” *^&#*^&@(*&!(@&(!*&(&! “

OK, on to google again. After some searching and a lot of effort in trying to refine my query, which was needed, because as opposed to what I actually found out (this being the #4 and #5 global support issues for VMWARE), information was scant. I did manage to find a couple of blogs that had some very brief and lacking in all technical detail, reviews of the recent VMWorld summit.

So with that hook, I then started to search on detailed info from that summit and managed to get a PPT file from one of the developers. And inside were all the details that I needed. Or thought that I needed. Because with any system as complicated as VMWARE, definitions of words and correct semantics can make if very difficult to get a clear grasp of one problem, versus a slight variation of it. And even a slight change can come with very different procedures to use and using the wrong ones could make a problem worse. First rule – do no more harm.

I then went to the page that was titled “Expanding the size of a VMDK with an existing Snapshot”. I did not know if this meant, “how to expand a VMDK with an existing snapshot and keep it intact”, or “How to recover from a monumental screw up that only an idiot would do, when expecting vmkfstools to do all due diligence for him and has fucked up the VMDK that happened to have a current and active snapshot that wasn’t committed to the main VMDK file first”

I assumed it meant the latter, being “tech support” and “high rating”… if it was documentary of a feature or process it would have been, well, better documented.

The procedure is this:

- After I was an idiot and issued this command to cause all the problems:

“vmkfstools –X 220G Virtual-Disk.vmdk”

- Check the “Virtual-Disk.vmdk” file with vi and look for the following lines:

RW 482344960 VMFS “Virtual-Disk-000001-delta.vmdk”

- Now check the “Virtual-Disk-000001.vmdk” file and look for the following lines:

RW 209715200 VMFS “Virtual-Disk-000001-delta.vmdk”

What we now know is the current RW value on the newly expanded “Virtual-Disk.vmdk” and it is 482344960. We want to ‘trick’ the system into thinking that the expand never happened. So we then go and replace that value with the one we got from the delta vmdk. So we replace 492344960 with 209715200.

- Now we need to commit all snap shots:

“vmware-cmd /vmfs/volumes/VMFSVOLUME/VPS/VPS.vmx removesnapshots”

Unfortunately I was not done yet. The system reported back that the virtual machine “VPS.vmx” did not have any snapshots present! “Ah ha” I thought. While this is not good, it is also the reason why vmkfstools went on and screwed everything in the start. There is a snapshot there – that is a fact – but the system does not believe so.

This is where global common VMWARE problem #5 comes in, “Corrupted .VMSD file”. In a nutshell this means that the file that tracks all this snapshot info (amongst other tid bits) is somehow compromised. So a new one is needed. This is also fairly simple once you know how:

- First rename the current VMSD file:

mv VPS.vmsd VPS.vmsd.old

- Now create a new snapshot to force the system to generate a new all emcompasing VMSD file:

“vmware-cmd VPS.vmx createsnapshot addedforrecovey “You are an IDIOT”

- Now commit all snapshots like we wanted to do before anyway. You have to commit them all:

“vmware-cmd VPS.vmx removesnapshots”

Now that all the snapshots are committed (the original one and the temp one we made to help recreate the VMSD file) we can continue the process of fixing up our expanding a disk issue. And this is as simple as running the initial vmkfstools expand command that we ran before, that caused all the problems. This is needed so that the correct RW values are set in Virtual-Disk.vmdk” because in the end, the virtual disk IS expanded already.

- So issue the command:

“vmware-cmd –X 220GB Virtual-Disk.vmdk”

In the end, I am NOT STUPID enough to try and expand a virtual disk with a snapshot. However if you DO SEE delta files in your file system, do not trust the VI3 clients snapshot manager if it says “No Snapshots present”. As a matter of caution, I would follow the process above to recreate a new VMSD file to be sure and commit the temporary and any other snapshots that may exist. Then you can go on and expand your disks.

Also, make sure that you have backups. While I did and they weren’t totally fresh and the client was not too upset when briefed of the situation, it could have been much worse.

ALWAYS BACKUP!

DON’T LET A JUNIOR TECH TOUCH THINGS!

TAKE THE TIME TO RELAX AND ASSES THE SITUATION BEFORE YOU POSSIBLY MAKE IT WORSE!

What is a mixed bag in my mind is the new ESX3i. It will be first released in the new Dell VORSO servers, originally due to ship this month and as confirmed by my DELL Sales manager, is slipping into the new year. These servers will not need hard disks (So what about core dumps, logs and swap?) but will have embedded flash with the hypervisor installed.

On the surface it looks like the Hypervisor is now part of the system board. It ain’t. So I don’t expect ESX3i to give any speed increase. What I do expect is that it will give a massive boost to deployment times. If you are deploying new servers at the rate of one or more a week, this is cool. Right now at CANDIS with blades, we have all the wiring already in place due to the chassis. But we still need to go in through the management card to setup the OS. Yes we can open the box and slide in the blade and have a new server fully setup physically. But not the software. From the looks of ESX3i, it goes a step further. Power on the system, it finds a VC server, configures itself, boots up and presents it’s resources to the farm, ready to be dynamically allocated by the VC server.

Oh, they also have smart power management that can power down systems when their computing resources are not needed. This is cool. Since we pay about .2 RMB per kilowatt of power, it does add up. The new low voltage XEON’s are great power savers, as are the new high efficiency power supplies. However add in those dynamic server on/off controls and you will be able to make a serious dent in your power bill. Oh, and keep the “Eco-Terrorists” happy too and thus maybe avoid a politically correct jihad aimed at you, unlike poor Apple Inc.

VMWARE

It is golden week here and all are away on break. So instead of forcing a staff member to come back, I thought I would take care of some stuff myself.

My problems started when a client who has a large advertising cluster, was running their main statistics database (for click fraud detection) on a Dell 1950 with only 1 SAS 15K drive.

I had suggested that this node, not being redundant like the tomcat servers be individually redundant, so DRAC card, redundant power and RAID.

Anyway, some new blades, Dell 1955′s arrived for the cluster and I thought, well, lets save the client some money, image the old 1950 DB server and load it onto a new 1955 server?

I thought this would be simple with Acronis.

No it wasn’t.

It turns out that a 3.5 Inch 73GB SAS drive is not the same size as a 2.5 Inch 73GB SAS drive. So I could not write my system image to the blades raid 1 array of 2 x 15K 73GB SAS drives.

Shite! If I reinstall the DB server it is not worth my time. Cheaper to buy the upgrade parts for the 1950.

Then I thought, well, I have LVM, I should be able to do this, after all I have used LVM before many times on large storage arrays.

So my goal was this, I needed a system image that was a couple hundred megs smaller than it was now, so it will go into the 1955′s ok.

This is where I added in VMWARE to the mix and made this an easy task. The steps are below:

1. Image the 1950 server to some ACRONIS TIB files somewhere. I used FTP
2. Image the TIB file to a new VPS made with a 73GB virtual disk
3. Create and attach a new 65GB virtual disk to the virtual machine
4. Image the MBR and /boot partitions using acronis onto the new 65GB virtual disk
5. Boot virtual machine with a rescue/live CD
6. Load FDISK for /dev/sdb and create a new LVM (Type 8e) partition in the remaining space on the 65GB virtual drive
7. Enter LVM with the “lvm” command
8. Activate all Volume Groups with the command “vgchange -a y”
9. EXIT out of LVM and then run this command to resize the EXT3 file system, “resize2fs /dev/VolGroup/LogVol00 40G”, you may have to run “e2fschk -f /dev/VolGroup00/LogVol00″ first too
10. Enter LVM again with the “lvm” command
11. Now we can reduce the Logical Volume that had the recently shrunk EXT3 file system on it with this command, “lvreduce LogVol00 -L 45G”
12. Because we are making a new custom boot image and we have already imaged over the /boot partition and MBR, we now want our old 73GB virtual drive to not have any of the same markings as our embryonic new 65GB virtual drive. To do this we need to change the Volume Group and Logical Volume names to something new:
13. lvrename VolGroup00 LogVol00 LogVol10
14. lvrename VolGroup00 LogVol01 LogVol11
15. lvchange LogVol10 -a n
16. lvchange LogVol11 -a n
17. vgchange VolGroup00 VolGroup10
18. Now we can create the new Logical Volumes and Volume Groups on the 65GB virtual disk in preparation for cloning our now 40GB EXT3 file system
19. Make a new Physical Volume first wit, “pvcreate /dev/sdb2″
20. pvscan
21. vgcreate VolGroup00 /dev/sdb2
22. vgscan
23. lvcreate VolGroup00 /dev/sdb2 -n LogVol01 -L 2G
24. lvcreate VolGroup00 /dev/sdb2 -n LogVol00 -L 50G
25. lvscan
26. Now we need to make our new LVM’s online and visible to the system so we run the command vgchange again, “vgchange -a y”
27. EXIT
28. Back at the command prompt we need to now setup our new SWAP partition, so we issue the command, “mkswap /dev/VolGroup00/LogVol01″
29. Now we can clone our old 40GB EXT3 partion that we shrunk to our new LVM which is larger than 40GB, but smaller than 65GB, so it will image onto the PE1955 2.5 SAS drive array
30. We use an old favourite for this, “dd if=/dev/VolGroup10/LogVol10 of=/dev/VolGroup00/LogVol00″
31. Once done, shutdown the VPS, boot up in Acronis and image the new 65GB virtual drive and then load it onto the PE1955 server. DONE!

I did all of this in single user mode so as to minimise the need for KUDZU to rescan and change all hardware. I can do this because of the hardware commonality in the 9th generation Dell servers.

Also once done, the new LVM will be smaller than the full capacity of the 73GB SAS 2.5 SAS drive. This is easily fixed while online. by making a new partition /dev/sda3 of LVM type (8e), making it into a Physical Volume, adding it to VolGroup00 and then extending the LogVol00 logical volume with the newly added extents. Once that is done, go back to the command prompt and use the ext2online command to finally expand your EXT3 partition to use all the space on the LVM.

Difference is, they are still paying too much $$$$ to over paid western IT staff that really don’t do any serious Computer Science or R+D. That’s where China comes in! Global economy people, get with it, or get left behind. Times they are a changing, as is technology. Put these two together and one had better not get too comfortable in their own job in this industry and this day and age.

I can’t believe that in Australia some IT workers still claim 100+ K salaries. While that is fine for a capable and experienced business analyst, but a web developer or any kind of programmer in general?? Desktop support staff? DBA’s? That is the real danger to this industry, over paid, self important IT geeks who believe that IT can only be done well in the west and that everything that comes out of India, Korea, Brazil or China must be dodgy or below quality.

FUD.

Some people are in for a rude shock in the coming years. Especially those without a diversified skill set that reaches beyond tech or who can’t fluently speak at least one foreign language (Chinese or Spanish being ideal candidates).