Most “IT Pro’s” in Australia actually have little to none creative or developmental capacity.

Looking over assets listings for a client I see countless line items for software, where tens of thousands of dollars have been spent on software that have better Open Source or SaaS alternatives available.

Discussions on LinkedIn boards talk about “What is best software for this?  Which vendor?”  Hardly a conversation about how “This is great” or “Or I used this library with this project and solved this problem while not spending $$ and adding to our own IP collection”.

It seems IT workers in this country are glorified order (procurement) takers and givers?  And those that actually can to the work and be creative are stuck at the bottom of the food chain and never listened too.

Is this how Silicon Alley or Valley works?  Or how Eastern Europe has surged forward? Or China, India and the Philippines?

Australia – you need to embrace INNOVATION and self driven productivity and not being glorified LEGO builders (NBN!!).  Otherwise I can’t see how Australia will have any IT industry left over?

And the next person that states that Australia or the way we do it has something to do with better “Quality” I am going to brain them for being so naive and self delusional.

And where is all the Open Source fun?  The JAVA and Python and Perl and what not?  Everything seems to be .Net and Microsoft.   I’m not silly enough to enquire about Ruby or Rails though.  :-s

Well apart from some systems using GiB and others GB and then some manufacturers using base10 and not base3 and saying that 1GB is 1000MB and not 1024 – I have found another one.

When making a file system that is not your root file system (/) or var (/var) or root (/root), etc – say a second drive or partition used for something else, try to format with the “-m 0″ command so that 5% is not reserved for the root user.

On a 1024GB array – say 1TB – that is a whopping ~52GB!!

Also if you will have many files in said system, also throw in the “-O dir_index” flag too, to enable hash tree’s for directory look ups – =big= speed increase.

Go read www.beijinglug.org for more.

Now if only IDG could put on a half decent Linux World instead of the token crap they excrete out like a constipated hippo.

Silicon Hutong

And the topic did strike a “Hey this is real man!!!” kind of chord with me. A cathartic resonance that shall never come from me with respect to the greater greenhouse effect crap while the infallible science is still not in. Nor came to me for any other “agenda” or “crusade of the world” that seemed to so very trouble all those students over in the Arts and Law faculties. That seemed to have so much time on their hands, relative to us poor saps in Engineering and Business school.

Politics aside, wastage for the sake of wastage though is not very good. And I have a rather poignant insider’s look at this issue that also manages (in my mind at least) to cross paths with this issue about China’s talent gap, or indeed this also about the talent gap.

What does resource wastage and power requirements have to do with a talent gap, or as they lament now in overly taxed and anti-entrepreneurial Australia – the “Brain Drain”?

Allow me to explain.

Servers cost a lot of money! Not only do you spend the 5-10K USD for the server and warranty, you then have colocation fees, cooling costs and power costs – this all adds up. And if you skimp on something like cooling, your equipment can age prematurely – especially bad in China given the tax depts allowed deprecation rates!

A good contact of mine who runs an IDC in New York city, mentioned how it is all about the power. Real estate? No worries, how much power will we need. So it seems pretty clear where things are going. Moore’s law, computing density increasing and an ever growing appetite for all things information.

With respect to the issues that David Wolf brought up, a major new trend that has been underway for the past 6 years or so and is trying to put a clamp on rising costs of infrastructure is Virtualisation. This technology allows for less resource wastage and consumables wastage by consolidating many lower end servers into fewer bigger more reliable and cost effective servers. Picking the low hanging fruit so to speak.

A further evolution of this that I mentioned about here is where the actual servers in a farm can be powered down too when not needed.

Now when it comes to electricity there are always two rates, the so called “Consumer Rate” and the “Industrial Rate”. And yes, data centre’s use the industrial rates, which while are relatively cheaper than consumer rates – are still not cheap in absolute terms. And if you go and use a nifty tool like this Dell capacity planner and then do some arithmatic, you can see what your costs will be. It is through this tool, that CANDIS deduced that paying the premium for higher efficiency power supplies and low voltage processors, actually did pay for themselves in outright power savings over a 2 year period. Even more if you factor in that less heat is better for stability and system longevity.

So where does the talent gap come into play? Well, in how many servers you need to do a job and knowing about virtualisation.

We had a client that was struggling to keep up and online with their 8 server cluster. They kept just purchasing new servers when things went bad and slowed down. Which seemed to happen all the time. Now I know this and they didn’t, is that an out of the box Linux system is not too tuned and the concurrent IP connection tracking vlaue in the kernel is a little conservative. So instead of them having 8 servers running at 20% capacity and still not managing the load on them as a cluster. Yours truly made a few changes and now 3 servers are doing the job, with better outward performance to the clusters users.

What made this situation all the more funny, is that in China, capital asset valuations are fair game for being compared to the salary of a team of workers to perform the same job (quality and reaction times not withstanding).

So, in summary, I will go out on a limb here and make a bold assertion.

“There are probably more servers than needed running in IDC’s across China, due to engineers not knowing how to admin them properly, not know how to architect systems properly and not knowing how to leverage new technology better.”

Anyway, I could go on. I just wanted to illustrate the nexus of those points and how they relate and intersect. Poorly deployed tech, poorly planned and architected tech and balooning operational costs and resource demands.

Seems that in the past you could be a lazy programmer and no one would notice due to the liberation of faster processors and free cycles. Or in China, buy a server is better than paying the for the expensive vendor support and optimisation.

However as one famous economist once mused…. “there is no such thing as a free lunch”

In any case, the short of it is, that we were juggling PERC4 cards around servers (PCI-X here, PCIe there..) and also complete raid 1 and raid 10 arrays too. The cards are supposed to “detect” the correct array type from the drives if the firmware was missing. Anyway, through a comedy of errors, it worked exactly 1/3 times. The other times we had to remember the exact settings of our arrays (stripe, etc) and how it was structured. So we could clear PERC cards and then recreate the arrays – taking special care to not initalise the new arrays.

So in the end, you can move arrays and channels about. And with LVM, even designations like /sda /sdb reording is also not an issue. However you should rely on good old fashioned hand held way of doing things. Before you start write down all the salient details of your arrays first.

The “gam_server” process. Set to identify when any file in the system is changed. A useful action that has benefits. But not when it does it 3-5 times per second and the sever is serving NFS and PostgreSQL!

To fix it, just ensure that somewhere in /etc/ (RedHat Base) or /etc/gamin/ (Debian Base) has a file called:

“gaminrc”

Add in one line for each file system type that you have. For example mine is now:

#

fsset ext3 poll 10

fsset nfs poll 10

fsset xfs poll 10

#

The daemon will now scan every 10 seconds. Of course for it to do that, make sure to invoke a “kill -s 9″ first!

This week, via the trickle down effect in the echelons of the China network, we got informed that a couple of our clients are not compliant and they need to become so, or remove their website.

We have had to remove some clients sites before because they did not have the correct ICP. They could not get one because they did not have the correct business licences for the type of work they did.

Anyway, while having an ICP is a delicate balance between business licences, residential permits, correct content, correctly licenced IDC and what not, it is a good thing to get done and host in China. To be official, legit and free to get on with doing business.

For the most part getting an ICP, or the existence of an ICP is not a barrier to hosting or a problem to comply with. It sounds a whole lot more draconian than it really is. However it is a hassle. For you China hands, “mafan si le”!

It is golden week here and all are away on break. So instead of forcing a staff member to come back, I thought I would take care of some stuff myself.

My problems started when a client who has a large advertising cluster, was running their main statistics database (for click fraud detection) on a Dell 1950 with only 1 SAS 15K drive.

I had suggested that this node, not being redundant like the tomcat servers be individually redundant, so DRAC card, redundant power and RAID.

Anyway, some new blades, Dell 1955′s arrived for the cluster and I thought, well, lets save the client some money, image the old 1950 DB server and load it onto a new 1955 server?

I thought this would be simple with Acronis.

No it wasn’t.

It turns out that a 3.5 Inch 73GB SAS drive is not the same size as a 2.5 Inch 73GB SAS drive. So I could not write my system image to the blades raid 1 array of 2 x 15K 73GB SAS drives.

Shite! If I reinstall the DB server it is not worth my time. Cheaper to buy the upgrade parts for the 1950.

Then I thought, well, I have LVM, I should be able to do this, after all I have used LVM before many times on large storage arrays.

So my goal was this, I needed a system image that was a couple hundred megs smaller than it was now, so it will go into the 1955′s ok.

This is where I added in VMWARE to the mix and made this an easy task. The steps are below:

1. Image the 1950 server to some ACRONIS TIB files somewhere. I used FTP
2. Image the TIB file to a new VPS made with a 73GB virtual disk
3. Create and attach a new 65GB virtual disk to the virtual machine
4. Image the MBR and /boot partitions using acronis onto the new 65GB virtual disk
5. Boot virtual machine with a rescue/live CD
6. Load FDISK for /dev/sdb and create a new LVM (Type 8e) partition in the remaining space on the 65GB virtual drive
7. Enter LVM with the “lvm” command
8. Activate all Volume Groups with the command “vgchange -a y”
9. EXIT out of LVM and then run this command to resize the EXT3 file system, “resize2fs /dev/VolGroup/LogVol00 40G”, you may have to run “e2fschk -f /dev/VolGroup00/LogVol00″ first too
10. Enter LVM again with the “lvm” command
11. Now we can reduce the Logical Volume that had the recently shrunk EXT3 file system on it with this command, “lvreduce LogVol00 -L 45G”
12. Because we are making a new custom boot image and we have already imaged over the /boot partition and MBR, we now want our old 73GB virtual drive to not have any of the same markings as our embryonic new 65GB virtual drive. To do this we need to change the Volume Group and Logical Volume names to something new:
13. lvrename VolGroup00 LogVol00 LogVol10
14. lvrename VolGroup00 LogVol01 LogVol11
15. lvchange LogVol10 -a n
16. lvchange LogVol11 -a n
17. vgchange VolGroup00 VolGroup10
18. Now we can create the new Logical Volumes and Volume Groups on the 65GB virtual disk in preparation for cloning our now 40GB EXT3 file system
19. Make a new Physical Volume first wit, “pvcreate /dev/sdb2″
20. pvscan
21. vgcreate VolGroup00 /dev/sdb2
22. vgscan
23. lvcreate VolGroup00 /dev/sdb2 -n LogVol01 -L 2G
24. lvcreate VolGroup00 /dev/sdb2 -n LogVol00 -L 50G
25. lvscan
26. Now we need to make our new LVM’s online and visible to the system so we run the command vgchange again, “vgchange -a y”
27. EXIT
28. Back at the command prompt we need to now setup our new SWAP partition, so we issue the command, “mkswap /dev/VolGroup00/LogVol01″
29. Now we can clone our old 40GB EXT3 partion that we shrunk to our new LVM which is larger than 40GB, but smaller than 65GB, so it will image onto the PE1955 2.5 SAS drive array
30. We use an old favourite for this, “dd if=/dev/VolGroup10/LogVol10 of=/dev/VolGroup00/LogVol00″
31. Once done, shutdown the VPS, boot up in Acronis and image the new 65GB virtual drive and then load it onto the PE1955 server. DONE!

I did all of this in single user mode so as to minimise the need for KUDZU to rescan and change all hardware. I can do this because of the hardware commonality in the 9th generation Dell servers.

Also once done, the new LVM will be smaller than the full capacity of the 73GB SAS 2.5 SAS drive. This is easily fixed while online. by making a new partition /dev/sda3 of LVM type (8e), making it into a Physical Volume, adding it to VolGroup00 and then extending the LogVol00 logical volume with the newly added extents. Once that is done, go back to the command prompt and use the ext2online command to finally expand your EXT3 partition to use all the space on the LVM.

Well, we came, we championed, we explained, we lobbied and we taught. We laughed, we gathered, we swapped tales and we helped clear up some misconceptions.

The sun was out, the sky was blue(ish – not bad by Beijing’s standards) and the students were happy and eager to learn what was going on and to quench their tech/FOSS thirsts. Tsing Hua got rained on by a heap of tech loving goodness!

Photo gallery after the jump….

Anyway, thanks to Google, RedHat, Red Flag, Mozilla and OpenMoko. They paid up cold hard cash to make it all happen. The rest of us did it FOSS style and donated servers, bandwidth, time and eagerness. But at the end of the day, cold hard currency talks!

Click HERE for more info from my previous post.

For a few days now the FIRST FEDORA MIRROR IN CHINA has been online. Proudly sponsored by CANDIS and urged on by the Beijing Linux Users Group.

There are still some tweaks to happen and more data sets to be added over time, but the “Releases” is a good start. And it would not be possible without the great help from Matt Domsch and the Fedora Mirrors List.

http://fedora.candishosting.com.cn