The following is a screen shot of a log from one client, who has a fairly large rack and a few servers for their thin client deployment. In this case the A/C failed and the servers shut down gracefully.

First noting the rising temperature and alerting CANDIS through our monitoring system and then when the failure threshold was met – issuing a shut down command to the OS and then powering off the server. All this happened on a long weekend and the final shut down was alerted to us as well.

Worked like clock work. Then again – I walk into many IDC’s in China and see flashing red failure LEDs and it seems they are forever on. Do SysAdmins in China not setup hardware monitoring? Do they not care? Why is this? Why would you want a failure to occur or interrupt your free time or your clients business?

I always try to get the best prices within the scope of common sense for clients. Mainly because as a integrated stack vendor, the pricing on one component can blow out the total price of the whole stack. And hardware is the first place to start (given an understanding of the management systems and risk credentials of the project at hand already).

One thorn in my side is mandated “Installation and Configuration” charges. Which are basically extortion attempts.

On the one hand no tech from SUN or IBM from their mainframe/mini computer division is going to know or be able to setup all variables in our system as we need – and as we will do over a long period of time – IE: the process of “Configuration” is Iterative and not finite to one small window of time. :-\

And going back to that whole vertical stack business. Why should a client pay a hardware vendor for support and then pay the integrator for support? These built in service charges make it harder for us to win business and sell anything as we have to protect ourselves as we will be doing the support – and that usually means that the client pays twice!

As much as these hardware firms like to chase the integrators and system builders as a great “Channel” to their products buyers in the market place. Mandatory setup and Configuration fees (Yes they apparently are mutually exclusive in charging if not in the vernacular of the Queen’s English!) are a great way of say “Up Yours” to said channel partners/integrators.

It is not yet clear who will get the Gold Medal – but the race is very competitive!

:-\

Well it is not until you actually start to UNDERSTAND it – that you get to appreciate it. All too often the pundits of tech in society (as nice as they are), like twit.tv, cnet.com and zdnet.com – have a great habit of talking about tech and service – but never having actually done it themselves or can appreciate it at a deeper and more sophisticated level.

Case in point, is the fact that many people go on about “I want a real app, not some web app”. Or, I need to have connectivity while I am in a cross country flight, etc. To me it just seems that some people can’t get with the times from either a tech standpoint, or even from another perspective, say costs.

I am using software as a service to highlight the missed nuances of many a lay person or tech pundits analysis of such situations and trends.

I look at all the effort that goes into running email infrastructure, the costs of running servers and IDC’s. And it is quite common now for people to expect email to work like a light switch (major oversimplification) or to be a single small cost in the case of hosting.

Why is it, that software then and other technologies like thin clients, that seek to change the usage and costing of hardware and software from a disparate and ad-hoc, per person/cpu based exercise to a usage and deployment style that is more akin to hosting and email services, that one encounters so much resistance?

Resistance to change from clients and resistance to change from vendors. Why do people still see computing and laptops and applications as objects that can be traded and bought and sold like shells and clams?

I know I asked the question, but I can’t answer it. I simply can’t understand it.

Where we must give some props to in this regard is Amazon, Google and Microsoft, for somewhat seeing the light and being industry leaders. Amazon, offer pay as you need it grid computing power, Google offers online applications and now Microsoft is also offering online applications.

To be fair though, Microsoft has had licensing schemes in place like the SPLA model for a while. Allowing people to just pay a monthly fee to use all the software that they need. Never having to worry about upgrade costs or “upgrade insurance” as some snake oil salesmen like to push.

Allow me to further expand my thoughts on this, with respect to some recent dealings of mine and in China and some questions that I can answer! I will do this as a little Q+A session with myself.

Richard The Interviewer:

“Richard, why would you not want to own your own software and hardware and put it inside your office or in your lap on a laptop?”

Me:

“Thank you for the thoughtful question Richard, allow me to answer on the issue of hardware first. You see Richard, the problem with hardware is that, well it is expensive! And tis fact is made even worse by the fact that it is outdated and superseded so quickly.

Richard The Interviewer:

“But surely even if something loses it’s value it is still useful? I have seen many an old car drive around leaving a trail of rust behind it?”

Me:

“Yes, to a point. The problem is that software gets more complicated and our needs get more demanding as we become more used to and comfortable with our technology. A better way to look at it, is like hand-me-down clothes. Servers and infrastructure to a degree can just be demoted down the line with new beasts coming in at the top for critical systems and then older ones being eventually upgraded – or converted into a VPS as they come up for expiry.”

“But to answer your question more accurately, it has to do with technologies like Virtualisation, economies of scale, operational costs and idle capacity. If you look at your infrastructure like a time share holiday house or corporate jet, it suddenly becomes a lot easier to handle, both on cash flow and on your exposure to depreciation, so there really becomes very little in the way of real world value propositions to hold on to that old junk – this is even without going into the issue of power usage!”

“For many years people liked to finance their equipment. That is a great way to do it, as it lets you amortise the costs over the actual useful life of the system and then throw it out and get a new one. That way, you can look at the leasing fee as a perpetual subscription model. Just make a payment every month and the system is always online, up to date and fault free.”

“Plus it does wonders for your tax exposure when you claim all those leasing payments back against your tax. Much better than doing it the Chinese way with a 5 year depreciation curve!”

Richard The Interviewer:

“OK, I get it. Why own it and pay for the depreciation of it – that will take longer to remove a residual book value than it does to remove any operational value. And just pay one fee once, per month for life and always have the tech taken care of.”

“But it seems that if I do this, I am still having to manage the equipment and run an IT dept, even though I am leasing the hardware?”

Me:

“Yes, so why manage it then if you have already outsourced the ownership to your financing company? That is the strange thing that happens. It is like financing a truck and then paying for a driver. Why not just use DHL or FedEx instead. They offer economies of scale and are better at it.”

Richard The Interviewer:

“But I need control!!! I am worried about privacy!”

Me:

“Control = expense. Digital technology and servers = no privacy. Your needs to control and keep data secure are very valid. But to be truthful, simply having equipment on site or in your control does not make it any more secure or better. These are separate issues. And companies never really think about them any further than just that. I have it, so it is safe. I can touch it, so it is safe. This is really an illusion of security and control.”

Richard The Interviewer:

“Ok, so what about software then? Software as a service….?”

Me:

“Well software is no different. Take all that I mentioned above and change hardware to software. The one exception being that software in most countries does not have any depreciation allowance. In China for example, software is always booked in as a single transaction once into your overhead. That is the tax law. Even if the software cost 1,000,000,000,000 USD and you intend to use it over 20 years, you can’t book the expense in and match it over many accounting periods. So you are much better off in China at least operationally by renting or using an ASP service.”

Richard The Interviewer:

“So in a single paragraph, what is your main issue then with these old dinosaurs who want to own, control, touch and carry their technology around with them all the time?”

Me:

“Do I have to be polite? Well, I would simply say, “What are you thinking?!” Get rid of the big software investments and yearly upgrades and all that hassle. Just hand over a list of needs to an ASP, get a copy of their SLA (service level agreement) and be done with it. Problem solved – for life!”

“As to hardware, well, the beauty of an ASP, is that you don’t need any!”

Richard The Interviewer:

“Well thanks for that Richard. I think I will go and study the intricacies of ASP’s, ISP’s, ESP’s, the tax code, our cash flow, the tech road maps of the various companies out there some more. And work out just why it is, that I feel so compelled to continue handing over cash by the fist full to continue in this rat race of upgrade cycles, repair windows and deployment plans.”

“IT that is like watching a TV or renting a movie online…..hmm, that would be useful and stress free…..seems tomorrow has been here for a few years already!”

Me:

“Maybe next time we can touch on issues like data security, data retention, corporate governance compliance, audit trails, business continuity, capacity budgeting, staff retention, staff acquisition, staff training, staff vetting, staff knowledge pooling and on demand resources?”

Richard The Interviewer:

“That would be good. Is that a whole load of hassle too?”

Me:

“Well it used to be, but it is all par for the course when dealing with a good ASP, ESP, ISP or any xSP really. Even more value!”

Silicon Hutong

And the topic did strike a “Hey this is real man!!!” kind of chord with me. A cathartic resonance that shall never come from me with respect to the greater greenhouse effect crap while the infallible science is still not in. Nor came to me for any other “agenda” or “crusade of the world” that seemed to so very trouble all those students over in the Arts and Law faculties. That seemed to have so much time on their hands, relative to us poor saps in Engineering and Business school.

Politics aside, wastage for the sake of wastage though is not very good. And I have a rather poignant insider’s look at this issue that also manages (in my mind at least) to cross paths with this issue about China’s talent gap, or indeed this also about the talent gap.

What does resource wastage and power requirements have to do with a talent gap, or as they lament now in overly taxed and anti-entrepreneurial Australia – the “Brain Drain”?

Allow me to explain.

Servers cost a lot of money! Not only do you spend the 5-10K USD for the server and warranty, you then have colocation fees, cooling costs and power costs – this all adds up. And if you skimp on something like cooling, your equipment can age prematurely – especially bad in China given the tax depts allowed deprecation rates!

A good contact of mine who runs an IDC in New York city, mentioned how it is all about the power. Real estate? No worries, how much power will we need. So it seems pretty clear where things are going. Moore’s law, computing density increasing and an ever growing appetite for all things information.

With respect to the issues that David Wolf brought up, a major new trend that has been underway for the past 6 years or so and is trying to put a clamp on rising costs of infrastructure is Virtualisation. This technology allows for less resource wastage and consumables wastage by consolidating many lower end servers into fewer bigger more reliable and cost effective servers. Picking the low hanging fruit so to speak.

A further evolution of this that I mentioned about here is where the actual servers in a farm can be powered down too when not needed.

Now when it comes to electricity there are always two rates, the so called “Consumer Rate” and the “Industrial Rate”. And yes, data centre’s use the industrial rates, which while are relatively cheaper than consumer rates – are still not cheap in absolute terms. And if you go and use a nifty tool like this Dell capacity planner and then do some arithmatic, you can see what your costs will be. It is through this tool, that CANDIS deduced that paying the premium for higher efficiency power supplies and low voltage processors, actually did pay for themselves in outright power savings over a 2 year period. Even more if you factor in that less heat is better for stability and system longevity.

So where does the talent gap come into play? Well, in how many servers you need to do a job and knowing about virtualisation.

We had a client that was struggling to keep up and online with their 8 server cluster. They kept just purchasing new servers when things went bad and slowed down. Which seemed to happen all the time. Now I know this and they didn’t, is that an out of the box Linux system is not too tuned and the concurrent IP connection tracking vlaue in the kernel is a little conservative. So instead of them having 8 servers running at 20% capacity and still not managing the load on them as a cluster. Yours truly made a few changes and now 3 servers are doing the job, with better outward performance to the clusters users.

What made this situation all the more funny, is that in China, capital asset valuations are fair game for being compared to the salary of a team of workers to perform the same job (quality and reaction times not withstanding).

So, in summary, I will go out on a limb here and make a bold assertion.

“There are probably more servers than needed running in IDC’s across China, due to engineers not knowing how to admin them properly, not know how to architect systems properly and not knowing how to leverage new technology better.”

Anyway, I could go on. I just wanted to illustrate the nexus of those points and how they relate and intersect. Poorly deployed tech, poorly planned and architected tech and balooning operational costs and resource demands.

Seems that in the past you could be a lazy programmer and no one would notice due to the liberation of faster processors and free cycles. Or in China, buy a server is better than paying the for the expensive vendor support and optimisation.

However as one famous economist once mused…. “there is no such thing as a free lunch”

Select from database ID where name = RICHARD;

Returns and ID of 55 for example.

Then go and do a query like this:

Select * from some_other_table where ID = 55;

Returns, “Sorry does not exist, time to die…..”

Well apparently indexes when corrupt – which is NOT SUPPOSED TO HAPPEN – can cause PostgreSQL to go all stupid and not do a table lookup for real. This happened to me. So I found this:

PlanetPostgresql

Turns out that a reindex and a full vacuum can do wonders – even though a full vacuum is not needed with autovacuum and indexes can’t get corrupted…..or so they say.

I have now added a system wide reindex maintenance plan for PostgreSQL every night. I know that MS-SQL server has an option for this with their maintenance jobs inside enterprise manager. Maybe someone should make an enterprise manager for PostgreSQL too?

In any case, the short of it is, that we were juggling PERC4 cards around servers (PCI-X here, PCIe there..) and also complete raid 1 and raid 10 arrays too. The cards are supposed to “detect” the correct array type from the drives if the firmware was missing. Anyway, through a comedy of errors, it worked exactly 1/3 times. The other times we had to remember the exact settings of our arrays (stripe, etc) and how it was structured. So we could clear PERC cards and then recreate the arrays – taking special care to not initalise the new arrays.

So in the end, you can move arrays and channels about. And with LVM, even designations like /sda /sdb reording is also not an issue. However you should rely on good old fashioned hand held way of doing things. Before you start write down all the salient details of your arrays first.

DRAC RESET

The “gam_server” process. Set to identify when any file in the system is changed. A useful action that has benefits. But not when it does it 3-5 times per second and the sever is serving NFS and PostgreSQL!

To fix it, just ensure that somewhere in /etc/ (RedHat Base) or /etc/gamin/ (Debian Base) has a file called:

“gaminrc”

Add in one line for each file system type that you have. For example mine is now:

#

fsset ext3 poll 10

fsset nfs poll 10

fsset xfs poll 10

#

The daemon will now scan every 10 seconds. Of course for it to do that, make sure to invoke a “kill -s 9″ first!

What is a mixed bag in my mind is the new ESX3i. It will be first released in the new Dell VORSO servers, originally due to ship this month and as confirmed by my DELL Sales manager, is slipping into the new year. These servers will not need hard disks (So what about core dumps, logs and swap?) but will have embedded flash with the hypervisor installed.

On the surface it looks like the Hypervisor is now part of the system board. It ain’t. So I don’t expect ESX3i to give any speed increase. What I do expect is that it will give a massive boost to deployment times. If you are deploying new servers at the rate of one or more a week, this is cool. Right now at CANDIS with blades, we have all the wiring already in place due to the chassis. But we still need to go in through the management card to setup the OS. Yes we can open the box and slide in the blade and have a new server fully setup physically. But not the software. From the looks of ESX3i, it goes a step further. Power on the system, it finds a VC server, configures itself, boots up and presents it’s resources to the farm, ready to be dynamically allocated by the VC server.

Oh, they also have smart power management that can power down systems when their computing resources are not needed. This is cool. Since we pay about .2 RMB per kilowatt of power, it does add up. The new low voltage XEON’s are great power savers, as are the new high efficiency power supplies. However add in those dynamic server on/off controls and you will be able to make a serious dent in your power bill. Oh, and keep the “Eco-Terrorists” happy too and thus maybe avoid a politically correct jihad aimed at you, unlike poor Apple Inc.

VMWARE

This week, via the trickle down effect in the echelons of the China network, we got informed that a couple of our clients are not compliant and they need to become so, or remove their website.

We have had to remove some clients sites before because they did not have the correct ICP. They could not get one because they did not have the correct business licences for the type of work they did.

Anyway, while having an ICP is a delicate balance between business licences, residential permits, correct content, correctly licenced IDC and what not, it is a good thing to get done and host in China. To be official, legit and free to get on with doing business.

For the most part getting an ICP, or the existence of an ICP is not a barrier to hosting or a problem to comply with. It sounds a whole lot more draconian than it really is. However it is a hassle. For you China hands, “mafan si le”!

It is golden week here and all are away on break. So instead of forcing a staff member to come back, I thought I would take care of some stuff myself.

My problems started when a client who has a large advertising cluster, was running their main statistics database (for click fraud detection) on a Dell 1950 with only 1 SAS 15K drive.

I had suggested that this node, not being redundant like the tomcat servers be individually redundant, so DRAC card, redundant power and RAID.

Anyway, some new blades, Dell 1955′s arrived for the cluster and I thought, well, lets save the client some money, image the old 1950 DB server and load it onto a new 1955 server?

I thought this would be simple with Acronis.

No it wasn’t.

It turns out that a 3.5 Inch 73GB SAS drive is not the same size as a 2.5 Inch 73GB SAS drive. So I could not write my system image to the blades raid 1 array of 2 x 15K 73GB SAS drives.

Shite! If I reinstall the DB server it is not worth my time. Cheaper to buy the upgrade parts for the 1950.

Then I thought, well, I have LVM, I should be able to do this, after all I have used LVM before many times on large storage arrays.

So my goal was this, I needed a system image that was a couple hundred megs smaller than it was now, so it will go into the 1955′s ok.

This is where I added in VMWARE to the mix and made this an easy task. The steps are below:

1. Image the 1950 server to some ACRONIS TIB files somewhere. I used FTP
2. Image the TIB file to a new VPS made with a 73GB virtual disk
3. Create and attach a new 65GB virtual disk to the virtual machine
4. Image the MBR and /boot partitions using acronis onto the new 65GB virtual disk
5. Boot virtual machine with a rescue/live CD
6. Load FDISK for /dev/sdb and create a new LVM (Type 8e) partition in the remaining space on the 65GB virtual drive
7. Enter LVM with the “lvm” command
8. Activate all Volume Groups with the command “vgchange -a y”
9. EXIT out of LVM and then run this command to resize the EXT3 file system, “resize2fs /dev/VolGroup/LogVol00 40G”, you may have to run “e2fschk -f /dev/VolGroup00/LogVol00″ first too
10. Enter LVM again with the “lvm” command
11. Now we can reduce the Logical Volume that had the recently shrunk EXT3 file system on it with this command, “lvreduce LogVol00 -L 45G”
12. Because we are making a new custom boot image and we have already imaged over the /boot partition and MBR, we now want our old 73GB virtual drive to not have any of the same markings as our embryonic new 65GB virtual drive. To do this we need to change the Volume Group and Logical Volume names to something new:
13. lvrename VolGroup00 LogVol00 LogVol10
14. lvrename VolGroup00 LogVol01 LogVol11
15. lvchange LogVol10 -a n
16. lvchange LogVol11 -a n
17. vgchange VolGroup00 VolGroup10
18. Now we can create the new Logical Volumes and Volume Groups on the 65GB virtual disk in preparation for cloning our now 40GB EXT3 file system
19. Make a new Physical Volume first wit, “pvcreate /dev/sdb2″
20. pvscan
21. vgcreate VolGroup00 /dev/sdb2
22. vgscan
23. lvcreate VolGroup00 /dev/sdb2 -n LogVol01 -L 2G
24. lvcreate VolGroup00 /dev/sdb2 -n LogVol00 -L 50G
25. lvscan
26. Now we need to make our new LVM’s online and visible to the system so we run the command vgchange again, “vgchange -a y”
27. EXIT
28. Back at the command prompt we need to now setup our new SWAP partition, so we issue the command, “mkswap /dev/VolGroup00/LogVol01″
29. Now we can clone our old 40GB EXT3 partion that we shrunk to our new LVM which is larger than 40GB, but smaller than 65GB, so it will image onto the PE1955 2.5 SAS drive array
30. We use an old favourite for this, “dd if=/dev/VolGroup10/LogVol10 of=/dev/VolGroup00/LogVol00″
31. Once done, shutdown the VPS, boot up in Acronis and image the new 65GB virtual drive and then load it onto the PE1955 server. DONE!

I did all of this in single user mode so as to minimise the need for KUDZU to rescan and change all hardware. I can do this because of the hardware commonality in the 9th generation Dell servers.

Also once done, the new LVM will be smaller than the full capacity of the 73GB SAS 2.5 SAS drive. This is easily fixed while online. by making a new partition /dev/sda3 of LVM type (8e), making it into a Physical Volume, adding it to VolGroup00 and then extending the LogVol00 logical volume with the newly added extents. Once that is done, go back to the command prompt and use the ext2online command to finally expand your EXT3 partition to use all the space on the LVM.