The following is a screen shot of a log from one client, who has a fairly large rack and a few servers for their thin client deployment. In this case the A/C failed and the servers shut down gracefully.

First noting the rising temperature and alerting CANDIS through our monitoring system and then when the failure threshold was met – issuing a shut down command to the OS and then powering off the server. All this happened on a long weekend and the final shut down was alerted to us as well.

Worked like clock work. Then again – I walk into many IDC’s in China and see flashing red failure LEDs and it seems they are forever on. Do SysAdmins in China not setup hardware monitoring? Do they not care? Why is this? Why would you want a failure to occur or interrupt your free time or your clients business?

I always try to get the best prices within the scope of common sense for clients. Mainly because as a integrated stack vendor, the pricing on one component can blow out the total price of the whole stack. And hardware is the first place to start (given an understanding of the management systems and risk credentials of the project at hand already).

One thorn in my side is mandated “Installation and Configuration” charges. Which are basically extortion attempts.

On the one hand no tech from SUN or IBM from their mainframe/mini computer division is going to know or be able to setup all variables in our system as we need – and as we will do over a long period of time – IE: the process of “Configuration” is Iterative and not finite to one small window of time. :-\

And going back to that whole vertical stack business. Why should a client pay a hardware vendor for support and then pay the integrator for support? These built in service charges make it harder for us to win business and sell anything as we have to protect ourselves as we will be doing the support – and that usually means that the client pays twice!

As much as these hardware firms like to chase the integrators and system builders as a great “Channel” to their products buyers in the market place. Mandatory setup and Configuration fees (Yes they apparently are mutually exclusive in charging if not in the vernacular of the Queen’s English!) are a great way of say “Up Yours” to said channel partners/integrators.

It is not yet clear who will get the Gold Medal – but the race is very competitive!

:-\

Well it is not until you actually start to UNDERSTAND it – that you get to appreciate it. All too often the pundits of tech in society (as nice as they are), like twit.tv, cnet.com and zdnet.com – have a great habit of talking about tech and service – but never having actually done it themselves or can appreciate it at a deeper and more sophisticated level.

Case in point, is the fact that many people go on about “I want a real app, not some web app”. Or, I need to have connectivity while I am in a cross country flight, etc. To me it just seems that some people can’t get with the times from either a tech standpoint, or even from another perspective, say costs.

I am using software as a service to highlight the missed nuances of many a lay person or tech pundits analysis of such situations and trends.

I look at all the effort that goes into running email infrastructure, the costs of running servers and IDC’s. And it is quite common now for people to expect email to work like a light switch (major oversimplification) or to be a single small cost in the case of hosting.

Why is it, that software then and other technologies like thin clients, that seek to change the usage and costing of hardware and software from a disparate and ad-hoc, per person/cpu based exercise to a usage and deployment style that is more akin to hosting and email services, that one encounters so much resistance?

Resistance to change from clients and resistance to change from vendors. Why do people still see computing and laptops and applications as objects that can be traded and bought and sold like shells and clams?

I know I asked the question, but I can’t answer it. I simply can’t understand it.

Where we must give some props to in this regard is Amazon, Google and Microsoft, for somewhat seeing the light and being industry leaders. Amazon, offer pay as you need it grid computing power, Google offers online applications and now Microsoft is also offering online applications.

To be fair though, Microsoft has had licensing schemes in place like the SPLA model for a while. Allowing people to just pay a monthly fee to use all the software that they need. Never having to worry about upgrade costs or “upgrade insurance” as some snake oil salesmen like to push.

Allow me to further expand my thoughts on this, with respect to some recent dealings of mine and in China and some questions that I can answer! I will do this as a little Q+A session with myself.

Richard The Interviewer:

“Richard, why would you not want to own your own software and hardware and put it inside your office or in your lap on a laptop?”

Me:

“Thank you for the thoughtful question Richard, allow me to answer on the issue of hardware first. You see Richard, the problem with hardware is that, well it is expensive! And tis fact is made even worse by the fact that it is outdated and superseded so quickly.

Richard The Interviewer:

“But surely even if something loses it’s value it is still useful? I have seen many an old car drive around leaving a trail of rust behind it?”

Me:

“Yes, to a point. The problem is that software gets more complicated and our needs get more demanding as we become more used to and comfortable with our technology. A better way to look at it, is like hand-me-down clothes. Servers and infrastructure to a degree can just be demoted down the line with new beasts coming in at the top for critical systems and then older ones being eventually upgraded – or converted into a VPS as they come up for expiry.”

“But to answer your question more accurately, it has to do with technologies like Virtualisation, economies of scale, operational costs and idle capacity. If you look at your infrastructure like a time share holiday house or corporate jet, it suddenly becomes a lot easier to handle, both on cash flow and on your exposure to depreciation, so there really becomes very little in the way of real world value propositions to hold on to that old junk – this is even without going into the issue of power usage!”

“For many years people liked to finance their equipment. That is a great way to do it, as it lets you amortise the costs over the actual useful life of the system and then throw it out and get a new one. That way, you can look at the leasing fee as a perpetual subscription model. Just make a payment every month and the system is always online, up to date and fault free.”

“Plus it does wonders for your tax exposure when you claim all those leasing payments back against your tax. Much better than doing it the Chinese way with a 5 year depreciation curve!”

Richard The Interviewer:

“OK, I get it. Why own it and pay for the depreciation of it – that will take longer to remove a residual book value than it does to remove any operational value. And just pay one fee once, per month for life and always have the tech taken care of.”

“But it seems that if I do this, I am still having to manage the equipment and run an IT dept, even though I am leasing the hardware?”

Me:

“Yes, so why manage it then if you have already outsourced the ownership to your financing company? That is the strange thing that happens. It is like financing a truck and then paying for a driver. Why not just use DHL or FedEx instead. They offer economies of scale and are better at it.”

Richard The Interviewer:

“But I need control!!! I am worried about privacy!”

Me:

“Control = expense. Digital technology and servers = no privacy. Your needs to control and keep data secure are very valid. But to be truthful, simply having equipment on site or in your control does not make it any more secure or better. These are separate issues. And companies never really think about them any further than just that. I have it, so it is safe. I can touch it, so it is safe. This is really an illusion of security and control.”

Richard The Interviewer:

“Ok, so what about software then? Software as a service….?”

Me:

“Well software is no different. Take all that I mentioned above and change hardware to software. The one exception being that software in most countries does not have any depreciation allowance. In China for example, software is always booked in as a single transaction once into your overhead. That is the tax law. Even if the software cost 1,000,000,000,000 USD and you intend to use it over 20 years, you can’t book the expense in and match it over many accounting periods. So you are much better off in China at least operationally by renting or using an ASP service.”

Richard The Interviewer:

“So in a single paragraph, what is your main issue then with these old dinosaurs who want to own, control, touch and carry their technology around with them all the time?”

Me:

“Do I have to be polite? Well, I would simply say, “What are you thinking?!” Get rid of the big software investments and yearly upgrades and all that hassle. Just hand over a list of needs to an ASP, get a copy of their SLA (service level agreement) and be done with it. Problem solved – for life!”

“As to hardware, well, the beauty of an ASP, is that you don’t need any!”

Richard The Interviewer:

“Well thanks for that Richard. I think I will go and study the intricacies of ASP’s, ISP’s, ESP’s, the tax code, our cash flow, the tech road maps of the various companies out there some more. And work out just why it is, that I feel so compelled to continue handing over cash by the fist full to continue in this rat race of upgrade cycles, repair windows and deployment plans.”

“IT that is like watching a TV or renting a movie online…..hmm, that would be useful and stress free…..seems tomorrow has been here for a few years already!”

Me:

“Maybe next time we can touch on issues like data security, data retention, corporate governance compliance, audit trails, business continuity, capacity budgeting, staff retention, staff acquisition, staff training, staff vetting, staff knowledge pooling and on demand resources?”

Richard The Interviewer:

“That would be good. Is that a whole load of hassle too?”

Me:

“Well it used to be, but it is all par for the course when dealing with a good ASP, ESP, ISP or any xSP really. Even more value!”

Silicon Hutong

And the topic did strike a “Hey this is real man!!!” kind of chord with me. A cathartic resonance that shall never come from me with respect to the greater greenhouse effect crap while the infallible science is still not in. Nor came to me for any other “agenda” or “crusade of the world” that seemed to so very trouble all those students over in the Arts and Law faculties. That seemed to have so much time on their hands, relative to us poor saps in Engineering and Business school.

Politics aside, wastage for the sake of wastage though is not very good. And I have a rather poignant insider’s look at this issue that also manages (in my mind at least) to cross paths with this issue about China’s talent gap, or indeed this also about the talent gap.

What does resource wastage and power requirements have to do with a talent gap, or as they lament now in overly taxed and anti-entrepreneurial Australia – the “Brain Drain”?

Allow me to explain.

Servers cost a lot of money! Not only do you spend the 5-10K USD for the server and warranty, you then have colocation fees, cooling costs and power costs – this all adds up. And if you skimp on something like cooling, your equipment can age prematurely – especially bad in China given the tax depts allowed deprecation rates!

A good contact of mine who runs an IDC in New York city, mentioned how it is all about the power. Real estate? No worries, how much power will we need. So it seems pretty clear where things are going. Moore’s law, computing density increasing and an ever growing appetite for all things information.

With respect to the issues that David Wolf brought up, a major new trend that has been underway for the past 6 years or so and is trying to put a clamp on rising costs of infrastructure is Virtualisation. This technology allows for less resource wastage and consumables wastage by consolidating many lower end servers into fewer bigger more reliable and cost effective servers. Picking the low hanging fruit so to speak.

A further evolution of this that I mentioned about here is where the actual servers in a farm can be powered down too when not needed.

Now when it comes to electricity there are always two rates, the so called “Consumer Rate” and the “Industrial Rate”. And yes, data centre’s use the industrial rates, which while are relatively cheaper than consumer rates – are still not cheap in absolute terms. And if you go and use a nifty tool like this Dell capacity planner and then do some arithmatic, you can see what your costs will be. It is through this tool, that CANDIS deduced that paying the premium for higher efficiency power supplies and low voltage processors, actually did pay for themselves in outright power savings over a 2 year period. Even more if you factor in that less heat is better for stability and system longevity.

So where does the talent gap come into play? Well, in how many servers you need to do a job and knowing about virtualisation.

We had a client that was struggling to keep up and online with their 8 server cluster. They kept just purchasing new servers when things went bad and slowed down. Which seemed to happen all the time. Now I know this and they didn’t, is that an out of the box Linux system is not too tuned and the concurrent IP connection tracking vlaue in the kernel is a little conservative. So instead of them having 8 servers running at 20% capacity and still not managing the load on them as a cluster. Yours truly made a few changes and now 3 servers are doing the job, with better outward performance to the clusters users.

What made this situation all the more funny, is that in China, capital asset valuations are fair game for being compared to the salary of a team of workers to perform the same job (quality and reaction times not withstanding).

So, in summary, I will go out on a limb here and make a bold assertion.

“There are probably more servers than needed running in IDC’s across China, due to engineers not knowing how to admin them properly, not know how to architect systems properly and not knowing how to leverage new technology better.”

Anyway, I could go on. I just wanted to illustrate the nexus of those points and how they relate and intersect. Poorly deployed tech, poorly planned and architected tech and balooning operational costs and resource demands.

Seems that in the past you could be a lazy programmer and no one would notice due to the liberation of faster processors and free cycles. Or in China, buy a server is better than paying the for the expensive vendor support and optimisation.

However as one famous economist once mused…. “there is no such thing as a free lunch”

Select from database ID where name = RICHARD;

Returns and ID of 55 for example.

Then go and do a query like this:

Select * from some_other_table where ID = 55;

Returns, “Sorry does not exist, time to die…..”

Well apparently indexes when corrupt – which is NOT SUPPOSED TO HAPPEN – can cause PostgreSQL to go all stupid and not do a table lookup for real. This happened to me. So I found this:

PlanetPostgresql

Turns out that a reindex and a full vacuum can do wonders – even though a full vacuum is not needed with autovacuum and indexes can’t get corrupted…..or so they say.

I have now added a system wide reindex maintenance plan for PostgreSQL every night. I know that MS-SQL server has an option for this with their maintenance jobs inside enterprise manager. Maybe someone should make an enterprise manager for PostgreSQL too?

In any case, the short of it is, that we were juggling PERC4 cards around servers (PCI-X here, PCIe there..) and also complete raid 1 and raid 10 arrays too. The cards are supposed to “detect” the correct array type from the drives if the firmware was missing. Anyway, through a comedy of errors, it worked exactly 1/3 times. The other times we had to remember the exact settings of our arrays (stripe, etc) and how it was structured. So we could clear PERC cards and then recreate the arrays – taking special care to not initalise the new arrays.

So in the end, you can move arrays and channels about. And with LVM, even designations like /sda /sdb reording is also not an issue. However you should rely on good old fashioned hand held way of doing things. Before you start write down all the salient details of your arrays first.

DRAC RESET

The “gam_server” process. Set to identify when any file in the system is changed. A useful action that has benefits. But not when it does it 3-5 times per second and the sever is serving NFS and PostgreSQL!

To fix it, just ensure that somewhere in /etc/ (RedHat Base) or /etc/gamin/ (Debian Base) has a file called:

“gaminrc”

Add in one line for each file system type that you have. For example mine is now:

#

fsset ext3 poll 10

fsset nfs poll 10

fsset xfs poll 10

#

The daemon will now scan every 10 seconds. Of course for it to do that, make sure to invoke a “kill -s 9″ first!