Well it is not until you actually start to UNDERSTAND it – that you get to appreciate it. All too often the pundits of tech in society (as nice as they are), like twit.tv, cnet.com and zdnet.com – have a great habit of talking about tech and service – but never having actually done it themselves or can appreciate it at a deeper and more sophisticated level.

Case in point, is the fact that many people go on about “I want a real app, not some web app”. Or, I need to have connectivity while I am in a cross country flight, etc. To me it just seems that some people can’t get with the times from either a tech standpoint, or even from another perspective, say costs.

I am using software as a service to highlight the missed nuances of many a lay person or tech pundits analysis of such situations and trends.

I look at all the effort that goes into running email infrastructure, the costs of running servers and IDC’s. And it is quite common now for people to expect email to work like a light switch (major oversimplification) or to be a single small cost in the case of hosting.

Why is it, that software then and other technologies like thin clients, that seek to change the usage and costing of hardware and software from a disparate and ad-hoc, per person/cpu based exercise to a usage and deployment style that is more akin to hosting and email services, that one encounters so much resistance?

Resistance to change from clients and resistance to change from vendors. Why do people still see computing and laptops and applications as objects that can be traded and bought and sold like shells and clams?

I know I asked the question, but I can’t answer it. I simply can’t understand it.

Where we must give some props to in this regard is Amazon, Google and Microsoft, for somewhat seeing the light and being industry leaders. Amazon, offer pay as you need it grid computing power, Google offers online applications and now Microsoft is also offering online applications.

To be fair though, Microsoft has had licensing schemes in place like the SPLA model for a while. Allowing people to just pay a monthly fee to use all the software that they need. Never having to worry about upgrade costs or “upgrade insurance” as some snake oil salesmen like to push.

Allow me to further expand my thoughts on this, with respect to some recent dealings of mine and in China and some questions that I can answer! I will do this as a little Q+A session with myself.

Richard The Interviewer:

“Richard, why would you not want to own your own software and hardware and put it inside your office or in your lap on a laptop?”

Me:

“Thank you for the thoughtful question Richard, allow me to answer on the issue of hardware first. You see Richard, the problem with hardware is that, well it is expensive! And tis fact is made even worse by the fact that it is outdated and superseded so quickly.

Richard The Interviewer:

“But surely even if something loses it’s value it is still useful? I have seen many an old car drive around leaving a trail of rust behind it?”

Me:

“Yes, to a point. The problem is that software gets more complicated and our needs get more demanding as we become more used to and comfortable with our technology. A better way to look at it, is like hand-me-down clothes. Servers and infrastructure to a degree can just be demoted down the line with new beasts coming in at the top for critical systems and then older ones being eventually upgraded – or converted into a VPS as they come up for expiry.”

“But to answer your question more accurately, it has to do with technologies like Virtualisation, economies of scale, operational costs and idle capacity. If you look at your infrastructure like a time share holiday house or corporate jet, it suddenly becomes a lot easier to handle, both on cash flow and on your exposure to depreciation, so there really becomes very little in the way of real world value propositions to hold on to that old junk – this is even without going into the issue of power usage!”

“For many years people liked to finance their equipment. That is a great way to do it, as it lets you amortise the costs over the actual useful life of the system and then throw it out and get a new one. That way, you can look at the leasing fee as a perpetual subscription model. Just make a payment every month and the system is always online, up to date and fault free.”

“Plus it does wonders for your tax exposure when you claim all those leasing payments back against your tax. Much better than doing it the Chinese way with a 5 year depreciation curve!”

Richard The Interviewer:

“OK, I get it. Why own it and pay for the depreciation of it – that will take longer to remove a residual book value than it does to remove any operational value. And just pay one fee once, per month for life and always have the tech taken care of.”

“But it seems that if I do this, I am still having to manage the equipment and run an IT dept, even though I am leasing the hardware?”

Me:

“Yes, so why manage it then if you have already outsourced the ownership to your financing company? That is the strange thing that happens. It is like financing a truck and then paying for a driver. Why not just use DHL or FedEx instead. They offer economies of scale and are better at it.”

Richard The Interviewer:

“But I need control!!! I am worried about privacy!”

Me:

“Control = expense. Digital technology and servers = no privacy. Your needs to control and keep data secure are very valid. But to be truthful, simply having equipment on site or in your control does not make it any more secure or better. These are separate issues. And companies never really think about them any further than just that. I have it, so it is safe. I can touch it, so it is safe. This is really an illusion of security and control.”

Richard The Interviewer:

“Ok, so what about software then? Software as a service….?”

Me:

“Well software is no different. Take all that I mentioned above and change hardware to software. The one exception being that software in most countries does not have any depreciation allowance. In China for example, software is always booked in as a single transaction once into your overhead. That is the tax law. Even if the software cost 1,000,000,000,000 USD and you intend to use it over 20 years, you can’t book the expense in and match it over many accounting periods. So you are much better off in China at least operationally by renting or using an ASP service.”

Richard The Interviewer:

“So in a single paragraph, what is your main issue then with these old dinosaurs who want to own, control, touch and carry their technology around with them all the time?”

Me:

“Do I have to be polite? Well, I would simply say, “What are you thinking?!” Get rid of the big software investments and yearly upgrades and all that hassle. Just hand over a list of needs to an ASP, get a copy of their SLA (service level agreement) and be done with it. Problem solved – for life!”

“As to hardware, well, the beauty of an ASP, is that you don’t need any!”

Richard The Interviewer:

“Well thanks for that Richard. I think I will go and study the intricacies of ASP’s, ISP’s, ESP’s, the tax code, our cash flow, the tech road maps of the various companies out there some more. And work out just why it is, that I feel so compelled to continue handing over cash by the fist full to continue in this rat race of upgrade cycles, repair windows and deployment plans.”

“IT that is like watching a TV or renting a movie online…..hmm, that would be useful and stress free…..seems tomorrow has been here for a few years already!”

Me:

“Maybe next time we can touch on issues like data security, data retention, corporate governance compliance, audit trails, business continuity, capacity budgeting, staff retention, staff acquisition, staff training, staff vetting, staff knowledge pooling and on demand resources?”

Richard The Interviewer:

“That would be good. Is that a whole load of hassle too?”

Me:

“Well it used to be, but it is all par for the course when dealing with a good ASP, ESP, ISP or any xSP really. Even more value!”

I used this excellent resource before.

Even the very rudimentary tweaks I did, and with a RAID array that is in a 90% rebuild rate, background initialisation, this 8.1 version is FAST! I have no idea why people use MySQL, it really is such a piece of crud.

Let me expand on this;

This type of backup file is advertised as “more convenient” and offers more options for restore time selective data restores, data re-ording, index tricks and the like. However no matter WHAT I did, it reported and sent a copy of the current pg_restore process and all the data being restored to standard output too!! This means that basically, I was going to have the same full text of 20GB worth of database data shoved down my SSH session!

Yes – this makes the whole affair much slower!

Luckily, being the lateral thinking kind of dude that I am, I never put all my eggs in one basket. That is just a recipe for data omelette.

I also had some plain text file dumps made from pg_dump. So I went into the the PostgreSQL template1 sessions and then used the “\i” command to import my big .SQL file. Perfect! Only important updates sent to std output and not the whole damn enchilada.

Disk restore is still going on. At a max through put of 20MByte a second with a 1000BaseT network, that is at best 1.2G a minute, 72GB an hour, so for 250GB approximately 3.5 hours. However I was still doing a raid array background initialisation, so even after setting the rebuild rate to 10%, the system still needs a lot of time to move the files back – I didn’t bother to do the maths, because 3.5 hours or 10 hours it would all breach my maintenance window.

Because my advertised downtime window to clients was rapidly approaching, I had no choice but to continue to allow the copy to proceed, but redirect the main cluster to access the data store via NFS over the network and bring services back online. I will then be able to do a RSYNC later in less than 1 hour to bring the haphazardly copied set on the main cluster in line with the now used and modified data store on my hot standby server.

First I noticed that the usual night time “Vacuum’s” that are needed to keep the PostgreSQL planar at it’s most efficient and indexes clean, was running right into the day time! It usually needed less than an hour for the 20GB database we currently have.

So after many failed attempts to get an online vacuum done. I stayed up really late, took the cluster down and did FULL vacuum. Full vacuum’s are slow, and you can’t run anything while they happen because they do full table locks, where as online vacuum’s do quick tuple/row level locks.

Anyway, database seemed speedier, but system was still sluggish. I have all data separated. Database files are on large RAID10 arrays with U320 SCSI drives spinning at 15K – split over TWO SCSI buses! Yeah tis fast. Big disks are used because it means relative to the size of the disk, more data is on the outer edge of the platters, that spin faster than the centre of the platters. I also keep PostgreSQL’s transaction log on a separate RAID1 array with 73GB 15K U320 drives as well with a 256MB battery backed cache.

Now database based operations were zippy again….the system while set by me to not rate a disk search at too high a cost due to the super speed disk IO that I have, still should get SOME data out of the cache and not run to the comparatively sloooow disks straight away. After doing this all was cool again database wise.

However operations that needed the data store to be accessed were still piss slow. So email ingestion, file usage, etc crawled. The data store which compliments the database data weighs in at about 250GB now. And this is on a single RAID1 array with 300GB U320 10K drives. I also remembered that over a 1000BaseT network to backup to a robotic tape library, the data store array (RAID1) maxes out at about 750MB a min. While the RAID10 with the database does it at 2700+MB a min.

So I thought, “lets add some more disks to the data store array”. I get more space, so relatively speaking more of the data is on the outside edges of all the platters (another topic for what one can do with LVM), the array has double the heads and spindles too – and some RAID0 goodness inside that RAID10 nested set.

Now Dell’s storage white paper from 2005 does state that a RAID1 to a RAID10 migration/raid level reconstruction is supported. RAID10 being two RAID1 arrays striped together in RAID0 (with the mirror part straddling 2 SCSI buses for speed and channel redundancy). However when I went into Open Manage after putting the drives in the chassis, expecting to be able to “reconstruct/migrate levels” to a RAID10, then pop into LVM, create some new data devices, add them to my Volume group and then expand my partition and then finally my file system – all while still being online……..I was greeted with only the option to reconstruct/migrate to RAID5 (Get ^*^*&^) or RAID0 (Uh, yeah, OK..).

So feeling quite annoyed now. I then went and made the two new drives into a new RAID1 set, thinking that I would then be able to add this to a final “nested” array with the current RAID1 set and make a RAID10 out of them. Well I thought it was working, but it wasn’t. All I managed to do was end up with my existing array being made into a RAID0 (heart attack!!) and the new array sitting there untouched. Also the Open Manage array management setup gave me no choice of stripe size. It defaulted to 64K. I prefer 128K to ensure the best chance of the records/tuples fitting into one whole stripe to maximise concurrency of head operations on different data records – and also because the PERC4 family of RAID cards does not suffer any penalty if a stripe size is too big for the data used – so why not eh?

So what am I doing now at 5:32 AM?

Rsyncing all data again to the hot standby server, will then power off the cluster, log in with the DRAC card, go into the BIOS of the RAID card and redo my friggin array from scratch, as RAID10 and 128K stripe and then boot up and copy my damn data back.

Not happy pappy. Not happy. I think it is SAN time…unless SAN’s can be this anal as well?

So what does this have to do with PostgreSQL upgrade? Well since the cluster is down anyway, I might as well go from 7.4 to 8.1. Get some of that auto vacuum goodness and lap up some of the apparent massive speed boosts in the 5 years of development between the two versions.