ICP, yes… but a “Cultural Understanding Course” ??!

My client explained to me that according to their current infrastructure provider, “since we are running a site that targeted the Chinese and had music and culture on it, in order to continue to host he needed to undertake a Cultural Understanding Course or just make a special payment of 10K RMB to avoid it.”  I’m paraphrasing but that is the gist of it.

Something didn’t smell right.  I have never known of any law or regulation in China that can be avoided simply by paying money to someone! 

In this case, it was the ISP asking for the money – not those in charge of the “Law” – which subsequent checking has failed to find any said regulation/requirement.

So what this all boils down to is a local company, trying to strong arm and exploit the naive foreigner.  No such suprise for me, this happens all the time.  The problem is not with the Chinese.  The problem is with the foreigners that come to China, exist here for 1 year, think they are China “Experts” and then proceed to do things that they wouldn’t dream of doing in their own country.

Now such stupidity can lead to being locked up in the extreme or losing a lot of money at the other end of the scale.  And no one industry is immune to these kinds of land mines.  It further proves the point that in China, you do get what you pay for (why are the average foreigners here so cheap?) and that experience counts.  And if you don’t have the latter, be man enough to admit that to yourself, your higher ups and co workers and seek out someone who does know and whom you can trust.  It is a lot less of a loss of face than being locked up or having to explain missing funds, or missed deadlines.

In general, educated people aren’t know it alls when it comes to their doctor and happily seek second opinions and often pay a premium for niche advice.  Why would any rational and educated business person in China operate any different?

It seems some foreigners in China do need a “Cultural Understanding Course” anyway – just of a different kind.  However that course takes between 5 and 10 years to complete and often involves not working inside a bubble wrapped MNC either.

Start Here

There has been a bit of a maelstrom recently regarding the use of .CN and ICP certificates all on top of the existing confusion.

So, to try and set the record straight this article is a nice little walk through the pages of history and clearly state the facts of the matter from the inception until the present. And hopefully not only explaining what the situation is, but also how it came about and the anthropology of how we got to the situation that we have today.

The Land that Time Forgot – 2005

There was a time where life was sweet, chocolate milk flowed from every faucet, a reasonably priced pork knuckle had crispy skin and the internet in China was (comparatively) more functional.

That time was 2005.

However, that was also the year that ICP, or Internet Content Provider, licenses first appeared. At first, like many new Chinese regulations, it was all bluster and wind. Combined with a good healthy respect for the lack of organizational ability of the people that often make these broad and sudden declarations – people just sat tight. Taking the view that, when they get their act together, they will point out to us where we are in breach and we will patch things on a case by case basis.

Even if legislators had wanted to force through compliance straight away – one couldn’t – nothing was set up and the ink was still wet on the paper. And like most things in China – subject to heavy blotting still.

People ask: “Do you remember the time and place that you were when you heard of the assassination of JFK or the death of Princess Di?” For some strange reason I remember the time and place I was when I heard about ICP too.

I was standing in our office’s mini machine room, kicking a very stubborn raised floor tile back into position and wondering why it couldn’t have been made to fit and work properly by those who created it in the first place before it got to me…

They got their Act Together – December 2008

Finally, at the end of 2008 the ISPs and IDCs started receiving the dreaded phone calls. Various time limits were given from a few hours to a week to get sites offline or registered. Sometimes lists were provided, but for every 20 sites identified, another 100 would be on the same IP (virtual hosting) with no request for an ICP made. One could therefore assume that DNS servers were being harvested to get an insight into the habits and traffic directions of users to find unregistered web sites. So the more obscure you were, the better it seemed.

In most cases one simply filled in the online registration form and was done with it. The only people who could not do this were dodgy businesses without clear registration in China, or were operating outside of their legally registered area. When discovered, if an ICP license was unobtainable (for whatever reason), the site had to be moved out of China.

It was also around this time that various hosting companies in China started to demand an ICP up front before putting a site up. While this was not strictly needed, it is how they decided to do things. By not needed, I mean that to apply for an ICP is not that hard and you could do that concurrently with the initial loading and development of a site. Later it became a requirement for all to do things in order.

There were some awkward moments too. You see many servers have “Remote Management Cards” as well as “IPMI/BMC Management Interfaces” – these all are web based. Company’s also have Extranets and Intranets. Which are also? Yup, web based.

The point that I am making is that there are a lot of web pages out there that are not web sites and are not for the public and indeed are all password protected. This however did not make them immune from cease and desist and take down threats. At first it started as a bit of a joke – “gee how ignorant can they be?” A good laugh at the policy
enforcers’ expense. Then it became annoying. Finally with enough heated phone calls and pleaded explanations via intermediaries, the enforcers gained the knowledge that these sorts of “Web Pages” are out there and can’t be “registered” or have an ICP code embedded in them.

The Shit Hits the Fan – mid-2009

While things progressed at a steady pace – downwards – in more ways than one – the ICP enforcement game started to become more prominent. Like clockwork, every FRIDAY afternoon, at 5.30PM, a call would come down the line – “You have 1 hour to remove the offending content or all of your IP’s and routes will be cut.”

I think it is important to also explain what “remove content” means, because in its nascent use in 2009 – we can see the genesis of the procedures that are causing confusion today.

Remove content means just that – remove ALL the content. It doesn’t matter WHAT the content is – EVERYTHING must be gone.

So…….? I hear you thinking.

Well simply using a control panel to “disable” a site is not enough. Because there would still be “Content” visible. “Site Disabled for Administrative Purposes” is just as infringing as anything else.

This then led to the next step – OK – remove the Virtual Host declarations. This also didn’t work. Because in the
absence of any defined VHOST the HTTPD daemon will serve the first declared VHOST on the server – which unto itself may be perfectly legit as a site.

However this is still “Content” and it has to be removed.

So one then removed all the files from these offending sites, leaving the VHOST declarations intact. This of course then causes a series 500 error page upon loading. And while this isn’t a page that exists or could be edited and is in fact generated automatically… as an informational error message that we find all over tech land (..PC Load Letter..?! )…

…this is still “Content” and therefore it has to be removed too.

The answer then was to simply place an empty index.html file into the virtual host root. Upon loading the domain a blank page is presented.

This is the accepted definition of “Remove Content”. So take note and don’t ask why? Or why not? Just accept it and get on with things. You’ll live longer.

So what is an ICP and who really needs one?

We have now laid out the history of the inception of the ICP and how it was deployed and some of the lifecycle of how it has been enforced and managed. Before we get to the current stage of this little grub’s metamorphous into the beautiful butterfly that we have today in early 2010, lets indulge in one last bit of confusion surrounding the ICP.

Who needs one?

Before we go into this section – please note that the details contained within are purely for historic value. They DO NOT REPRESENT CURRENT STATE OF AFFAIRS, and more importantly how the law is enforced is still changing. What it will do, is like the previous discussions give an understanding to what has happened in the past and how that has helped shape the situation that we have today, both for better and for worse…

Originally, the rules as laid out and explained by the people that you apply for an ICP with, was that an ICP is needed for all domains that list a Chinese address in the WHOIS, are located on a China assigned IP and are owned/operated by a Chinese business entity. The ICP for these types was free in principle. For eCommerce and blog/BBS sites there were additional requirements as well as a quite high “application security fee” for eCommerce sites – refundable after the application process was completed (minus a few charges here and there).

A local business actually grew out of this to lend companies money for the application process – against a fee of course.

So foreign WHOIS, foreign IP, foreign company and individuals, were all (read: not enforced, though not necessarily legally) exempted. This situation, changed for the worse and caused no end of confusion when people who had foreign companies and/or foreign WHOIS records over night were being told by the enforcers (the people on the other end of the line that tell you have 1 hour before all IP’s are cut) to get ICP’s for all sites on a China assigned IP regardless of anything else. Of course this made it next to impossible for some companies to acquire an ICP, especially since you would need a Chinese Business License to apply for one.

The internet by it’s nature is borderless and many organizations that operate in China have a web site here by mandate of the superior local speed and the crappy international speed. Individually owned and operated sites were still in the clear though.

So what is one to do if you are a foreign entity and don’t have a local entity, be it a Representative Office, JV, WFOE or other establishment and you want an ICP? How the law was enforced at the time would lead you to conclude that you didn’t need an ICP. Indeed if you tried to apply for one with the ICP application office, they’d also tell you that you didn’t need one.

However let’s not be forgetting that there are those people on the end of the phone line at 5.30PM on a Friday…….so what is one to do? Well smart money is on listening to those with the biggest stick and actionable power.

This led to a somewhat dodgy and messy situation of foreign companies or web sites then trying to proxy or slipstream into an ICP via a local entity. The most common was the eager and “long term not in your best interests but only ours” local IDC/ISP’s that would, for a fee, let you use their ICP.

The ramifications of this ranged from dubious to outright illegal in some cases. Many of these ISP’s/IDC’s were basically taking quick profit from these hapless foreigners (oh how many a hapless foreigner, be it inside or outside China manages to get caught with their pants down around their ankles) who would eventually be found and taken down anyway after paying up their yearlong (non-cancelable of course) hosting contract.

A crackdown finally happened and hundreds of unregistered and dodgy data centers and ISP’s were shut down late last year. Many a foreign company who was lured in this way though, got their stuff online, did the big launch and started to spend money of advertising, only to be shut down one day out of the blue, usually finding out because their clients complain, not because their local IDC/ISP knew about or could bother to inform prior to taking action.

Back to our Time Line – it is late 2009

So, by the very end of 2009, for better and worse the internet in China is cleaned up a lot. The dodgy sites are gone, the dodgy data centers and ISP’s offline. Everyone is obeying, respecting and following the law and requirements of the ICP.

While it remains a problem with technologies fairly new to China, like geographic load balancing, private networks, IP aliasing, storage and cloud fabrics that further blur the line of “Server” and “IP” to be something far more ethereal and harder to pin down, in general the rule of thumb is – do what you need to do to make the tech work. But if content is being served to the public, from a China assigned IP, it must have an ICP, registered by the owners legal Chinese entity.

Thus the ICP issue was finally settled and the rules and regulations crystal clear at the end of 2009 – or so one would think. Though one never got a clearer or more definitive statement of what was needed and who was required to have one. Pragmatically speaking though, the actions of the enforcers spoke loudly and clearly. And the legitimate industry players listened.

Post Script – Time of Domain Crack Down

So, now towards the end of Q1, 2010, things are almost settled – again. By settled, I don’t mean ideal, I mean, quantifiable. This is where the domain name saga starts. First off there was a ruling that no Chinese citizen is allowed to own ANY domain name. As a result domains that could not be verified as being owned by a business in all TLD’s were being excised from the registries very quickly by local registrars.

Now whether or not this is in line with current ICANN rules or other TLD rules – if you are a Chinese registrar and are being told that certain domain name sales are now illegal… what is one to do? Well smart money is on listening to those with the biggest stick and actionable power. Am I repeating myself?

Of course if one has a domain registered overseas in any domain but .CN – then you are safe – no one can take your domain away from you. If on the other hand you do have a .CN domain name, registered anywhere in the world – you now have a problem. The Chinese Academy of Science – who makes the rules for .CN domains on behalf of the Ministries – are in the process of enforcing their specific understanding of these rules, rules that now are seeming to settle in.

What has finally settled in the last few weeks (everyone now agrees on a common understanding of the intent and purpose of the law – at least it seems that way) is that basically all .CN domains have to be registered to a Chinese registered company and furthermore the business name and .CN domain names must have a proven similarity or relationship.

This is not all that new of an idea. auDA along with a lot of other TLD authorities saw the shambles that went on in the .COM space with domain squatters and trade mark disputes and wanted to avoid the same fate. China is merely bringing control of it’s own .CN in line with how other countries manage their TLD’s – at least in part.

This is a good move if you ask me, as many a domain name here is “taken” and put up with a nice for sale parking sign. Domain squatters and opportunists have run rampant in China with the comparatively cheap cost to register a domain name.

During this whole process of the facts settling in, hosting companies received emails from the registrars that are on the ball and were given key dates and pre filled in PDF’s of the legal forms for convenience. Clients that took the cheap and ill advised registrar route when it comes to business domain name registration (let alone .CN) got nothing. There was an uproar and the EU got involved and started to solicit for information (complaints) from businesses.

The key dates came and went and nothing happened.

So now what does one do if one wants to register a new .CN domain name?

REQUIREMENTS

Domain name applicants need to submit the formal paper based application material as well when making the online application to the registrar.

The application material includes the original application form with Chinese branch business seal, company business license (photocopy), and registrant ID (photocopy).

Individuals are not allowed to register.

China Branch: means international companies, enterprises, and organizations establish within
mainland China (PRC), a wholly owned or share controlling entity: including a branch, a subsidiary or a representative office which is having the same “name” as the parent entity. The local entity must register with the relevant Chinese authorities.

For example: IBM should use “IBM China” to register IBM.cn

Chinese employee: the person submitting the application on behalf of the registrant (the entity), must be a Chinese citizen and has a valid PRC personal ID.

Once all the above is done, the domain name must also be pointed at a China assigned IP and left content free (ie blank HTML page) for 20 working days – so one month of normal days. Then – barring anything going wrong – you will be issued the ICP and can start to put up content.

And what does one do if one already has a registered .CN domain name? My advice is get your China business registration sorted out quick smart, or go and get the next best .COM or similar domain – though you’ll still need an ICP to host inside China!

UPDATE 22nd March 2010

Domains must now resolve to nothing.  So no A records allowed.  The blank page era has just been blotted.  Expect more changes to happen as the preparation phase takes place during the implementation phase.

Cross Posted to CANDIS BLOG

We do have one client though, who has a social media site based around book reviews, summaries and content quality. Well it is a recipe for problems. Today we had a govt take down notice because a book was summarised by some Chinese speaking person somewhere in the world.

Now the content in question was not even on the site or servers – it was just a summary – smells a lot like a certain search engine that is having troubles recently in China (I am referring to google) even though the same search queries on BAIDU return the same results almost and the same … problems….but I digress.

In any case as is par the course for China – if you are going to provide media and content, then you know what the rules are. Like em or hate…you must obey them and basically make sure that if you have a lot of content that you moderate it. Please don’t confuse moderate with censorship. People can always find what they want and view what they want all around the world. But if you run a server in China – conform to the laws.

And if you want to serve the Chinese people, then you pretty much HAVE to host inside China….which means also that you must have a business in China….which means you are already obeying the laws in China…. why wouldn’t your online presence be any different.

I don’t want to sound like a China apologist – because I am not. But I hate the added stress and work because someone got sloppy and pushed their risk over onto me.

Humph!

http://drupal.org/node/357715

- Grid Computing

- China wide redundant “Storage Fabric”

- China Optimised International and Local Routing

- Geographic IP and IDC Failover

- More thin client goodness

BTW – never use Windows Server 2008. It sucks. It has “VISTA” written all over it….literally in all the help files and what not. It is a true dog of an OS…but the Terminal Services support is WAAAAAY better. Not that W2K3 was bad – this is just even more fluid – especially under virtualisation.

Will update soon.

Well it is not until you actually start to UNDERSTAND it – that you get to appreciate it. All too often the pundits of tech in society (as nice as they are), like twit.tv, cnet.com and zdnet.com – have a great habit of talking about tech and service – but never having actually done it themselves or can appreciate it at a deeper and more sophisticated level.

Case in point, is the fact that many people go on about “I want a real app, not some web app”. Or, I need to have connectivity while I am in a cross country flight, etc. To me it just seems that some people can’t get with the times from either a tech standpoint, or even from another perspective, say costs.

I am using software as a service to highlight the missed nuances of many a lay person or tech pundits analysis of such situations and trends.

I look at all the effort that goes into running email infrastructure, the costs of running servers and IDC’s. And it is quite common now for people to expect email to work like a light switch (major oversimplification) or to be a single small cost in the case of hosting.

Why is it, that software then and other technologies like thin clients, that seek to change the usage and costing of hardware and software from a disparate and ad-hoc, per person/cpu based exercise to a usage and deployment style that is more akin to hosting and email services, that one encounters so much resistance?

Resistance to change from clients and resistance to change from vendors. Why do people still see computing and laptops and applications as objects that can be traded and bought and sold like shells and clams?

I know I asked the question, but I can’t answer it. I simply can’t understand it.

Where we must give some props to in this regard is Amazon, Google and Microsoft, for somewhat seeing the light and being industry leaders. Amazon, offer pay as you need it grid computing power, Google offers online applications and now Microsoft is also offering online applications.

To be fair though, Microsoft has had licensing schemes in place like the SPLA model for a while. Allowing people to just pay a monthly fee to use all the software that they need. Never having to worry about upgrade costs or “upgrade insurance” as some snake oil salesmen like to push.

Allow me to further expand my thoughts on this, with respect to some recent dealings of mine and in China and some questions that I can answer! I will do this as a little Q+A session with myself.

Richard The Interviewer:

“Richard, why would you not want to own your own software and hardware and put it inside your office or in your lap on a laptop?”

Me:

“Thank you for the thoughtful question Richard, allow me to answer on the issue of hardware first. You see Richard, the problem with hardware is that, well it is expensive! And tis fact is made even worse by the fact that it is outdated and superseded so quickly.

Richard The Interviewer:

“But surely even if something loses it’s value it is still useful? I have seen many an old car drive around leaving a trail of rust behind it?”

Me:

“Yes, to a point. The problem is that software gets more complicated and our needs get more demanding as we become more used to and comfortable with our technology. A better way to look at it, is like hand-me-down clothes. Servers and infrastructure to a degree can just be demoted down the line with new beasts coming in at the top for critical systems and then older ones being eventually upgraded – or converted into a VPS as they come up for expiry.”

“But to answer your question more accurately, it has to do with technologies like Virtualisation, economies of scale, operational costs and idle capacity. If you look at your infrastructure like a time share holiday house or corporate jet, it suddenly becomes a lot easier to handle, both on cash flow and on your exposure to depreciation, so there really becomes very little in the way of real world value propositions to hold on to that old junk – this is even without going into the issue of power usage!”

“For many years people liked to finance their equipment. That is a great way to do it, as it lets you amortise the costs over the actual useful life of the system and then throw it out and get a new one. That way, you can look at the leasing fee as a perpetual subscription model. Just make a payment every month and the system is always online, up to date and fault free.”

“Plus it does wonders for your tax exposure when you claim all those leasing payments back against your tax. Much better than doing it the Chinese way with a 5 year depreciation curve!”

Richard The Interviewer:

“OK, I get it. Why own it and pay for the depreciation of it – that will take longer to remove a residual book value than it does to remove any operational value. And just pay one fee once, per month for life and always have the tech taken care of.”

“But it seems that if I do this, I am still having to manage the equipment and run an IT dept, even though I am leasing the hardware?”

Me:

“Yes, so why manage it then if you have already outsourced the ownership to your financing company? That is the strange thing that happens. It is like financing a truck and then paying for a driver. Why not just use DHL or FedEx instead. They offer economies of scale and are better at it.”

Richard The Interviewer:

“But I need control!!! I am worried about privacy!”

Me:

“Control = expense. Digital technology and servers = no privacy. Your needs to control and keep data secure are very valid. But to be truthful, simply having equipment on site or in your control does not make it any more secure or better. These are separate issues. And companies never really think about them any further than just that. I have it, so it is safe. I can touch it, so it is safe. This is really an illusion of security and control.”

Richard The Interviewer:

“Ok, so what about software then? Software as a service….?”

Me:

“Well software is no different. Take all that I mentioned above and change hardware to software. The one exception being that software in most countries does not have any depreciation allowance. In China for example, software is always booked in as a single transaction once into your overhead. That is the tax law. Even if the software cost 1,000,000,000,000 USD and you intend to use it over 20 years, you can’t book the expense in and match it over many accounting periods. So you are much better off in China at least operationally by renting or using an ASP service.”

Richard The Interviewer:

“So in a single paragraph, what is your main issue then with these old dinosaurs who want to own, control, touch and carry their technology around with them all the time?”

Me:

“Do I have to be polite? Well, I would simply say, “What are you thinking?!” Get rid of the big software investments and yearly upgrades and all that hassle. Just hand over a list of needs to an ASP, get a copy of their SLA (service level agreement) and be done with it. Problem solved – for life!”

“As to hardware, well, the beauty of an ASP, is that you don’t need any!”

Richard The Interviewer:

“Well thanks for that Richard. I think I will go and study the intricacies of ASP’s, ISP’s, ESP’s, the tax code, our cash flow, the tech road maps of the various companies out there some more. And work out just why it is, that I feel so compelled to continue handing over cash by the fist full to continue in this rat race of upgrade cycles, repair windows and deployment plans.”

“IT that is like watching a TV or renting a movie online…..hmm, that would be useful and stress free…..seems tomorrow has been here for a few years already!”

Me:

“Maybe next time we can touch on issues like data security, data retention, corporate governance compliance, audit trails, business continuity, capacity budgeting, staff retention, staff acquisition, staff training, staff vetting, staff knowledge pooling and on demand resources?”

Richard The Interviewer:

“That would be good. Is that a whole load of hassle too?”

Me:

“Well it used to be, but it is all par for the course when dealing with a good ASP, ESP, ISP or any xSP really. Even more value!”

I hate to bitch. And I hate to appear as if I am drumming up business or trying to screw other businesses. I am just trying to call it as I see it as a public service.

We get maybe 3-4 clients a week transfer to us from a very prominent and large Chinese domain registrar and hoster, I won’t say the name. But your guess is probably correct.

Apart from the issues mentioned above, the latest fiasco involves again transferring domain names AND DNS.

Issue One:

Anyone who has worked in IT for a while and manages IT, or at least has half a brain, can understand the confusion that a lower case “L” and the numeral “One” and and upper case “I” can cause. l1I in some fonts.

I have had a domain transfer request reject THREE TIMES because no matter how clear I write down the authorisation codes, that are issued by said registrar – when they eventually get the transfer request from my new registrar they reject the very same code! :-\

C’mon already, think people! Or at the very least set your random number generator to exclude those characters. God knows when I get a password failure, I take 2 mins and check for things like the above or even a six “6″ and a lower case B “b” too, before I go back whinging that the code “didn’t work…”

It reminds me of people that when you call a wrong number, just say, sorry wrong number and hang up! Thus requiring me to call them back AGAIN to see if it was indeed the wrong number (as given to me) or if I just dialled it wrong. How hard is it to get the logic of checking what number the wrong caller was trying to call before hanging up??

Issue Two:

DNS server re-delegation. I won’t go into the details here. If you are in the business, you will get the subtleties of it. The dance that involves old and new mail servers, old and new DNS servers, propagation times, TTL values, client usage, abandoned/orphaned email, etc..

Point is, you need to plan it right.

We do.

So after the usual 500 pages of chopped and stamped paper that has gone through the usual 500 people of whom no one can take responsibility to make a decision, they (aforementioned very large and famous Chinese registrar and hoster) agree to a 9PM Friday night change.

So what do they do come 9PM Friday night?

Nothing.

Then to make it worse – hey at least we could have rescheduled. They just assumed it would be OK to go ahead and change it anyway on Monday morning.  More likely they would not want to admit to an error so as to not “lose face” and in the process lose much more face by turning things into a dog’s breakfast.   FUBAR!

The lesson to be learned? One of the very largest and most famous Chinese registrars and hosters, does not even know themselves the intricacies of their products and services and the importance of the correct and attendant management of them.

As I mentioned before, don’t risk you company’s DNS or domain name by letting them manage it. If someone can’t take DNS and registration seriously – then they are a serious risk to your business continuity.

This week, via the trickle down effect in the echelons of the China network, we got informed that a couple of our clients are not compliant and they need to become so, or remove their website.

We have had to remove some clients sites before because they did not have the correct ICP. They could not get one because they did not have the correct business licences for the type of work they did.

Anyway, while having an ICP is a delicate balance between business licences, residential permits, correct content, correctly licenced IDC and what not, it is a good thing to get done and host in China. To be official, legit and free to get on with doing business.

For the most part getting an ICP, or the existence of an ICP is not a barrier to hosting or a problem to comply with. It sounds a whole lot more draconian than it really is. However it is a hassle. For you China hands, “mafan si le”!