Start Here

There has been a bit of a maelstrom recently regarding the use of .CN and ICP certificates all on top of the existing confusion.

So, to try and set the record straight this article is a nice little walk through the pages of history and clearly state the facts of the matter from the inception until the present. And hopefully not only explaining what the situation is, but also how it came about and the anthropology of how we got to the situation that we have today.

The Land that Time Forgot – 2005

There was a time where life was sweet, chocolate milk flowed from every faucet, a reasonably priced pork knuckle had crispy skin and the internet in China was (comparatively) more functional.

That time was 2005.

However, that was also the year that ICP, or Internet Content Provider, licenses first appeared. At first, like many new Chinese regulations, it was all bluster and wind. Combined with a good healthy respect for the lack of organizational ability of the people that often make these broad and sudden declarations – people just sat tight. Taking the view that, when they get their act together, they will point out to us where we are in breach and we will patch things on a case by case basis.

Even if legislators had wanted to force through compliance straight away – one couldn’t – nothing was set up and the ink was still wet on the paper. And like most things in China – subject to heavy blotting still.

People ask: “Do you remember the time and place that you were when you heard of the assassination of JFK or the death of Princess Di?” For some strange reason I remember the time and place I was when I heard about ICP too.

I was standing in our office’s mini machine room, kicking a very stubborn raised floor tile back into position and wondering why it couldn’t have been made to fit and work properly by those who created it in the first place before it got to me…

They got their Act Together – December 2008

Finally, at the end of 2008 the ISPs and IDCs started receiving the dreaded phone calls. Various time limits were given from a few hours to a week to get sites offline or registered. Sometimes lists were provided, but for every 20 sites identified, another 100 would be on the same IP (virtual hosting) with no request for an ICP made. One could therefore assume that DNS servers were being harvested to get an insight into the habits and traffic directions of users to find unregistered web sites. So the more obscure you were, the better it seemed.

In most cases one simply filled in the online registration form and was done with it. The only people who could not do this were dodgy businesses without clear registration in China, or were operating outside of their legally registered area. When discovered, if an ICP license was unobtainable (for whatever reason), the site had to be moved out of China.

It was also around this time that various hosting companies in China started to demand an ICP up front before putting a site up. While this was not strictly needed, it is how they decided to do things. By not needed, I mean that to apply for an ICP is not that hard and you could do that concurrently with the initial loading and development of a site. Later it became a requirement for all to do things in order.

There were some awkward moments too. You see many servers have “Remote Management Cards” as well as “IPMI/BMC Management Interfaces” – these all are web based. Company’s also have Extranets and Intranets. Which are also? Yup, web based.

The point that I am making is that there are a lot of web pages out there that are not web sites and are not for the public and indeed are all password protected. This however did not make them immune from cease and desist and take down threats. At first it started as a bit of a joke – “gee how ignorant can they be?” A good laugh at the policy
enforcers’ expense. Then it became annoying. Finally with enough heated phone calls and pleaded explanations via intermediaries, the enforcers gained the knowledge that these sorts of “Web Pages” are out there and can’t be “registered” or have an ICP code embedded in them.

The Shit Hits the Fan – mid-2009

While things progressed at a steady pace – downwards – in more ways than one – the ICP enforcement game started to become more prominent. Like clockwork, every FRIDAY afternoon, at 5.30PM, a call would come down the line – “You have 1 hour to remove the offending content or all of your IP’s and routes will be cut.”

I think it is important to also explain what “remove content” means, because in its nascent use in 2009 – we can see the genesis of the procedures that are causing confusion today.

Remove content means just that – remove ALL the content. It doesn’t matter WHAT the content is – EVERYTHING must be gone.

So…….? I hear you thinking.

Well simply using a control panel to “disable” a site is not enough. Because there would still be “Content” visible. “Site Disabled for Administrative Purposes” is just as infringing as anything else.

This then led to the next step – OK – remove the Virtual Host declarations. This also didn’t work. Because in the
absence of any defined VHOST the HTTPD daemon will serve the first declared VHOST on the server – which unto itself may be perfectly legit as a site.

However this is still “Content” and it has to be removed.

So one then removed all the files from these offending sites, leaving the VHOST declarations intact. This of course then causes a series 500 error page upon loading. And while this isn’t a page that exists or could be edited and is in fact generated automatically… as an informational error message that we find all over tech land (..PC Load Letter..?! )…

…this is still “Content” and therefore it has to be removed too.

The answer then was to simply place an empty index.html file into the virtual host root. Upon loading the domain a blank page is presented.

This is the accepted definition of “Remove Content”. So take note and don’t ask why? Or why not? Just accept it and get on with things. You’ll live longer.

So what is an ICP and who really needs one?

We have now laid out the history of the inception of the ICP and how it was deployed and some of the lifecycle of how it has been enforced and managed. Before we get to the current stage of this little grub’s metamorphous into the beautiful butterfly that we have today in early 2010, lets indulge in one last bit of confusion surrounding the ICP.

Who needs one?

Before we go into this section – please note that the details contained within are purely for historic value. They DO NOT REPRESENT CURRENT STATE OF AFFAIRS, and more importantly how the law is enforced is still changing. What it will do, is like the previous discussions give an understanding to what has happened in the past and how that has helped shape the situation that we have today, both for better and for worse…

Originally, the rules as laid out and explained by the people that you apply for an ICP with, was that an ICP is needed for all domains that list a Chinese address in the WHOIS, are located on a China assigned IP and are owned/operated by a Chinese business entity. The ICP for these types was free in principle. For eCommerce and blog/BBS sites there were additional requirements as well as a quite high “application security fee” for eCommerce sites – refundable after the application process was completed (minus a few charges here and there).

A local business actually grew out of this to lend companies money for the application process – against a fee of course.

So foreign WHOIS, foreign IP, foreign company and individuals, were all (read: not enforced, though not necessarily legally) exempted. This situation, changed for the worse and caused no end of confusion when people who had foreign companies and/or foreign WHOIS records over night were being told by the enforcers (the people on the other end of the line that tell you have 1 hour before all IP’s are cut) to get ICP’s for all sites on a China assigned IP regardless of anything else. Of course this made it next to impossible for some companies to acquire an ICP, especially since you would need a Chinese Business License to apply for one.

The internet by it’s nature is borderless and many organizations that operate in China have a web site here by mandate of the superior local speed and the crappy international speed. Individually owned and operated sites were still in the clear though.

So what is one to do if you are a foreign entity and don’t have a local entity, be it a Representative Office, JV, WFOE or other establishment and you want an ICP? How the law was enforced at the time would lead you to conclude that you didn’t need an ICP. Indeed if you tried to apply for one with the ICP application office, they’d also tell you that you didn’t need one.

However let’s not be forgetting that there are those people on the end of the phone line at 5.30PM on a Friday…….so what is one to do? Well smart money is on listening to those with the biggest stick and actionable power.

This led to a somewhat dodgy and messy situation of foreign companies or web sites then trying to proxy or slipstream into an ICP via a local entity. The most common was the eager and “long term not in your best interests but only ours” local IDC/ISP’s that would, for a fee, let you use their ICP.

The ramifications of this ranged from dubious to outright illegal in some cases. Many of these ISP’s/IDC’s were basically taking quick profit from these hapless foreigners (oh how many a hapless foreigner, be it inside or outside China manages to get caught with their pants down around their ankles) who would eventually be found and taken down anyway after paying up their yearlong (non-cancelable of course) hosting contract.

A crackdown finally happened and hundreds of unregistered and dodgy data centers and ISP’s were shut down late last year. Many a foreign company who was lured in this way though, got their stuff online, did the big launch and started to spend money of advertising, only to be shut down one day out of the blue, usually finding out because their clients complain, not because their local IDC/ISP knew about or could bother to inform prior to taking action.

Back to our Time Line – it is late 2009

So, by the very end of 2009, for better and worse the internet in China is cleaned up a lot. The dodgy sites are gone, the dodgy data centers and ISP’s offline. Everyone is obeying, respecting and following the law and requirements of the ICP.

While it remains a problem with technologies fairly new to China, like geographic load balancing, private networks, IP aliasing, storage and cloud fabrics that further blur the line of “Server” and “IP” to be something far more ethereal and harder to pin down, in general the rule of thumb is – do what you need to do to make the tech work. But if content is being served to the public, from a China assigned IP, it must have an ICP, registered by the owners legal Chinese entity.

Thus the ICP issue was finally settled and the rules and regulations crystal clear at the end of 2009 – or so one would think. Though one never got a clearer or more definitive statement of what was needed and who was required to have one. Pragmatically speaking though, the actions of the enforcers spoke loudly and clearly. And the legitimate industry players listened.

Post Script – Time of Domain Crack Down

So, now towards the end of Q1, 2010, things are almost settled – again. By settled, I don’t mean ideal, I mean, quantifiable. This is where the domain name saga starts. First off there was a ruling that no Chinese citizen is allowed to own ANY domain name. As a result domains that could not be verified as being owned by a business in all TLD’s were being excised from the registries very quickly by local registrars.

Now whether or not this is in line with current ICANN rules or other TLD rules – if you are a Chinese registrar and are being told that certain domain name sales are now illegal… what is one to do? Well smart money is on listening to those with the biggest stick and actionable power. Am I repeating myself?

Of course if one has a domain registered overseas in any domain but .CN – then you are safe – no one can take your domain away from you. If on the other hand you do have a .CN domain name, registered anywhere in the world – you now have a problem. The Chinese Academy of Science – who makes the rules for .CN domains on behalf of the Ministries – are in the process of enforcing their specific understanding of these rules, rules that now are seeming to settle in.

What has finally settled in the last few weeks (everyone now agrees on a common understanding of the intent and purpose of the law – at least it seems that way) is that basically all .CN domains have to be registered to a Chinese registered company and furthermore the business name and .CN domain names must have a proven similarity or relationship.

This is not all that new of an idea. auDA along with a lot of other TLD authorities saw the shambles that went on in the .COM space with domain squatters and trade mark disputes and wanted to avoid the same fate. China is merely bringing control of it’s own .CN in line with how other countries manage their TLD’s – at least in part.

This is a good move if you ask me, as many a domain name here is “taken” and put up with a nice for sale parking sign. Domain squatters and opportunists have run rampant in China with the comparatively cheap cost to register a domain name.

During this whole process of the facts settling in, hosting companies received emails from the registrars that are on the ball and were given key dates and pre filled in PDF’s of the legal forms for convenience. Clients that took the cheap and ill advised registrar route when it comes to business domain name registration (let alone .CN) got nothing. There was an uproar and the EU got involved and started to solicit for information (complaints) from businesses.

The key dates came and went and nothing happened.

So now what does one do if one wants to register a new .CN domain name?

REQUIREMENTS

Domain name applicants need to submit the formal paper based application material as well when making the online application to the registrar.

The application material includes the original application form with Chinese branch business seal, company business license (photocopy), and registrant ID (photocopy).

Individuals are not allowed to register.

China Branch: means international companies, enterprises, and organizations establish within
mainland China (PRC), a wholly owned or share controlling entity: including a branch, a subsidiary or a representative office which is having the same “name” as the parent entity. The local entity must register with the relevant Chinese authorities.

For example: IBM should use “IBM China” to register IBM.cn

Chinese employee: the person submitting the application on behalf of the registrant (the entity), must be a Chinese citizen and has a valid PRC personal ID.

Once all the above is done, the domain name must also be pointed at a China assigned IP and left content free (ie blank HTML page) for 20 working days – so one month of normal days. Then – barring anything going wrong – you will be issued the ICP and can start to put up content.

And what does one do if one already has a registered .CN domain name? My advice is get your China business registration sorted out quick smart, or go and get the next best .COM or similar domain – though you’ll still need an ICP to host inside China!

UPDATE 22nd March 2010

Domains must now resolve to nothing.  So no A records allowed.  The blank page era has just been blotted.  Expect more changes to happen as the preparation phase takes place during the implementation phase.

Cross Posted to CANDIS BLOG

Silicon Hutong

And the topic did strike a “Hey this is real man!!!” kind of chord with me. A cathartic resonance that shall never come from me with respect to the greater greenhouse effect crap while the infallible science is still not in. Nor came to me for any other “agenda” or “crusade of the world” that seemed to so very trouble all those students over in the Arts and Law faculties. That seemed to have so much time on their hands, relative to us poor saps in Engineering and Business school.

Politics aside, wastage for the sake of wastage though is not very good. And I have a rather poignant insider’s look at this issue that also manages (in my mind at least) to cross paths with this issue about China’s talent gap, or indeed this also about the talent gap.

What does resource wastage and power requirements have to do with a talent gap, or as they lament now in overly taxed and anti-entrepreneurial Australia – the “Brain Drain”?

Allow me to explain.

Servers cost a lot of money! Not only do you spend the 5-10K USD for the server and warranty, you then have colocation fees, cooling costs and power costs – this all adds up. And if you skimp on something like cooling, your equipment can age prematurely – especially bad in China given the tax depts allowed deprecation rates!

A good contact of mine who runs an IDC in New York city, mentioned how it is all about the power. Real estate? No worries, how much power will we need. So it seems pretty clear where things are going. Moore’s law, computing density increasing and an ever growing appetite for all things information.

With respect to the issues that David Wolf brought up, a major new trend that has been underway for the past 6 years or so and is trying to put a clamp on rising costs of infrastructure is Virtualisation. This technology allows for less resource wastage and consumables wastage by consolidating many lower end servers into fewer bigger more reliable and cost effective servers. Picking the low hanging fruit so to speak.

A further evolution of this that I mentioned about here is where the actual servers in a farm can be powered down too when not needed.

Now when it comes to electricity there are always two rates, the so called “Consumer Rate” and the “Industrial Rate”. And yes, data centre’s use the industrial rates, which while are relatively cheaper than consumer rates – are still not cheap in absolute terms. And if you go and use a nifty tool like this Dell capacity planner and then do some arithmatic, you can see what your costs will be. It is through this tool, that CANDIS deduced that paying the premium for higher efficiency power supplies and low voltage processors, actually did pay for themselves in outright power savings over a 2 year period. Even more if you factor in that less heat is better for stability and system longevity.

So where does the talent gap come into play? Well, in how many servers you need to do a job and knowing about virtualisation.

We had a client that was struggling to keep up and online with their 8 server cluster. They kept just purchasing new servers when things went bad and slowed down. Which seemed to happen all the time. Now I know this and they didn’t, is that an out of the box Linux system is not too tuned and the concurrent IP connection tracking vlaue in the kernel is a little conservative. So instead of them having 8 servers running at 20% capacity and still not managing the load on them as a cluster. Yours truly made a few changes and now 3 servers are doing the job, with better outward performance to the clusters users.

What made this situation all the more funny, is that in China, capital asset valuations are fair game for being compared to the salary of a team of workers to perform the same job (quality and reaction times not withstanding).

So, in summary, I will go out on a limb here and make a bold assertion.

“There are probably more servers than needed running in IDC’s across China, due to engineers not knowing how to admin them properly, not know how to architect systems properly and not knowing how to leverage new technology better.”

Anyway, I could go on. I just wanted to illustrate the nexus of those points and how they relate and intersect. Poorly deployed tech, poorly planned and architected tech and balooning operational costs and resource demands.

Seems that in the past you could be a lazy programmer and no one would notice due to the liberation of faster processors and free cycles. Or in China, buy a server is better than paying the for the expensive vendor support and optimisation.

However as one famous economist once mused…. “there is no such thing as a free lunch”

This place had official Olympic logo’s all over the place! And nice toilets – a rudimentary western yard stick when measuring quality in China.

This IDC has some big name sites in it. In front of our servers are some SOHU.com servers, Baidu was in the next room – past the hand print scanner – and sina.com was a few rows over too.

Great cooling, but still no sign of people getting “Hot Aisle/Cool Aisle” design. This IDC used overhead conduits instead of a raised floor. I guess it is all down to preference on that one. They do offer as standard two power circuits per rack, but only 16A per circuit. So forget about putting 60 blades in 6 chassis in these racks. One rack we saw already had an official warning on it about being careful when adding another server due to power use. Yet this rack only had 8 Dell PE2950′s in it?? Seems a bit wasteful on space to me.

Overall, very impressive. Apart from the hand scanners, the quality is the same as CANDIS’ current facilities. With some extras and minus some others. The good news is that most of what you would want is standard and then only a few missing things need be added. Where as we had to build a lot of stuff to get to the standard we wanted.

Oh well, in a city with as many cars and people as Beijing……location, location, location. We are in the CBD, in the embassy district. This place is out with the cows…… and no camera’s were allowed and I needed my passport and caused much fanfare being a “Foreigner” and all.

But in the end, I must say VERY PROFESSIONAL, as if anyone is doubting China’s commitment to out do Sydney (as evidenced by all my compatriots here doing a double take for Beijing) as the “greatest Olympics ever”.