WAN Replicated Service Availability?

[Rick Reineman]

I've been grinding this problem around in my head for awhile then it 
occurred to me that this may be a good forum to pose the question.

How to deploy a series of replicated services (web, ldap and oracle) 
across a wide geographical area with the goal being survivability in the 
event of a significant Internet outage?

In other words if (for example) the East coast of the US looses Internet 
connectivity due to some catastrophe, the rest of the country should 
still have access to (at least) one of the replicated servers that are 
still available.

With round robin DNS we can hand out IP's to servers in turn but there 
is no validation that a service is available.

A load balancer can do some service availability evaluation but we have 
the possibility of a load balancer in the geographically affected area.

A router can determine availability of a path or paths to an IP, but not 
a service (as far as I know).  It wouldn't take long for a complex 
routing issue to be outside of my current ability to understand though.

I know people do replicated services all the time but I get the idea 
they are not planning for the same sort of network unavailability that I 
want to.  Most replicated services I am aware of are in one co-lo site 
with a load balancer, maybe highly available load balancers.

Any ideas or comments?

Thanks,
Rick
-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Rick Reineman                                         IT Systems Manager
Lawrence Livermore National Laboratory - NAI/Q/CAS    reineman1 at llnl.gov