Some people believe it was Canada, others think it was the US. Yesterday a big blackout affected close to 11 to 17 states of the eastern region of the United States. How do we stop this event? Should we rely on other powerplants? Should states have their own eleterical grid?

By the way: Thank God Our Lights Are back. They came back at 5:23 this morning. Which was 13 hours after it went out.

-Holt-

There is no way to stop it, only minimize it. We are better off being on the grid system as power plants trip off line more often than the grid system fails. If you are not on a grid, and your local plant fails, you lose electricity til the power plant gets back on line.
Being on a grid, you will usually still have electricity thanks to the supplies from other plants.

For maximum reliability, states/regions should have completely redundant connections to two or more sources of power either of which would be sufficient to handle 90th-percentile loads. It's the same philosophy as a high-availability computer system, where you'd have two machines connected over two separate paths to two controllers in a disk array which has multiple internal connections and redundant disks inside. In something as important as electrical power there should be no single point of failure (SPOF, pronounced just as it looks by people who spend a lot of time talking about these things) whose loss can make the whole system fail. You can even extend that by saying there should be no combination of N failures that can take the whole system down.

As I understand it, the problem that led to the current blackout was lack of sufficient "hot" connections to alternative providers. Power could have been "borrowed" from the mid-Atlantic sector, for example, except that the initial response had overloaded a bunch of breakers and such that had to be manually reset. That's an incorrect initial response, the result of a bad design for which someone should be fired. The system should have started drawing power from neighbors, through multiple circuits to avoid overload, much sooner.

I was watching Larry King last night & Former Sec. of Energy Bill Richardson was on and he said that the mass blackout was caused because of Antiquated Technology in the United States power grid.

Bill Richardson said that we have old power grids and technology that needs to be updated to meet the loads of the higher populations of today.


Also, there is a story that the Blackout started in Ohio

I guess most people here weren't affected (or it didn't make an impression on them...)

I got power back at 8am (16 hours after it went out).

It is funny that everyone is pointing the finger at everyone else.

I think the root cause was the failsafe systems in the nuclear power plants. 9 nuclear plants were taken offline due to instability in the power grid. That is a LOT of lost power, which overtaxed any conventional plants still on-line.

Whatever the initial cause, the massive blackout was due to the loss of those nuclear plants, so maybe the cause was TOO many safety precautions.

For the more technically inclined, I've posted an examination of the power outage from a network-design perspective on my own website. The main point is the same as I made above, so I won't repeat the whole thing here, but some folks might find it interesting.

Edited out unauthorized link.

That is what a grid system does. All the power plants in a region, and there are more than you might think, supply power to the grid. One or 2 dropping off line will not affect the grid, unless it is severely overloaded beforehand. That is more than double or triple redundancy.
Nuclear power plants are required to have greater safety constraints than non-nuclear, and that isn't going to change any time soon.
BTW, since it is AC electricity, all the plants have to be synchronized with respect to the phase of the frequency. Try to switch on to the grid out of phase and watch what happens!
The system did work, just not as well as hoped for. If it had failed completely the problem would have spread to many other states. But the region affected is densely populated so more people per square mile were affected.
And, just try to get permits and public support for new power plants and new transmission lines in the Northeast, where all the democrats are!
The fact that the problem wasn't worse is a testimony to the efforts of all the engineers out there who do their best to prevent these kinds of things from happening.

That would be true of a properly designed and operated grid/network, but the fact that it didn't work out that way shows that the grid was in fact not properly designed and/or operated.



For redundancy to do any good, it has to extend all through the system. You can have all the power plants you want, but if they all converge on one possible point of failure between the provider and the consumer then what you have is a non-redundant system.



And those factoids are relevant because...?



That's not necessarily true if the grids in those other states were properly designed and had sufficient internal capacity. It's ridiculous to say that "the system worked" just because the failure didn't cause something even worse to happen. Might as well say that the system on the Exxon Valdez worked because it "only" spilled half of its oil and didn't do so near a heavily populated coastline. The system failed. Large areas that did not need to be blacked out, that would not have been blacked out if the system had responded to the supply outage properly, were blacked out.



That's completely gratuitous Democrat-bashing, don't you think? There are more than a few Republicans and Libertarians and independents living in those same states. Can you prove that the problem was lack of facilities rather than improper use of those facilities, and that the lack is in any way connected to voter demographics?



There are plenty of skilled and hard-working engineers out there in the trenches doing what they can to improve the situation, no doubt. That doesn't change the fact that the people making system-level decisions screwed up.

I might not be an electrical engineer, but I know more than a little about networks in a different context. You've shown nothing to indicate that power is so different from data that the same principles of redundancy and routing around failures don't apply.

If you aren't an electrical engineer, what are you? Certainly not well versed in the generation and distribution of electricity, that is for sure. Networks in the computer world are not the same as networks in electrical grids. Entirely different purposes, one is a type of communication, the other is power distribution. The two are not even close enough to be apples and oranges.
And in smaller outage situations, the controllers/operators can route around problem areas. This incident was much too large for instant fault correction.
Rather than take offense when someone challenges your position, try learning something about the technology being discussed before posting your ideas on how to make it better. It is all too easy to challenge the design of something you obviously know nothing about. Jane Fonda was good at that.

Amlord made a statement about 9 nuclear plants shutting down, suggesting that perhaps they might have too many saftey systems. The Three Mile Island incident is the driving force for the many saftey systems and we are stuck with that.

BTW, I have been involved in power generation (nuclear) for a very long time, and altho not directly involved in the distribution side, I do know quite a bit about it.

I was watching Larry King last night as well and Senator Hillary Clinton really went after President Bush *which didn't bother me * She said that this admin. isn't doing enough for our power. She also stated that this President isn't doing everything he can to prevent this in the future. I also think it's funny that they point the fingers at everyone else so they don't have to deal with it. I think everyone should just point their own finger at theirselves because this is POWER we're dealing with. It's not like a huge travisty however, it's a huge inconvience. To point a finger at everyone else is just a cop out, you just deal and move on. I agree with Senator Clinton and the governors on the show. Let's just get this behind us and try to get some sort of system to prevent this in the future *instead of wasting all our time, money, and men/women on Saddam*

OK, then how about you be constructive and explain why basic principles of redundancy and routing around failure don't apply to power grids. As far as I know, it doesn't matter whether you're routing electrons or water or apples or oranges; routing is routing, graph theory is graph theory, network flow diagrams are network flow diagrams. To put it bluntly, I don't believe you that they are that different, nor should I without proof. Instead of offering empty insults and appeals to authority, why don't you explain what makes this case so special? It's not common knowledge; I think people have a right to an explanation before they just take your word for it.

I agree with Platypus, Bill. If we were all expected to be experts before posting, we wouldn't have much of a forum. Please help us to understand this instead of being insulting. We're willing to learn

why don't you explain what makes this case so special? It's not common knowledge; I think people have a right to an explanation before they just take your word for it.
[QUOTE]

Do you think that someone who has spent the better part of 2 years learning how a Nuclear Power Plant works, and how to operate it safely, can transfer even the basics of that knowledge to untrained persons in a chat forum?

Grid technology is just as complicated. Likewise network technology.
It is not just an ideal situation for the safety and backup systems to work, they DO work most of the time, a lot more than we know. But SOMETIMES events occur in such a way that the system does not work. That is when it becomes a special case. The cause could be old equipment, operator error, a surge on the system due to lightning, or a dip due to a major fault that was not foreseen by the designers (truck driving into a transformer), it could be a lot of things. Keep watching the news and you may find out.

You are right that it is not common knowledge, nothing highly technical is, not even data networks. And you do not have a right to an explanation, if you won't be able to understand it.

BTW, I have SOME experience in basic data networks, tho it was a long time ago in a Navy far, far away. It was the late 70's when only the military was messing with the kind of things that eventually became the internet. We had data coming in from Naval Facilities all over the Atlantic to Norfolk, VA, data that would tell us where Russian submarines were lurking, and I had to help keep it running.

Don't take my word for it, go to one of the Electric Utilities Distribution control rooms near you and ask them to explain why there isn't enough protection and proper design and redundancy. The next thing you will hear is, "Security, report to the control room". They don't have the time to teach you what took them years to learn either.

David Cook, from NERC has been urging for energy legislation for at least two years now:

http://energycommerce.house.gov/107/Hearin...387/Cook640.htm

I heard him on the radio saying that for two years now he's been saying that it's not a matter of "IF" but, "WHEN" a major blackout would happen.

I'll try and not politicize it, but the bills have not made it through congress because many supporters of alternative energy and conservation have shot the bills down. I can see why it wasn't on anyone's top list of priorities, the last blackout that even comes close to this one was in 1965. Maybe now they can see the effects of a blackout of that magnitude and sway some of the people that were on the fence.

I know we still don't have a definitive cause of the blackout, but if it has anything to do with what Cook has been stating for a while, they'll have to pay a little more attention to what he's been saying all this time.

While I agree that alternative energy resources would be better, we cannot just sit here and hope they implement these plans before we have a major blackout while our demand for energy grows exponentially.

Absolutely! We should be re-building/modernizing those parts of the grid that need it, and doing the conservation/alternatives at the same time.
Maybe now Congress will devote some real time and effort to our energy issues.

The Utility that I used to work for, until last October, just spent a lot of money over 5 years modernizing their communications, including Microwave Radio hops throughout the state, mobile communications, data networks, etc. The next and possibly last thing on their list is that part of communications related to grid/transmission line protection, relaying, and control. And much of that is already done since it will use some of the increased capacity of the Microwave and network systems already updated.
I know that other utilities have been doing at least the microwave links and the mobile, because the FCC forced them into it by taking some of their frequencies away and requiring them to move up in the various frequency bands. Our microwave system went from primarily 2 GigHz analog to 6 GigHz digital equipment. Our mobile radio system has been going from VHF/UHF to 800MHz trunking systems. And our data went from mostly wires to mostly fiber optics.

But this is in Arizona, anybody have first hand knowledge of what the eastern utilities are doing?

Do you think we need to know about details of fission reactions and cooling and whatnot to understand how the grid works? I don't. Perhaps you could at least explain the connection between the two, even if you don't want to explain the details on one side of that connection.



Yes, it's just as complicated, and it's separate. The grid is supposed to work no matter how the power is generated. The problem is in the grid, not the plant, so we need to know about the grid, not the plant. What's so special about the grid itself that makes it so different from any other kind of network? The only place where your domain-specific knowledge seems relevant is in why several nuclear plants shut down, but as far as the grid is concerned why doesn't matter.



Then again, that might not happen if they knew that I do have a decade of training and experience in designing very similar types of networks. They'd probably throw you out too, knowing that what you do within a plant has even less to do with how to design the grid. They'd probably throw any of us out, just on general principle, so maybe we should just ditch the ad hominem and get back to the facts.

That's certainly a better explanation than all the looneys I was hearing on the radio claiming it was terrorists.

What's next, we're going to be blaming bad storms and hurricanes on terrorists?

If we are talking about the GRID, and relevant facts, how did the Exxon Valdez get involved?

The GRID is not seperate from generation if a power plant can trip part of the grid. Why? Power plants are loads on the Grid as well as suppliers to the grid, especially Nuclear. Off-site power, the power plant's load on the grid, is required as a safety measure, as well as on-site diesel generators to insure power to emergency core cooling features. We don't want another Three Mile Island fiasco, altho that one was caused primarily by operator error in the face of automated emergency systems doing what should be done. The operators turned off the emergency cooling system, thus the melt down. I don't remember if that affected the local grid or not.

An example of grid redundancy that works well is "circular" transmission lines around a city. Power can come to your nearby substation from either direction, so if there is a break at one point, it is easily isolated to within a short distance of the fault, and the rest of the circle can still have power, unless that one point happens to be where a single power plant attaches to it. But that is usually not the case. Here in the Phoenix area we have power from the Nuclear power plant east of town, some coal fired plants, several combined cycle natural gas powered turbine generators, and even some Hydro from Salt River Project. So we have multiple taps into the circle for supply, thus any one can drop off line and the loop will not be affected. BTW, only a portion of the nuclear plant's output goes to the loop, a great deal of it goes to Texas, New Mexico, and California.

Examles of unforeseen problems that you cannot design around are: The saboteurs who tried to topple transmission line towers with explosives back in the 80's. Or the poor guy who drove a crane next to incoming power lines at Unit 1 of Palo Verde Nuclear Generating Station, and set the truck brakes but not the "table" brake. As he stepped off the crane, the boom rotated and made contact with the power lines. The stabilizing legs were out, and there was one humongous, continuous arc from the foot of one of the legs to the ground. I believe the employee was fired on the spot. Saw that one myself, right outside the shop door, about 100 yards away. And for some reason there was no automatic trip, someone had to go manually isolate the section of line involved.

Situations of the magnitude that occurred yesterday are not common, so it would not be a good idea to hastily condemn designs that work well almost all the time. If you ask the utility operators, they can probably show you a long list of the times that the system worked perfectly, and only a few customers were ever aware that power has been lost somewhere.
Yes, it can be better, but so far no one has offered to step up and pay the expenses involved.

I listen to the radio pretty much all day and haven't heard anyone, besides a news reporter say anything about terrorists, and even then the statement was, "We cannot rule out that it may have been a terrorist attack". That doesn't sound very looney. I understand there's an investigation into the matter, which is probably a good idea. All areas should be looked into because if it was a terrorist attack and it happens again, there's going to be a whole lot of finger pointing.

I wonder why you bothered to enlighten us at all, then, Bill.

Platypus prefaced his comments by stating his background was related to network routing, and as far as I can tell never claimed to be an expert on power routing.

You have, however, asserted that you are more qualified than the rest of us without any real supporting proof of that.

I tend to agree with Platypus' opinion in that the system clearly failed. What is the system's goal? Adequate distribution of power to end users. Did the end users have adequate power? No.

The system failed, plain and simple.

How do we fix it? I guess I'm not qualified to speculate.

Mike

It was an example of how seriously flawed the "could have been worse, we'll pretend it's OK" argument is. It was addressed toward the philosophical side of the debate, not the technical side.



That's one example of exactly the kind of redundancy I discussed in the article on my site, which you apparently didn't read, and which obviously was not implemented correctly in the eastern grid. Would you say the Phoenix system worked if the nuke plant went down and the switchover to coal/NG/hydro sources wasn't executed properly? That's the situation most nearly parallel to what happened in the Northeast: one source went down, other sources were available, but those other sources could not be used because an unnecessary overload had cut off connectivity to them. How does losing a power source cause an overload elsewhere in the system unless someone messed up?



How can that not be designed around? Are you saying that there is a single point of failure, and that's OK? Or are you talking about multiple failures, strategically placed by someone who knew enough about the topology of the power grid to cause a complete split?



I'll condemn any design that "works well most of the time" when the consequences of not working are so severe and a design that works even more of the time is well known to anybody who took a one-semester class on network theory. A ship that stays afloat in flat water most of the time is not well designed.



If you're saying that there are known flaws in the current system, which remain unsolved for financial or political reasons, I don't think we disagree. It sounded like you were claiming there are technical reasons why we couldn't have a more robust system.

I think the power system worked as designed. The outages were caused by the shutdown of systems that experienced overload, just as the design called for.

Power failure clues point to Ohio

Using the term "failure" is relative. What was the system designed to do in this instance? Protect the equipment/grid or continue to supply power? The system was DESIGNED to shut down, which is what it did.

Now, should we change what we expect from the system? I think the answer is probably yes. Electricity in this day is more than just a convenience, it is a necessity.

For Mike,
Were you enlightened by my last post?
It is as clear as I can explain how the grids DO have redundancy, and that there is no such thing as perfect or total design that will cover all possible circumstances. Just expecting such a thing is evidence that you are not aware of the technical difficulties involved.

PlatyPus implied several times, without foundation, that the system is not designed properly, as though he knows? NO system is perfect, and most of the time the system, whether it is a power grid, or a computer network, takes care of problems without us even being aware of them. So one time it fails in such a way that we can see it and now it is due to poor design? Yes, it failed, but can you or any other here name one system that never fails?

You want my credentials? How about a resume', and will that be with or without an independent audit? I will compare mine to anyone here if necessary.
How do you know that Platypus has 10 years experience with networks? And how does that experience automatically transfer to grid technology?
BTW, nothing personal here, but I once worked with a guy who claimed to have 20 years experience in electronics repair, but his boss said that it was 2 years experience 10 times. (And he was being generous)


I am sorry if I appeared to be condescending. I am not an expert, but I have spent a lot of years in the power industry, in operating, maintenance, and technical support.

I tried to do a little informing here, and it was refused.
This is no longer a debate, it is a whizzing contest, and I will not participate in it any longer.

Not without foundation, Bill. I mentioned the concept of a single point of failure (SPOF) earlier. Are you familiar with the concept of high availability? A highly available system can be designed and implemented so there is no SPOF - i.e. it can survive any single failure without disruption of the service it provides to the rest of the world. It's not perfect, there might be performance degradation and it can still be taken down by multiple concurrent failures, there might even be "hiccups" as it adjusts, but no single failure can render it inoperative. That's the only standard I'm applying to the eastern power grid, and it's a standard I'm used to meeting in the systems I design so I don't think I'm being unfair.

Edited to add...

A visitor to my website hinted at a couple of possible explanations for how a power source failing might cause an overload elsewhere. One is that, if a power source dies due to a short, it can draw more power from the grid than its neighbors are equipped to provide. The other has to do with the need for power sources to be phase-synchronized; when a source goes offline (or, presumably, when it comes online) the change can cause its neighbors to fall out of sync. Either could in theory lead to a cascading failure, though I have to wonder a little. Is it really not possible to isolate things so that a short doesn't propagate, or to resynchronize sources that are out of phase? Maybe it's possible, but decreases operational efficiency and is therefore not done by deregulated profit-driven power companies. It would be interesting to hear what a real expert would say about such theories, but even if they're reasonable theories they seem like weak excuses for a failure of this magnitude.

SPOF is NOT a foreign concept to me. Neither is the concept of high availability. Both concepts are part of the driving force behind redundancy of controls on the grid and within power plants. Did I not mention that I worked in a Nuclear environment? A single unit at Palo Verde Nuclear generates $1,000,000 in revenue per day, and there are 3 units. I think high availability is an important issue there as well. BTW, back in the early 80's, I spent a year co-authoring a report done for the NRC on the issue of instrument failure at commercial nuclear power plants. The source material was Instrument and Control Equipment Failure reports that the NRC required all Nuclear plants to submit so they could study them and determine what might be done to improve saftey and reliability with respect to the operation of Nuclear Power Plants.
Just reading the reports was an eye opener. The number of ways that failures can occur is astounding. Maintenance personnel error was one of the big ones. I defy anyone to design any system that requires periodic maintenance/adjustment by humans that will be completely failure free. And the NRC requires periodic maintenance/adjustment.
How does that factor in to data networks? If you have people messing with your working system for no good reason other than some controlling governmental authority says so, your reliability might suffer a bit as well.

Your statement about a system that can be designed without SPOF may be valid when applied to the comparatively simple, environmentally protected networks you work on. But not the GRID. The SMALL PORTION of the eastern grid that FAILED has hundreds of power plants and half a million miles of transmission lines, all of which are exposed to the elements, not neatly stashed away inside conduit or air conditioned enclosures. Got those numbers, or"factoids", off the news last night. Is the perspective changing yet?

YES, I followed your link to your web site long ago, before posting the first time, and it says nothing relevant until someone who knows more about this topic enters the fray. That is the visitor to your web site, as follows.




I already mentioned phase synchronizing, and the fact that power sources are also loads, but you must have missed those parts in your haste to rebut.

Do the math if you want to understand the phase issue. At 60HZ, and assuming a 20 degree phase error, you don't have much time to correct an error if someone tries to attach their plant to the grid out of phase. Nothing is fast enough to overcome operator errors of this type. And a "hiccup" in your kind of network has circuits designed to correct it, a hiccup in a power line carrying many thousands of kilowatts is a bit more than a timing glitch, and it isn't as easy to correct it on the fly. My guess is, it is probably technically impossible to get 100% total reliability, but surely economically impossible. Some things you can throw all the money and talent you have at it, and the problems remain unsolved.
We will never have a BLIP FREE world, no matter how much we wish it.

Yes, it WOULD be nice if a REAL expert that YOU KNOW and RESPECT was to enter the argument. Perhaps Jaime knows one?

I said that I was out of this, but your last post points out your faults in not reading others input while you accuse others of not reading your links.

It could be worse, we could have Jane Fonda as our technical consultant.

And did they all fail independently, at the same time? Was there a hurricane or some other large-scale event that caused all of these simultaneous failures but somehow failed to make the news? Or was a single failure allowed to cascade through the system?



You mentioned phase synchronization, but not how a failure of one power source can cause others to lose sync, and not how that can cascade. Still haven't. I still don't believe it's necessary for things to be that way.



That's about a millisecond, and a millisecond is a very long time for a modern control system. It's hundreds of thousands of instructions for the kind of microprocessor typically used in embedded real-time applications. Other devices - such as this one designed for high-power switching - can respond in five microseconds.

I know there's a possibility that there's a technical reason why a single failure cannot be prevented from cascading across a million-square-mile region. I find it highly unlikely, but I can be persuaded if someone with more relevant knowledge would only try presenting the facts.

Bill - PLEASE stop being SO condescending. It's apparent you know know A LOT on this issue, probably more than anyone on the forum. No one here is saying otherwise. However, we still have the right to question things even if it appears ignorant to you. This is a way we can educate ourselves on this. In order for this debate to continue, we need to conduct ourselves in a civil manner (and this last sentence is relevant to everyone).

Here's another tidbit, this time from a Boston Globe article about the possible origin of the blackout.

1. A 345kV power line south of Cleveland tripped at 3:06pm.
2. Two more lines east and south of the first tripped at 3:32pm and 3:41pm respectively.
3. Two more lines to the southeast (a little over half-way to Pittsburgh) tripped at 3:46pm and 4:06pm
4. Power swings were noted throughout eastern US and Canada

Is anybody going to stand up and say it's impossible to design a system that could respond to the original outage in less than an hour and prevent more widespread blackouts? I think that's ridiculous, and I'm not alone. The president of the North American Electricity Reliability Council is quoted in the article as being upset and embarrassed that "I failed at my job" of preventing such an outage. Bill Richardson, former Secretary of Energy, is quoted in an editorial as saying that we have a "third-world electricity grid". Charles Curtis, a former DoE undersecretary is quoted in the Houston Chronicle as saying "it's apparent that the defensive mechanisms did not work in this case". Some smart folks from Siemens seem to believe that better technology could produce a more robust grid. The folks at the Electric Power Research Institute and the Consortium for Electric Reliability Technology Solutions seem to have some ideas too. One of those ideas is DC transmission, which pretty much makes the phase problem go away.

The fact is that the technology - both hardware and software - for a more reliable power grid does exist, and a bunch of experts have been talking about the need for years. The only real obstacles are political and financial, not technical.

And just what is Platypus doing, Jaime? Do you even read his posts, or is he automatically your best buddy. And I was TRYING to educate, without being condescending, but neither of you will listen. If you want to be fair here, find an independent person to audit this thread.



Patently stupid, the inherent losses of a DC circuit over short distances will cause you to need a power plant every few miles. Transformers don't work on DC, and neither does 99% of all the appliances in the world. The DC vs AC issue was settled a long time ago when Edison and Tesla had their fight over this issue. Tesla was a genius, but Edison was right. Look it up if you want FACTS.

Boston Globe article, they most likely tripped because they were unable to supply the LOAD that was dropped by the previous plants that tripped! Of course they didn't fail independently, they failed when the load expected exceeded the capacity available from that plant. How difficult is this to understand? This isn't low current data, it is very high current POWER. Look up OHM's Law and the associated power formulas. PPus should know them, does he understand them?
And I don't have the patience to try to explain how phase synchronization works, especially to someone who doesn't really want to know. But it has to do with making sure that the power plant coming on line is peaking on the AC cycle in sync with the grid. Look it up if you want more.
One person says "third world grid" and you buy it? How about the guy who said our grid, altho not perfect, is the model for the rest of the world. Miss that one?
Is Bill Richardson an engineer? NO, HE IS NOT, he is a political appointee with a law degree. If he tells you your network design is flawed, will you listen to him, now that you know that he has to clue technically speaking?
Siemens folks are right, not that they don't have an incentive to say such a thing. There is always better technology around the corner. Do you have one to those $10,000 plasma TV screens hanging on your wall yet? You said it yourself, it is too expensive.

Milliseconds is a LONG time for a mechanical relay that lives outside and is subject to corrosion, and there are no solid state relays that can handle extremely high current steady state, much less during the process of interruption or initiation of power. Think SURGE current, and then look it up if you REALLY want to know.
The one mentioned in your link is NEW, and not widely available yet. It will be a great thing when implemented though. BTW, Field Effect devices don't like to be near lightning strikes, or even static electricity, if I remember correctly.

I'll do my best, but I think that a bit of finger-pointing is in order. It strikes me that the most salient contribution to the debate so far has come from goamerica:
I would also accept, based on my own reading, that Platypus is correct in stating that "the technology... for a more reliable power grid does exist" and that "experts have been talking about the need for years" - though those experts may not include Bill55AZ.

If that is the case - and I am open to correction by the astronomically better-informed here - it seems to me that the answer to Juber's question - How do we stop this event (or, presumably, prevent similar events from occurring in the future)? - lies with our legislature and our administration. As with the interstate highway system, what Richardson described as our "third world grid" is, well, a national problem - and is, perhaps, one of the few issues which absolutely demands a federal solution.

Just over two years ago, Rep. Sam Farr of California offered an amendment to the FY02 Energy and Water appropriations bill that would have provided funding for electric power grid improvements. Congressional Republicans refused to allow the amendment on the bill.

The amendment would have authorized $350 million to fund direct loans and loan guarantees to improve electric power transmission systems throughout the US. From Rep. Farr in the Congressional record:
The amendment was designed to deal with all of the major power grid vulnerabilities - and was voted down three times: on straight party lines in the House Appropriations Committee, on straight party lines the House Rules Committee, and on straight party lines on the floor of the full House [Roll Call Vote #169, 6/20/01].

From the Associated Press following the Appropriations Committee vote:
From a House press release of June 20, 2001:
Two years ago, supporters of the Farr Amendment cited studies by the Energy Department showing that the national grid was in desperate need of upgrades; the Bush Administration lobbied against it and the Republicans voted it down. Two days ago, President Bush said "We'll have time to look at it and determine whether or not our grid needs to be modernized. I happen to think it does, and have said so all along." It was yet another shameless lie. I wouldn't be surprised if we saw the Farr Amendment revived - though, this time, as the DeLay Amendment (with Republicans accusing Democrats of having previously obstructed similar legislation and the "liberal media" obediently swallowing it).

If we are to prevent a similar "blip" or "hiccup" (which cost New York City alone $750 million in lost revenue, $40 million in lost tax revenue, and $10 million in overtime pay - and three American lives) from occurring in the future, we need a non-partisan Congress with some responsible Republicans and a few Democrats with a bit of backbone, we need the Enron-friendly National Energy Plan thrown out - and we need the Bush junta out of the White House.

Either that or Juber, Amlord, and everyone else in the greater Cleveland area had better stop using their AC - and PCs.

I don't want to further stir the flames in this debate, but:

Bill Richardson is a Clinton political appointee, a former lawyer, and is in no way an expert in electrical engineering. I have already heard others ask, on the news, why didn't he push for modernization when he was in charge? Perhaps he did, and he ran into the political mess that Wertz discusses.
Richardson's statement about a "3rd world power grid" has already been dismisssed by others as a political attack on the current administration. When the politicians run out of wind, then maybe the engineers will have their say, and we will get some better info.
I never said that new technology isn't available, I tried to say that it is not as simple to effect as a data network.
I am not an expert, but the training and experience I have in physics, nuclear power plant operations, electrical circuits within the power plants (mini-grids) etc. puts me in a good position to know that nothing as huge as this is as simple as some want to say it is. There is an AP story in the paper this morning that says the Nuclear plants on the affected grids are just now getting back on line. That comes as no surprise to me, as I have been there and done that, and Commercial Nuke plants cannot be turned back on like a switch. Part of the problem is due to the laws of nuclear physics, part is due to regulatory issues, or the laws of man.
When the dust settles, you can bet that there will be several books written by opportunist hacks and pundits from both sides of the political fence blaming everyone but their own party, and neither side will give the utilities and the engineers the credit due for having a system that works as well as it does 99.999% of the time.
And it is likely that once everyone has their power back, congress will quickly put this issue on the back burner AGAIN. I hope not, but time will tell.
And what will WE, the consuming public do?
We will continue using as much as we are willing to pay for, not thinking one minute about the consequences until they bite us in the butt. I wonder what it will take before conservation of energy in our homes becomes an issue we are willing to address.
Many of us have unrealistic expectations of the capabilities of the technical/scientific world. We tend to think that the impossible is possible, if only the giant, invisible conspiracies would just allow it to happen. The only conspiracy here is our unwillingness to pay the price required to do what is achievable, and to stop whining about not having perfection in all those things that make our lives comfortable.

The people studying the problem have identified....drum roll please....

HUMAN ERROR

as the root cause.

Human Error Likely Cause of Blackout, Timeline Says

Amlord's post of Human Error as the cause of the Blackout of 2003 reminds me of a documentry of the human error that caused the Chernoybl incident.

Now i know that the Blackout is not as serious as the Chernoybl incident, but we need to ensure that human error doesn't cause another blackout.