SamYeager2016 wrote:

Because you're just a customer, not THE customer. The company carrying out the outsourcing is always going to do things that are in their financial interest but which may not necessarily be in the best interests of the company that outsourced the work e.g. less regular testing of disaster recovery, cut down QA work to speed implementation.

If outsourcing saves money something has to go, a number of non-union companies outsource, staff salaries are not always the biggest savings.
My company previously had an entire building dedicated to IT equipment and staff, imagine the amount of money saved by not having that building and its expenses.

Recall the days when IT Professionals sat in the boardrooms, they were replaced by accountants as finances were more important.
That's just business life

kalvado wrote:

I agree, anyone with a valid driver license should be allowed to fly 777, period!

Well if you want to go that far for an example..............this is a.net after all
However, it does make you wonder why airlines have training when they do not make a/c. One should simply get training from the OEM and be good to go.

xmyai wrote:

I guess you have never worked on complex IT systems? SOPs only go so far. There will be occasions when there isn't a SOP that covers what you are trying to do or if there is one, it doesn't work as desired. It takes expertise based on knowledge gained over a long period of time, ideally involving commissioning the systems in the first place, to fix such issues.

par13del wrote:

3. If the job is outsourced to a company that lacks competence, then those doing the outsourcing have issues themselves, and expecting them to have competent management to supervise the outsourced company is a bit much

This is one of the issues with outsourcing. It's not always possible to know how good the outsourcing company are when the contract is signed.

As for outsourcing in general, reading another forum today someone claimed that all the big UK banks, bar one, are now moving their core IT systems back in house because they are not happy with the quality of service provided by the out sourcing companies.

So the company doing the outsourcing is just using economics to select the partner...nothing is ever 100%, that what companies tell their customers and apparently they also apply that to themselves.

As for returning in house, I suspect in most cases they lost money which they have on paper to take to the shareholders to justify the return, just as they had to show the numbers to justify the outsource.

Sometimes, some things just don't work out, not always someone being good or bad.

par13del wrote:

xmyai wrote:

I guess you have never worked on complex IT systems? SOPs only go so far. There will be occasions when there isn't a SOP that covers what you are trying to do or if there is one, it doesn't work as desired. It takes expertise based on knowledge gained over a long period of time, ideally involving commissioning the systems in the first place, to fix such issues.

par13del wrote:

3. If the job is outsourced to a company that lacks competence, then those doing the outsourcing have issues themselves, and expecting them to have competent management to supervise the outsourced company is a bit much

This is one of the issues with outsourcing. It's not always possible to know how good the outsourcing company are when the contract is signed.

As for outsourcing in general, reading another forum today someone claimed that all the big UK banks, bar one, are now moving their core IT systems back in house because they are not happy with the quality of service provided by the out sourcing companies.

So the company doing the outsourcing is just using economics to select the partner...nothing is ever 100%, that what companies tell their customers and apparently they also apply that to themselves.

As for returning in house, I suspect in most cases they lost money which they have on paper to take to the shareholders to justify the return, just as they had to show the numbers to justify the outsource.

Sometimes, some things just don't work out, not always someone being good or bad.

Interesting how when it comes to money, you are black and white with taking it to shareholders and justifying the return. However when it comes to why outsourcing doesn't work in many cases, you are vague saying "some things just don't work out". Strange as there are good reasons. Pity that IT people aren't better at communicating our concerns and allow companies like BA to put the whole business in jeopardy when they out source IT systems that are critical to the core business.

par13del wrote:

If outsourcing saves money something has to go, a number of non-union companies outsource, staff salaries are not always the biggest savings.
My company previously had an entire building dedicated to IT equipment and staff, imagine the amount of money saved by not having that building and its expenses.

And now lets stop and think about the building comment.
My take is that there was a certain amount of IT hardware which is no longer there. Why? I can see a few options;
1. It is updated to something much newer and smaller, which can still do the same job
2. There is still same amount of hardware running elsewhere
3. No update, but amount of hardware underwent cost-cutting review.

first one is definitely possible, especially if shared facilities - e.g. Amazon datacenters - are involved. Then cost cutting may be real, but I doubt savings are as great as shareholders would love to see. Reliability should be really great, though, as things would be in the hands of top notch professionals. I would say this is a wise way to outsource, although sometimes misunderstanding will be still fairly expensive.
Second.. Most likely equipment is running in a cheaper building elsewhere - and at the end of the day you get what you pay for.
Third. Yes, and now you can use the savings to pay customers like BA will be doing.

Anyone from BA ops know if BA will be using spare 777's and / or 744's on Euro routes to clear passenger backlogs ?

Why couldn't airlines transport the entire system into the cloud to get the required redundancy?

Planeflyer wrote:

Why couldn't airlines transport the entire system into the cloud to get the required redundancy?

Not too long ago, the Amazon Cloud service was down for a full day. So much about redundancy.

Planeflyer wrote:

Why couldn't airlines transport the entire system into the cloud to get the required redundancy?

Because you wouldn't and besides, do you really want to hand control of IT systems that are critical to your business over to another party? By all means do it for non-critical systems but not ones that effect your core business.

The Times is reporting this weekend's BA IT failure chaos is going to cost them £150m +.
Wouldn't be surprised if IAG shares take a big hit on Tuesday.

I am an AWS client for 6 years and have never had one service outage. Many firms who process and store way more data than BA run their entire operations on AWS. Amazon being one of them.

manny wrote:

redroo wrote:

We had two of every application on different servers in different buildings at least 30 miles apart. All the applications were designed to fail over onto the backup within a minute or so. This whole setup was regularly tested using multiple scenarios, every couple of months... e.g. could you flip onto backup, stay on it, then flip back to production; what would happen if the network in primary location died; what would happen if primary was blown up by a terrorist. It cost a lot of money for this setup, but we knew it would cost a ton more if we couldn't operate for even a minute. There is no excuse for a system outage like this.

Its very poor decision making and planning to have your backup facility only 30 miles away. They need to be further apart. Like hundreds if not thousands of miles away because in case of a major storm or some unforeseen incident both centers might not be available.

That wasn't the only DR system. We could failover any of the major system to the worlds major financial centres without much of a fuss. Usually LON, NYC and a few places in ASIA.

My point is that most companies don't even do that because it's too expensive

xmyai wrote:

Interesting how when it comes to money, you are black and white with taking it to shareholders and justifying the return. However when it comes to why outsourcing doesn't work in many cases, you are vague saying "some things just don't work out". Strange as there are good reasons. Pity that IT people aren't better at communicating our concerns and allow companies like BA to put the whole business in jeopardy when they out source IT systems that are critical to the core business.

Well, what is the primary reason that management usually gives for outsourcing, cost savings and efficiency. Ever hear them saying they are trying to break unions, lower wages or because they are financial persons and have no clue about IT it is easier for them to manage a company versus an IT dept, those are some of your vague reasons. Outsourcing any sections of BA IT infrastructure was a major undertaking, what do you think they took to the board and shareholders for justification, which was required?

In the 70-80's IT / Technical personnel were in the boardrooms, they were ejected in favour of financial experts as a general trend due to costly projects and failures, that is still the case today. The communication point is valid, but it is just a new trend where engineers and or technical persons are now doing financial courses, in time if it continues it will allow them to communicate more effectively with the "suits" in the boardrooms.

redroo wrote:

manny wrote:

redroo wrote:

We had two of every application on different servers in different buildings at least 30 miles apart. All the applications were designed to fail over onto the backup within a minute or so. This whole setup was regularly tested using multiple scenarios, every couple of months... e.g. could you flip onto backup, stay on it, then flip back to production; what would happen if the network in primary location died; what would happen if primary was blown up by a terrorist. It cost a lot of money for this setup, but we knew it would cost a ton more if we couldn't operate for even a minute. There is no excuse for a system outage like this.

Its very poor decision making and planning to have your backup facility only 30 miles away. They need to be further apart. Like hundreds if not thousands of miles away because in case of a major storm or some unforeseen incident both centers might not be available.

That wasn't the only DR system. We could failover any of the major system to the worlds major financial centres without much of a fuss. Usually LON, NYC and a few places in ASIA.

My point is that most companies don't even do that because it's too expensive

It depends on the type of application, nature of business, how critical it is and in most situations Govt regulations that make the decision makers in any business to move towards making those investments. That is why we always need smart regulations that make sense.

redroo wrote:

My point is that most companies don't even do that because it's too expensive

Most companies have "redundancy," they just don't test it adequately because that is very hard. All the dependencies of modern systems make true reliability difficult.

par13del wrote:

In the 70-80's IT / Technical personnel were in the boardrooms, they were ejected in favour of financial experts as a general trend due to costly projects and failures, that is still the case today. The communication point is valid, but it is just a new trend where engineers and or technical persons are now doing financial courses, in time if it continues it will allow them to communicate more effectively with the "suits" in the boardrooms.

The suits are coming around. With breaches and failures affecting just about every industry in the past year or two, boardrooms are paying far more attention to IT. They're not appointing the CIO to the board just yet, but they meet with the head of IT regularly and they're more involved in big projects than they used to. The number of board meetings my team has attended has significantly increased across all industries. Board members may not understand everything, but they know they need to pay more attention.

The second part of your comment is spot on. We have to learn how to tune our presentations to focus more on financial outcomes, and less on technical details...

Planeflyer wrote:

I am an AWS client for 6 years and have never had one service outage. Many firms who process and store way more data than BA run their entire operations on AWS. Amazon being one of them.

Amazon had a very public, very large, very stupid (read oooppsss wrong key) outage, not too long ago. It may not have affected you, but it severely affected Netflix, among other websites.

No one will ever claim to have perfect redundancy because the people in charge know human errors happen, bad luck is a thing, suppliers are no more perfect than they are, and acts of God do occur.

par13del wrote:

In the 70-80's IT / Technical personnel were in the boardrooms, they were ejected in favour of financial experts as a general trend due to costly projects and failures, that is still the case today. The communication point is valid, but it is just a new trend where engineers and or technical persons are now doing financial courses, in time if it continues it will allow them to communicate more effectively with the "suits" in the boardrooms.

IMO removing IT people from the boardroom was a mistake given how critical IT is to so many businesses today. As for communication perhaps most IT people speak a different, more technical language from your average board members who comes from a finance background, so don't fit in too well especially when the board are so often dominated by finance types?

There's also the issue of how well IT people play the "politics" game. I am very geeky but I'm definitely no political animal. Many of the other technical people I've known are the same.

Doing finance courses should help technical people but they might also want to attend politics and drama classes too. Not sure if your typical geek will have the will to do that though! Maybe the finance types should be taking IT courses given how critical IT is to so many businesses today...

Interestingly the FANG (Facebook, Amazon, Netflix, Google - now Alphabet) companies which will have more IT representation on the board are doing exceedingly well!

Blue flyer,

As you correctly point out perfect redundancy is impossible.

I'm not advocating a service such a AWS for an airline but do wonder if it might be the better way to go.

After all who knows more about moder mission critical IT? AWS or BA?

Leave IT to those Fluent in it so BA can focus on only what they can do best: get passengers into and of LHR better than anyone else

The amount of redundancy is depended on the money you spend on it. In the case of BA it was shown that they spend to little on it. It seems that failure of power supply pulled the whole system down. So there was no second installation in a different location, mirroring the main data center, able to take over.
Usually after such a disaster people start thinking, if the cost reduction due to outsourcing and limited redundancy, was worse the price if some disaster strikes and you loose millions.

TC957 wrote:

Anyone from BA ops know if BA will be using spare 777's and / or 744's on Euro routes to clear passenger backlogs ?

i doubt they have any spare after all this!

mjoelnir wrote:

So there was no second installation in a different location, mirroring the main data center, able to take over.

Can't make that assumption. Wouldn't be the first time an incident at the primary site exposes a flaw in the failover process to a secondary site... Maybe it wasn't tested, maybe it couldn't be tested... Or maybe it was, and passed, and yet still failed when needed... All are possible at this point.

This fiasco raises several questions which, if I was a BA share holder, I would want answers to :

- why the IT systems fell over in this first place
- why were so many critical business functions affected
- why has it taken BA's IT staff or the out sourced staff so long to recover the systems

The real question is why doesn't BA management just outsource itself? I'm sure they could find a management consulting outfit could come in and do a better job for far less money. Oh, you think there's something important and unique about BA's management philosophy that can't be replicated by an outside firm? Well, the same can be said of IT. So much of what an airline does depends on not just the presence of IT, but the way it's applied. There's countless little decisions being made every day that shape what the customer sees and what the customer can and cannot do. That's what you lose when you outsource it all. And, for the record, I don't work in the airline IT industry.

SamYeager2016 wrote:

par13del wrote:

1. Usually out-sourcing companies have their own management, so why exactly does the company doing the outsourcing need competent management, you only need an accountant to verify the charges and ensure cost are being saved.Most time, the local management undermines the activities of the outsourced company.

Because you're just a customer, not THE customer. The company carrying out the outsourcing is always going to do things that are in their financial interest but which may not necessarily be in the best interests of the company that outsourced the work e.g. less regular testing of disaster recovery, cut down QA work to speed implementation. The company outsourcing needs people that understand the implications of proposed changes sometimes known as an "Intelligent Customer" function. Equally the work that is outsourced is unlikely to be static i.e. there will be enhancements to existing systems, new systems being developed that may be replacing all or part of existing systems and which will always be interfacing with existing systems.

Don't get me wrong, I'm not against outsourcing as such but it involves a lot of work initially as well as continuing work subsequently to make sure your company's interests are fully taken into account.

Been there, done that, got the t-shirt. Big boss decides project X will be outsourced and I will be the minder of the outsourced team. But wait, I'm a software engineer with no training in managing others or finance or what not. Don't worry, they'll do exactly what we want, says he, we're a big company and we're paying them a lot of money, so they'll do just what we want them to do. HA HA HA HA HA. They do what is in their company's best interest. The people in the trenches aren't graded by me, they're graded by their local managers who of course work for a different company. They bid low so they don't have the staffing to make our project go any faster than if we did it in house. Add in the overhead of the lawyers to write the contracts and the managers to negotiate the contract and everyone involved is losing except the managers and the lawyers. Hmm, funny how that works, at least for a while...

jgch wrote:

@xmyai - good post! This brings it to the point.

Outsourcing as concept basically works. But it requires a very well managed hand over project and an active management afterwards. In the cases I have experienced, neither was the case.
I have seen a number of outsourcing/offshoring projects. None of them gave the expected ROI or quality gain thet was expected.
During the hand over phase, the local experts are made redundant. Hence their motivation is down to a minimum and they have all the possibilities to sabotage the project by telling the contractors staff only 90% of the truth. The missing 10% everyone forgets and no one realizes it was missing during the handover.
The managers on the other hand have often only a rough idea what the ground staff is really doing, so they have not the detailed know how to manage the project. Once everything is outsourced, these guys make sure to leave asap. Their successors have to clean up the left over crap.

When it comes to the day to day work, the outsourced companies loose their know how, hence becoming more and more dependent on their contractor. They then have a very rough idea on what is going on - hence they have lost control.

I agree on the lack of motivation but disagree on the intent to sabotage. There's no need to sabotage. Even a person with the best intentions in the world cannot communicate everything that in an ideal world should have been communicated. Add to that things that people don't communicate due to embarrassment rather than malice (like, oh yeah, we never did get that part of the system working the way we should have, etc) and it's certain that you won't ever communicate 100% of the information.

However it is hard to understate the obvious truth that the people doing the hand off are not motivated to do the hand off because they know they're about to get the sack. They and everyone around them are primarily looking for their next job and whatever remaining low-quality cycles are left go to the hand off. The good ones find work before the hand off is done so everything gets compressed. The managers are quick to declare victory to make the best out of a bad situation so the truth gets buried till it's no longer possible to avoid the truth.

xmyai wrote:

Because you wouldn't and besides, do you really want to hand control of IT systems that are critical to your business over to another party? By all means do it for non-critical systems but not ones that effect your core business.

When you outsource, you are handing off control of IT systems that are critical to your business over to another party. Sure, you own the result, but you no longer have the capability to change the result.

xmyai wrote:

This fiasco raises several questions which, if I was a BA share holder, I would want answers to :

- why the IT systems fell over in this first place
Said to be a power outage, which presumably caused hard failures in the power supplies of various computers due to transient voltage/current spikes.
- why were so many critical business functions affected
These systems are highly inter-dependent. It's easy to add dependencies and hard to remove them. It takes extreme engineering discipline to avoid this problem.
- why has it taken BA's IT staff or the out sourced staff so long to recover the systems
See above.

Revelation wrote:

Been there, done that, got the t-shirt. Big boss decides project X will be outsourced and I will be the minder of the outsourced team. But wait, I'm a software engineer with no training in managing others or finance or what not. Don't worry, they'll do exactly what we want, says he, we're a big company and we're paying them a lot of money, so they'll do just what we want them to do. HA HA HA HA HA. They do what is in their company's best interest. The people in the trenches aren't graded by me, they're graded by their local managers who of course work for a different company. They bid low so they don't have the staffing to make our project go any faster than if we did it in house. Add in the overhead of the lawyers to write the contracts and the managers to negotiate the contract and everyone involved is losing except the managers and the lawyers. Hmm, funny how that works, at least for a while...

......for a minute there I thought I wrote this and forgot that I did.

Alex Crux, the CEO behind BA's thrift drive, refuses to resign...

http://www.bbc.co.uk/news/uk-40083778

Revelation wrote:

When you outsource, you are handing off control of IT systems that are critical to your business over to another party. Sure, you own the result, but you no longer have the capability to change the result.

I think this is the major failing of outsourcing once the dust has settled and the hand over is complete. Disregard for a second all the miscellaneous items that were not fully documented and handed over, in the ongoing operation how they are dealt with is where the frustration arrives leading to eventual failures, sometimes dramatic because no one is now focused on "fixing" the problems. Staff do not respond to two masters, senior management who are now in the position of get this done and get that done, are in no mood to understand why their staff and not getting the outsource vendor staff to perform as expected, they on the other hand, as long as they do what their management tells them to do are good, in between is the customer who really could care less, and if they have the chance, take their business elsewhere when thing go tits up.

JannEejit wrote:

Alex Crux, the CEO behind BA's thrift drive, refuses to resign...

http://www.bbc.co.uk/news/uk-40083778

... because he has the board's full backing since his real job is to wring the last penny out of every pound and take the bullets when things like this happen.

From the brief interview of Alex Cruz I saw on TV the claim is that there was a "power surge" that stopped all the messages between various systems. I thought big international companies smoothed external power to prevent exactly that type of issue.

The board is just as accountable as Cruzy. Cruzy just speaks for them all.

I also think Walsh is just using Cruzy as a puppet to force more cost cutting down the throat of BA. One more high profile disaster like this and I think shareholders will demand action, possibly along with Al Baker. I think the days of this current management of BA are numbered. People will vote with their wallets.

blueflyer wrote:

mjoelnir wrote:

So there was no second installation in a different location, mirroring the main data center, able to take over.

Can't make that assumption. Wouldn't be the first time an incident at the primary site exposes a flaw in the failover process to a secondary site... Maybe it wasn't tested, maybe it couldn't be tested... Or maybe it was, and passed, and yet still failed when needed... All are possible at this point.

I can make that assumption, at the minimum the second installation was not able to take over.

SamYeager2016 wrote:

From the brief interview of Alex Cruz I saw on TV the claim is that there was a "power surge" that stopped all the messages between various systems. I thought big international companies smoothed external power to prevent exactly that type of issue.

That is correct, someone bought this up in an earlier post, however it is an expensive bit of kit to install, and men in suits don't like expense, unless they are the ones claiming it back from the company.
Alex Cruz says he will not resign but he is damaged goods, did he show his face at T5, might have missed it. Give it a while and he may quietly slip away.

Planeflyer wrote:

Leave IT to those Fluent in it so BA can focus on only what they can do best: get passengers into and of LHR better than anyone else

The crux is that BA needs their own IT to improve what they can do best. IMHO, it makes more sense for BA to outsource the flying than its high-level IT.
Every airline should consider itself a software company that just so happens to fly people from A to B. Unfortunately, I think it's going to take a while for this to happen. Ryanair seems to be the furthest ahead in that regard.

Without IT BA cannot operate. To outsource IT means you lose control of the business. Its that simple.

SamYeager2016 wrote:

From the brief interview of Alex Cruz I saw on TV the claim is that there was a "power surge" that stopped all the messages between various systems. I thought big international companies smoothed external power to prevent exactly that type of issue.

Surge protection has its limits. There is no such thing as infinite surge prevention. You purchase what you need for the anticipated spikes. There is a reason why we talk of a hundred year flood. Same thing happens to power networks, especially when lightening is involved. It'll be interesting to know what the root cause of all of this really was, if we ever do find out.

par13del wrote:

Revelation wrote:

When you outsource, you are handing off control of IT systems that are critical to your business over to another party. Sure, you own the result, but you no longer have the capability to change the result.

I think this is the major failing of outsourcing once the dust has settled and the hand over is complete. Disregard for a second all the miscellaneous items that were not fully documented and handed over, in the ongoing operation how they are dealt with is where the frustration arrives leading to eventual failures, sometimes dramatic because no one is now focused on "fixing" the problems. Staff do not respond to two masters, senior management who are now in the position of get this done and get that done, are in no mood to understand why their staff and not getting the outsource vendor staff to perform as expected, they on the other hand, as long as they do what their management tells them to do are good, in between is the customer who really could care less, and if they have the chance, take their business elsewhere when thing go tits up.

Yes, you've characterized it well. Many of my fellow tech co-workers are very frustrated because their job is no longer developing technology. Their jobs have become managing outsourced workers who have little motivation to do what they're being asked to do because they know in the end the outcome is pre-ordained: upper management on both sides will accept their work and whitewash any short comings because if they do otherwise it'll boomerang back on them. The built-in bias is to "make it work" and everyone in the loop knows it. It all works out until/unless you get an event that shows the shortcomings vividly, but almost always the upper management who made the decisions has cashed in their bonuses and moved on, and others are stuck dealing with the aftermath. As you point out, no one owns the big picture view any more. Everything is chunked up into contracted tasks. The in-house team no longer has the ability to change the outcomes, so they have to live with the results. There is no "impedence mismatching" to deal with the cases where the chunks of contracted work don't line up well. The contractors know the customer no longer has the staffing or the knowledge base to bring work back in house so each future contract costs more and delivers less.

In my case I'm very thankful because I'm no longer in that environment. I now work for a small company that to me seems to me to be a time capsule. They haven't outsourced any of their engineering functions. They have offshore engineers in lower cost localities but they're all employees of our firm. No one can be bothered with management fads like Agile, and I couldn't be happier about it. We do things the old school way, get great results, and make good money. The firm is privately held, mostly by the founders, and they don't seem to even be thinking of "modernizing", thank god. I hope things keep going this way till I'm ready for retirement. Far less stress than my last employment situation. Oh, and by the way, lots of grey hairs working there, and not a bit of ageism. Management seems to understand the value of hiring experienced people and paying them a decent wage. Like I said, it's something of a time capsule, and I'm thankful for it.

Revelation wrote:

Surge protection has its limits. There is no such thing as infinite surge prevention. You purchase what you need for the anticipated spikes. There is a reason why we talk of a hundred year flood. Same thing happens to power networks, especially when lightening is involved. It'll be interesting to know what the root cause of all of this really was, if we ever do find out.

It's worth noting that on Saturday morning at roughly the time of the failure or before, there was a band of intense thunderstorm activity in the vicinity of London. Whilst not the storm of the century, it looks like it wasn't insignificant either with frequent, multiple lightning strikes.

See the historical data here... http://en.blitzortung.org/live_lightning_maps.php

Extremely poor show from BA in any case!

Rgds.

SamYeager2016 wrote:

From the brief interview of Alex Cruz I saw on TV the claim is that there was a "power surge" that stopped all the messages between various systems. I thought big international companies smoothed external power to prevent exactly that type of issue.

The more i hear about this the more it is obvious this has nothing to do with IT. No software can protect you from such events. It seems the company providing power to BA needs to invest in better and more modern equipment,

manny wrote:

SamYeager2016 wrote:

From the brief interview of Alex Cruz I saw on TV the claim is that there was a "power surge" that stopped all the messages between various systems. I thought big international companies smoothed external power to prevent exactly that type of issue.

The more i hear about this the more it is obvious this has nothing to do with IT. No software can protect you from such events. It seems the company providing power to BA needs to invest in better and more modern equipment,

It also seems that BA needs to invest in some redundancies, as this looks to be a single point of failure event.

manny wrote:

It seems the company providing power to BA needs to invest in better and more modern equipment,

There's limits to what they can do. In the extreme, they could bury all transmission lines and put all sub-stations into giant Faraday cages, but to do that they'd have to pass on the costs to all the customers.

AWACSooner wrote:

It also seems that BA needs to invest in some redundancies, as this looks to be a single point of failure event.

Sadly it seems they made redundancies in the IT staff already.

[ Making someone redundant is the British equivalent of the US term "laying someone off" ... ]

None of us know exactly what happened, but it seems to be more than just an electrical issue. It seems to be an electrical issue that presumably destroyed some equipment in a critical location, followed by a poor recovery plan, or a poorly executed recovery plan, or no recovery plan at all.

The more I read the more I think BA could benefit from a cloud system. Cloud vendors have forgotten more about IT than any individual company will ever know.

JannEejit wrote:

Alex Crux, the CEO behind BA's thrift drive, refuses to resign...

http://www.bbc.co.uk/news/uk-40083778

Manuel has got a nerve, large swathes of BA middle and upper management are on yearly contracts, don't perform, your out.
With some exceptions.......
(I bet he'll get another big fat bonus, then they wonder why morale is in the toilet).

Worst BA CEO since Alying and just in a year.

manny wrote:

SamYeager2016 wrote:

From the brief interview of Alex Cruz I saw on TV the claim is that there was a "power surge" that stopped all the messages between various systems. I thought big international companies smoothed external power to prevent exactly that type of issue.

The more i hear about this the more it is obvious this has nothing to do with IT. No software can protect you from such events. It seems the company providing power to BA needs to invest in better and more modern equipment,

Redundant mains power supplies, redundant data centers, redundant availability zones consisting of multiple data centers...

Plenty of ways to protect against a power surge taking it your entire data center.

The claim mentioned on TV is so vague as to be laughable. No power surge is going to kill your alternative data center, even if you do have to trigger the failover manually.

This sort of outage simply shouldnt happen in this day and age, its pure incompetence that it has.

The ironic thing is that BA will probably switch outsourcing contractors now, citing performance issues (in reality more embarrassment issues and a need to blame someone disposable) but that wont solve anything - its just BA waving goodbye to yet more institutional knowledge earned the hard way...

moo wrote:

manny wrote:

SamYeager2016 wrote:

From the brief interview of Alex Cruz I saw on TV the claim is that there was a "power surge" that stopped all the messages between various systems. I thought big international companies smoothed external power to prevent exactly that type of issue.

The more i hear about this the more it is obvious this has nothing to do with IT. No software can protect you from such events. It seems the company providing power to BA needs to invest in better and more modern equipment,

Redundant mains power supplies, redundant data centers, redundant availability zones consisting of multiple data centers...

Plenty of ways to protect against a power surge taking it your entire data center.

The claim mentioned on TV is so vague as to be laughable. No power surge is going to kill your alternative data center, even if you do have to trigger the failover manually.

This sort of outage simply shouldnt happen in this day and age, its pure incompetence that it has.

I heard something that was shocking from someone i know who worked as a vendor for BA IT. Because i do not have proof if this is really true, I hope someone here can confirm that. But once upon a time the BA data center was very close to the end of a runway at LHR. If that is indeed true than it is shocking and shows how callous and incompetent BA has been with its IT infrastructure setup and services.

You never put a data center in a path that could be in harms way. You just don't do something like that.

Without insider information, we can infer practically nothing from just one data point. The outage may have been caused by incompetence, cost cutting, outsourcing, insufficient redundancy, inadequate testing of failover and disaster recovery procedures, or just bad luck. For all we know, BA may have the best-designed and best-tested, most redundant and most resilient IT systems and infrastructure of all airlines in the world, and still may have beeen hit by a combination of unforseen and unforseeable events resulting in an unpreventable outage. Not that I believe this, but it's not impossible.

Possibly the story about the power supply problem is a red herring. Most likely there was a complex chain of events where the power outage played only a small part, if any, but it sounds believable.

I think something doesn't quite add up about the "power surge" claim. Power companies for BA headquarters and for LHR report no power surge in those areas at that time, and experts also doubt the power surge claim, see below:
https://www.ft.com/content/35cb574e-451c-11e7-8519-9f94ee97d996
https://www.theguardian.com/business/2017/may/30/british-airways-it-failure-experts-doubt-power-surge-claim
(Note: there are other easily Google-able news stories out there if people find issue with these particular sources.)

I feel for the BA employees on the front line! Through no fault of their own they are assailed by thousands who are instant IT and airline experts. While this is a very bad couple days for BA, people need to look at the scale of these intricate systems and appreciate that S@#& happens sometimes! We get so accustomed to everything working flawlessly 100% of the time that we forget that humans made these things and humans are not perfect. Hopefully BA will figure out what caused this and we can all learn from these mistakes.

fly4ever78 wrote:

I feel for the BA employees on the front line! Through no fault of their own they are assailed by thousands who are instant IT and airline experts. While this is a very bad couple days for BA, people need to look at the scale of these intricate systems and appreciate that S@#& happens sometimes!

"S@#& happens" to everybody but it's how you respond to the problem(s) that matters. In this case BA's response was badly lacking, in part because the IT failure was so all encompassing and lasted for so long. If BA had responded well, including early and frequent communications from senior management, then the flak they received would not have been so bad.

It now seems that his job is safe.

http://www.bbc.co.uk/news/business-40117382

"Mr Walsh said it was unfair to try to blame Mr Cruz because the problem was not of his making."

I have a BA question which doesn't really merit its own thread but it could be IT related.

I have tried to check in for a BA flight tomorrow LHR to Palermo.

I successfuly checked in my travelling companion and a seat has been allocated. When it came to my turn a message came up saying it isn't possible to check in for this flight and do it at the airport tomorrow - which is a pain as it requires extra time at the airport and the flight is at 0735.

Being 'clever' I thought I'd try checking in via the app on my phone to see if this made any differnt only this time the message came up that my passport wasnt valid for travel [or words to that effect] even though it has several months before it expires. I reentered the passport details on the BA main website but still not allowed to check in.

Are there still residual issues from last weeks melt down or should I be worried about why it would let my companion check in but not me?

Does your passport have at least 6 months remaining on it upon your rtn to LHR....?