Showing posts with label Perhaps They Should Have Tested More. Show all posts
Showing posts with label Perhaps They Should Have Tested More. Show all posts

May 30, 2017

Perhaps They Should Have Tested More - The Schiaparelli Mars Lander

Schiaparelli Lander


Bugs have consequences! 

The wrong bugs can crash your Mars lander, cost you a lot of money and cost you a lot of confidence in your planned Mars missions, as the European Space Agency found out.

Back in October 2016, the ESA attempted to land the Schiaparelli lander on the surface of Mars. But software bugs caused the craft to crash and to produce a rather impressive crater.

Here lies Schiaparelli

Recently, an independent investigation has concluded that insufficient parachute modeling, inadequate handling of alerts, and an “insufficient approach to Failure Detection, Isolation and Recovery and design robustness" were to blame for the failure. #copyrightjoestrazzere

Essentially, the software thought the spacecraft was closer to the ground than it actually was, and released the parachute and shell early. When the control system shut down, the craft was still 3.7 km in the air - oops! This resulted in a 34 second free-fall, subsequent crash at a speed of 540 km/h, and the destruction of Schiaparelli.

In spite of the failure in the planned soft landing, the Europoean Space Agency declared the mission "a success". Maybe they were just really excited about the brand new Martian crater they discovered?


Perhaps the European Space Agency should have tested more.


See Also:

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm an experienced Quality Assurance professional.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
2 comments
Labels:

September 1, 2015

Perhaps They Should Have Tested More - SunGard


not the actual           BNY Mellon       logo *         

Perhaps SunGard and/or BNY Mellon should have tested more


If you've been watching the Market the past few weeks, you may have been concerned by all the volatility and heavy activity occurring.

And if your funds' NAV (Net Asset Value) was reported through BNY Mellon, you may have been even more concerned to learn that the NAVs at the end of the day's trading were delayed.

That's apparently because an operating system upgrade by SunGard (which hosts the fund account platform of BNY Mellon) caused a malfunction - on both the production platform, and the backup platform!

  • SunGard hosts the fund account platform of BNY Mellon
  • A malfunction in this platform disrupted the net asset values (NAVs) of BNY Mellon mutual funds and exchange-traded funds
  • Impacted 66 of BNY Mellon's US fund accounting clients
  • For up to 1200 funds valued at over $400 billion, the reporting of the NAVs at the end of the day's trading was delayed.
  • The timing was especially poor, due to the heavy market activity in the days that followed.
  • The problems may have led BNY Mellon to calculate inaccurate prices during the market's volatility
  • BNY Mellon's clients were likely receiving imprecise information
  • Per SunGard "issue appears to have been caused by an unforeseen complication resulting from an operating system change."
  • SunGard said it followed standard testing procedure for the implementation of this change, but the update still corrupted both BNY Mellon's primary and backup environments
  • BNY Mellon's chairman and CEO Gerald Hassell said the full restoration of SunGard's platform "has taken far longer than any of us would have expected."
  • Dozens of companies rely on Bank of New York data to determine the correct value for their funds
  • Bank officials say the problem began on Aug. 24, just as U.S. markets suffered their worst single-day loss since 2011

Perhaps it was just bad timing. 

It's nice that SunGard followed their standard testing procedures, but perhaps they weren't adequate. And perhaps it's not a good idea to make changes to the Backup environment, until you are completely sure the Production environment is actually working correctly.

Perhaps SunGard (and perhaps even BNY Mellon) should have tested more.

See also:


* actual BNY Mellon logo was removed per demand from BNY Mellon
"In connection to the Bank of New York Mellon Corporation’s proprietary rights over its famous trademarks we are notifying you of the following:

It has come to our attention that our trademark(s) appears as a metatag, keyword, visible or hidden text on the web site(s) located at:
http://www.allthingsquality.com/2015/09/perhaps-they-should-have-tested-more.html

without having obtained prior written authorization from The Bank of New York Mellon Corporation. This practice infringes upon our exclusive intellectual property rights."
This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm an experienced Quality Assurance professional.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

August 19, 2015

Perhaps They Should Have Tested More - Dilbert

http://dilbert.com/strip/2015-08-19

The "Internet of Things" is a scary place:
  • New brain stimulator device had a software glitch
  • Momentary homicidal mania is a side-effect
  • One dead in cafeteria brawl
Perhaps Dilbert should have tested more?

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm an experienced Quality Assurance professional.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels: ,

November 4, 2014

Perhaps They Should Have Tested More - Microsoft Band


Is it 12;45? Or 7:46? Is it Tuesday yet?


Microsoft's entry in the wearable fitness tech category is less than a week old, and already the first bug appears - and it's a Daylight Saving Time bug! copyrightjoestrazzere

  • Band did not correctly synchronize with Microsoft Health
  • Forgot to keep up as clocks fall back
  • Users could find themselves stuck between two times
  • Corrected automatically on November 3rd


I'm pretty sure this Standard Time versus Daylight Saving Time wasn't just recently invented, right? It's been "Spring Forward" and "Fall Back" for a while, right?

Perhaps Microsoft doesn't think Daylight Saving is Healthy. Perhaps in their enthusiasm to get the Band on the market, they just skipped this part of their test plans. Perhaps, someday, vendors will consistently remember to check for things like Daylight Saving Time and Leap Years, rather than follow up buggy versions with press releases and patches.

Perhaps Microsoft should have tested more.



See also:


This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
1 comments
Labels:

October 29, 2014

Perhaps They Should Have Tested More - Audi

Perhaps Audi Should Have Tested More... Earlier


This is another story about faulty airbags - but not the shrapnel-shooting airbag design by Takata. These airbags are faulty due to software bugs.

So if you own an Audi A4, Avant or Allroad vehicle, you might want to contact your dealer. A software problem may prevent your front airbags from deploying in a crash. copyrightjoestrazzere
  • Audi is recalling about 850,000 cars
  • Worldwide recall
  • Defective software could cause airbags to fail to deploy properly
  • An improper algorithm/coding within the airbag control module,
  • Front-seat occupants may not have the intended protection
  • Affects the Audi A4 sedans, Avant and AllRoad versions from 2011 through October 2014.
  • The software update takes about 20 minutes
  • The problem was discovered during routine quality assurance testing

It's nice that Audi discovered the problem recently. It's nice that the problem was discovered during routine testing.

Perhaps Audi should have tested more - perhaps before releasing the faulty software beginning 3 years ago.




This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

October 22, 2014

Perhaps They Should Have Tested More - Intrado 911

Perhaps Intrado Should Have Tested More



You could think of it as the "40 million call bug".

A newly-released study by the Federal Communications Commission explains how a software glitch caused a massive outage in 911 services last April 9th. At the center of the disruption was a system maintained by a Longmont Colorado-based company - Intrado. copyrightjoestrazzere

Each of the calls to the Intrado system had a counter assigned to it. Unfortunately, once the number of calls reached 40 million, the system ran out of counters and stopped accepting new calls.

Intrado's servers noticed that they had run out of counters. But the "out of counters" log entries were categorized as a "low priority" state, and thus no alerts were sent to any humans.

  • When the 40 millionth call was received the 911 system stopped working
  • Affected 81 call dispatch centers.
  • More than 11 million people across seven states were affected
  • Entire state of Washington was denied access to 911 services
  • The outage continued for six hours
  • More than 6,600 people tried, and failed, to reach help
  • The out-of-counters condition was logged as a low-level incident because that was the default level

Perhaps Intrado thought that embedding a magic limit in the sofware was a good idea. Perhaps Intrado figured that the limit of 40 million counters would never be reached. Perhaps they figured that even when they ran out of counters, it wasn't very important. Perhaps all of the 6,600 people who couldn't reach 911 emergency services didn't really need help quickly.

Perhaps Intrado should have tested more.

See Also:

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

October 15, 2014

Perhaps They Should Have Tested More - Nielsen


Perhaps Nielsen should have tested more



When ABC's "Dancing with the Stars" premiered this season, the show got a big bump in the ratings. Researchers for competing networks were skeptical, since two big television markets didn't even air the show, as it was pre-empted by a broadcast of a NFL Football game. copyrightjoestrazzere

When they contacted Nielsen, the company rechecked its methodology. They found that not only were those particular numbers inflated, but that months worth of ratings were incorrect, due to a software glitch introduced in March.

  • Nielsen controls almost all of the television ratings measurement market
  • On October 6, Nielsen discovered an error in their TV ratings, which had been present since March 2
  • Nielsen misattributed some viewership results
  • Errors resulted in incorrect data over a period of seven months
  • Advertisers and networks rely on Nielsen to provide the metrics used to conduct their ad sales business
  • The error wound up benefiting ABC, while negatively impacting others
  • Billions of dollars of advertising are based on Nielsen's ratings
  • Some television and advertising executives have called Nielsen's methodology antiquated
  • Some have questioned it's ability to measure the ways people currently watch television
  • Nielsen says it intends to recalibrate its ratings starting only from August 18
  • When the bug was made public, Nielsen's stock dropped, while that of a competitor went up 15 percent

Perhaps this black mark won't cause advertisers to rely more on competitors' products rather than Nielsen. Perhaps the ratings inaccuracies weren't enough to matter much in the long run. Perhaps there will never be another Dancing Star as good as Emmitt Smith. But perhaps Nielsen should have tested more.

See also:
http://www.adweek.com/news/television/nielsen-mistake-means-ratings-are-even-lower-we-thought-160719
This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

May 24, 2014

Perhaps They Should Have Tested More - Nest Labs

Nest Protect - Don't Make Waves!


In April, Nest Labs determined that their Nest Protect smoke alarm system's "Wave" feature could cause problems if people were "waving" when a fire occurs. The Consumer Product Safety Commission is now calling this a "recall", even though the devices could be updated over the internet to disable this feature.
copyrightjoestrazzere
  • About 440,000 units will be recalled
  • Nest can fix the problem with a software update.
  • Not a physical recall so much as a software update alert
  • Glitch that can stop the alarm from sounding.
  • The Nest Protect could misinterpret people waving their hands and turn off if there was a real fire
  • Activity near the product during a fire can prevent the alarm from immediately sounding when the Nest Wave feature is enabled
  • If your device is not connected to a Nest account, or you’ve taken it offline, you should “immediately” reconnect it so the company can remotely disable the feature.
  • If you can’t connect to Wi-Fi, you should stop using the device altogether

On April 3rd, Tony Fadell, the CEO of Nest Labs wrote:
At Nest, we conduct regular, rigorous tests to ensure that our products are the highest quality. During recent laboratory testing of the Nest Protect smoke alarm, we observed a unique combination of circumstances that caused us to question whether the Nest Wave (a feature that enables you to turn off your alarm with a wave of the hand) could be unintentionally activated. This could delay an alarm going off if there was a real fire. 
We identified this problem ourselves and are not aware of any customers who have experienced this, but the fact that it could even potentially happen is extremely important to me and I want to address it immediately. 
We feel that the best and safest thing to do is to immediately disable the Nest Wave feature to resolve the issue and remove any safety concerns. While we fix Nest Wave, we have also halted sales of all new Nest Protect alarms to ensure no one buys an alarm that needs an immediate update.
...
Once we have a solution that ensures Nest Wave works as intended, we will update our software to turn this feature back on. This will only happen after extensive testing and once we have received approval from safety agencies in the US, Canada and UK. We expect this to take at least two or three months and we’ll continue to update you as we have more information.

So if you own a Nest Protect, you can nod, wink, or say "Hi". But don't wave at it, whatever you do! (Just kidding. Seriously though, disable the Nest Wave feature immediately!)

Next time Nest Labs releases a product, I assume they'll check the ramifications of waving at it before they begin sales. Perhaps this time, they should have wave tested more?

[Full Disclosure: I own a Nest Learning Thermostat (see: http://www.allthingsquality.com/2014/02/our-nest-in-maine.html). So if you are in Maine, and you see me waving furiously at my thermostat, you'll know that I'm not crazy - I'm just conducting my own testing. So far, no bugs found.]

See Also:

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

April 23, 2014

Perhaps They Should Have Tested More - CTB/McGraw Hill



Perhaps CTB/McGraw Hill Should Have Tested More


For the second year in a row, students taking online high-stakes standardized tests in Oklahoma had to stop because of  problems in the software. copyrightjoestrazzere
  • About 8,100 Oklahoma students experienced problems
  • Many could never make it past the sign-in screen
  • Many were unexpectedly kicked out in the middle of the test
  • In spite of problems last year, the state renewed its contract with the company for more than $13 million this year in large part because there was not enough time to solicit bids and select another vendor
  • The Education Department is considering switching to pencil and paper testing
  • This appears to be 100 percent a failing of CTB

Like last year, CTB/McGraw Hill gets an "F" on their report card.

Perhaps next year they will do their homework and test more.

See also:

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

January 29, 2014

Perhaps They Should Have Tested More - GM


Not a GM Product


Recently General Motors recalled over 370,000 full-size pickup trucks due to a software problem that appears to have already caused 11 fires: copyrightjoestrazzere
  • Affects 2014 Chevrolet Silverados and GMC Sierras
  • Could lead to overheating, potentially leading to engine compartment fire
  • GM has reports of 11 fires related to this bug
  • Until it is fixed, owners are advised not to let the truck idle
  • Problem uncovered in part because the recent extreme cold led to more idling
  • The 2014 Silverado is one of three finalists for North American Truck of the Year

Perhaps GM was idling too much, when they could have tested more.


See also:


Image courtesy of zirconicusso / FreeDigitalPhotos.net
This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

January 9, 2014

Perhaps They Should Have Tested More in 2013

Perhaps They Should Have Tested More in 2013


2013 saw some "interesting" bugs show up in the news.  Perhaps these companies should have tested more?


The Chicago Board Options Exchange (CBOE) experienced a shutdown lasting 3.5 hours, paralyzing the biggest U.S. venue for options trading.
Perhaps They Should Have Tested More - Chicago Board Options Exchange
http://www.allthingsquality.com/2013/05/perhaps-they-should-have-tested-more.html

A $4,000 toilet, the Satis was released with a hard-coded bluetooth PIN of "0000".
Perhaps They Should Have Tested More - Lixil's Satis Toilet
http://www.allthingsquality.com/2013/08/perhaps-they-should-have-tested-more.html

When iOS7 was released, it wasn't long before someone discovered a way to bypass the lock screen, and access all photos contained on the iPhone.
Perhaps They Should Have Tested More - Apple iOS 7
http://www.allthingsquality.com/2013/09/perhaps-they-should-have-tested-more.html

Honda was forced to recall over 344,000 minivans to correct a problem with the software controlling the vehicle's stability control system.
Perhaps They Should Have Tested More - Honda
http://www.allthingsquality.com/2013/11/perhaps-they-should-have-tested-more.html

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels: ,

November 6, 2013

Perhaps They Should Have Tested More - Honda

Perhaps Honda Should Have Tested More

We continue to see a never-ending stream of software problems turning up in automobiles.

This time, Honda is recalling over 344,000 minivans to correct a problem with the software controlling the vehicle's stability control system.
  • Honda Motor Co. is recalling over 344,000 Odyssey minivans
  • Corrects a problem in the stability control software
  • The vehicle may suddenly and unexpectedly brake hard
  • Worse - the brake lights won't go on when this happens
  • Honda cannot replace the software
  • The fix itself won't actually be available until Spring of 2014
  • Requires a new part to fix the software problem
copyrightjoestrazzere
Good news! There is a workaround:
  • After starting, don't move the car for a while
  • Before shifting out of park, make sure the wheels point straight ahead
  • Keep the wheels straight for the first few feet as you move
  • If you must start your car with the wheels turned, drive slowly to a place where you can stop the car and start it with the wheels straight
It's nice to have a workaround when the bug can't be fixed for many months. The workaround seems rather convoluted in some cases though. Do I need to rub my stomach while patting my head, too?

Perhaps they should have tested more.

Also see:


This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

September 20, 2013

Perhaps They Should Have Tested More - Apple iOS 7

Hmm, was that Down, Down, Left, A, Up, Right? Or Left, Left, Down, B, Right, Up?

A new method for bypassing the passcode on an iOS7 lock screen has been discovered. This allows an unauthorized intruder access to all photos, and the ability to post them to channels such as Twitter, Facebook, Flickr, or even to email them. copyrightjoestrazzere

  • The iOS 7 lock screen can be bypassed with a series of gesture techniques
  • Up - Hold - Cancel - Doubleclick - Hold
  • Affects iPhones, iPod Touch, and iPads
  • Grants unintended access to Mail, Photos, Twitter and more
  • Found by Canary Islands-based soldier Jose Rodriguez
  • Rodriguez also found lock screen flaws in earlier versions of iOS

It's probably not a big deal. After all, I'm sure nobody ever has photos on their i-Devices that they wouldn't want everyone in the world to see, right?

Still, this isn't the first time that someone has found low-tech ways to bypass the lock screen.

Fool me once, shame on you. Fool me twice, shame on me. Fool me three times - perhaps they should have tested more.

Also see:
http://www.forbes.com/sites/andygreenberg/2013/09/19/ios-7-bug-lets-anyone-bypass-iphones-lockscreen-to-hijack-photos-email-or-twitter/
http://www.zdnet.com/how-to-fix-the-ios-7-lock-screen-bypass-flaw-7000020913/
http://appleinsider.com/articles/13/09/19/apples-control-center-used-to-bypass-ios-7-passcode-lock
http://lifehacker.com/ios-7-bug-allows-thieves-to-bypass-your-lock-screen-1350894097
http://allthingsd.com/20130919/yes-apple-is-working-on-a-fix-for-the-ios-7-lock-screen-hack/
http://techcrunch.com/2013/09/19/ios-7-lock-screen-vulnerability-discovered-gives-access-to-photos-and-social-sharing/
http://www.macrumors.com/2013/09/19/ios-7-lock-screen-vulnerability-gives-access-to-photos-email/

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
5 comments
Labels:

August 8, 2013

Perhaps They Should Have Tested More - Lixil's Satis Toilet


Not actually the Satis toilet
(But probably not subject to being hacked, either)

Retailing for over $4,000, the Satis toilet manufactured by Japanese firm Lixil is state-of-the-art in every way. Features include: copyrightjoestrazzere
  • automatic flushing
  • music
  • deodorizing fragrance release
  • a heated seat
  • a massage feature (hmm, I'm not sure I want to know more about this)
  • soft lighting
  • a bidet function
  • the ability to be hacked from any android phone with bluetooth capability

Call me a Luddite, but to me, controlling my toilet via an Android app doesn't seem like the smartest idea anyway. Do people really want to track their poop? Then, to use a hard-coded Bluetooth PIN of "0000" means that clearly someone wasn't thinking this one through to its logical conclusion.

According to Trustwave's Spiderlabs information security experts:
The "My Satis" Android application has a hard-coded Bluetooth PIN of "0000"
as can be seen in the following line of decompiled code from the
application: 
BluetoothDevice localBluetoothDevice =
BluetoothManager.getInstance().execPairing(paramString, "0000")
I can just see it now. Toilets gone wild! Unexpected bidet action... overpowering fragrance... constant flushing... overheating... overly aggressive massage (hmm)... poop counts through the roof... the mind boggles!

The message is clear folks: If you are going to open up your device to external control (via Bluetooth, or via the Internet, whatever), you need to take security seriously.

Before they get too flushed with success, piss away their profits, or start to see any corporate goodwill go down the drain, perhaps Lixil needs to invest in some Google-style "Testing on the toilet" here?

I shudder to think what the onrushing "internet of things" will bring!

See also:
0 comments
Labels: ,

May 2, 2013

Perhaps They Should Have Tested More - Chicago Board Options Exchange


Perhaps the Chicago Board Options Exchange should have tested more



The Chicago Board Options Exchange (CBOE) experienced a shutdown lasting 3.5 hours last Thursday, April 25th, with residual effects lasting through Friday morning. copyrightjoestrazzere
  • The outage came as CBOE celebrated its 40th anniversary last Friday.
  • The interruption left some auction processes unavailable to traders until Friday morning.
  • A software glitch shut it down for three hours, wrecking trades and shaking confidence.
  • Glitch was related to the planned reconfiguration of the exchange’s systems.
  • Root of Thursday's problem is seen as being changes made to the software underpinning its systems that handle "complex" orders.
  • Outage paralyzed the biggest U.S. venue for options trading.
  • The shutdown left customers unable to manage positions in some of the options industry's most heavily traded contracts, which are available only at CBOE. 
  • Traders could not deal in CBOE’s S&P 500 Index options contract, the most active US index option, as well as its popular Vix index.
  • The market's shutdown for more than three hours created chaos for customers wanting to trade the exchange's proprietary contracts, such as options on the Standard & Poor's 500-stock index and the VIX, and left them without key hedging tools..
  • Consequences could have been much worse had the failure occurred on a busier, or more volatile, day.
  • CBOE Holdings Inc. technology staff knew of software issues in the hours before the largest U.S. options exchange suffered a three-hour outage.
  • “We have determined that the catalyst was preliminary staging work related to the planned reconfiguration of our systems in preparation for extended trading hours on CBOE Futures Exchange and eventually CBOE options,” the company said in a letter to traders. “It was this staging work, and not a systems upgrade or new systems load, that exposed and triggered a design flaw in the existing messaging infrastructure configuration.”
  • The solution has to be more investments on creaky IT infrastructure.
  • CBOE and its peers won't get away with that "it-doesn't-happen-much-around-here" explanation for very long.

Perhaps there was no way to avoid this problem (although some reports indicate that CBOE's technical staff knew of problems in advance of Thursday outage.)

Perhaps, as Chairman and Chief Executive Officer Bill Brodsky wrote in a follow-up letter to traders "Unfortunately, the nature of a software bug is sometimes only identifiable once the system is operationally ready."

But, perhaps CBOE should have tested more.



CBOE Says Software Bug Resolved

CBOE Letter to traders explaining what happened

CBOE Preaches to Vegas Choir as ‘Glitch’ Crashes Exchange

CBOE Staff Knew of Problems in Advance

CBOE details software bug that shut down options trading

CBOE's Systems Preparation for After Hours Caused Outage

How a "Harlem Shake" shook Wall Street

Time for a Reboot For Tech on Street



And yet another CBOE outage just one week later!
Thursday's isolated delay "was triggered by a product maintenance function being performed for a symbol change to one of the classes in the cluster," CBOE CEO William Brodsky and CEO-designate Ed Tilly said in a memo to clients.
"The software maintenance function was recently updated in preparation for the extended trading hours initiative."

CBOE has 2nd outage in a week
http://www.chicagotribune.com/business/breaking/chi-cboe-2nd-outage-20130502,0,2170747.story

CBOE defers start of extended trading day after recent outages
http://www.baltimoresun.com/business/sns-rt-us-cboe-delay-brodskybre9420k4-20130503,0,7588326.story



This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
4 comments
Labels:

January 2, 2013

Perhaps They Should Have Tested More in 2012



2012 saw more "interesting" bugs show up in the news.  Perhaps these companies should have tested more?


A system designed to help Mitt Romney volunteers record who voted, and to identify those who hadn't for follow-up, apparently failed miserably. It probably didn't cost Romney the election.
Perhaps They Should Have Tested More - Romney Project Orca
http://www.allthingsquality.com/2012/11/perhaps-they-should-have-tested-more.html

When Apple released its new iOS 6 software update, they chose to replace Google Maps with Apple Maps. Hilarity, unhappy users, and a VP firing, ensued. copyrightjoestrazzere
Perhaps They Should Have Tested More - Apple iOS 6 Maps
http://www.allthingsquality.com/2012/09/perhaps-they-should-have-tested-more.html

Knight Capital rolled out a change to its trading software - and in the process lost $440 million. Knight's logo refers to them as "The Science of Trading. The Standard of Trust."
Perhaps They Should Have Tested More - Knight Capital
http://www.allthingsquality.com/2012/08/perhaps-they-should-have-tested-more.html

When accounts were transferred from LaSalle Bank to Bank of America, a software bug allowed a man to withdraw $1,543,104 more than he owned. He gambled the money away.
Perhaps They Should Have Tested More - Bank Of America
http://www.allthingsquality.com/2012/06/perhaps-they-should-have-tested-more.html

In a precursor of bugs to come, the Mitt Romney campaign released a new iPhone app titled "A Better Amercia". Hey, it was only a minor typo, right?
Perhaps They Should Have Tested More - Mitt Romney Campaign
http://www.allthingsquality.com/2012/05/perhaps-they-should-have-tested-more_30.html

When Facebook went public in May, NASDAQ's IPO system went loopy even after "thousands of hours" of testing. Maybe thousands and one would have done the trick.
Perhaps They Should Have Tested More - Nasdaq
http://www.allthingsquality.com/2012/05/perhaps-they-should-have-tested-more.html

On February 29th, Windows Azure experienced about 8 hours of downtime, demonstrating that Microsoft still hasn't quite got the hang of this whole Leap Day thing. Maybe next time.
Perhaps They Should Have Tested More - Windows Azure
http://www.allthingsquality.com/2012/03/perhaps-they-should-have-tested-more.html

The Russian Mars probe Phobos-Grunt crashed back to Earth in January, after a programming error forced it into "safe mode". Oh well - a paltry budget of $165 million apparently doesn't allow for much testing.
Perhaps They Should Have Tested More - Russian Space Agency
http://www.allthingsquality.com/2012/02/perhaps-they-should-have-tested-more.html

The company founder of Fisker Automotive personally called every owner to apologize for bugs in the software controls of their electric "Karma" sedan. It was the second time in two months Fisker cars were recalled. "Hello? Yes, this is Henrick again..." 
Perhaps They Should Have Tested More - Fisker Automotive
http://www.allthingsquality.com/2012/01/perhaps-they-should-have-tested-more.html

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

November 9, 2012

Perhaps They Should Have Tested More - Romney Project Orca

Perhaps the Project Orca team should have tested more?

Described as a "Killer Whale Project", a system designed to help volunteers record who voted, and to identify those who hadn’t for follow-up, apparently failed miserably. copyrightjoestrazzere
  • Designed to digitize and streamline the poll watchers' strike lists
  • Not only did Project Orca not improve the process, it may have actually hindered it. 
  • Checklist in instruction packet listed the same item twice, and missed a critically important step
  • Didn't auto-forward users who didn't know to add an S to the http:// protocol in the app's URL
  • PINs that were required to login and download the voter lists didn't work and couldn't be reset
  • In one area, the head of the Republican Party plus 10 volunteers were all locked out
  • All of Colorado had the wrong PINs
  • In North Carolina every pin was wrong and not fixed until 6PM
  • The system went down for a half hour during peak voting, but for hundreds or more, it never worked all day
  • The system projected every swing state as pink or red
  • As of 4 p.m., Orca still projected a Romney victory of somewhere between 290 and 300 electoral votes
  • An aide said Orca had pretty much crashed in the heat of the action
  • By pushing Orca into the most important swing states they drew 37,000 volunteers away from more battle-tested methods and left local campaign offices flying blind

According to a Romney campaign release from just before the election:
"Project ORCA is a massive undertaking – the Republican Party’s newest, unprecedented and most technologically advanced plan to win the 2012 presidential election.
...
We have tweaked and improved Project ORCA throughout primary, so going into the general, we had several ideas and more time to incorporate those ideas into a system that would work nationally."

Perhaps (with all those ideas and more time), they should have cast a vote or two for more testing?


See also:
This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
1 comments
Labels:

September 24, 2012

Perhaps They Should Have Tested More - Apple iOS 6 Maps


Show me the way to go home
I'm tired and I want to go to bed
I had a little drink about one release ago
And it's gone right to my head

Apple's new iOS 6 software update replaces Google Maps with Apple Maps as expected, but so far, unhappy users are panning the change in websites and online forums. copyrightjoestrazzere

Apple is receiving massive backlash from users around the globe, who report that Apple’s replacement maps, “Apple Maps,” are riddled with strange glitches, inaccurate direction and location data, and fall short of Google Maps.
  • Called "an unsightly blemish on what is otherwise a beautiful OS"
  • Train stations have gone missing
  • The Sears Tower in Chicago has been moved to a smaller tower
  • In Ireland, Dublin has been given a new airport where a farm called Airfield currently stands.
  • Searches for London mostly tend to redirect users to the small city of London in Ontario, Canada.
  • Stratford Upon Avon disappeared
Perhaps Apple should have tested more.



An update:

In http://www.apple.com/letter-from-tim-cook-on-maps/ Apple's CEO writes:
At Apple, we strive to make world-class products that deliver the best experience possible to our customers. With the launch of our new Maps last week, we fell short on this commitment. We are extremely sorry for the frustration this has caused our customers and we are doing everything we can to make Maps better.

I think I know one thing they can do to make Maps better - test more!


A further update:

The former head of Apple's Maps team, a 10-year veteran, was fired.
Some might argue that he should have been tarred and feathered, but Apple has, reportedly, restrained itself and merely fired iOS 6 Maps chief Richard Williamson.

Perhaps his replacement will want to test more.

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
7 comments
Labels:

August 7, 2012

Perhaps They Should Have Tested More - Knight Capital

Perhaps Knight Capital Should Have Tested More

Last Wednesday, Knight Capital rolled out a change to its trading software. It contained bugs that caused unauthorized offers to buy and sell shares. Knight had to sell stocks that it accidentally bought - causing a $440 million loss. They considered filing for bankruptcy, but eventually settled on a painful rescue plan. copyrightjoestrazzere
  • A technology error that occurred when the firm unveiled new trading software
  • Chaos ensued
  • Unauthorized trades were made
  • Knight was left holding at least $4.5 billion worth of securities it hadn't planned to buy
  • Loss of $440 Million when they were sold
  • Considered filing Chapter 11 bankruptcy
  • Agreed to allow investors to purchase $400 Million of its stock at about $1.50 a piece, after trading over $10 the prior week
  • Rescue deal will hugely dilute existing shareholders
  • Firm's market-making responsibilities have been temporarily reassigned
  • SEC is examining the company for potential legal violations

Perhaps Knight Capital should have tested more.

According to Reuters, "Wall Street banks and brokers are poring over their trading systems and rethinking the way they test software to make sure they don't become the next Knight Capital Group". That sounds good.

And yet, as written in Smart Money, "These kinds of mistakes can happen to anyone," said James Angel, associate professor at Georgetown University's McDonough School of Business.

An analysis by Nanex Research concludes that a test program was accidentally released into production:
"We believe Knight accidentally released the test software they used to verify that their market making software functioned properly, into NYSE's live system."
http://www.highfrequencytraders.com/article/1330/knight-capital-group-software-glitch-raises-questions-over-technology-markets
http://articles.marketwatch.com/2012-08-02/markets/32990523_1_knight-capital-shares-capital-base-financial-sector
http://www.washingtonpost.com/national/glitch-at-knight-capital-mars-trading-in-dozens-of-stocks-the-latest-technical-snafu/2012/08/01/gJQATjs6PX_story.html
http://www.nanex.net/aqck2/3525.html
This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
0 comments
Labels:

June 22, 2012

Perhaps They Should Have Tested More - Bank Of America

Perhaps Bank of America should have tested more



Due to a software bug, a 55 year old man was allowed to withdraw far more than his bank account contained while using Bank of America ATM machines. He subsequently gambled away all of the money. copyrightjoestrazzere


  • Bug occurred while accounts were being transferred from LaSalle Bank to Bank of America after an acquisition
  • Status of account was changed to allow unlimited withdrawals (Is that actually a valid status? If so, where can I get me one of those?)
  • Able to withdraw $1,543,104 over a 15-day period
  • Lost all the money gambling
  • Prosecutors in the case place some of the blame with Bank of America for allowing him to withdraw the funds in the first place, and are recommending only 15 months in jail
  • Some think the offender should be freed, because the bank was negligent


Perhaps the man was a gambling addict with no control over his behavior. Perhaps the bank was negligent, and the man should be freed.

And perhaps Bank of America should have tested more.


See also:


This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.
1 comments
Labels:
Subscribe to: Posts (Atom)