QA Leader's Checklist

So I'll be moving to a new job in a few weeks.

I'll be the Director of Quality Assurance at a company which has no QA department yet.

Since I tend to write a lot of lists, I've started a list of "things I need to learn, think about, ask about; people and groups I need to talk with". A few of these things are specific to my situation, but most apply to any QA Lead role.

I shared my initial list with my friends at QAForums.com.  They helped me expand the list even more.

---

Important Stakeholders

• Customers
• QA
• QC
• Engineering
• Product Management
• IT
• Customer Support
• Training
• Marketing
• Sales
• Sales Support
• Documentation
• Management
• Others

Products

• Current Products
  
  • Manuals
  • Online Documentation
  • Training
• Product History
• Upcoming Products
• Releases
• Supported Releases

Technologies

• Front End (Flash, AJAX, etc)
• Application Servers
• Database (SQL Server, Oracle, etc)
• Operating Systems (Windows, Unix, Linux, etc)
• Development Languages/Tools (Java, etc)

Platforms

• Desktop Platforms
• Server Platforms
• Browser Platforms
  
  • IE
  • Firefox?
  • AOL?
  • Opera?
  • Other?
• Supported E-Mail Clients
  
  • OutLook
  • AOL?
  • Other?

Processes

• QA Processeses
• Development Processes
• Release Processes
  
  • When do releases occur
  • how often
  • Service Level Agreements
• Product Introduction Processes
• Backup Processes
• Disaster Recovery / Business Continuance
• Problem Escalation Processes
• Hiring

Tools

• Document Collaboration
• Issue Tracking
• Test Management 
• Requirements Tracking
• Specification Tracking
• Time Management

Schedules

• Release Schedules
• Internal Releases
• External Releases
• Beta Releases
• Meeting Schedules
• Customer Commitments

Requirements

• Functional Requirements
• Performance/Load Requirements
• Security Requirements
• Regulatory Requirements
• Release Criteria

QA Infrastructure

• Lab
• Hardware
• Software
• Test Automation
• Load/Performance Tool
• Issue Tracking
• Test Plans
• Reporting
• QA Business Plan
• QA Mission Statement
• QA Vision Statement

Metrics

• Current
• Proposed

Domain Knowledge

• Glossary of Terms
• Training Classes
• Reading Materials
• Other sources

Budget

• QA
• Overall Engineering

Company

• History
• Directions
• Phone Numbers
• Web Site
• Intranet/Portal
• Facilities
• Parking
• Company Culture
  
  • Dress Code
  • Working offsite
  • Current culture of quality (or lack thereof)
  • Mission Statement
  • Vision Statement

General Office Procedures

• E-Mail Addresses
• E-Mail Lists
• IMs
• Payroll
• Meeting Rooms
• Supplies
• Human Resources
• Mailing Address
• Business Cards

People

• Who holds the power
• Who are the influencers
• Who will help you
• Who will hinder you
• Who do you need to influence
• Who organises the social events

Puzzling Response to a Bug Report

One person on my team was testing a new Troubleshooting feature, which attempted to rank similar items together on a single page, based on their "relatedness".

The feature had no written requirements at all, and was only recently implemented and deemed "testable".

Some of the results produced by this feature were seemingly unpredictable.

So he wrote an issue report demonstrating what appeared to be totally inconsistent results.

The developer replied this way:

Rankings are not human understandable. Don't try. Operates as designed. Not a bug.

Bear in mind that this was an Architect-level developer, not some Entry-level coder.


When there are no requirements, any solution will suffice.

Win32 API IsTextUnicode() is not foolproof - a simple demonstration

From the Aftermarket Pipes blog (http://apipes.blogspot.com/2006/06/this-api-can-break.html)

---

Over at WinCustomize, someone thought they'd found an Easter Egg in the Windows Notepad application. If you:

1. Open Notepad
2. Type the text "this app can break" (without quotes)
3. Save the file
4. Re-open the file in Notepad

Notepad displays seemingly-random Chinese characters, or boxes if your default Notepad font doesn't support those characters.

It's not an Easter egg (even though it seems like a funny one), and as it turns out, Notepad writes the file correctly. It's only when Notepad reads the file back in that it seems to lose its mind.

But we can't even blame Notepad: it's a limitation of Windows itself, specifically the Windows function that Notepad uses to figure out if a text file is Unicode or not.

You see, text files containing Unicode (more correctly, UTF-16-encoded Unicode) are supposed to start with a "Byte-Order Mark" (BOM), which is a two-byte flag that tells a reader how the following UTF-16 data is encoded. Given that these two bytes are exceedingly unlikely to occur at the beginning of an ASCII text file, it's commonly used to tell whether a text file is encoded in UTF-16.

But plenty of applications don't bother writing this marker at the beginning of a UTF-16-encoded file. So what's an app like Notepad to do?

Windows helpfully provides a function called IsTextUnicode()--you pass it some data, and it tells you whether it's UTF-16-encoded or not.

Sorta.

It actually runs a couple of heuristics over the first 256 bytes of the data and provides its best guess. As it turns out, these tests aren't terribly reliable for very short ASCII strings that contain an even number of lower-case letters, like "this app can break", or more appropriately, "this api can break".

The documentation for IsTextUnicode says:

These tests are not foolproof. The statistical tests assume certain amounts of variation between low and high bytes in a string, and some ASCII strings can slip through. For example, if lpBuffer points to the ASCII string 0x41, 0x0A, 0x0D, 0x1D (A ^Z), the string passes the IS_TEXT_UNICODE_STATISTICS test, though failure would be preferable.

Indeed.

As a wise man once said, "In the face of ambiguity, refuse the temptation to guess."

Some Strings for Pasting

Often, I want to enter strings into input fields of particular lengths or contents.

I use handy strings like these to copy to the clipboard and paste into the fields:

---

10 Characters

1234567890
123456789+
1234567.89
12345.6789
$123456.78
$12,345.67
12,345,678
123,456.78

abcdefghi+
abcdefghiX
ABCDEFGHI+
ABCDEFGHIx

Lorem ipsu
WWWWWWWWWZ

*3*5*7*10*

---------+
~!@#$%^&*X

---

100 Characters

123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789+
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
123456789+123456789+123456789+123456789+123456789+123456789+123456789+123456789+123456789+123456789+

abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstu+
abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuZ

ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTU+
ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUz

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore

WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWZ

*3*5*7*9*12*15*18*21*24*27*30*33*36*39*42*45*48*51*54*57*60*63*66*69*72*75*78*81*84*87*90*93*96*100*
---------------------------------------------------------------------------------------------------+

~!@#$%^&*()_+{}|:"<>?`-=[];',./~!@#$%^&*()_+{}|:"<>?`-=[];',./~!@#$%^&*()_+{}|:"<>?`-=[];',./~!@#$%X

---

1000 Characters

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas ac consequat dui. Quisque erat turpis, rhoncus ut imperdiet euismod, iaculis a quam. Quisque malesuada, velit quis viverra gravida, nunc leo tempor arcu, vitae pulvinar sapien dui ut neque. Suspendisse potenti. Vestibulum arcu mauris, varius et egestas ut, fringilla quis lorem. Integer ullamcorper gravida varius. Nam eget mauris justo. Nullam in risus mauris, ac elementum libero. Morbi molestie elit quis tellus tempor in lobortis nibh tempus. Quisque dapibus, dolor facilisis varius bibendum, nunc dui adipiscing leo, eget imperdiet arcu lectus et velit. Nulla cursus dictum elit dapibus laoreet. Nullam sagittis enim id risus eleifend ornare. In porttitor quam ac ante interdum in dictum urna pretium. Aliquam consectetur eleifend erat, sed rhoncus urna condimentum quis. Sed porta rhoncus justo, ut varius arcu pellentesque a. Fusce cursus dolor eu leo iaculis non dapibus erat sodales. Sed quis purus orci. Sed eget mauris amet

*3*5*7*9*12*15*18*21*24*27*30*33*36*39*42*45*48*51*54*57*60*63*66*69*72*75*78*81*84*87*90*93*96*99*103*107*111*115*119*123*127*131*135*139*143*147*151*155*159*163*167*171*175*179*183*187*191*195*199*203*207*211*215*219*223*227*231*235*239*243*247*251*255*259*263*267*271*275*279*283*287*291*295*299*303*307*311*315*319*323*327*331*335*339*343*347*351*355*359*363*367*371*375*379*383*387*391*395*399*403*407*411*415*419*423*427*431*435*439*443*447*451*455*459*463*467*471*475*479*483*487*491*495*499*503*507*511*515*519*523*527*531*535*539*543*547*551*555*559*563*567*571*575*579*583*587*591*595*599*603*607*611*615*619*623*627*631*635*639*643*647*651*655*659*663*667*671*675*679*683*687*691*695*699*703*707*711*715*719*723*727*731*735*739*743*747*751*755*759*763*767*771*775*779*783*787*791*795*799*803*807*811*815*819*823*827*831*835*839*843*847*851*855*859*863*867*871*875*879*883*887*891*895*899*903*907*911*915*919*923*927*931*935*939*943*947*951*955*959*963*967*971*975*979*983*987*991*995*1000*

---

All Alpha Characters

abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ

---

All Special Characters

~!@#$%^&*()_+{}|:"<>?`-=[];',./

---

Others

1
12
123
1,234
12,345
123,456
1,234,567
12,345,678
123,456,789
1,234,567,890

1.23
12.34
123.45
1,234.56
12,345.67
123,456.78
1,234,567.89
12,345,678.90

$1.23
$12.34
$123.45
$1,234.56
$12,345.67
$123,456.78
$1,234,567.89
$12,345,678.90

Perhaps They Should Have Tested More - Cox Communications

Thursday, June 08, 2006

Cox customers seethe

A software glitch leaves thousands of customers without phone service for most of the day Wednesday.

Andrew Kantor

At least 9,000 Cox Communications residential and business telephone customers in the Roanoke Valley lost their service early Wednesday morning, thanks to a software upgrade gone awry.

It took Cox until early evening to trace the glitch, meaning throughout the afternoon the company had little information to provide its customers in Roanoke, Roanoke County and Vinton.

"We are having some difficulties," Cox spokeswoman Stacie Vest conceded at 2:30 p.m. "We did experience an outage," but in midafternoon, "we have not figured out what caused it."
Having heard that from a Cox customer service representative, Tony Williams, president of General Truck Body Co. in Roanoke, asked, "Do they have chimpanzees working on it? They've been working on it for five hours."

At the time, the company knew of about 9,000 telephone routers that were not working, but that number was growing as more customers contacted the company. By 5 p.m., though, it had traced the problem and was beginning to bring users back online.

Routers connect customers' telephones to the Cox network, which in turn connects to the telephone system. Like a personal computer, they do their magic through software. And, like a personal computer, adding or changing that software can sometimes cause problems.
Unlike a PC, though, Cox updates the software on its customers' routers through the system. If there's a problem with that software, every customer can be affected.

Although many home users might not have been affected because they were at work, businesses that rely on the Cox service had a different story.

Williams switched his business line from Verizon to Cox about four months ago and is beginning to think he made a mistake.

"They were more expensive," he said of Verizon, "but we never had this."

Williams discovered his phones were out when he got to work and used his cellphone to contact Cox. He said the company couldn't tell him what the problem was nor when it would be fixed.

After several calls without result, Williams' frustration mounted.

"I told them, 'You're costing us thousands of dollars,' " he said. "This hasn't happened to me in all the years I've been here."

Williams, not surprisingly, expects people to be able to reach him. Instead, the calls were sent directly to his voice mail. He and other customers were able to check voice mail from working phones, but had no way of knowing if a message was waiting without checking. For a business, that's a problem.

"It could be very costly because we have no contact with our customers," Williams said.
And this isn't the first time he's lost his Cox service. In fact, he said, it's the third or fourth time -- but the earlier incidents lasted only an hour or two.

"We have had several software upgrade glitches that have caused both small numbers and larger numbers of customer to go out," Vest admitted.

One of those was on March 30, when approximately 10 percent of the company's phone and high-speed data customers lost service for most of the day because of a botched software upgrade.

Cox would not disclose how many telephone subscribers it has in the Roanoke region, but Vest said that Wednesday's outage affected "a significant number of customers."

None of which meant all that much to Williams. "You might be able to stand it one time," he said, but, "if you never know if you're going to have phone service or not, that's not good."

Cox said it will issue refunds on a case by case basis, when customers request them.

Mack Cooper, who owns G & M Upholstering & Furniture Repair in Roanoke, was one such customer. He's already received one refund from the company because of previous outages on his home phone, but he now wants another one -- and he plans to drop Cox and stick with his cellphone.

"I've just had crappy service," he said, despite repeated calls to the company. "It still doesn't work. It's been a year."

He said he often doesn't get a dial tone, and callers report being cut off after only a few seconds.

"I should never have switched from Verizon," he said, and he's glad he doesn't use Cox at work. "It's not reliable," he said. "It's just not reliable."

Magic 8-Ball Answers

When the answer "42" isn't appropriate, it's time to break out the Magic 8-Ball for answers to foolish questions:

Signs point to yes.

Yes.

Reply hazy, try again.

Without a doubt.

My sources say no.

As I see it, yes.

You may rely on it.

Concentrate and ask again.

Outlook not so good.

It is decidedly so.

Better not tell you now.

Very doubtful.

Yes - definitely.

It is certain.

Cannot predict now.

Most likely.

Ask again later.

My reply is no.

Outlook good.

Don't count on it.

Perhaps They Should Have Tested More - All Binary Searches???

Nearly all binary searches are broken?

Guess I had better add "Search a billion-element-array" to my Test Cases...

---

(From: http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html)

Extra, Extra - Read All About It: Nearly All Binary Searches and Mergesorts are Broken

6/02/2006 08:34:00 AM

Posted by Joshua Bloch, Software EngineerI remember vividly Jon Bentley's first Algorithms lecture at CMU, where he asked all of us incoming Ph.D. students to write a binary search, and then dissected one of our implementations in front of the class. Of course it was broken, as were most of our implementations. This made a real impression on me, as did the treatment of this material in his wonderful Programming Pearls (Addison-Wesley, 1986; Second Edition, 2000). The key lesson was to carefully consider the invariants in your programs.

Fast forward to 2006. I was shocked to learn that the binary search program that Bentley proved correct and subsequently tested in Chapter 5 of Programming Pearls contains a bug. Once I tell you what the it is, you will understand why it escaped detection for two decades. Lest you think I'm picking on Bentley, let me tell you how I discovered the bug: The version of binary search that I wrote for the JDK contained the same bug. It was reported to Sun recently when it broke someone's program, after lying in wait for nine years or so.

So what's the bug? Here's a standard binary search, in Java. (It's one that I wrote for the java.util.Arrays):

1:     public static int binarySearch(int[] a, int key) {
2:         int low = 0;
3:         int high = a.length - 1;
4: 
5:         while (low <= high) {
6:             int mid = (low + high) / 2;
7:             int midVal = a[mid];
8:
9:             if (midVal < key)
10:                 low = mid + 1;
11:             else if (midVal > key)
12:                 high = mid - 1;
13:             else
14:                 return mid; // key found
15:         }
16:         return -(low + 1);  // key not found.
17:     }

The bug is in this line:

6:             int mid =(low + high) / 2;

In Programming Pearls Bentley says that the analogous line "sets m to the average of l and u, truncated down to the nearest integer." On the face of it, this assertion might appear correct, but it fails for large values of the int variables low and high. Specifically, it fails if the sum of low and high is greater than the maximum positive int value (2³¹ - 1). The sum overflows to a negative value, and the value stays negative when divided by two. In C this causes an array index out of bounds with unpredictable results. In Java, it throws ArrayIndexOutOfBoundsException.

This bug can manifest itself for arrays whose length (in elements) is 2³⁰ or greater (roughly a billion elements). This was inconceivable back in the '80s, when Programming Pearls was written, but it is common these days at Google and other places. In Programming Pearls, Bentley says "While the first binary search was published in 1946, the first binary search that works correctly for all values of n did not appear until 1962." The truth is, very few correct versions have ever been published, at least in mainstream programming languages.

So what's the best way to fix the bug? Here's one way:

6:             int mid = low + ((high - low) / 2);

Probably faster, and arguably as clear is:

6:             int mid = (low + high) >>> 1;

In C and C++ (where you don't have the >>> operator), you can do this:

6:             mid = ((unsigned) (low + high)) >> 1;

And now we know the binary search is bug-free, right? Well, we strongly suspect so, but we don't know. It is not sufficient merely to prove a program correct; you have to test it too. Moreover, to be really certain that a program is correct, you have to test it for all possible input values, but this is seldom feasible. With concurrent programs, it's even worse: You have to test for all internal states, which is, for all practical purposes, impossible.

The binary-search bug applies equally to mergesort, and to other divide-and-conquer algorithms. If you have any code that implements one of these algorithms, fix it now before it blows up. The general lesson that I take away from this bug is humility: It is hard to write even the smallest piece of code correctly, and our whole world runs on big, complex pieces of code.

We programmers need all the help we can get, and we should never assume otherwise. Careful design is great. Testing is great. Formal methods are great. Code reviews are great. Static analysis is great. But none of these things alone are sufficient to eliminate bugs: They will always be with us. A bug can exist for half a century despite our best efforts to exterminate it. We must program carefully, defensively, and remain ever vigilant.

Resources

• Programming Pearls - Highly recommended. Get a copy today!
• The Sun bug report describing this bug in the JDK
• A 2003 paper by Salvatore Ruggieri discussing a related problem - The problem is a bit more general but perhaps less interesting: the average of two numbers of arbitrary sign. The paper does not discuss performance, and its solution is not fast enough for use in the inner loop of a mergesort.

Perhaps They Should Have Tested More - AOL Mail Outage

AOL Suffers E-Mail Outage

Users trying to send and receive e-mail were prevented from doing so starting around 11 a.m. EDT on Thursday. The service was back up about 4 p.m., a spokesman said.

By Barbara Darrow,  CRN
June 1, 2006
URL: http://www.informationweek.com/story/showArticle.jhtml?articleID=188700949

America Online's e-mail experienced a major glitch Thursday.

Users trying to send and receive e-mail were impacted starting at around 11 a.m. EDT. The service was back up about 4 p.m., a spokesman for the Dulles, Va. based Time Warner subsidiary said Thursday afternoon.

He attributed the problem to a software glitch. Users with aol.com and AIM.com addresses were affected as were netscape.com and compuserve.com users.

America Online says it has some 25 million e-mail users.

CRN learned of the outage from an AOL customer who was informed earlier today by the service provider that it had experienced a "massive internal system failure" and was asked to suspend bulk mailings immediately.

Problems started surfacing yesterday and became a major problem on Thursday, this source said.

"What this means for you is that e-mail delivered into AOL will likely not be received by your subscribers," the source said.

The AOL spokesman contested the use of the term "massive internal system failure" and instead referred to the root cause as a software glitch. "We have recovered. E-mail has stabilized. There is no current impact on users," he said.

A Bug By Any Other Name

So what do we call Software Bugs?

Here are a few terms I've seen or heard:

• Aberration
• Abend
• Anomaly 
• Blemish
• Blooper
• Blunder
• Boner
• Botch
• Bug
• Complaint
• Defect
• Deficiency
• Deviation 
• Err 
• Erratum
• Erroneousness
• Error
• Failing
• Failure
• Fault
• Flaw
• Foul-up
• Fumble
• Glitch
• Imperfection
• Inaccuracy 
• Inadequacy
• Incident
• Incorrectness
• Irregularity
• Issue
• Lapse
• Loophole
• Miscalculation
• Misconception
• Miscue
• Misplay
• Misstep
• Mistake
• Mixup 
• Noncompliance 
• Oops
• Oversight
• Problem
• Shortcoming 
• Slip
• Slip-up
• Snafu
• Stumble
• Tripup 
• Typo
• Variation
• Weakness
• Wrongdoing

Can you add to the list?

9/7/2010: Chad Patrick on SQAForums.com jokingly suggests one more term:

• Feature 

2/27/2011: Added a few terms from Gerald M. Weinberg's "Why Software Gets In Trouble".
6/30/2011: Added "Foul-up" after I saw it used in the Wall Street Journal about some errors in Southwest Airlines systems. 

From NASA: 100 Lessons Learned for Project Managers

I stumbled across this list while browsing around.  To me, a lot of these lessons apply well to Software QA.  (And yes, I know there are more than 100 lessons here)

---

From: http://appl.nasa.gov/ask/issues/14/practices/ask14_lessons_madden.html and http://www.altisinc.com/Links/100_Rules.html

Jerry Madden, Associate Director of the Flight Projects Directorate at NASA's Goddard Space Flight Center, collected these gems of wisdom over a number of years from various unidentified sources. Rod Stewart of Mobile Data Services in Huntsville, Alabama edited and updated them. I found and kept a copy of these during the early nineties. Over time, the NASA link disappeared, so this version of the rules was posted to make sure that it stayed available.

Jerry was apparently well respected at NASA, and his list continued to circulate. One NASA site includes these comments: "Madden retired from NASA in 1995 as Associate Director of Flight Projects at Goddard Space Flight Center. Considered by many of his peers to be one of NASA's premiere project managers, Madden's reputation for frank, on-target observations of project management continues to be celebrated today, as his list of lessons is handed down to a new generation of project managers. Naturally, not all of Madden's wisdom made it into his '100 Lessons Learned for Project Managers.' Marty Davis, who worked under Madden at Goddard, recalls one of the unwritten lessons: 'Show up early for all meetings; they may be serving doughnuts.'"

---

1. There is no such thing as previously flown hardware, i.e., the people who build the next unit probably never saw the previous unit; there are probably minor changes; the operational environment has probably changed; and the people who check the unit out will in most cases not understand the unit or the test equipment.
2. Most equipment works "as built," i.e., not as the designer planned. This is due to layout of the design, poor understanding on the designer's part, or poor understanding of component specifications.
3. The source of most problems is people but damned if they will admit it. Know the people working on your project, so you know what the real weak spots are.
4. Most managers succeed on the strength and skill of their staff.
5. A manager who is his own systems engineer or financial manager is one who will probably try to do open heart surgery on himself.
6. One must pay attention to workaholics -- if they get going in the wrong direction, they can do a lot of damage in a short time -- it is possible to overload them, causing premature burnout, but hard to determine if the load is too much, since much of it is self-generated. It is important to make sure such people take enough time off and that the workload does not exceed 1-1/4 to 1-1/2 times what is normal.
7. NASA programs compete for budget funds -- they do not compete with each other, i.e., you never attack any other program or NASA work with the idea you should get their funding. Sell what you have on its own merit.
8. Contractors respond well to the customer who pays attention to what they are doing, but not too well to the customer that continually second-guesses their activity. The basic rule is: a customer is always right, but the cost will escalate if a customer always has things done his way, instead of the way the contractor had planned. The ground rule is never change a contractor's plans unless they are flawed or too costly, i.e., the old saying, "better is the enemy of good."
9. Never undercut your staff in public, i.e. don't make decisions on work that you have given them to do in public meetings. Even if you direct a change, never take the responsibility for implementing away from your staff.
10. The project has many resources within itself. There probably are five-to-ten system engineers considering all the contractors and instrument developers. This is a powerful resource that can be used to attack problems.
11. Know who the decision makers on the program are. It may be someone on the outside who has the ear of Congress, or the Administrator, or the Associate Administrator, or one of the scientists -- or someone in the chain of command -- whoever they are, try to get a line of communication to them on a formal or informal basis.
12. You and the program manager should work as a team. The program manager is your advocate at NASA HQ and must be tied in to the decision-making and should aid your efforts to be tied in too.
13. A project manager should visit everyone who is building anything for his project at least once, should know all the managers on his project (both government and contractor), and know the integration team members. People like to know that the project manager is interested in their work, and the best proof is for the manager to visit them and see first hand what they are doing.
14. Never ask management to make a decision that you can make. Assume you have the authority to make decisions unless you know there is a document that states unequivocally that you cannot.
15. Wrong decisions made early can be salvaged, but "right" decisions made late cannot.
16. Never make excuses; instead, present plans of actions to be taken.
17. Never try to get even for some slight by another project. It is not good form -- it puts you on the same level as the other person--and often ends up hindering the project getting done.
18. If you cultivate too much egotism, you may find it difficult to change your position -- especially if your personnel tell you that you are wrong. You should instill an attitude on the project whereby your personnel know they can tell you of wrong decisions.
19. One of the advantages of NASA in the early days was the fact that everyone knew that the facts that we were absolutely sure of could be wrong.
20. Managers who rely on the paperwork to do the reporting of activities are known failures.
21. Not all successful managers are competent and not all failed managers are incompetent. Luck still plays a part in success or failure, but luck favors the competent, hard-working manager.
22. If you have a problem that requires the addition of people to solve, you should approach recruiting people like a cook who has under-salted, i.e., a little at a time.
23. A project manager must know what motivates the project contractors, i.e., their award system, their fiscal system, their policies, and their company culture.
24. Other than original budget information prior to the President's submittal to Congress, there is probably no secret information on the project -- so don't treat anything like it is secret. Everyone does better if they can see the whole picture, so don't hide any of it from anyone.
25. Know the resources of your center and if possible other centers. Other centers, if they have the resources, are normally happy to help. It is always surprising how much good help one can get by just asking.
26. Contractors tend to size up their government counterparts, and staff their part of the project accordingly. If they think yours are clunkers, they will take their poorer people to put on your project.
27. Documentation does not take the place of knowledge. There is a great difference in what is supposed to be, what is thought to have been, and what the reality is. Documents are normally a static picture in time which is outdated rapidly.
28. Remember who the customer is and what his objectives are, i.e., check with him when you go to change anything of significance.
29. In case of a failure:
    
    • Make a timeline of events and include everything that is known;
    • Put down known facts -- check every theory against them;
    • Don't beat the data until it confesses, i.e., know when to stop trying to force-fit a scenario;
    • Do not arrive at a conclusion too rapidly. Make sure any deviation from the norm is explained--remember the wrong conclusion is prologue to the next failure;
    • Know when to stop.
30. Remember the boss has the right to make decisions, even if you think they are wrong. Tell the boss what you think but, if he still wants it done his way, do your best to make sure the outcome is successful.
31. Redundancy in hardware can be a fiction. We are adept at building things to be identical so that if one fails, the other will also fail. Make sure all hardware is treated in a build as if it were one of a kind and needed for mission success.
32. Don't be afraid to fail or you will not succeed, but always work at your skill to recover. Part of that skill is knowing who can help.
33. Experience may be fine but testing is better. Knowing something will work never takes the place of proving that it will.
34. People have reasons for doing things the way they do them. Most people want to do a good job, and if they don't, the problem is they probably don't know how or exactly what is expected.
35. The boss may not know how to do the work, but he has to know what he wants. The boss had better find out what he expects and wants, if he doesn't know. A blind leader tends to go in circles.
36. A puzzle is hard to discern from just one piece, so don't be surprised if team members deprived of information reach the wrong conclusion.
37. Reviews are for the reviewed and not the reviewer. The review is a failure if the reviewed learn nothing from it.
38. The amount of reviews and reports are proportional to management's understanding, i.e., the less management knows or understands the activities, the more it requires reviews and reports. It is necessary in this type of environment to make sure the data is presented so that the average person, slightly familiar with activities, can understand it. Keeping the data simple and clear never insults anyone's intelligence.
39. In olden times, engineers had hands-on experience, technicians understood how the electronics worked and what it was supposed to do, and layout technicians knew too-but today only the computer knows for sure, and it's not talking.
40. Not using modern techniques like computer systems is a great mistake, but forgetting the computer simulates thinking is still greater.
41. Management principles are still the same. It is just the tools that have changed. You still should find the right people to do the work and get out of the way so they can do it.
42. It is mainly the incompetent that don't like to show off their work.
43. Whoever you deal with, deal fairly. Space is not a big playing field. You may be surprised how often you have to work with the same people. Better they respect you than carry a grudge.
44. Mistakes are all right, but failure is not. Failure is just a mistake you can't recover from; therefore, try to create contingency plans and alternate approaches for the items or plans that have high risk.
45. You cannot be ignorant of the language of the area you manage or with that of areas with which you interface. Education is a must for the modern manager. There are simple courses available to learn computerese, communicationese, and all the rest of the modern ese's of the world. You can't manage if you don't understand what is being said or written.
46. Most international meetings are held in English. This is a foreign language to most participants such as Americans, Germans, Italians, etc. It is important to have adequate discussions so that there are no misinterpretations of what is said.
47. NASA Management Instructions (NMIs) are written by another NASA employee like yourself; therefore, challenge them if they don't make sense. It is possible another NASA employee will rewrite them or waive them for you.
48. A working meeting has about six people attending. Meetings larger than this are for information transfer.
49. Being friendly with a contractor is fine -- being a friend of a contractor is dangerous to your objectivity.
50. The old NASA pushed the limits of technology and science; therefore, it did not worry about "requirements creep" or over-runs. The new NASA has to work as if all are fixed price; therefore, "requirements creep" has become a deadly sin.
51. Many managers, just because they have the scientists under contract on their project, forget that the scientists are their customers and many times have easier access to top management than the managers do.
52. Most scientists are rational unless you endanger their chance to do their experiment. They will work with you if they believe you are telling them the truth. This includes reducing their own plans.
53. Cooperative efforts require good communications and early warning systems. A project manager should try to keep his partners aware of what is going on and should be the one who tells them first of any rumor or actual changes in plan. The partners should be consulted before things are put in final form, even if they only have a small piece of the action. A project manager who blindsides his partners will be treated in kind and will be considered a person of no integrity.
54. All problems are solvable in time, so make sure you have enough schedule contingency -- if you don't, the next project manager that takes your place will.
55. The number of reviews is increasing but the knowledge transfer remains the same; therefore, all your charts and presentation material should be constructed with this fact in mind. This means you should be able to construct a set of slides that only needs to be shuffled from presentation to presentation.
56. Just because you give monthly reports, don't think that you can abbreviate anything in a yearly report. If management understood the monthlies, they wouldn't need a yearly.
57. Abbreviations are getting to be a pain. Each project now has a few thousand. This calls on senior management to know a couple hundred thousand. Use them sparingly in presentations unless your objective is to confuse.
58. Occasionally things go right--the lesson learned here is: Try to duplicate that which works.
59. Running does not take the place of thinking. For yourself, you must take time to smell the roses. For your work, you must take time to understand the consequences of your actions.
60. Sometimes the best thing to do is nothing. It is also occasionally the best help you can give. Just listening is all that is needed on many occasions. You may be the boss but, if you constantly have to solve someone's problems, you are working for him.
61. We have developed a set of people whose self interest is more paramount than the work or at least it appears so to older managers. It appears to the older managers that the newer ones are more interested in form than in substance. The question is are old managers right or just old.
62. One problem new managers face is that everyone wants to solve their problems. Old managers were told by senior management -- "solve your damn problems; that is what we hired you to do."
63. Remember, it is often easier to do foolish paperwork than to fight the need for it. Fight only if it is a global issue which will save much future work.
64. Know your management -- some like a good joke; others only like a joke if they tell it.
65. Integrity means your subordinates trust you.
66. You cannot watch everything. What you can watch is the people. They have to know you will not accept a poor job.
67. Next year is always the year with adequate funding and schedule -- next year arrives on the 50th year of your career.
68. The first sign of trouble comes from the schedule or the cost curve. Engineers are the last to know they are in trouble. Engineers are born optimists.
69. External reviews are scheduled at the worst possible time: therefore, keep an up-to-date set of technical data so that you can rapidly respond. Having to update business data should be cause for dismissal.
70. Hide nothing from the reviewers. Their reputation and yours is on the line. Expose all the warts and pimples. Don't offer excuses -- just state facts.
71. NASA is establishing a set of reviewers and a set of reviews. Once firmly established, the system will fight to stay alive, so make the most of it. Try to find a way for the reviews to work for you.
72. Knowledge is often confounded by test. Computer models have hidden flaws, not the least of which is poor input data.
73. Today one must push the state of the art: be within budget, take risks, not fail, and be on time. Strangely, all these are consistent as long, as the ground rules, such as funding profile and schedule, are established up front and maintained.
74. Most of yesteryear's projects overran because of poor estimates and not because of mistakes. Getting better estimates may not lower cost but will improve NASA's business reputation. Actually, there is a high probability that the cost of getting better estimates will increase cost and assure a higher profit to industry, unless the fee is reduced to reflect lower risk on the part of industry. A better reputation is necessary in the present environment.
75. A scientific proposal takes about 9 months to put together. It takes NASA HQ about 9 months to a year to select the winning proposals. Then, it takes 3 to 4 years to sell the program. This means 5 to 6 years after the initial thoughts, the real work starts. Managers, for some strange reason, do not understand why a scientist wants to build something different than proposed. Managers are strange people.
76. There are rare times when only one man can do the job. These are in technical areas that are more art and skill than normal. Cherish these people and employ their services when necessary as soon as possible. Getting the work done by someone else takes two to three times longer, and the product is normally below standard.
77. Software now has taken on all the parameters of hardware, i.e., requirement creep, high percent-age of flight mission cost, need for quality control, need for validation procedures, etc. It has the added feature that it is hard as hell to determine it is not flawed. Get the basic system working and then add the bells and whistles. Never throw away a version that works even if you have all the confidence in the world the newer version works. It is necessary to have contingency plans for software.
78. History is prologue. There has not been a project yet that has not had a parts problem despite all the qualification and testing done on parts. Time and being prepared to react are the only safeguards.
79. Award fee is a good tool that puts discipline both on the contractor and the government. The score given represents the status of the project as well as the management skills of both parties. The Performance Measurement System (PMS) should be used to verify the scores. Consistent poor scores require senior management intervention to determine the reason. Consistent good scores, which are consistent with PMS, reflect a well-run project, but if these scores are not consistent with the PMS, senior management must take action to find out why.
80. A project manager is not the monitor of the work but is to be the driver. In award fee situations, the government personnel should be making every effort possible to make sure the contractor gets a high score, i.e., be on schedule and produce good work. Contractors don't fail, NASA does, and that is why one must be proactive in support. This is also why a low score damages the government project manager as much as the contractor's manager because it means he is not doing his job.
81. There is no greater motivation than giving a-good person his piece of the puzzle to control but a pat on the back or an award helps.
82. Morale of the contractor's personnel is important to a government manager. Just as you don't want to buy a car built by disgruntled employees, you don't want to buy flight hardware built by them. You should take an active role in motivating all personnel on the project.
83. People who monitor work and don't help get it done, never seem to know exactly what is going on.
84. Never assume someone knows something or has done something unless you have asked them. Even the obvious is overlooked or ignored on occasion -- especially in a high-stress activity.
85. Don't assume you know why senior management has done something. If you feel you need to know, ask. You get some amazing answers that will dumbfound you.
86. If you have someone who doesn't look, ask, and analyze, ask them to transfer.
87. Bastards, gentlemen, and ladies can be project manager. Lost souls, procrastinators, and wishy-washers cannot.
88. A person's time is very important. You must be careful as a manager that you realize the value of other people's time, i.e., work you hand out and meetings should be necessary. You must, where possible, shield your staff from unnecessary work, i.e., some requests should be ignored or a refusal sent to the requester.
89. A good technician, quality inspector, and straw boss are more important in obtaining a good product than all the paper and reviews.
90. The seeds of problems are laid down early. Initial planning is the most vital part of a project. Review of most failed projects or of project problems indicates that the disasters were well planned to happen from the start.
91. A comfortable project manager is one waiting for his next assignment or one on the verge of failure. Security is not normal to project management.
92. Remember, the President, Congress, OMB, NASA HQ, senior center management, and your customers all have jobs to do. All you have to do is keep them all happy.
93. Always try to negotiate your internal support at the lowest level. What you want is the support of the person doing the work, and the closer you can get to him in negotiations the better.
94. Whoever said beggars can't be choosers doesn't understand project management. Many times it is better to trust to luck than to get known poor support.
95. Remember your contractor has a tendency to have a one-to-one interface with your staff; so every member of your staff costs you at least one person (about a 1/4 of million) on the contract per year.
96. There is only one solution to a weak project manager in industry -- get rid of him fast. The main job of a project manager in industry is to keep the customer happy. Make sure the one working with you knows that "on schedule, on cost, and a good product" -- not flattery -- is all that makes you happy.
97. Talk is not cheap. The best way to understand a personnel or technical problem is to talk to the right people. Lack of talk at the right levels is deadly.
98. Projects require teamwork to succeed. Remember most teams have a coach and not a boss, but the coach still has to call some of the plays.
99. In the rush to get things done, it is always important to remember who you work for. Blindsiding the boss will not be to your benefit in the long run. Over-engineering is common. Engineers like puzzles and mazes -- try to make them keep their designs simple.
100. Never make a decision from a cartoon. Look at the actual hardware or what real information is available, such as layouts. Too much time is wasted by people trying to cure a cartoon whose function is to explain the principle.
101. An Agency's age can be estimated by the number of reports and meetings it has. The older it gets, the more the paperwork increases and the less product is delivered per dollar. Many people have suggested that an Agency self-destruct every 25 years and be reborn starting from scratch.
102. False starts are normal in today's environment. More than ever, in this type of environment, one must keep an ear open for the starting gun and be prepared to move out in quick and orderly fashion once it is sounded. In the past, too many false starts have resulted in the project not hearing the real starting gun or jumping off and falling on its face.
103. The pioneering phase of NASA is mostly done, if not actually by fiat. This means the difficult and more important work has started. This work requires more discipline, but there should still be room for innovation.
104. There are still some individuals who think important decisions are made in meetings. This is rarely the case. Normally, the decision-makers meet over lunch or have a brief meeting to decide the issue and than (at a meeting called to discuss the issue) make it appear that the decision is made as a result of this discussion.
105. In political decisions, do not look for logic -- look for politics.
106. Interagency agreements are hard to make even if there is no conflict in the responsibilities and the requirements do satisfy both parties. Conflict in these areas normally leads to failure no matter how hard the people involved try to make an agreement.
107. In dealing with international partners, the usual strategy is to go 1 day early, meet with your counterpart, discuss all issues to be brought up at a meeting, arrive at an agreeable response (or a decision to table the issue for later discussion), and agree not to take any firm positions on any new issues brought up at the meeting. This makes it appear to the rest of the world that you and your counterpart are of one mind and that the work is in good hands. All disputes are held behind closed doors with the minimum number of participants.
108. Gentlemen and ladies can get things done just as well as bastards. What is needed is a strong will and respect -- not "strong arm" tactics. It must be admitted that the latter does work but leaves a residue that has to be cleaned up.
109. Though most of us in our youth have heard the poem that states "for want of a nail the race was lost," few of us realize that most space failures have a similar origin. It is the commonplace items that tend to be overlooked and thus do us in. The tough and difficult tasks are normally done well. The simple and easy tasks seem to be the ones done sloppily.
110. In the "old NASA," a job done within schedule and cost was deemed to be simple. The present NASA wants to push the start of the art, be innovative, and be a risk taker but stay on schedule and cost. One gets the feeling that either the new jobs will be simple or that the reign of saints has finally occurred.
111. Meetings, meetings -- A Projects Manager's staff meeting should last 5 minutes minimum -- 1 hour max -- less than 5 minutes and you probably didn't need the meeting -- longer than 1 hour, it becomes a bull session.
112. Taking too many people to visit a contractor or other government agency puts them in the entertainment business -- not the space hardware or software business.
113. Too many engineers get in the habit of supporting support contractors and of using them as a crutch. In many cases it is getting to the point where one has to wonder who is who.
114. Reviews, meetings, and reality have little in common.
115. You should always check to see how long a change or action takes to get to the implementer -- this time should be measured in hours and not days.
116. Let your staff argue you into doing something even if you intended to do it anyway. It gives them the feeling that they won one! There are a lot of advantages to gamesmanship as long as no one detects the game.
117. Some contractors are good, some are bad, but they seem to change places over time, making the past no guarantee of the future; thus, constant vigilance is a project requirement.
118. It is rare that a contractor or instrumentor does not know your budget and does not intend to get every bit of it from you. This is why you have to constantly pay attention to the manpower they use and to judge their activities in order to assure that they are not overloading the system.
119. People tend to ask for what they think they can get and not what they need. On GRO the specs for photomultiplier tubes were based on the engineering units performance on all parameters. One parameter, though made in the engineering tubes, was difficult to obtain in the flight tubes.
120. It was a meaningless parameter put in only because the engineering tubes met it. Finally, after about 9 months of sweat and tears, this was recognized and deleted so we could get the flight tubes.
121. Today one must get an honest bid -- one which is accurate to 15 percent. On GRO, with TRW the only bidder and with them knowing it, we all got what we believed to be an honest bid that was off by about 18 to 20 percent at the finish. The main area of overrun was the structure. TRW had never built one this large or heavy before. We estimated that the structure would require 600 drawings, multiplied this by 1.25 to get 750 and rounded to 800 to estimate the cost. It took 1,186 drawings. It is normally not the complex systems that get you, so beware when you estimate the cost -- especially if there is no experience base.
122. Too much cost data on a proposal can blind you to the real risks or forgotten items. On a project we thoroughly knew, we spent 6 months of government and contractor time validating the cost, had rooms full of data, and presented our findings to Headquarters. Two weeks later, the contractor found an "Oh I forgot" that costs $30 million. One should look at how past programs spent their money to try to avoid these traps.
123. On GRO we sort of estimated we needed about 20 percent contingency on previously flown subsystems and about 40 percent to 50 percent on new ones. The ratio was about right except the order was reversed.
124. There are some small companies that make the same subsystem correctly every time because the same people do it. There are some large companies that can never make the same unit correctly every time because different people do the work each time. Heritage should be questioned when the people doing the work all have peach fuzz on their faces.
125. Too many project managers think a spoken agreement carries the same weight as one put in writing. It doesn't. People vanish and change positions. Important decisions must be documented.
126. Make sure everyone knows what the requirements are and understands them. Much easier to say than do. On GRO we stated quite clearly that the scientific instruments had to take 18g in a specific axis. Everyone understood the requirement but until the mechanical test on EGRET no one stood up and said it was impossible to meet it. The thermal specification for the momentum wheels required that they run 5 degrees colder than normal limits to make the spacecraft thermal engineers life easier. No one stood up until after 9 months of failure in the test program to say that the grease used changes state if taken that cold, and would not recover when brought back to higher temperature. You have to have the right people look at requirements. A bunch of managers and salesmen nodding agreement to requirements should not make you feel safe.
127. Too many people at Headquarters believe the myth that you can reduce the food to the horse every day till you get a horse that requires no food. They try to do the same with projects, which eventually end up as dead as the horse.
128. The project manager who is the smartest man on his project has done a lousy job of recruitment.