Category: Software Testing

Everyone On The Delivery Team Is Responsible For Quality


When it comes to software quality assurance, most people think of phase at the end of a project. This testing phase is usually performed by “quality assurance” testers. The traditional project delivery model is to blame for this. As someone who worked as a software tester for years, I know about the pains of doing all testing at the end.

The problem of course is that when testing happens at the end, after a development phase, it is too late. What you are doing is trying to inspect quality in, instead of building quality in.

Lean manufacturing and lean software development is about building quality in. This was one of W. Edwards Deming’s, a pioneer in the lean and quality movement, motto’s . To build quality in, testing should take place all throughout the project. Testing early reduces the cost of defects because the earlier you catch defects, the cheaper they are to fix.

For this reason, it is well advised for software delivery teams to apply lean principles and automate as much as possible. Automation shouldn’t only be a set of regression tests that run at the end. Automation should take place all throughout the entire software development and release process.

Automation can begin in development using test driven development (TDD). For more on TDD, see my post on why automated testing is needed early and often.

After TDD, automation can be applied from build and deploy to acceptance tests.

This brings us to the process of Continuous Delivery. Continuous delivery is all about creating reliable software releases through build, test, and deployment automation. Today, more software organizations are reaping the benefits of continuous delivery, yet for most it’s still a pipe dream.

People think the idea of automating everything is too daunting, or can’t be done for their application. More and more we are now seeing that continuous delivery can be done, for any application or organization, no matter the size. A good way to start is by looking at your current release process, and finding the biggest bottlenecks. You can then start to automate over time, targeting the bottlenecks first.

By automating the whole software release process, you build quality in. This makes everybody on the delivery team responsible for quality, not just a “QA” team who tests at the end. For manual software testers, continuous delivery enables them to focus on exploratory testing.

There are many other benefits to TDD and continuous delivery, but perhaps the greatest is the reduction of stress for delivery teams. If you’ve gone through a production deployment process, using manual processes and ad-hoc configuration techniques, you know what I’m talking about.

For those who already use lean development techniques this information shouldn’t be news to you. If you still work in an environment that uses SDLC processes with little to no automation, the topics I’ve covered only scratch the surface.

Some great books I recommend on lean software development and continuous delivery:

By using continuous delivery and automating as much as possible, everyone on the delivery team is responsible for quality.

About the Author: Mike MacIsaac is the president and principal consultant for MacIsaac Consulting. Mike provides leadership as an IT Project and Program Manager as well as an Agile Scrum Master. Follow Mike on Twitter @MikeMacIsaac or subscribe to Mike’s blog.

 

Going from Software QA to IT Project Management

Software QA

My journey in the world of IT began as a quality assurance analyst. For UPS, I would test and certify applications used in operations. I spent most of my time in the test lab with the rest of the lab rats. Testing software back then was the easy part, the hard part was building the right platform. Performing a ground up install of MS Windows Pro could take almost half a day and if it failed, you’d have to start over again. If you were lucky you got to use a ghost OS image, but even those presented their own challenges.
 
Later in my career I would go on to focus less on hardware and more on testing software. With Accenture I worked along side large testing teams on huge multi year waterfall software projects. We would create test plans consisting of hundreds of test scenarios. We then charged off to complete all the testing in a matter of weeks. Anyone who worked in QA (or dev) can attest to the agony of the waterfall project death march experience. Pressure from management, late nights in war rooms, and working weekends combined to make the QA role a nightmare.
 
Fast forward to the arrival of Agile and my role in QA became tolerable again. Instead of having a mountain of code dumped on me for testing, now I could test small chunks of software in two week sprints. The only problem was (and still is) that many organizations struggle building shippable software in sprints. This results in companies that are waterfall masked as Agile.
 
After almost a decade working in QA I decided I to get into project management. I always enjoyed connecting with people and I felt project management better fit my strengths. Through the guidance of mentors, lots of studying and hard work, I was able to become an associate project manager. After time I was able to work on larger projects, consulting for large companies.
 
Making the transition into project management was a great career choice for me. Project management much better aligns with who I am. For you it may be different. Your passion might be in QA and there’s nothing wrong with that.
 
If you are someone who works in QA and you’re considering get into project management, my advice is to go for it. Software QA analysts make great project managers because they are battle tested. I discussed this more in my last post “The identity crisis of the IT project manager“.
One thing’s for sure, if you’re not happy in your current role, don’t settle!

About the Author: Mike MacIsaac is the owner and principal consultant for MacIsaac Consulting. Mike provides leadership as an IT Project and Program Manager as well as an Agile Scrum Master.

Follow Mike on Twitter @MikeMacIsaac or subscribe to Mike’s blog.

What happens to UAT in the Agile world?

uat

UAT (user acceptance testing) is often the final activity that takes place in a software development project. Typically, a group of system users will test to ensure all works as expected and there are no missed defects. The users are usually non technical, they are actual users of the product in production. For example, a new code change to a POS system might need a cashier to perform some of the user acceptance testing.

Unlike traditional waterfall projects, Agile methods delivers software in iterative chunks. The Scrum framework develops production ready software in 2-4 week increments (sprints). Each sprint ends with a demo or review with the customer to ensure the product is right. The stakeholders are heavily engaged during the software development process.

This leaves us to beg the question, is UAT required in the Agile world? As someone who spent many years as a software tester before working as a Scrum Master, here’s my take. For most projects I think there is tremendous value in having the product users do some testing. In Agile this may look a little different because you won’t have a UAT phase, but the principle is the same. Let the actual users of the product give a test run before you ship to production.

Could you imagine if Boeing put planes into production before pilots performed test flights? I don’t think you’d ever hear Boeing say, “our QA signed off, but we never had actual pilots do any testing”.  Of course not. The same concept should hold true for most software development projects.

In the Agile world, users can perform their UAT scenarios during the Agile sprints. The users can partner with the dev and QA resources for help and guidance.

Now, here’s the caveat. There are some occasions where UAT may not be required. It’s up to the team and project to make those decisions. For minor enhancements and defect fixes, etc, product demos alone may be enough. For more complex software projects, I think you’d be well advised to not forgo the value of UAT. What do you think?

For more content like this, subscribe to the MacIsaac Consulting Blog.

To contact us about our services, click here.

 

System integration testing, a 6 step survival guide

system integration testing

So you’ve gotten through your planning, design, development and application testing. Everything looks good, the team feels confident, and the project is on track. There’s only one problem. Up until this point you’ve been looking at your code changes with tunnel vision. Application testing is like fixing a bike tire while it’s not connected to the bike. The real testing begins when the wheel gets reconnected to the bike. Once reconnected, then you’ll see the how the fixed tire works with all other parts of the bike.

I’ve been through many large software projects, both as a project manager and as a tester. Often the most difficult part of completing a software project is end to end system integration testing. This is particularly true if you’re not using an Agile model.

System integration testing is where the rubber meets the road. If you work for companies like Facebook, Spotify, or Twitter you are lucky. These companies use modern Agile software delivery methods that make integration testing less painful. Automation, DevOps, and continuous delivery is the way of the future.

Unfortunately most companies still build software the old fashion way. They create one huge pile of software then throw it over the fence to QA nearing project end. This leaves integration testing only a matter of weeks to complete and meet a tight deadline.

If you don’t do a great job planning for integration testing, you’ll be in for real trouble. If you do a great job planning, you’ll still most likely be in for some pain.

Here are 6 steps to help you survive system integration testing so you can be successful.

  1. Plan enough contingency time – This might be the most important point. Give the team enough time to do the test effort. If you know it takes 1-2 weeks just to get your environment and data working right, plan for that time. In system integration testing, many roadblocks occur. Data, configuration and environment issues can  block testing. Plan for the unexpected.
  2. Create test scenarios with the help of system experts – Once you know all the affected systems, find out who the technical experts are for each system. Then schedule white board meetings to draw out end to end flow test scenarios. The system experts will be invaluable helping you to write end to end scenarios.
  3. Get all your data setup in advance – There’s nothing more frustrating than finding out none of your data works on day 1 of testing. Setup your data in advance and check to see its setup right before you do your testing.
  4. Run connectivity tests – For large projects, take at least 1 week to test environment connections before the start of your SIT. Often companies use shared testing environments and configurations are always being changed. Make sure all your configurations and connections are setup. If you don’t do this, you’ll think you’re running into defects only to discover an application is pointing to the wrong environment.
  5. Use a war room – All throughout system integration testing you should have a room where all key resources sit together, all day. The key players usually consist of developers, testers, a PM, and system and environment experts. As soon as a test runs into an issue, a developer should be there to look into it immediately.
  6. Have a test lead manage the effort – Last but not least is having a good test lead. In my opinion, test leads have one of the most difficult jobs when it comes to managing integration testing. The job requires both testing expertise, project management skills, and tremendous grit. Often test leads have a lot of pressure put on them to meet deadlines. A good test lead will rise to the challenge.

For more content like this, subscribe to the MacIsaac Consulting Blog.

To contact us about our services, click here.

Root cause analysis – Why toddlers are better at it than us

root cause analysis

If you’ve ever talked to a toddler, you’ve been hit with an endless amounts of whys…

Dad, why do you go to work? So I can make money. Why do you need to make money? So we can eat. Why do we need to eat? And on and on it goes.

Well, it turns out we need to learn from toddlers. Their relentless, and annoying, asking of why is what we should be doing for root cause analysis.  Working in technology, we deal with problems all the time. When we’re confronted with problems, the majority of us don’t take the time to do a good root cause analysis. This results in problems coming back to haunt us.

For example, say an application stops working. The team begins to triage the issue, and discovers a service stopped running. The service gets restarted, application relaunched and all is well. Or so the team thought. The next day the application stops working again. After inspection, the team finds again the agent needed to be restarted.

The agent stopping was only a symptom, and not the root cause. This is why we need to use “the 5 whys” when confronted with a problem. By asking why 5 times, you’ll dig deep enough to get to the core of the problem.

Repeating why five times is a process that was first used by Taiichi Ohno in the Toyota Production System. It is now a common practice in Lean software development.

In the example I gave above, the 5 why scenario might look like this:

  1. Why did the application stop working? Because the service stopped.
  2. Whey did the service stop? Joe performed an install and forgot to restart all the agents.
  3. Why did Joe forget to restart all the agents? Because there is no documented process about restarting agents.
  4. Why is there no documented process? Nobody took the time to do it.
  5. Why didn’t anyone take the time to do it. Everyone is busy fighting fires, nobody has the time.

In the above scenario, by asking why 5 times you get to the root of the problem.

Creator of XP Kent Beck writes: “After Five Whys, you find the people problem lying at the heart of the defect (and it’s almost always a people problem).

For more content like this, subscribe to the MacIsaac Consulting Blog.

To contact us about our services, click here.

References: Beck, K (With Gamma, E.). (2005) Extreme Programming Explained, Upper Saddle River, NJ, Pearson Education, Inc

 

 

 

 

Automated testing – The bottom line why it’s needed early and often

You need automated testing early and often because it reduces the cost of defects. “Defect Cost Increase (DCI) is one of the few empirical verified truths about software development: the sooner you find a defect, the cheaper it is to fix”(Beck, 2005). In a traditional waterfall project, all testing happens at the end of the project.

The waterfall based late testing approach results in high defect cost increase. As a former SQA analyst, I have experienced the pain that goes along with this process. Late nights working in WAR rooms, constant stress and continual red status reports. It’s the classic project death march experience.

In Agile and Lean software development, testing happens early and often. The team builds working software in short iterations. Testing is part of the software development process. In the below caption, Figure 14 shows the waterfall style of testing, compared to figure 15 which shows a more agile testing process.

FullSizeRender

One of the more popular and effective approaches is test driven development (TDD). Using TDD, the developer will first write an automated test. This test will ensure the basic functionality of the code works. After writing the test, he’ll first run it to see that it fails. Then, he’ll write his code, and run the test again to ensure the test passes.

By using TDD, the developer has already taken steps to ensure quality is built up front. The code then goes to QA, where integration, user experience, or performance testing happens. Just as the developer has created his automated test, so should the QA resource.

A high performing QA team will have an automated regression suite ready to run as soon as new code is ready. It is crucial that core functionality regression tests run first, before new feature testing. The idea is, make sure the new code hasn’t broken anything first, then begin the new feature testing. Most organizations have this backwards, doing all the regression testing at the end.

To put this all together you need the right tools, technology and resources. Whether it be JUnit, QTP, or Selenium, find the automation technology that’s right for your organization. Remember, the goal is to reduce defects and their cost.

Many organizations have avoided implementing test automation due to the expense. Ask any software quality team that uses automation and they’ll tell you the ROI is well worth it. If your organization builds software, you must use test automation. By not using it, you’re operating on an unsustainable model. It’s unsustainable because your regression test suite will continue to grow.  You cannot sustain a growing regression suite without automation.

For more content like this, subscribe to the MacIsaac Consulting Blog.

To contact us about our services, click here.

References: Beck, K (With Gamma, E.). (2005) Extreme Programming Explained, Upper Saddle River, NJ, Pearson Education, Inc

 

 

 

 

 

Quality and the consumer – The 3 corners of quality

quality

Defining what quality means is difficult. On this subject, the great statistician Walter Stewart stated: “The difficulty in defining quality is to translate future needs of the user into measurable characteriscts, so that a product can be designed and turned out to give satisfaction at a price that the suer will pay” (Shewhart, 1931).

In software development, we tend to focus on a series of defined tests. If all our tests pass, we have developed a quality product ready to be shipped. If the product is shipped and the result is a high volume of production defects, the product was not of good quality.

The challenge with the old way of Waterfall software delivery, and only relying on our tests,  is that the consumer of the product is involved too late in the product development process. The consumer is perhaps the most important aspect of developing quality software.

For this reason, we need to involve the consumer all throughout the development process. This way, we build quality up front.

The below figure shows the three corners of quality model put forth by Edwards Deming.

drawit-diagram-4

What we have learned with the application of Agile, is the importance of learning from the customer. Deming talked about this back in the 1970s, but it wasn’t until the early 1990s that it was applied for software development.

On learning from the consumer, Deming writes: “The main use of consumer research should be to feed consumer reactions back into the design of the product, so that management can anticipate changing demands and requirements and set economical production levels. Consumer research takes the pulse of the consumer’s reactions and demands, and seeks explanations for the findings” (Deming, 1982).

What Deming describes is exactly what takes place during the Scrum sprint reviews. During the sprint review, we show production ready software to the consumer. The consumer can then give immediate feedback. This allows us to make changes to the design of the product.

The old way of delivering software looked like this:

drawit-diagram-5

The new way of delivering software using Agile ensures we build quality into the product up front. We build production ready software in short iterations, then review it with the customer. The new way of delivering software looks like this:

drawit-diagram-6

For more content like this, subscribe to the MacIsaac Consulting Blog.

To contact us about our services, click here.

 

 

 

 

 

Edwards Deming and quality control

W._Edwards_Deming

W. Edwards Deming was a quality theorist, statistician, author and management consultant. He was influential in the practice of quality management. Deming’s most urgent message was for management. He stressed that management create a system where each worker could do a good job and take pride in their work.

What is quality? The International Organization for Standardization (ISO9000) defines quality as “The degree to which a set of inherent characteristics fulfills requirements” (Project Management Institute, 2013, p.228). Quality is one of the most important factors for any business to be successful.

“Quality is more than just a statistical analysis tool for manufacturing lines. When done right, quality should encompass the entire enterprise” (Stevenson, 2009, p.406). An increase in quality results in better productivity, higher morale, and increased customer satisfaction.

Edwards Deming was a visionary who understood the importance of quality management.  His concepts helped Japan rise to become a world economic power.

History

Like most of the major contributors in the area of quality, Deming studied engineering. He would go on to get a PhD in mathematical physics from Yale in 1928. Deming was a professor of statistics at New York University’s graduate school of business. He later went to Japan after WWII to help improve their quality and productivity. He used statistical analysis and a systems approach towards quality.

Deming’s teachings and models were successful in Japan. To this day the Japanese give out an award for quality named the Deming prize. One of the causes for Deming’s popularity was his list of 14 points for quality. He was also an advocate of the Plan-Do-Check-Act (PDCA) cycle.

Deming lived from 1900 – 1993.  His theories had great success in Japan, but he didn’t have much popularity in the US until the 1980s. In the 80s he worked with American Automobile companies to help improve their quality.

Management and the system

One of Deming’s key messages was that the cause for poor quality was not the employees, it was the system. He emphasized that it was management’s responsibility to fix the system. “According to Deming, 85 percent of the quality problems on a project are attributable to the management environment and the system in which the team works” (Mulcahy, 2013, p. 297).

Appreciation for the system refers to everyone within a company working to achieve optimization. “Deming believed that knowledge comes from theory, and that learning cannot occur within an organization without a theory of knowledge” (Stevenson 2009, p 409).

Deming was an adversary of the annual performance review process within companies. He thought the arbitrary annual review process was managing by fear. He believed it robbed employees of their internal motivation. Deming believed managements greatest challenge was motivating workers to achieve a common goal.

The Plan Do Study Act (PDSA) Cycle

The concept of the PDSA cycle was introduced to Deming by Walter Shewhart of Bell Laboratories in New York. The first step in the process is to put together a plan by formulating a theory and creating success metrics. Next comes the execution of that plan (the Do step). Following execution is studying the results. The last step is to act based on the results found and put in place adjustments if needed.

The Five Deadly Diseases of Management

Edwards Deming recognized five deadly diseases of management which impede quality improvements. The five disease are as follows:

1.      Lack of constancy of purpose. This means that leadership needs to understand exactly why they are in business. They must have a clear mission that has a market for the future and is understood throughout the organization. Firms need to plan for the future and have a long term definition of goals.

2.      Short term thinking and emphasis on short term profits. Too much focus on quarterly dividends and raising the company stock. Not enough focus on quality and long term planning.

3.      The annual system of rating/performance review. The yearly employee review has a negative effect on long term planning as well as team work. Employees get bitter when they do not receive high performance ratings. The system is arbitrary, unjust and demoralizing to employees.

4.      Management that has no roots in the company. They lack knowledge and understanding of operational problems.

5.      Use of visible figures only for management. Management needs to  understand there are  unknowable figures.  Understand the importance of  the multiplying effect of a happy or unhappy customer (Deming, 1984).

14 Points for Quality

Perhaps the most well know of Edwards Deming’s teachings were his 14 points for management. Below are the 14 points as stated from the Deming Institute (https://www.deming.org):

1. Create constancy of purpose toward improvement of product and service, with the aim to become competitive and to stay in business, and to provide jobs.

2. Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.

3. Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.

4. End the practice of awarding business on the basis of price tag. Instead, minimize total cost. Move toward a single supplier for any one item, on a long-term relationship of loyalty and trust.

5. Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.

6. Institute training on the job.

7. Institute leadership (see Point 12 and Ch. 8). The aim of supervision should be to help people and machines and gadgets to do a better job. Supervision of management is in need of overhaul, as well as supervision of production workers.

8. Drive out fear, so that everyone may work effectively for the company (see Ch. 3).

9. Break down barriers between departments. People in research, design, sales, and production must work as a team, to foresee problems of production and in use that may be encountered with the product or service.

10. Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.

Eliminate work standards (quotas) on the factory floor. Substitute leadership.

Eliminate management by objective. Eliminate management by numbers, numerical goals. Substitute leadership.

11. Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality.

12. Remove barriers that rob people in management and in engineering of their right to pride of workmanship. This means, inter alia, abolishment of the annual or merit rating and of management by objective.

13. Institute a vigorous program of education and self-improvement.

14. Put everybody in the company to work to accomplish the transformation. The transformation is everybody’s job.

For more content like this, subscribe to the MacIsaac Consulting Blog.

To contact us about our services, click here.

 

Stevenson, W. (2009). Operations Management, New York, NY: McGraw-Hill

Mulcahy, R. (2013) PMP Exam Prep, Minnetonka, MN: RMC Publications, Inc.

Project Management Institute (2013) Project Management Body Of Knowledge (5th ed.). Newtown Square, PA: Project Management Institute

Deming, E. (1984) Dr Deming – The Five Deadly Diseases [online Video]. Retrieved from the Deming Institute YouTube website: https://www.youtube.com/watch?v=ehMAwIHGN0Y

Deming, E. (1984) W. Edward Deming Quality Guru [online Video]. Retrieved from YouTube at https://www.youtube.com/watch?v=YQpY3lnljBE

5 attributes of the best manual software QA testers

software testers

With the increase of test automation, let’s not forget the importance of manual testers. No matter how advanced test automation becomes, we will always have a need for manual testers. As someone who worked in software QA, I know how important good manual QA resources are. They can be the difference maker on a project. Here are 5 attributes that make up the best manual software testers:

  1. They are technical – Just because they are not writing code does not mean they are not technical. The best QA analysts are technical.The more technical they are, the better. They have no problem digging through logs, working with XML and using SQL. SQL is a software QA analysts best friend. Before they throw defects over the fence to dev, they do their due diligence to pinpoint the root cause.
  2. They log clear defects – It takes time to log clear defects. This is when QA needs to partner with dev and provide dev with as much helpful information as possible. A defect not written well will say something like “login is not working”.  A well written defect provides a clear description, screenshots and steps to reproduce. If available, error or log information will also be in the defect.
  3. They take ownership and drive resolution – Testers run into many road blocks. Data, environment, and configuration problems happen all the time in test environments. Average  QA analysts will give up when they hit a snag. They will look to the project manager or someone else to get things unstuck. The best QA analysts will take ownership of the issue and drive resolution. Whether they have to track people down or escalate, they get things resolved.
  4. They emphasize end to end testing – It’s easy for testers to focus only on their application. It’s harder to analyze all the system impacts of a code change and perform a full integration test. The best QA analysts want to test the full end to end user experience.
  5. They won’t compromise on quality– Software testing is tough work. The testers are usually under a lot of pressure to complete their testing. Team leads and project managers breath down their neck and put the heat on. Particularly in the traditional waterfall SDLC, testers get the short end of the stick. Under this type of pressure, there is temptation to cut quality to appease project leads. The best testers will never do this. No matter how much pressure they face, they will not put their stamp of approval until they feel code is ready. They take pride in their quality standards, which is what makes them the best.

Powered by WordPress & Theme by Anders Norén