As testing professionals who make a living looking for bugs, it can feel even more daunting when the software problem comes from a mistake that was our fault.
Most of the time, it’s easy to dust yourself off and move on while learning from the situation, and yet, sometimes, a mistake will haunt you forever. It’s also a blunder that makes you double-check in the future while becoming a better software quality champion.
We asked some of our nearest and dearest in the Provar Community about the testing mistake that still haunts them. In addition, we will share five tips to help make future testing less scary.
Remember User Profiles
Not every new feature will be available for every user. That’s why it’s extra essential to make sure you’re checking that end users have access to the features that they are anticipating. The feature you’ve created might work beautifully, but can everyone who should be able to access it use it?
We created a new product for a lower price with fewer features. During go-live, we pushed the new access control on the prod DB – to realize no customers could access their service. The call center experienced it and was swamped. Luckily, it was a DB fix to update. — Jesper Ottosen, 20+ years of testing
Don’t forget to check access and user profiles as an end-to-end test to verify that all your hard work can be seen.
Ask the Difficult Questions
You should never assume. Test the tester. All of these things are good advice. Maybe the most important thing about testing is if you see something weird, ask questions … lots of questions.
The memory that haunts me to this day regarding testing had to do with interactions with people. We had a report of fraud on a test account in production. I was asked to investigate this further. I dug in deep and found a handful of things we needed to address, but we ran automation tests in production every 15 minutes. We had a test case transacting real company money for $2 in that test plan. Typically, we reverse any transaction performed in production, so real money doesn’t get transacted. But, this particular automation test wasn’t performing a reversal. Looking into the history, this has been running for the past 3 years.
Doing some math:
60 minutes / 15 frequencyOfTest = 4 times per hour
4 * 24 = 96 times a day
96 * 365 days in a year = 35,040 times a year
35,040 * 3 years = 105,120 times. This test has been executed in the last 3 years
105,120 * $2 = $210,240 that has been wasted of our company’s money just executing a test case
The first thing I did was get our automation team to stop that automation from continuing to trigger every 15 minutes since there wasn’t a need for that. From here, I took this to my QA Director, our Engineering VP, and we talked to our fraud department and some managers in finance. No one knew the correct approach to address this and reverse these transactions since they had already been posted, and many of these were years ago. I then emailed my Finance Executive, CC’d my Director and Engineering VP, and explained the scenario in depth. I wrote to all of the parties I’ve been in touch with that we were out $200k if we didn’t find a resolution for this. The response I got will forever frustrate me … “Talk to your boss.” — Joe, 4 years testing.
You might not get a helpful answer, but at least you’ve learned to keep an eye out!
Setting Yourself Up for Success
Not every day is a peak mental acuity day. Maybe you were up late the night before, helping your kids with their science project or watching the last few episodes of your favorite TV show. Personal distractions can drain our focus just as quickly as a heavy workload.
On one of my last major projects, I often tested late at night when the devs had pushed changes and UAT was up-to-date. Anyone who knows me can attest that my brain has run out of usable brain cells by then, so I’m not sure why I ever thought this was a good idea. This particular sprint’s story was only a net-new addition to existing functionality. Nothing should have changed, so in my haste to complete the work, I gave my tests a run and passed them with flying colors.
The following day, I woke to an email from the BA asking if I had passed the wrong story because he’d found many bugs and logged against it. The new changes broke the old functionality, which I had not looked at, and tumbled everything that relied on those previous paths. Oops. — Tara, 10 years testing
Scheduling out tasks so you’re not in a rush is always a recommended practice. Set yourself up for success by ensuring you have the right resources and time to do the job.
Use All the Test Steps
I know complacency is one of the greatest enemies of testing, especially when we’re doing the same tasks repeatedly. It’s easy to cut corners or to think you did something because you’ve done it a million times.
I forgot to change the emails of contacts in a Sandbox and sent emails to all contacts while doing regression testing. It was all contacts. — Andrew, 7 years in software quality
Make sure you follow your test steps to the letter. At best, you might add a variable you don’t need, but the worst-case scenario could be a public blunder.
Own It
Tests pass for years without so much as a thought. Until they don’t. I once joined a team that assumed their tests had passed, but no one was looking at the output to see over 70% failure rates.
The year was 2009, and I had recently joined the first IT firm as a software tester. Working as a [beginner] in the software testing industry, I was assigned testing of a mail utility for a large banking project. Unbeknownst to me, the testing process included a detailed and lengthy mailbox setup on a local server. The weather in Bangalore was gloomier than usual, and my friends started logging out of the office as the darkness of the night descended upon us. I asked a senior tester, “Do you know how to set up the mail server for this test case?” he responded: “Don’t worry at all; that test case is pretty old and always works, so go ahead and mark it passed.” At that moment, I made one of the testing mistakes, which still haunts me. I marked the test “passed” without actually running it … the next day, we had an unusual meeting in which our manager highlighted that the mail utility test had failed in the latest build and the test case was not working. — Robin, 13 years testing
A test that no one checks the outcome of is not a test at all. It’s just a time-wasting activity.
It’s Not So Scary
Testing isn’t always so scary, and even the most seasoned professional can have questionable moments that will haunt them. Take heed and remember no mistake cannot be recovered from. As John Wooden reminds us, “The team that makes the most mistakes usually wins because doers make mistakes.”
I like to remind people that it’s always important to do things at the right pace to help avoid the worst mistakes, but to err is human. We hope this resource reminds you to share your mistakes openly (even the scary ones), and help build a culture of continuous and inclusive learning for your teams.
Have a testing mistake that still haunts you? Share this spooky testing resource on social and let us know your story! For more on Provar Community and various test strategy resources, including its Community forum, visit the Community page.
