All About the Apps

Right on, Brighton! TestBash Brighton 2016 Recap

Right on, Brighton! TestBash Brighton 2016 Recap



While I was in the UK for the Internet of Things Forum, I took the opportunity to go down to Brighton for the one-day TestBash (Brighton) 2016 Conference, run by the Ministry of Testing. I got to meet a lot of inspiring people, and hear some fantastic talks about software testing. These stories ranged from being the sole tester in a product team, to the challenges of testing smart algorithms. Here’s my summary of the conference (the videos of the presentations should be available soon at the TestBash Dojo).


The Brighton DomeThe Brighton Dome

The conference was held at the Brighton Dome, and was actually the last day in a week of software testing-related activities, including training courses and a series of half-day workshops on test automation, architecture, agile testing, load testing, API testing, and lots of other essential topics.

After registration, and meeting up with my colleagues Simon Norrington and Kashif Husain, the first item on the agenda was attending Lean Coffee, where people get together in small groups to talk about whatever topics they cared to raise. Helena Jeret-Mäe put up a list of the topics on GitHub (with word clouds!), which offered a fascinating insight into the minds of the people doing testing today.


Although the conference was sponsored (Hewlett Packard Enterprise had the honor of being one of the sponsors), there were no booths to see product demonstrations, which is common in larger conferences. At TestBash, everyone is encouraged to use the breaks for networking, to meet new people and talk about common interests.

The event was chaired by Vernon Richards, who did an amazing job, keeping everyone on schedule, and making us all feel welcome and at home.

Building the Right Thing: How Testers can Help

Lisa Crispin and Emma Armstrong started their session by giving everyone a blank piece of A4 paper, and challenged us to make something that flies through the air. That was all the information we were given. Some people created elaborate paper airplanes, while others just scrunched it into a ball. The point of the talk was about understanding what to deliver, and getting to the end-game gradually while continuously delivering real value to the user. Emma used Henrik Kniberg’s example of building a car:

Lisa Crispin and Emma ArmstrongLisa Crispin and Emma Armstrong

You could start with a tire, and build up to a full car via an axle, then a car body, then the car. But instead think about delivering something real at each stage: a skateboard, then a scooter, then a bike, then a motorbike, then a car. That’s much more useful, because people will use them and give you concrete feedback. Rather than the product owner starting with the problem that needs to be solved, we often see a description of an implementation. This ends up with a lack of shared understanding in the team, and wasted time in rework down the line. Testers tend to think in terms of the bigger picture, this approach challenges the team to get a better understanding of the purpose of the feature, or product. A key message from this experience is to experiment, measure, and learn, and engage the whole team while building in quality.

Testing or Hacking? Real Advice on Effective Security Testing Strategies

Dan Billing.jpg

Dan Billing made everyone sit up by announcing that our apps and systems are being hacked RIGHT NOW! Everyone is at risk from hackers, which means that security must be part of the conversation. Responses such as “security testing is out of scope”, “we don’t have time for that”, etc., are not acceptable. He recommends thinking of hackers as users – after all, they are using your system, although perhaps not in ways you intended.

Rather than thinking of security as a non-functional issue, it becomes very functional. You need to start to think like a hacker, just as you think of your other end users. He suggested ten things to consider in security testing:

  1. Consider the scope
  2. Know your technology stack
  3. Understand your weaknesses
  4. Power up by using tools…
  5. …and use these tools effectively
  6. Scan, verify, explore – again, and again and again
  7. Be (occasionally) evil
  8. Don't do it alone
  9. Be clear, be heard
  10. Be determined.

A Pairing Experiment

Katrina Clokie has the most amazing slides! She draws them by hand, and they look great. But they’re also really good at helping her to deliver her message.

Katrina Clokie and one of her hand-drawn slidesKatrina Clokie and one of her hand-drawn slides

She made the journey all the way from New Zealand to talk about how she introduced pair testing at work, and coached her testers through the process of learning about pair testing and becoming effective with it. Some trial and error with the team resulted in an effective implementation of pair testing which involves:

  • Pair testing sessions that are one-hour long
  • There is one session every two weeks
  • Pairing should be cross-team
  • The session must be hands-on, with both testers taking turns at the keyboard

She started with an experiment, and advised the audience to give it a try. You can learn more about pair testing on her blog.

Model Fatigue and How to Break It

John Stevenson started out with some techno music. His talk was about how we adopt testing models, yet just stick with the same old concepts. We have the option of turning things about, changing things, challenging things, adapting things – but most of us don’t move out of our comfort zone. Just like the techno music that he started with, you CAN remix your approach to testing. He advises using things like SCAMPER (Substitute, Combine, Adapt, Modify, Put to other use, Eliminate, Reverse) and FCC CUTS VIDS for different tours to take through the application (Feature, Complexity, Claims, Configuration, User, Testability, Scenario, Variability, Interoperability, Data, Structure). Take a model, change it, and make it your own. And change it up again. Just do it, you can take the blame (or credit) afterwards. You’ll end up finding more bugs, and you might even find live bugs that your customers haven’t yet discovered.

Accepting Ignorance – The Force of a Good Tester

Patrick Prill showing us how much we don't know about ignorancePatrick Prill showing us how much we don't know about ignorance

Patrick Prill started his talk by citing James Bach’s fascinating conversation with a student on the definition of integration testing. He asked a number of follow-on questions, about the definitions of common terms such as ‘test case’, ‘performance testing’, ‘quality’, ‘testing’, and ‘agile’. He showed us the definition of the word ‘ignorance’ – “Lack of knowledge or information”, then he went through the definitions of each word in the definition a few times, to come up with “To be aware that you have insufficient facts, skills or understanding about a topic and remain curious to constantly improve and further discover what you don’t know.” This was a great example of a deep dive that surfaced facets of a concept that you might not be aware of. While this happens all the time in testing, and we need to at least be aware that we’re ignorant. As Socrates said, “the only true wisdom is in knowing you know nothing”. So challenge that ignorance: perform risk assessments and test planning, evolve test charters, do deep testing, and report on what you tested – and on what you didn’t. You need skills and knowledge to explore the unknown, and you need to find the borders of your skills and knowledge. Most importantly, you need to make the decision to improve.

Test/QA – A Gatekeeper’s Experience

Michael Wansley (you might know him as Wanz, or TeeWanz) is a Grammy award winning hip-hop artist. But he’s also a software tester. His talk, which was delivered without any slides, was about the controversial concept of the tester being the gatekeeper. He was speaking from his experience at Microsoft, where he was a tester on the Windows team, working on numerous versions from Window 95 to Windows 7. Whether or not you identify with the tester as a gatekeeper, his message was relevant to everyone. As a tester, you need to know what’s going on with the product, which should be, as Wanz’s boss once put it, “so simple that my grandmother could use it.” You need to understand where you are in the process, but remember that it’s all about the user. Testers are power users, but most users aren’t. Look at the product you’re testing from their perspective.

Having All Your Testers Code: It doesn't have to be a Big Deal

Anna Baik  and Andrew Morton tag-teamed about their experience of introducing test automation into their company before funding ran out. The company was looking to scale, but they could only do that with full stack automation. To understand more, one stayed working on the sprint, on their own, while the other did some experiments. But they swapped regularly, because if you have one expert on your team, you want to ensure that they don’t remain the only expert on the team. They also had a problem with back seat drivers, and lots of people wanting to control the team’s direction. To resolve that, they got support from management, and hammered out some really clear goals with the stakeholders. The automation tools they chose were Watir for automation and Cucumber for behavior-driven development, and set up a Continuous Integration (CI) pipeline, which made a huge difference. Interestingly, the biggest problem that they had with automation was not the code – it was learning CI to foster a sense of safety when working. To maximize visibility, they adopted a Kanban approach. At the end of the day, an automation is a project just like any other. As long as the team is aligned and heading in the same direction, the members can keep learning and improving.

Do Testers Need a Thick Skin, or Should We Admit We’re Simply Human?

Nicola Sedgwick bared her soul in this presentation. When there’s a report of a bug in production, one of the first questions is “Was this tested??”. The tester’s reaction is often “What did I miss??”, followed by wondering if you are any good at all at your job. Looking at it rationally, we know that no one is looking to place blame. We also are confident in our testing process. But when emotions are involved, rationality goes out the window. We think it’s our own fault, and that we might even get fired because of it. And having someone say "I found a bug just by opening a page and looking at it. I could be a tester!" doesn’t help. She noted that there are many talks about how testers should work with developers, or product owners, and other key people in a development project. But no one is talking to these people on how they can work with testers! She observed that stress can be a blessing or a curse, depending on how you deal with it. The key to her message is that communication is key. When faced with the fight or flight response, flight is easiest. But sometimes testers need to push back, and she noted that she found herself fighting on behalf of her team. But persistence paid off - she was thanked afterwards by other members of her team, who were on her side, although they never said anything in the meetings. This is because they had gone down the flight route, but after seeing someone fight, their approach was challenged.

Smart Algorithms – Are We Ready For This?

Bill Matthews spoke about the rise of applications that have particularly smart algorithms at their heart. For example, he uploaded an image of himself to an online app that estimates your age based on your picture. But when he uploaded a second, identical picture, but with a different resolution, it estimated a different age (neither of the ages were actually correct!). There are many entertaining applications like this, such as the one that shows you what dog you look like.

Apparently I’m a Boerboel – a dog with a protective nature, which can cause damage when jumping (due to its size), sociable and obedient only after trainingApparently I’m a Boerboel – a dog with a protective nature, which can cause damage when jumping (due to its size), sociable and obedient only after training

But these algorithms have serious applications behind them too, and have more complexity than testers are used to. They use big data, and simulate cognitive processes rather than procedural processes, so the system’s behavior adapts and evolves over time. If system behavior emerges, rather than being specified, it makes it very difficult to test.


Bill Matthews.jpg


Nowhere to hide: Adjusting to being a team’s sole tester

Nicola Owen, with nowhere to hide...Nicola Owen, with nowhere to hide...

Nicola Owen told her story of going from one of ten testers in a large on-going project, to being the sole tester within a small product team. The first project had a dedicated testing team, and each area of the product was tested by an average of two to four testers. Success was measured by the number of tests that you could write, and how many you managed to execute. A big benefit was having other people to bounce ideas off. She also felt safe from the consequences of the testing, because there was always someone else to ‘blame’ if a bug was found. But then she moved to a different context. She became part of an agile team working on a brand new product, where she was the only tester. The team adopted a continuous delivery approach, and made their own decisions about how to test. Suddenly, there was nowhere to hide! The team made extensive use of mind maps, so that they could easily identify and add scenarios that they hadn’t thought of. But they also avoided unnecessary documentation. Being the team’s only tester encouraged her to become a better tester. She became a more rounded tester, contributing to and learning from the huge ecosystem of meetups, conferences, blogs, discussion forums, courses, and social media. She summed up the main differences between the two positions: “In the first context, I was tester who happened to be in a team. In the second context, I was a team member who tested!”

The 99 Second Talks

The last part of the conference was the 99 second talks. Anyone who had something to say, on any topic, could register to speak for up to 99 seconds – and not a second longer, otherwise you get an air-horn blast. It’s a great idea – if you’re thinking about speaking at a conference, and you’re not sure about talking to a large audience, it’s an ideal opportunity to dip your feet in the water. If you’ve arranged a testing conference, and you want to let people know about it, you’ve got a captive audience of testers right in front of you. And if you just want to rap, rant about beards, or stage a 99-second sit-in, or just wear a tutu on stage (yes, all these things happened), you’re welcome!


The energy at TestBash was amazing. The organization was impeccable, the speakers and the audience were all friendly and everyone was ready to share their stories and experiences. The quality of the talks was really high across the board. Thanks to the Ministry of Testing people, who clearly worked really hard behind the scenes. I’d strongly recommend going to a TestBash. You’ll learn something, meet new people, and have a great time.


Were you at TestBash? Are you planning on going to the next one? Let me know by leaving a comment in the box below.

  • Future of Testing
About the Author


Malcolm is a researcher in Hewlett Packard Enterprise's Software's Application Delivery Management group. You can find him on Twitter as @MalcolmIsaacs

New Member.

Did you see all those tshirts fly off the tables?  There were none left!  Thanks for sponsoring! :)