A visual test strategy

Originally published on Assurity’s website in 2016 here.

A visual test strategy

We use pictures to tell stories. We share photos with family and friends to let them know what’s happening in our lives. We use pictorial instructions and how-to videos to learn new skills and we inform our Agile teams and stakeholders of progress and delivery with our visual management boards.

Many of us learn new ideas and information best when they are presented visually. However, in our workplaces, much of the information and guidance is only available in written form. While there are people who prefer to learn from written documents, for visual learners these documents can be impenetrable or just plain boring. Significant time and effort can go into writing strategy and planning documents which fail to engage their audience and the intended outcomes are never fully realised. We need to use a variety of techniques and media to take as much of our audience on a journey as possible.

I worked with the talented Fairfax Product Technology team as Test Chapter Lead last year. In collaboration with their Agile coach Jaume Durany, we ran two visualisation experiments which proved more successful than we ever anticipated in communicating our test strategy and gaining engagement across our teams.

Background to the test strategy

Jamie McIndoe Big Thought

Me (far left) with the Fairfax testers (l-r) Yuliya, Daphne, Bjorn (via Hangout), Ardian and Rupesh.

At Fairfax, I developed and implemented a strategy to guide testing of our features and stories. The strategy was for all of the Product Technology team, testers and non-testers alike. It was aimed at encouraging greater agility and cross-functionality in testing.

The testing approaches needed for new features or stories are contextual to the combination of products, features, technologies in use and the team members involved. Implementing a prescriptive, process-oriented test strategy for Fairfax would neither have suited the wide range of testing approaches needed, nor encouraged increased communication and collaboration within the teams.

Working with the testers and with feedback from Assurity colleagues such as Aaron Hodder, I pared the strategy back to a set of five principles to guide Fairfax’s skilled and collaborative teams in testing.

The central principle in the strategy is ‘Testing as a Team’, with the other principles (‘Shifting Left’, ‘Test Lean’, ‘Shifting Right’ and testers providing ‘Quality Assistance’) supporting delivery of it. The aim is for teams to be thinking and talking about testing throughout development and for every team member to be able to contribute to testing. This principle recognises that testing is collaborative and that many skill sets can be involved.

Using guiding principles also recognises the testers’ talent and experience, empowering them to determine the best approach for each testing task and trusting them with the responsibility to guide their teams to delivering better quality products.

The strategy is covered in more detail in the test strategy video below or in the Making Stuff article on the strategy.

Experiment 1: The visual test strategy

1pg

With ‘Testing as a Team’ being central to the Fairfax test strategy, there needed to be buy-in across the Product Technology team for the strategy to be successfully implemented.

I wanted to experiment with different delivery approaches beyond just writing a strategy document and was inspired by Lynne Cazaly’s work to try to visualise the strategy. Visualising the strategy would communicate the principles that make up the strategy in an engaging and easy-to-understand way.

While I am limited in my own artistic skills, I was able to sketch out basic visual representations of each of the principles. I then collaborated with Jaume Durany on refining the visual ideas and producing the finished illustration. Jaume is a Fairfax Agile coach with a passion for visualisation and strong drawing skills.

The visual test strategy captured our guiding principles for testing in a one-page illustration – the idea being for the test strategy to be displayed on team walls and other shared team spaces, enabling it to be used as a reference in stand-ups and team discussions on testing.

The visualisation communicated enough about each of the principles, such that team members could work with the strategy without having to have read the detail behind it. This was an important factor for engaging the non-testers, who may have been otherwise reluctant to try to understand and use the strategy.

The visual test strategy has been well-received by the Fairfax teams. It has informed and reminded the teams of the strategy principles and helped with conversations and collaboration on testing.

In particular, the central concept of ‘Testing as a Team’ has been adopted by the teams, with everyone getting actively involved in testing to some degree. This involvement has ranged from crowd-sourced help with mobile device testing, to non-testers leading test delivery for stories (with tester support) and whole product teams testing together for complex releases.

Experiment 2: The visual test strategy – a short film


The Visual Test Strategy video – illustrated by Jaume Durany, narrated and edited by Jamie McIndoe

The visual test strategy helps to communicate the test strategy, but it doesn’t capture everything. My article introducing the strategy covered much of the background and this information is also on the team’s wiki. However, this won’t be the best form of communication for everyone.

As the visualisation approach with the test strategy had proved to be engaging, Jaume and I experimented with another visual medium – video – for communicating the next level of strategy detail to a wider audience.

We were inspired to take this approach by the concept of ‘micro-learning’, which focuses on delivering education in small, accessible and easy-to-digest chunks. In micro-learning, the focus is on delivering learning in mediums that your audience readily uses. With video, we could keep the test strategy visual and communicate detail in a format that would be more appealing than a wiki document.

We wanted the video to be visually engaging for viewers and hit upon the idea of recording the illustration of the strategy and have the illustration emerging as narration was covering the background of the strategy. The end product was a six-minute video introducing the strategy.

The video has been well-received by those who’ve viewed it so far, both inside and out of Fairfax. Perhaps the most enthusiastic reactions have been from my Assurity colleagues. Many of these colleagues work outside of testing, but after they watched the video we’ve been able to discuss the strategy in some depth. This has demonstrated to me that the key messages are getting communicated in a way that genuinely captures people’s attention. By introducing the strategy in this way, we’ve reached and engaged people both inside and outside of Fairfax who were never likely to willingly read a testing article or document.

There’s also been real excitement about the micro-learning approach we took with the video and we’re looking at sharing other ideas and learning across Assurity using this type of approach. This ranges from technical training to company communications.

Looking back on the experiments

I believe that both the visual test strategy and its accompanying video have been successful experiments in communication and engagement. The visualisation experiments have enabled more widespread interest, understanding and buy-in to the test strategy than the written documentation alone would have.

Having the strategy based on a set of straightforward principles helped to make it accessible, understandable and, most importantly, drawable! However, I believe you could distil most strategies to a set of core messages which could be visualised. Even if this visualisation only skims the surface, it should help to capture the attention of a wider audience and encourage these people to learn more about the strategy.

The success of our visual work does not mean we should neglect written documentation. There are many learning styles and, just as there are people who learn best from visualisations, there are also people who learn best from reading written documentation. We need to understand our audience and how they prefer to learn. To effectively engage with a diverse group of people, we’ll need to use a range of communication and learning approaches.

Advertisements

Agile in the Real World

I’ve recently been collaborating with Cat McRae on our project at NZQA. Our collaboration has included visuals to help guide our team, supporting each other’s project work and talks.

Cat invited me to help her present an IIBA talk introducing Agile project life to a class of around 100 first-year Information Systems students studying business analysis at Victoria University.

victalk

Cat talked about her day-to-day work, how our project functioned, what aspects were Agile and what weren’t, and her experiences as a recent graduate. I supported Cat’s topics by covering some of the agile, cross-functional and visual aspects of the project, from my role as our project’s scrum-master. Finally we both covered the visuals we’ve created to support our team.

This was the first public talk I’ve been involved in where I’ve not used slide notes (Cat is already a pro at the no-notes talk). I was surprised and delighted that not only could I get my points across without notes, but I was a more compelling speaker when I simply talked on a topic I enjoy discussing anyway!

This was a really enjoyable talk, that hopefully helped a few first-years better understand how project life works. Cat and I are planning further collaborations on visuals and talks … watch this space!

Musical accompaniment:

The lead track from The Bangles’ ace debut EP, which can be heard in full on Ladies and Gentlemen… The Bangles!

Visual Test Strategy – A Short Film

In Agile projects we use visualisation to communicate and engage with our teams and stakeholders. This is effective for high-level information, but detail is often left to written documentation which many people (especially visual learners) would prefer to avoid.

To share the detail behind Fairfax Product Technology’s Visual Test Strategy, we’ve experimented with a micro-learning approach and produced a video version of the strategy.

Early feedback on the video has been very positive, especially from people outside of Fairfax. The video has been particularly effective with my Assurity colleagues who have viewed it so far. Viewings of the video have lead to in-depth discussions on both the messages and the medium of delivery. We’re now looking at experimenting with a similar approach for sharing a range of learning and ideas across Assurity.

The Visual Test Strategy

1pg
The Fairfax Product Technology’s Visual Test Strategy

The Visual Test Strategy captured our guiding principles for testing in a one-page illustration. This approach was used as an experiment to communicate the strategy and principles in an easy-to-understand and engaging way.  The idea being for the test strategy to be displayed on team walls and other shared team spaces, to be used as a reference in stand-ups and team discussions during development.

The visual test strategy was well received by the team and it has helped them understand how to approach Testing throughout development. In particular, the central concept of “Testing as a Team” has been adopted by the teams, with everyone getting actively involved in Testing to some degree. This involvement ranges from crowd-sourced help with mobile device testing to non-Testers leading Test delivery for stories (with Tester support).

The Visual Test Strategy Video

The illustration helps to communicate the test strategy in an engaging way, but it doesn’t capture everything. My article introducing the strategy covered much of the background and this information is also on the team’s wiki. While many team members are now familiar with the strategy and its background via this documentation, it isn’t going to be the best form of communication for everyone.

Many of us are visual learners, which was what made our original test strategy approach so effective. To communicate the background to the strategy to a wider audience, we experimented with the illustration in another visual medium and produced a short video version of it.

Jaume Durany (the strategy’s illustrator) and I were inspired to take this approach by the concept of “micro-learning”, which focuses on delivering education in small, accessible and easy-to-digest chunks. In micro-learning, the focus is on delivering learning in mediums that your audience readily uses. With video we could keep the test strategy visual and communicate detail in a format that was more engaging than a wiki document.

We wanted the video to be visually engaging for viewers and hit upon the idea of recording the illustration of the strategy and have the illustration emerging as the background of the strategy was being discussed. The end product is this 6 minute video introducing the strategy


The Visual Test Strategy video – illustrated by Jaume Durany, narrated and edited by Jamie McIndoe

The Early Reviews …

Our strategy video has been shared with the Fairfax team and some of my Assurity colleagues.

The video has been well-received by those who’ve viewed it so far, but perhaps the most enthusiastic reactions have been from my Assurity colleagues. Many of these colleagues work outside of Testing, but after they’ve watched the video we’ve been able to discuss the strategy in some depth. This has demonstrated to me that the key messages are getting communicated in a way that genuinely engages people. By introducing the strategy in this way, we’ve reached and engaged people both inside and outside of Fairfax who were never likely to willingly read a Testing article or document.

There’s also been real excitement about the micro-learning approach we took with the video and we’re looking at sharing other ideas and learning across Assurity using this type of approach. This ranges from technical training to company communications.

The Making of …

vts_uncropped
The uncropped video view prior to editing

An advantage of working with video for us was that we could work with free and readily available tools. The video editing tools did take some learning, but there were plenty of useful learning resources available which helped me understand how to do what I wanted quickly. The tools we used were:

  • Our team’s stationery supplies!
  • Video was captured on my phone, positioned on a tripod, recording at 1080p
    • We couldn’t get a close-up view on the illustration that looked good enough during recording. We instead recorded at a wider angle that included a good view of the illustration and cropped the video in editing
  • My narration was recorded on my phone’s voice memo recorder
    • I narrated to a roughly even timeline for each part of the strategy to help the video flow better.
    • The video was edited to fit the narration
  • The video was formatted and edited using open-source software, particularly
    • VirtualDub, used for cropping the video to focus solely on the illustration
    • Blender, for all video editing and rendering. There is a detailed video tutorial series which helped me understand the tool quickly

The illustration and narration took some time to refine and record but, including learning the tools and doing the editing, the video was produced in around 6 hours effort.

Hopefully I’ll have further experiments in micro-learning to report on soon!

 

Bowie: Words & Music

Bowie1

It’s been a tough few days since the news broke of Bowie’s death. I still feel incredibly sad (and still in some degree of shock) at the loss of an artist  whose music I’ve loved since I was young and who my own children have been growing up listening to. There’s a real sadness at an era passing and the loss of a truly unique and inspiring artist who was commited to following his own path, right to the end.

I’ve tried to write a proper tribute to Bowie and what he meant to me, but I’m not articulate enough to genuinely add something to the many articles written in the last week. Instead I’ve done what I can do OK, compiling and referencing great writing on Bowie and sharing a playlist of my perspective on Bowie’s music … if nothing else it will be something I can refer back to …

Bowie Books:

Low (33 1/3 series) – Hugo Wilcken

My favourite from the excellent 33 1/3 series (a series made for music obsessives like myself!), I’ve read this dozens of times. It’s a short, but engrossing, read on Bowie’s seminal Low album. Starting with the incredible song Station to Station and covering the Berlin period and beyond while looking at the 11 tracks that make up the album.

Pushing Ahead of the Dame / Rebel Rebel – Chris O’Leary (@bowiesongs)

Chris’s blog Pushing Ahead of the Dame is a richly detailed analysis of Bowie’s body of work. I’ve spent countless hours reading through the stories, the influences and the impacts of Bowie’s songs. His first book compiling and refining that analysis covers Bowie’s career up to 1976, finishing with an incredible piece on perhaps Bowie’s greatest work – the song Station to Station

Tony ViscontiThe Autobiography: Bowie, Bolan and the Brooklyn Boy

Tony Visconti is Bowie’s longest-standing, and probably most important, collaborator – working from Bowie’s 2nd album in 1969 to Blackstar released last week. This is a really enjoyable read on Visconti’s life with plenty of Bowie anecdotes.

Bowie Articles / Tributes:

* My wife and I were lucky to be able to see Bowie at this show. It was in fact sort of a “honeymoon” thing for us, being the week after we got married. It was an amazing, career-spanning, show that not even a severe rainstorm could dampen.

Bowie Music

Some obvious choices here … and hopefully a deep cut or two to introduce. This playlist starts with my rough top 10 (for today at least)

The playlist starts with Ashes to Ashes which is probably my absolute favourite song by any artist … it’s easily one of my 2 or 3 most played songs anyway.

 

 

CukeUp Australia – The Workshops

Following on from my previous post on the top talks at CukeUp Australia 2015, this post covers what I learnt from two of the workshops.

There were 4  workshop sessions over the two days, with two options available at each time. The workshops were 90 minutes of hands-on practical learning and they helped to balance the conference between talks and getting stuck in and learning new practical skills.

I’ll be detailing what I learnt in the following workshops:
Paul Rayner’s session on using design techniques in BDD and
Matt Wynne’s session on example mapping

BDD in the Key of Design – Integrating DDD and UXD (Paul Rayner)

IMG_20151119_142113

Resources:
You can download the participant’s workbook for Paul’s workshop here.

Related:
Behaviour-Driven Development Combined with Domain-Driven Design
Layers of abstraction: combining BDD and UX

Techniques covered:
Personas
Wireframes
Set-Based Design
Event-Storming

Paul’s workshop covered a range of techniques and ideas from User-Experience Design (UXD) and Domain-Driven Design (DDD) which can help with Behaviour-Driven Development.

Personas

The techniques we tried in our workshop included the use of Personas to help understand users and develop valuable scenarios.

In our case we worked on fleshing out the persona of “Bob”, a user of the library website that was the system our exercises were based on. We started with some basics like deciding his age and family. However we quickly dived into much more detailm including his preferences in DVDs and why his travel for work combined with his kids’ love of books was causing him to get so many library fines!

Not only was it fun developing Bob’s persona, but it made the exercise of developing scenarios much more like storytelling. I found working with a persona made creating new scenarios and thinking outside of the “obvious” scenarios much easier than if we had been developing against a dry set of business requirements.

Wireframes

In developing our scenarios for Bob, we also used simple wireframes to help facilitate the discussion. Starting with an example wireframe of a fines & fees summary screen, we worked on wireframes for Bob paying them.

Having a sketched implementation helped to make the scenario more real. We could work with Bob as a persona and imagine him working through paying his library fines, including asking ourselves questions of design from his perspective – how could we make the payments process simple? what options would he want for payment? how would he have confidence in the payments being secure?

As we worked with the wireframes we noticed parts that we initially thought fit, didn’t really fit on closer inspection, and we also noticed where there were gaps in the process.

Set-Based Design

We split into pairs while working on the wireframes and came back to our table to discuss the wireframes we’d developed. This turned out to be an exercise in the technique of set-based design.

Set-based design is an approach where instead of working on solutions straight away, we work to develop a number of options and narrow in on the eventual solution. In our case, each table had 3-4 options from the wireframe exercise as our starting set of options.

The benefits of this approach were obvious from our exercise. When working as a group (especially with an unfamiliar domain and unfamiliar people) it can be very easy to fall into “groupthink” and move to reach consensus before critically evaluating all the options. Coming back as a group with multiple different perspectives on the same business problem, we could compare our approaches, identifying the common parts where there was wider agreement and discussing the differences in working out a combined group design.

Event-storming

For our final exercise we tried the technique of Event-Storming.

We worked as a team on telling the story of how our persona Bob had accumulated library fines to a level where his card got blocked. We brainstormed events that could have lead to that situation and laid them out in the order the group agreed on. This gave us a timeline of events that modelled a simple workflow through our system. While we ran short on time to evolve alternate workflows we’d already started to find events that didn’t fit our main workflow, but could be the seeds of alternate workflows.

IMG_20151119_144457

I found event-storming a natural extension of the work we’d done with personas, being another approach based on story telling. It was an easy technique for our group to pick up and start modelling workflows collaboratively, with little direction required.

and more …
Some other ideas looked at in Paul’s workshop included:

  • Ubiquitous language
    • Naming concepts in your business domain to make the implicit explicit. In our Library example this included concepts such as an account being “deactivated” and a card being “blocked”.
  • Timeboxing
    • Our exercises were timeboxed and that help us to keep focus and not over-engineer ideas
  • Using free text space in feature files
    • Paul’s session was one of many in CukeUp (including Matt Wynne’s workshop) which emphasised the value using the Feature description free text area in a Cucumber feature file to detail business rules and other relevant details.
    • This approach helps to set better context for scenarios without having to make the scenarios themselves too wordy.
  • Discovery within BDD 
    • We focused on the Discovery phase in the workshop, developing and exploring options, rather than trying to come up with solutions and formalise scenarios straight away.IMG_20151119_145451
    • An overview of cycles within BDD

There was a lot covered in Paul’s workshop, and while nothing was covered in great depth it was an informative and fun sampler of a range of techniques for further investigation and use.

The team I currently work closest with is relatively quiet and I think that techniques such as set-based design could help in giving all the team an opportunity to develop options, rather than having the more outspoken members driving discussion.

Example Mapping (Matt Wynne)

Resources:
The slides from Matt’s workshop are shared here.
Matt’s Example Mapping introduction.

Related:
Example Mapping Can Save Your Three Amigos
Example Mapping – Steering the conversation

IMG_20151119_114711

Matt’s workshop focused on Example Mapping, a structured technique to get the most out of “3 Amigos” sessions and effectively break down user stories into rules / acceptance criteria and examples.

IMG_20151119_115306

Before getting into trying Example Mapping, we looked at the purpose of a 3 Amigos session and the types of outcomes we’re looking for from them.

Starting with a story, we’re aiming to:

  • define / refine rules or acceptance criteria for the story
  • produce examples to confirm those rules are met
  • identify questions and get them answered
  • potentially split the story or create additional stories

with an overall goal of creating a shared understanding of the story.

What Matt highlighted is that while we have these outcomes in mind, we often focus much of our effort in 3 Amigos sessions on writing nicely structured Gherkin (Given/When/Then) examples at a cost of focusing on discovery.

IMG_20151119_114911

Our first exercise in the workshop looked at the relationship between rules and examples. We worked on developing rules for a “strong” password. Our task was to create 3 rules and try to communicate the rules to another team using only examples.

While some of the rules teams chose were straightforward to pick from the examples (e.g. a minimum number of characters) others were too complex or too obscure and we found we didn’t have enough examples to adequately communicate all the rules.

This exercise emphasised how BDD is about much more than examples, and that it can be hard to know if you’ve provided complete coverage of rules with examples.

Example Mapping

Example mapping is a technique to structure the conversations in 3 Amigos sessions – mapping the rules and examples for a story and to identifying the questions we need answered.

In our exercise we worked through with different coloured cards for each element in our maps to make the purpose of each card clear – the story (yellow), questions (red), examples (green), rules (blue).

IMG_20151119_120320

The mapping is simple – the story goes at the top, below that is a row of the rules, for each rule we have a column of examples, with questions to the side of the main map.

We developed an example map for getting a refund on a kettle. A mundane and seemingly straightforward story which evolved quickly into a complex scenario over the course of the workshop.<

Rules generated examples and examples generated rules.

As we created examples for the initial set of rules, we found some examples we created generated questions for our product owner. Some of these question lead to refined or additional examples, and some lead to new rules being defined. I forget whether it was Matt or one of the participants who said it, but this process was summed up nicely as examples start conversations, rules close them

As the map of examples and rules grew, it also became apparent that the scope of the story may be too large … the fact that we were quickly running out of physical space on our table for the map was a good indicator! While we didn’t have time in the exercise to actually split the story, our map had clear lines along which we could do the split and once a split was done we would have had well-developed starter maps for those stories.

So we hada story (via an example map) generating more stories.

IMG_20151119_121857 - Copy.jpg

Our team’s rapidly growing example map. We generated a lot of questions (pink cards) and as those questions were answered (the pile at the bottom of the picture) the map grew wider and deeper

While a simple technique, I found example mapping was highly effective in facilitating the team’s conversation. It clearly highlighted where there were gaps in examples coverage and in our understanding of the rules.

I’ve discussed Example Mapping with members of the teams I work with and there is interest in trying it for some of our stories. In particular, we are thinking the structure of Example Mapping may suit teams who tend to lose focus in 3 Amigos sessions.

CukeUp Australia – The Talks

CukeUpAu

Myself and 3 colleagues from Assurity (Scottie, Isham & Darren) recently spent two days in Sydney at the CukeUp conference.

 IMG_20151119_084131

For me, the conference worked really well because of the small scope of topics that most of the talks and workshops were based on. This meant that you got a much deeper understanding of the topics as ideas were discussed, built on and sometimes challenged over the conference.

The only real negative about the conference, was the record heatwave that hit Sydney at the same time as it was on. The temperature hit 42 degrees on the 2nd afternoon which made for a pretty tough environment to think and learn in, while we were in a historic cell block building with no aircon…

At least the heat provided an opportunity to introduce some Cucumber basics with a very relevant example of driving aircon to a more comfortable temperature!

IMG_20151120_091709_crop

Because there were so many insightful talks and workshops to learn new skills in, I’ve split my highlights blog into this one covering the talks and a forthcoming one diving into what I learnt from the workshops.

Beyond BDD (Matt Wynne)

IMG_20151120_163114

Starting at the end. In spite of the stifling afternoon heat, Matt wrapped up the conference with an energising talk that pulled together many of the threads and themes covered at the conference and also looked forward.

Matt looked at different levels of BDD development using a burned toast analogy. This was based on W. Edwards Demings’ “You burn it, I’ll scrape it” quote on tolerating systems that produce substandard results instead of fixing the process. At lower levels of BDD development (i.e. low collaboration, poor communication) the process results in a high amount of bugs and rework.

221

When collaboration is built into the development process (e.g. with 3 Amigos sessions defining examples) more quality is built in from the start. However, it’s when teams take a test-first approach that quality is really baked into the process and teams are really “doing” BDD.

247

… However, a test-first approach has a flipside … BDD can become addictive and automated test suites can get too large, too bloated and too brittle (I’ve certainly seen this with automation efforts that have started with the best of intentions, but the team hasn’t kept control of test quality, relevance and performance)

254

This is often the point at which it becomes easy to blame the tool, rather than the process that led you to this point.

The key here is you still have to do software design. Matt highlighted how testing and design are inextricably linked – quality design relies on great feedback from Testers and Developers who understand testing themselves.

260

The less code the better and that includes automated test code. With good design you require less automated tests.

The shallower your tests can be (lower in the application stack) the more targeted they can be. Shallower tests are faster to run and when they fail, you know why they’ve failed.

271

Matt finished with a quote from David West’s Object Thinking Model the problem well enough, and the solution will take care of itself

The 10 Do’s and 500* Don’ts of Automated Acceptance Testing (Alister Scott)

IMG_20151120_154323

I’ve been following Alister’s blog (Watirmelon.com) and using his testing pyramids to help illustrate automation concepts for some time now, so I was very keen to hear what he had to say.

His talk reinforced some ideas covered in the conference, but also challenged some other ideas and assumptions. “DO run end-to-end tests in production”, in particular generated a lot of discussion in the Q&A. This approach is about maximising your returns on developing test automation, by using the tests wherever they offer value.

Alister delved into scenarios where running end-to-end tests in production has worked for him. The main scenario discussed was from when he worked with Domino’s Pizza and they ran automated end-to-end against the production ordering system. Using test-specific data in the orders meant they could be easily be separated from real orders and automatically cancelled.

The 10 DO’s covered were:
  1. DO differentiate between acceptance and end-to-end tests
  2. DO specify intention over implementation
  3. DO write independent automated tests
  4. DO automate (just) enough to eliminate manual regression testing
  5. DO manual story / exploratory testing
  6. DO ensure your whole team is responsible for your auto tests
  7. DO run your tests on every commit
  8. DO run your tests in parallel
  9. DO run your tests against a single browser
  10. DO run end-to-end tests in production

and Alister wrapped the talk nicely with his 1 Don’t …

IMG_20151120_154927

Alister’s blogged the full text of his talk here

Keynote: When your testing is in a pickle (Anne-Marie Charrett)

IMG_20151120_095004

In her keynote, Anne-Marie Charrett emphasised the need to focus on bridging communication gaps and enabling collaboration. This was challenging the situations where BDD or Cucumber get adopted without really understanding what problems you’re trying to solve.

The keynote covered 4 “insights on Cucumbers” which provided wisdom beyond Cucumber or BDD

#1 – It’s not just a Cucumber

We need to focus on what problems are we trying to solve, rather than the tool itself.

“Is fast feedback what we really want from automation? more important is quality feedback”

#2 – Cucumbers can’t talk

Don’t look for Cucumber to solve your communication and collaboration problems.

“A lot of problems that occur in BDD are people problems. Not tooling problems”

“the problems we are trying to solve are for people.”

#3 – Cucumbers make lousy hammers

The tool is only as good as the person using it and the specific context they’re working in.

“Do you really need a tool for Given / When / Then if you’re trying to improve communication and collaboration. Maybe start with a document, use that approach to get a shared understanding”

Included in this part of Anne-Marie’s talk were two extremely useful summary slides on questions for testers to ask in “3 Amigos” sessions and important skills for Testers.

IMG_20151120_095227

Questions for Testers to ask in 3 Amigos sessions

  • Variables
  • Data
  • Scenarios
  • Outputs
  • Business impacts
  • Technical complexity – e.g. cross-system test examples, which ones are really useful? where is the value in these tests?
  • Does it have to be automated?

IMG_2163

Important Testing Skills
  • Risk Identification
  • Analysis of the system
  • Business model
  • Learning mindset
  • Challenging assumptions
  • Logical reasoning
  • Strategic thinking
#4 – Cucumbers make great pickles

Anne-Marie used the metaphor of Cucumber as a base camp on your way to the final destination of a quality product. It can help you along the way, but implementing it is only a step towards your actual goal.

Twelve BDD Anti-Patterns: Stories from the Trenches about how NOT to do Behaviour Driven Development (John Smart)

IMG_20151119_100934

What I liked most about John’s session was that while he was covering anti-patterns in BDD, the focus was on balance – i.e. understand what the anti-patterns are, but take them as guidelines and direction for improving your process, rather than absolute rules.

016

John discussed 3 types of balance:

  • Timing (#1-3) When do you write your requirements – too early? too late? are the right people involved? are you providing time for feedback?
  • Pitch(#4-11) How are you pitching these requirements? Are you giving enough information to develop the features, but also not locking the implementation into a specific solution? Are you focusing on features that will actually make a difference?
  • Feedback (#12) Are you providing the most relevant and meaningful feedback to the right people?

BDD Anti-Patterns

  1. Out to lunch – you need time from the right people, to ensure you’re delivering value.
  2. Cucumber salad – where detailed requirements are being written as gherkin scenarios up front. The more work done in isolation up front, the more likely you’re heading down the wrong path and wasting effort.
  3. Cucumber as a Test Tool If you are just using Cucumber as a test automation tool, you aren’t doing BDD. Cucumber is a collaboration and communication tool and there are better automation tools out there if all you want to do is test automation.
  4. Aimless requirements – Where there is no clear business goal. Make sure the level of requirement abstraction is higher than the action being used.
  5. No shit Sherlock – don’t waste time on the obvious, prioritise the uncertain
  6. Eyes on the screen – Examples tied to implementation, meaning the example has to change if the implementation does. Examples are about what you are trying to achieve, not how you are trying to achieve it
  7. Top-heavy scenarios – when too many scenarios are automated as UI-tests. UI tests are slow and brittle compared to tests down the stack. Only automate against the UI where the scenario needs UI interaction.
  8. Not having all the cards – poorly-defined inputs and outcomes. The purpose and value of scenarios is unclear.
  9. Scenario overload – the more information you have, the less information you actually communicate.
  10. Cucumber test scripts – writing step-by-step test scripts in Gherkin
  11. Tech-speak scenarios – scenarios are essentially regression tests, rather than examples relating to business goals.
  12. Incommunicado scenarios – you want to provide reports that are meaningful to the business.

John’s BDD anti-patterns slides are shared here

The first edition …

My name is Jamie and I’m a Test Lead / Engineer / Coach from Wellington, New Zealand. I’m great at talking about (rambling on about …) my experiences and ideas but I’ve always struggled with writing these things down!

One thing I do a lot (and I’m not sure if it’s good or bad) is that I draw a lot of parallels between my other interests and my work … expect to see building and agile, music (hence geetar/guitar) and automation, football and testing, my family and all of the above being intertwined in the same article.

I’ve been working for a consultancy called Assurity for 8 years and have been working at Stuff product technology as test chapter lead for since the start of the year, this role and team are amongst the best I’ve ever worked with and I’ll be making sure I share some of my experiences working there soon 🙂

This post’s tenuous music link … Roxy Music – Editions of You