The other week I facilitated a session at the UK Test Management Summit.
I presented the topic of moving to weekly releases from 8 month releases.
I talked about some of the challenges we faced, the support we had, the outcomes and the reasons for needing to make this change.
It actually turned in to a questions and answers sessions and despite my efforts to facilitate a discussion it continued down the route of questions and answers. It seems people were very interested in some of the technicalities of how we did this move with a product with a large code base of both new and legacy code (and that my facilitation skills need some fine tuning).
Here are some of the ideas.
We had a vision
Our vision was weekly releases.
It was a vision that everyone in the team (the wider team of more than just development) knew about and was fundamentally working towards.
This vision was clear and tangible.
We could measure whether we achieved it or not and we could clearly articulate the reasons behind moving to weekly releases.
We knew where we were
We knew exactly where we were and we knew where we were going. We just had to identify and break down the obstacles and head towards our destination.
We had a mantra (or guiding principle)
The mantra was “if it hurts – keep doing it”
We knew that pain was innevitable but suffering was optional.
We could endure the pain and do nothing about it (or turn around) or we could endure the pain until we made it stop by moving past it.
We knew the journey would be painful but we believed in the vision and kept going to overcome a number of giant hurdles.
Why would we do it?
We needed to release our product more frequently because we operate in a fast moving environment.
Our markets can shift quickly and we needed to remain responsive.
We also hated major releases. Major feature and product releases are typically painful, in a way that doesn’t lead to a better world for us or our customers. There are typically always issues or mis-matched expectations with major releases, some issues bigger than others. So we decided to stop doing them.
The feedback loop between building a feature and the customer using it was measured in months not days meaning we had long gaps between coding and validation of our designs and implementations.
What hurdles did we face?
The major challenge when moving to more frequent releases (we didn’t move from 8 months to weekly overnight btw) was working out what needed to be built. This meant us re-organising to ensure we always had a good customer and business steer on what was important.
It took a few months to get the clarity but it’s been an exceptional help in being able to release our product to our customers.
We also had a challenge in adopting agile across all teams and ensuring we had a consistent approach to what we did. It wasn’t plain sailing but we pushed through and were able to run a fairly smooth agile operation. We’re probably more scrumban than scrum now but we’re still learning and still evolving and still working towards reducing waste.
We had a major challenge in releasing what we had built. We were a business based around large releases and it required strong relationships to form between Dev and Ops to ensure we could flow software out to live.
What enablers did we have?
We had a major architectural and service design that aided in rapid deployments; our business offering of true cloud. This means the system had just one multi-tenanted version. We had no bespoke versions of the product to support and this enables us to offer a great service, but also a great mechanisms to roll products out.
We owned all of our own code and the clouds we deploy to. This enabled us to make the changes we needed to without relying on third party suppliers. We could also roll software to our own clouds and architect these clouds to allow for web balancing and clever routing.
We had a growing DevOps relationship meaning we could consider these perspectives of the business together and prepare our plans in unison to allow smoother roll outs and a growing mix of skills and opinions in to the designs.
What changes took place to testing?
One of my main drivers leading the testing was to ensure that everyone took the responsibility of testing seriously.
Everyone in the development team tests. We started to build frameworks and implementations that allowed selenium and specflow testing to be done during development. We encouraged pairing between devs and testers and we ensured that each team (typically 4/5 programmers and a tester) would work through the stories together. Testing is everyone’s responsibility.
Testing is done at all stages in the lifecycle. We do TDD, Acceptance Test Driven Development and lots of exploratory testing.
We do a couple of days of pre-production testing with the wider business to prove the features and catch issues. We also test our system in live using automation to ensure the user experience is as good as it can be. We started to publish these results to our website so our customers (and prospective customers) could see the state of our system and the experience they would be getting.
We started to use techniques like KeyStoning to ensure bigger features could be worked on across deployments. This changed the approach to testing because testers have to adapt their mindsets from testing entire features to testing small incremental changes.
Why we love it
Releasing often is demanding but in a good way. The pressure is there to produce. The challenge we have is in balancing this pressure so as not to push too hard too often but have enough pressure to deliver. We don’t want to burn out but we want to ship.
We exceed the expectations of our customers and we can deliver value quickly. In an industry that has releases measured in months (sometimes years) we’re bucking the trend.
As a development team we get to see our work in production. This gives us validation that we are building something that is being used. Ever worked on a project that never actually shipped? Me too. We now see none of that.
It’s been tough getting to where we are now but we’ve had amazing support from inside and outside of the business which has helped us to really push ahead and set new markers of excellence in our business domain. We’ve still got lots to get done and lots to learn but that’s why we come to work in the mornings.
These are just a few of the factors that have helped us to push forward. There are companies releasing more often, and some releasing less often to good effect. Each business has a release cadence that works for them and their customers.
Did I mention We’re Recruiting?
I got asked the other day how I come up with ideas for talks/blogs, how I think through these ideas and how I go about preparing for talks. I’ll take this opportunity to add a short side note of how I do this. This approach may not work for you.
I firstly create a central topic idea in a mind map (I use XMind).
I then brainstorm ideas around the central topic. After the brainstorm I go through the map and re-arrange, delete, add and rename until I feel I have a story to tell.
I then start planning the order and structure of the story. Every story has a beginning, a middle and an end.
I start by writing the beginning and then the end. The middle is the detail of the presentation.
I then doodle, sketch and plot.
I then move to my presentation tool of choice. In this case it is PowerPoint – sometimes it is Prezi.
The presentation typically takes a long time to prep, even for a very short intro like this. This is because I don’t like including too much text in my slides and also because I think simple, but attractive slides can add some impact to the topic. So I spend some time making sure they are right. Saying that, no amount of gloss in the slides will help with a bad/poor/boring story.