How to get involved earlier in the software development life cycle: be involved!

Once the process helps us to focus on fewer things and enables us to collaborate and test as a team, I as a “tester” will have more time. And I should spend that time where the product is actually baked: I should get more involved with the developers.

Many QAs are a bit hesitant to move somewhere close to the actual application code. But they shouldn’t be: our main contribution is not that we can code. There are other people who are much more specialised in. We call them developer. They are much better coders. And as such, much better suited to write automated tests. What is then left for a QA?

We bring very strong analytical skills. One of them can analyse all tests on all levels from a strategic point of view. What is the share of unit- integration and end-to-end tests? Where do we cover what business logic, where do we have gaps in our test coverage.

Many QAs who specialize in Front-End-Test-Automation write lots and lots of End-To-End / User-Journey tests. These tests are typically the most flaky, hard to maintain and cost intensive tests you can imagine. Hence, we usually advocate to add as few of them as late as possible.

Screen Shot 2019-02-15 at 20.09.11.png
Picture: 100 End-2-End test to rule them all

Instead QAs should aim to understand the big picture of the system architecture: what services do we have? How are they connected? How are they split? Does each service have a distinct capability? If so what is it and how does it relate to you business?

Once you figured that out you can assess how to test each of these services or domains in your service independently. If this works have a look at the communication between the domains and ensure this. If all of this is covered you may want to add a slight hint of an end-to-end test on top.

Screen Shot 2019-02-15 at 20.11.06.png

Obviously, the second approach is much more difficult – but that is what you are for and what you contribute in the team? You are the one to keep the big picture and consult your team members where to add what test in what way. What is the key assertion in a given test case? What is the right level for it? With a strong coder and your analytical abilities in testing you can ensure that things are working while they are implemented. That does not only improve quality early on, it also significantly decreases the time you need to spend testing (and reporting defects) afterwards.

Still, some defects will be released. No matter how much money you invest, there is no way to ensure bug free software at all (even if you are the NASA and spend more than 320 Million $ on a project). The second thing you can figure out with your dev team is how to identify and catch them. With the lean process (see above) that you established you can be sure to ship a potential fix very fast. The way to detect them is a helpful monitoring setup. This involves to visualise the amount of errors, as much as server/database request (and possibly a deviation to 24hours before). If you go to real professional levels you want to think about anomaly detection so that your system can notify you on its own once something is off.

Screen Shot 2019-02-15 at 20.12.49.png

The last open question is how to react to breaking changes that you may have accidentally released to production. We are running a two fold strategy here. We try to minimize our time-to-market for bug fixes and mitigate risks of other larger issues with feature toggles. Let me go into some details:

Usually, in a classic release management process you have a plan how to do your releases and if there are major problems afterwards you execute a predefined rollback. If this is – for one reason or the other – not possible there is usually a hot-fix-release-branch-deploy process defined by someone somewhere. Here is the problem: If you need a hot fix then your team is probably already on fire. In this very moment you need to follow a process that most people are unfamiliar with which usually bypasses a lot of security measures you have previously established in your release cycle. That is quite a bad habit concerning the production problems one has in this very moment.

Our goal is to make the fasted possible release process our standard process. Thus we drive our teams to deploy every single commit to production. That also means that every commit has to be shippable – with enough tests to make sure our product is still working. This is baking quality in already!

Screen Shot 2019-02-15 at 20.15.03.png
wikimedia.org

Still, things will break. But with a quick way to react and deploy a fix we do not even need rollback strategies any more. But being able to deploy a hot fix very quickly implies that you can also quickly analyse the root cause. But that is not always true. If you know what commit was faulty you can of course deploy a revert. But sometimes a new feature in it’s complexity across stories and commits is just not working right. Thus, we work a lot with feature toggles, making sure that all new functionalities are toggled off in production. We also make sure that we can toggle those features independent of deployments. Thus, we decouple the technical risk of a deploy with the business risk of a feature toggle. This reduces our needs for reverts by about 90% and most deployments run automatically and without problems. Every few days a new feature is toggled on. People are then usually more aware of the situation and monitor the apps behaviour more closely (at least they should). Problems can then be identified and either quickly be fixed with a tiny commit or, if you encounter major blockers, you toggle the feature off again.

In conclusion, we have way fewer way less troublesome releases, while we can activate new features in a very controlled way. Thus, we do not only deliver value fast, we also achieve higher quality at the same time. However, a very experienced and disciplined team is needed to work on such a high level of quality commit by commit.

How we do “Quality” at ThoughtWorks Germany

Thanks to the other QAs in ThoughtWorks Germany for contributing thoughts to this topic over the past months: Sarah@DizMario, @Nadineheidrich and @bratlingQA


What is testing?

Testing is a method to analyze the quality of a given software. It is a method that is applied after the software is developed. If the software has an insufficient level of quality, yet another cycle of development & testing is needed to increase and measure the quality again.

Testing is bug detection.

Metaphor: Testing is like putting chocolade on a muffin, but exclusively after baking it.

What is QA?

There are other methods that can improve / increase the quality of a software while it is developed. Those methods decrease the amount of cycles of development & testing that are required  to reach a certain level of quality.

QA is bug prevention.

10797887872_IMG_1353
We want to consider chocolade while baking and only put some additional on top.

How we are testing

In addition to the tools and processes that allow us to build high quality software from the first line of code (see below), we have the highest standards for testing software. As mentioned before,  we are well aware that testing can only analyze a software’s current state and show the presence of issues/defects. Another cycle of development is needed to actually improve the quality. That leads to the known problem: whenever the testing is “successful”, the cycle time of stories increases and the delivery of a new feature needs to be postponed.

To minimize this delay, we apply the most efficient testing approaches to provide fast feedback for developers. This reduces the overall time to market of new features. With these methods we are able to reduce the cycle times significantly while improving the overall quality of the software in different projects:

  • Our tests are fast and effective. We run Unit-, Integration and End-to-End tests in a well shaped testing pyramid. This allows us to quickly check if our application behaves as expected. Writing the right tests on the right level reduces the time we need for regression tests from days to minutes. This includes – amongst other things – a 100% automation of regression tests.
  • Of all the tests in the pyramid, we take special care of the integration tests (of different services) to assure the architecture’s resilience. One of our favorites are the consumer driven contract tests. They allow different teams to work independent with loosely coupled services while ensuring that the entire system behaves well altogether.
  • For us, testing is an integrated activity within the software development team and not an independent, separate discipline. There are two ways to get a story tested:
    1. When a story needs QA attention, you move the ticket into a “QA” or “ready for QA” column. The person who is in the role of the Quality Analyst then picks up the story as soon as possible.
      (this is push & role-focused ⇒ that is the Scrum-way with experts in the team)
    2. When a story needs QA attention, you look out for the capability in the team. Any person with some capabilities in testing (often but not always the QA) rotates into the story.
      (this is pull & capability-focused ⇒ that is the Kanban-way with cross-functional people in the team)

      Guess what. I prefer the 2nd approach. The 2nd one is real team-play. And it decreases the cycle time of a single story and thus increases your velocity! Devs will learn (more) about testing and QAs can pair on the programming part, eg. to sort unit- and integration tests into the pyramid ⇒ baking the chocolade inside.

  • For exploratory testing we do not always apply the same standard methods but acknowledge the individual context we are in. Only then can we make use of the various advantages of different test methods. We find all kinds of tools in our box: Behaviour Driven Testing, Acceptance Testing, End-To-End Testing, Scenario based Testing, User Journey Testing, Integration Testing, System Testing, Risk based Testing, Penetration (Security) Testing, UX testing, performance Testing, Guerrilla Testing…

How we do QA

As mentioned before, we really need to learn all about testing. And its a mastery to study. However, its only a (small) part of our job and every day live. Besides testing we look into other things, as we know that good software is only the first step towards a high quality product.

We create a culture in the team, where the aspect of quality is an important asset for each team member. In this context we can address the team’s current needs with a wide set of processes, frameworks and tools that we as ThoughtWorkers already use or create if they do not yet exist (e.g. Selenium).

We acknowledge that we cannot build defect free software.  Hence, we focus on defect prevention and establishing an overall quality mindset. We assure a high level of quality in software through tools and processes that allow us to prevent defects and find errors fast:

  • Well designed services to ensure a resilient architecture. There are so many things to work on if you want to improve the resilience. You can have the best software without bugs. It wont help you if your servers are down for the bigger part of the day. From a QA point of view, we are interested in circuit breaker (self-healing systems), feature toggles, well designed APIs and a kick-ass monitoring:
  • Monitoring! This is so important. And so many people think that “only” Ops should care. What a misunderstanding. Constant monitoring of all services and environments is a shared discipline to be able to react quickly on any arising issue. No matter how good you test (see above), some defects will slip through to production. The best way to reduce their impact is a combination of a good monitoring and quick deployment. If we are able to release a fix fast (= best case: 20 min after a bug is found), we can reduce the impact significantly. This is what monitoring is for. Learn more in this podcast.
Screen Shot 2019-02-15 at 20.12.49
  • We have a strong focus on Continuous Integration and Continuous Deployments best practices, such as fully automated regression testing. You can read all about it in the other post as well as talk in Ljubljana.
  • Test Driven Development for high test coverage and fast feedback during development is also an important thing to notice. While most developers know that a tests are written first in TDD, not all know about the testing pyramid, much less of its benefits. Hence, a pairing of QA and dev while practicing TDD can be of high value. This is how you bake quality in!
10798083424_IMG_1367
  • I just mentioned the consistent pair programming. Pairing allows best designs and fewer defects from the beginning. Make sure to rotate frequently and across the roles. Pairing is a general activity for most tasks in a team. Pair-programming is just one of them.
  • We love Feature Toggles™. They give us maximal control over the features in production and canary releases. A very easy method I usually use is to give chocolate while the standup to the pair that implemented a toggle the previous day. This is a fun way to talk about it, remember it and give a sweet incentive to build it in when it was forgotten. If you find the time to use them it will make your lives much easier. You will not need rollback strategies any more and it is a very, very good safety net. Quality “assurance” at its best!

The mix for the win.

Of course we combine the two aspects of QA and testing. And this is the biggest challenge for us. Where to focus at what point of time. Where do we need more attention and what part of the application / system / team is running smooth? Ultimately,  we try to pick the right tools and create the right mindset to build a high quality product.

A Guide to Testing Microservices

Over the last couple of months I have often been asked about micro services, especially how to test them. Building a small (e.g. a micro) service usually requires a different approach towards the architecture and towards the services themselves. In many cases also the team structures changes (like Conway’s Law, just the other way around). The question that arises then is how we can “compensate” for it while testing. Or to be more precise: how to adopt our way of working to ensure a high quality software.

There are three major themes that need to be taken care of when we think about testing micro services:

  1. The many times referenced testing pyramid principle apply here as much as for a monolith. And not just for one single service. The principles are as important when it comes to testing different service’s interactions.
  2. Automation plays a key role in order to speed up your build and deployment pipelines. Maybe this applies even more for micro services than for other architectures.
  3. Its not easy to start with a good micro service landscape and there certainly is some overhead in setting things up (the first time). But the benefit of small and independent services is the fast feedback you can receive.

The second and third points are mentioned most of the time, when someone talks or writes about working with micro services. The testing pyramid is not really new and there is not too much detail about the testing of those little services. So we will look into this even more, as indeed, there are some pitfalls you should avoid in order to have a properly testable service landscape.

If we have a look at a monolith, things are straight forward: it is pretty clear where and how to apply the testing pyramid. The basic components of our services are unit-tested, their interaction is integration tested and then there is probably one end to end test around the entire service.

Screen Shot 2019-02-15 at 20.52.18.png
The testing pyramid for a micro service. The scheme of the service is by Toby Clemson from martinfowler.com

Now a “micro” service is (in this abstraction) nothing but a small monolith. In other words: treat the service itself as you always did. Be sure that the unit test coverage is really good. Find the places where you need to test the integration and then put as few as possible (in some cases: 0) end-to-end tests on top.

Up to here, there is nothing breaking new. But any project dealing with only 1 micro service has no big issues with testing. The real problems arise, once you have two, three or many services. The big difference to the monolith-world is that you have a lot more interaction via the HTTP clients. Lets have a look at an example.

Lets assume that the product we want to build is some kind of messaging system. There are three mayor features (see green boxes in the image):

  1. The dashboard page where the user is going after login and sees a greeting. On this dashboard, also unread messages are displayed.
  2. The inbox itself, all messages can be viewed here
  3. The entire messaging system has some super strong encryption.
Screen Shot 2019-02-15 at 20.53.08.png

Although those features are very clear, we have a look at the business domains. The first domain we can identify is the user (“Finn”, the black circles). The user has a dashboard and an inbox. The second domain are the messages (yellow circles). They are displayed on the dashboard and in the inbox. The messages are also encrypted. The third domain is the encryption itself (blue circle). The encryption only works with messages.

Those circles represent the domains. If we cut our system according to the circles, we will end up with a clear domain for every service.

Service A deals with the user (and probably owns the html). Service B takes care of the messages. And Service C is doing the de-/encryption. For each single one of those services we apply – as described the testing pyramid. Then we put them together and view the entire system. And we consult our pyramid once more:

Screen Shot 2019-02-15 at 20.53.48.png
(side note: the level of tests we are discussing here is completely independent of the tools you may use. The integration tests can be PACT tests or CDCs, they could be written in Selenium or you could use Appium. But we do not want to talk about single tools here, rather about the general concept)

The single service itself is the unit. We know its properly tested and the unit is working as expected. There are a lot of tests in place to make sure this is the case. The base of the pyramid is ok.

The integration becomes very tricky. This is new and was/is not needed in the world of monolithic applications. There is one thing you really – really! – need to be aware of: test the interaction itself – and not two services.
If you have to get two services up and running to make sure that one is working properly it does not sound too bad. But if you scale and you have to get 27 services up and running to test one it just will not work. Too much complexity for a test run.
The challenge is that you will want (need) to test the integration of “your” services without the other ones. An approach that works really well are the consumer driven contract tests. It will blow the scope of this post to explain the concept here again – so I wrote a different post only about those CDCs. Make sure to dig into this!

If you got the trick with the integration, then the rest is “easy”. You will run the standard set of end-to-end tests. And once you have a good feeling about the integration and contract tests, then you will probably also reduce the amount of costly end-to-end tests. If you got all of this in place it would be perfect.

So much for the theory.

While this seems to be a simple principle, the real world looks very different. In many cases I have seen micro services being carved out of an already existing monolith. Furthermore, this usually happens under time constraints and has to be done fast.

This is exactly where a major error occurs, which afterwards propagates through the entire software development cycle: When teams start with the first, small service, the business domain is often unclear. As a result, the system with the Services A, B and C is designed not as above but very different. Remember the three major feature of our examples? Giving just a quick thought about cutting the services, many teams will cut their architecture by those features:

Screen Shot 2019-02-15 at 20.54.43.png

One service handles the dashboard (and thus needs to know about user and encrypted messages, black circle). On service handles the inbox and displays messages (and thus need to know about the user, too, as well as the encrypted messages, yellow circle). The encryption holds/stores all the messages (blue circle).

That means, that the domains of “messages” and “users” is not contained to one service, but instead propagates through the system. And this is the critical point: if we now have a look at how to unit test those domains, the units spread across the services. We need two or three services to write the unit tests. Writing integration tests becomes incredibly complex – or more accurate: hell. Then its also no wonder that some people then tend to leave the integration tests aside and rather cover the entire system by end-to-end tests. The result will look similar to this:

Screen Shot 2019-02-15 at 20.55.31.png

In this situation, the tests are most likely very unstable: as already indicated in the picture there is no clear scheme of what to test where. The integration point of the domains are randomly somewhere in the services. It will be difficult to mock things. If all services are always needed to be available it usually leads to “flakiness” of tests in pre-production environments and it becomes very hard – if not impossible – to test on a local machine. For every fail of the end-to-end tests someone needs to check the error log, in order to find out if it is a “real” error or not. Let me repeat, this is gonna be incredibly complex – or more accurate: hell.

If we then start to automate the entire thing… (you remember: fast feedback and so on) we will end up with a workload that is way beyond what we experienced with the monolith. At this point of time it is perfectly reasonable to do a reality check, whether or not things became easier with the micro services. For all teams that chose this “approach” that I have worked with, we actually figured that we did not improve compared to a monolith. But realizing this is important and valuable:

If your finding is, that it got more complicated then better stop and think how to improve. Because otherwise, as soon as you start to build more and more services and scale your application, it will only get worse. Thus, do yourself a favor and spend a lot of time thinking about the domain split. It will be easier tot test – and scalable!

Screen Shot 2019-02-15 at 20.56.13.png

And at last, the biggest advice I can give to people – teams (!) – that start working with smaller services is to

“Test” in time.

What do I mean? The team needs to be involved early:
Engage with the people designing the services. Talk to the business and understand the domains your are about to form in your services. And make sure that all people in your team have a clear understanding what you are doing and why you are doing this. This is the time where our role stretches far out of the testing area. This allows all of the former testers to grow out of the test manager role. We can make a real difference on the flexibility that our service landscape will provide to our business.  Please make sure, that your team and your business benefits from the approach to micro service. Show them where the pitfalls are. And guide them.

To summarize:

  1. Apply the testing pyramid. Make testing cheap and reliable. (as usual)
  2. Automate everything (that you can). Each manual step is a real show blocker in a micro service environment.
  3. Make it fast. Value the fast feedback from quick running pipelines. Be flexible.
  4. Start in time. Get a clear picture about your business domains and how you think it is – should be – reflected it in your services.

The beauty is: once testing is easy and helpful its a real cool amplifier when you continue to build and improve your system. Have fun 🙂

Are we only Test Manager?

This is a translation of the original blog post that I wrote with Diana Kruse, Natalie Volk and Torsten Mangner. While writing this blog once more just in another language, I took the liberty of adding some personal notes here and there.


In every development team at otto.de there is at least one tester / test manager / QA… or however you would call the person, who is shaping the mindset for quality.

Until recently, “test manager” was the dominating description at Otto – a very rigid and bureaucratic term. Although the intention was good to emphasize that we do not only execute tests, but we also manage them! Meanwhile, even managing tests is only a very small part of the value we deliver.

In a bigger workshop Finn and Natalie, two of our “test managers” picked up on this contradiction and worked things out. We were sure that it would not be sufficient to write “agile” in front of test manager. Hence, we developed a new understanding of our role that looks and feels like this:

coaching1.jpg
We are the teams’ Quality Coaches

We support the teams to understand “quality” as a collective responsibility. We achieve this by working intensively with all roles of the team rather than talk about generic concepts. We establish knowledge and practical approaches regarding the topic of quality.

lifecycle.jpg

We See Through the Entire Story Live Cycle

Together with the team we take care that our high standards of quality are regarded long before the development of our product starts: We suggest alternative solutions during the conception of the story and indicate potential risks. We avoid edge-case problems later on by thinking about them while writing the story. We pair with developers, so that we know that the right things are tested in the right place. Thus, we have more time to talks to our stakeholders and users during the review. With the right monitoring and alerting we are able to observe our software in production.

continuous

We Drive Continuous Delivery / Continuous Deployment

One central goal is to deploy software to the production environment as risk free as possible. Therefore we try to change as little of our codebase as possible and roll out every single commit automatically. We are using feature toggles, to switch on new functionality independent of these deployments. This has two major advantages: we can roll out our software to customers (almost) at the speed of light and get fast feedback for new developed features.

pyramid.jpg

We are Balancing the Test Methods of the Testing Pyramid

We know how to test what on which level of the testing pyramid. We use this concept to create a lot of fast unit tests, a moderate number of integration tests and as few end-to-end tests as possible. This does not only speed up our pipelines but it makes our tests more stable, more reliable and easier to maintain.

Additionally, in our tool box we can find all kinds of tests (acceptance tests, feature tests, exploratory tests), methods (eg. test first, BDD) and frameworks (like Selenium or RSpec). We know how to use those tests, methods and frameworks on all levels of the testing pyramid.

(as a side note: this indeed implies to run eg. Selenium tests on a unit test level if applicable)

agile.jpg

We Help the Team to Choose the Right Methods for a High Quality Product

Being specialists, we know all (dis) advantages of different methods and can help the team to benefit from the advantages. We learned that pairing will enable knowledge transfer, communication, faster delivery and higher quality.  Besides pairing, test driven development is one of the key factors to create a high quality product from the beginning.

Flexible software can only emerge from flexible structures. This is why we are not dogmatic about processes and methods but decide together with the team what mix of processes we really need to get our job done.

pairing.jpg

We are Active in Pairing

We do not only encourage our developers to pair, we also have fun pairing ourselves. In tThis way we can point to problems even while the code is being written. To avoid finding all edge cases only during development we also like to pair and communicate with Business Analysts, UX-Designers and Product Owners. Together with the operations people we will monitor our software in production environment.

The pairing with different people and different roles allows us to further develop our technical as well as domain knowledge.

challenging.jpg

We Represent Different Perspectives

By taking on different positions we prevent unidirectional discussions. We try to avoid typical biases by challenging assumptions about processes, methods, features and architectures. This enables us – from time to time – to show a different solution or an alternative way to solve a given problem. It helps us to reduce systematic errors, money pits and to objectively evaluate risks while developing our software.

communication.jpg

We are Communication Acrobats

We are the information hub for all kinds of things inside and outside the teams. We make special, constructive use of the grapevine, a phenomenon that practically occurs in every company with more that 7 people.

We are enablers for communication. This may be the communication of a pair of developers, between many or all team members or between teams throughout the organization.  By facilitating this coordination we can reduce obscurities about features or integrations of systems and hence get our software into a deliverable state faster.


After developing this role, we engaged more and more of our “agile test managers” with this concept. They were so enthusiastic about it, that they wanted to apply for the job once more right away. The only thing missing was a good name: As in every cross functional team we have different specialists and one of those people is the driving force for high quality we found the perfect name: the Quality Specialist.

specialist.jpg

(Side note: the German term for “quality assurance” (QA) is “QualitätsSicherung” (QS). Using the same abbreviation made it even easier to adopt the new term.)

Quality Specialist is a very well fitting name for this stretched role. Although we are broad generalists, our core value lies in shaping a quality mindset and a culture of quality in a team.

Those were the first steps on a very exciting journey. The next thing to do is talking with other roles in order to find out how this new comprehension of the role changes our daily work. Furthermore, almost no one fulfills this role description today. Thus, we need to grow, level up and reflect on our development. The most fun part is that we can learn a million things in different domains from different people.