50 points from the book_How Google tests Software

50 points from the book: How Google tests software
Book by James A. Whittaker, Jason Arbon, and Jeff Carollo
Originally published: 2012
1. At Google, software testing is part of a centralized organization called Engineering Productivity
2. The Google testing blog – http://googletesting.blogspot.com/2011/01/how-google-tests-
software.html
3. GTAC - GTAC is the Google Test Automation Conference (www.GTAc.biz) - a staple conference
on the industry testing circuit
4. Google is known for its simple and straightforward approach to software. Perhaps this book is in
line with that reputation. We (Google Testers) approach the problems of scale, complexity, and
mass usage.
5. At times, testing is interwoven with development to the point that the two practices are
indistinguishable from each other, and at other times, it is so completely independent that
developers aren’t even aware it is going on.
6. Testing must not create friction that slows down innovation and development
7. Google Test is no million-man army. We are small and elite Special Forces that have to depend
on superior tactics and advanced weaponry to stand a fighting chance at success. As with
military Special Forces, it is this scarcity of resources that forms the base of our secret sauce.
The absence of plenty forces us to get good at prioritizing, or as Larry Page puts it: “Scarcity
brings clarity.”
8. Quality is not equal to test. Quality is achieved by putting development and testing into a
blender and mixing them until one is indistinguishable from the other. Get it right from the
beginning or you’ve created a permanent mess.
9. Test isn’t a separate practice; it’s part and parcel of the development process itself. If a product
breaks in the field, the first point of escalation is the developer who created the problem, not
the tester who didn’t catch it.
10. Development and testing are inseparable from the Google development mindset. Testing must
be an unavoidable aspect of development, and the marriage of development and testing is
where quality is achieved.

11. Motto is “you build it, you break it”
12. The software engineer (SWE) is the traditional developer role. SWEs write a lot of test code,
including test-driven design (TDD), unit tests.
13. SETs (The software engineer in test) are partners in the SWE codebase, but are more concerned
with increasing quality and test coverage than adding new features or increasing performance.
SETs write code that allows SWEs to test their features. SET was the first role that combined the
engineering flavor of a developer with the quality-minded role of a tester
14. The TE (test engineer) role puts testing on behalf of the user first. TEs organize the overall
quality practices, interpret test results, drive test execution, and build end-to-end test
automation. User-focused testing is the job of the Google TE.
15. Google SWEs are feature developers. Google SETs are test developers. Google TEs are user
developers.
16. TEs perform a lot of testing and manage coordination among other TEs, contract testers, crowd
sourced testers, dogfooders, beta users, and early adopters.
17. The term dogfood is used by most software companies in the U.S. to denote internal adoption of
software that is not yet released. The phrase “eating your own dogfood” is meant to convey the
idea that if you make a product to sell to someone else, you should be willing to use it yourself
to find out if it is any good.
18. As a single team, senior managers tend to come from program management or development
and not testing ranks. In the push to ship, priorities often favor getting features complete and
other fit-and-finish tasks over core quality. The tendency for such organizational structures is for
testing to be subservient to development.
19. Testers are assigned to product teams by Engineering Productivity leads who act strategically
based on the priority, complexity, and needs of the product team in comparison to other
product teams. This creates a balance of resources against actual and not perceived need. As a
central resource, good ideas and practices tend to get adopted companywide.
20. There’s no faster way of moving innovations in test than moving the actual innovators.
21. Google is a company full of testers who understand client, web, browser, and mobile
technologies, and who can program effectively in multiple languages and on a variety of
platforms.
22. This is what we did with Gmail, a product that kept its beta tag for four years. That tag was our
warning to users that it was still being perfected. We removed the beta tag only when we
reached our goal of 99.99 percent uptime for a real user’s email data.
23. Google often builds the “minimum useful product” as an initial version and then quickly iterates
successive versions allowing for internal and user feedback and careful consideration of quality
with every small step. Products proceed through canary, development, testing, beta, and
release channels before making it to users.
24. Canary Channel: This is used for daily builds we suspect aren’t fit for release. In general, only
engineers (developer and testers) and managers working on the product pull builds from the
canary channel.

25. The Android team goes one step further, and has its core development team’s phone
continually running on the nearly daily build. The thought is that they will be unlikely to check in
bad code it if impacts the ability to call home.
26. Dev Channel: This is what developers use for their day-to-day work. These are generally weekly
builds that have sustained successful usage and passed some set of tests.
27. Test Channel: This is essentially the best build of the month in terms of the one that passes the
most sustained testing. At some point, a Test Channel build becomes stable enough to be used
internally companywide and sometimes given to external collaborators and partners who would
benefit from an early look at the product.
28. Beta Channel or Release Channel: These builds are stable Test Channel builds that have survived
internal usage and pass every quality bar the team sets.
29. This crawl, walk, run approach gives us the chance to run tests and experiment on our
applications early and obtain feedback from real human beings in addition to all the automation
we run in each of these channels every day.
30. Types of Tests: Instead of distinguishing between code, integration, and system testing, Google
uses the language of small, medium, and large tests (not to be confused with t-shirt sizing
language of estimation among the agile community), emphasizing scope over form.
31. Small tests: The question a small test attempts to answer is, “Does this code do what it is
supposed to do?” They are most likely written by a SWE, less often by an SET, and hardly ever by
TEs. TEs rarely write small tests but might run them when they are trying to diagnose a
particular failure.
32. Medium tests: The question a medium test attempts to answer is, “Does a set of near neighbor
functions interoperate with each other the way they are supposed to?” .These are usually
automated and involve two or more interacting features. SETs drive the development of these
tests early in the product cycle as individual features are completed and SWEs are heavily
involved in writing, debugging, and maintaining the actual tests.
33. Large tests: The question a large test attempts to answer is, “Does the product operate the way
a user would expect and produce the desired results?” All three roles are involved in writing
large tests and everything from automation to exploratory testing can be the vehicle to
accomplish them.
34. The original purpose of using small, medium, and large was to standardize terms that so many
testers brought in from other employers where smoke tests, BVTs, integrations tests, and so on
had multiple and conflicting meanings. Those terms had so much baggage that it was felt that
new ones were needed
35. The mix between automated and manual testing definitely favors the former for all three sizes
of tests. If it can be automated and the problem doesn’t require human cleverness and intuition,
then it should be automated.
36. For example, if an automated test breaks, the system determines the last code change that is
the most likely culprit, sends email to its authors, and files a bug automatically. The ongoing
effort to automate to within the “last inch of the human mind” is currently the design spec for
the next generation of test-engineering tools Google builds.

37. It becomes necessary to distinguish between a feature developer and a test developer. For
feature code, the mindset is creating and considering users, use cases, and workflow. For test
code, the mindset is about breaking and writing code that will draw out cases that disrupt the
user and his workflow.
38. It is interesting to note that even with client software, Google attempts to make updating
frequent and reliable with the development of an “auto update” feature, a necessary feature for
any client application.
39. “The Life of” is a tip of the hat to a series of Google internal courses on how Search and Ads
work. Google has courses for Nooglers (new Google employees) called Life of a Query that
details the dynamics of how a search query is implemented and Life of a Dollar to show how Ads
work.
40. Google is one company and has one easily searchable (of course!) source repository. This
companywide sharing of resources has enabled the development of a rich set of shared code
libraries and services.
41. “Twenty percent time” is what Googlers call side projects. It’s more than a concept; it is an
official structure that allows any Googler to take a day a week and work on something besides
his day job. The idea is that you spend four days a week earning your salary and a day a week to
experiment and innovate. It is entirely optional and some former Googlers have claimed the
idea is a myth. In our experience, the concept is real and all three of us have been involved in 20
percent projects. In fact, many of the tools outlined in this book are the result of 20 percent
efforts that eventually turned into real, funded products.
42. Engineers get a lot of credit for writing a service that is picked up by multiple teams. Reuse is
rewarded far more than complexity or cleverness. If an engineer comes up with a better way of
doing something, he is tasked with refactoring all existing libraries and assisting dependent
projects to migrate to the new libraries.
43. Google takes code reviews seriously, and, especially with common code, developers must have
all their code reviewed by someone with a “readability” in the relevant programming language.
Readabilities exist for C++, Java, Python, and JavaScript: Google’s four primary languages.
44. The most common of which is Google’s often touted “peer bonus” benefit. Any engineer whose
work is positively impacted by another engineer can serve a peer bonus as a thank you.
Managers additionally have access to other bonus structures. The idea is that impactful
community work be positively reinforced so it keeps happening!
45. If a bug occurs on a tester’s machine, chances are it will reproduce on a developer’s machine
and in production. Finally, for each programming language Google uses, there is exactly one
compiler, which is well maintained and constantly tested against the one Linux distribution.
46. Keeping it simple and uniform is a specific goal of the Google platform: a common Linux
distribution for engineering workstations and production deployment machines; a centrally
managed set of common, core libraries; a common source, build, and test infrastructure; a
single compiler for each core programming language; language independent, common build
specification; and a culture that respects and rewards the maintenance of these shared
resources.

47. The only local test labs Google maintains outside this common infrastructure are for Android
and Chrome OS where various flavors of hardware must be kept on hand to exercise a new
build.
48. Overall flow of build process:
 Write a class or set of functions for a service in one or more source files and
make sure all the code compiles.
 Identify a library build target for this new service.
 Write a unit test that imports the library, mocks out its nontrivial dependencies,
and executes the most interesting code paths with the most interesting inputs.
 Create a test build target for the unit test.
 Build and run the test target, making necessary changes until all the tests pass
cleanly.
 Run all required static analysis tools that check for style guide compliance and a
suite of common problems.
 Send the resulting code out for code review, make appropriate changes, and
rerun all the unit tests.
49. Test is part of the functionality, and buggy tests are functional bugs and must be fixed. This
ensures that new functionality does not break existing functionality and that any code
modifications do not break any tests.
50. SETs Role: It creates an equal footing between feature developers and test developers that is
productive and lends credibility to all types of testing; including manual and exploratory testing
that occurs later in the process and is performed by a different set of engineers.
51. SETs: SETs sit side by side with feature developers (literally, it is the goal to have SWEs and SETs
co-located). It is fair to characterize test as just another feature of the application and SETs as
the owner of the testing feature. SETs also participate in reviewing code written by SWEs and
vice versa.
 In other words, both SWEs and SETs answer coding questions. SETs are expected to nail
a set of testing questions as well.
 Test is just another feature of the application, and SETs are the owner of the testing
feature.
 There is a cross section of these communities, SWE-leaning SETs, and SET-leaning SWEs,
who are some of our best engineers and make up some of our most effective product
teams.

50 points from the book_How Google tests Software

More Related Content

What's hot (20)

Similar to 50 points from the book_How Google tests Software (20)

50 points from the book_How Google tests Software