20161121 goat2016 - ken mcmillan - bas lessons learned - all notes

These notes deliberately left free of chocolate
1
Ken McMillan | Gatineau Ottawa Agile Tour
2016 | 2016Nov21

In this culture, people say yes way too often
2
2016 | 2016Nov21

3
2016 | 2016Nov21

Tender notices always in top 10
Contract history always in top 25
4
2016 | 2016Nov21

5
2016 | 2016Nov21

6
2016 | 2016Nov21

7
2016 | 2016Nov21

8
2016 | 2016Nov21

1. Make many mistakes often – not a surprising approach to an Agilist
2. Say no often (remember Steve Jobs?)
9
2016 | 2016Nov21

--------------------
Do not allow a long-horizon project management methodology to override practical,
business-driven outcomes with early, demonstrable results. If you assume you know
more about the problem than you really do, you risk committing to a solution path
too early.
Risks to manage (warning signs):
Expending a great deal of energy, time, and money prior to seeing any tangible
stakeholder-facing deliverables.
Committing to highly described (function and cost) outcomes in the distant future
(longer than 6 months away).
Suggesting that all risks can be identified and documented prior to project
commencement.
Stakeholder resistance to engagement due to the long or unknown amount of time
prior to a tangible result.
Stakeholder resistance to engagement due to the overly-early descriptive solution
provided with little input on their part.
10
2016 | 2016Nov21

What we did:
We engaged key stakeholders early and often. We provided tangible, usable examples
to guide discussion and build confidence.
We talked to key stakeholders to discover and document their perceptions of their
needs, problems, and desired outcomes.
We did not necessarily talk about requirements as stakeholders are likely uncertain
about end-state solutions.
We assumed seeing is believing and used working proof-of-concept demonstrations
and pilots to help stakeholders envision outcomes and discover needs.
We learned to be more tolerant of our own mistakes and used post mortems to gain
insight and discover what the problems really were.
We sought incremental funding and attempted to have useful deliverables such that
early termination would still have a positive outcome.
What we recommend:
Adopt an open and flexible approach incorporating problem discovery in the early
stages of a project.
Consider early engagement of stakeholders via demonstrable and tangible outcomes.
Be prepared to control the stakeholder desire to jump ahead during problem
discovery and move towards solutions.
Be prepared to control your own tendency to assume you understand the problem or
business environment better than the stakeholder.
Anecdotal example(s):
Early in the program we developed a simple concept to combine a number of
Government of Canada legacy services into one single web site. We believed this new
site should focus on helping business people, allowing them to easily find information
to help them do business with the Government of Canada. This user-task oriented
approach was in contrast with the incumbent practice of telling stakeholders what we
considered important in Government of Canada terms. It was client-driven.
Intuitively, we believed that changing both the what (a single website) and the how (a
new way to organize and present the information) would be a difficult transition,
particularly for our internal stakeholders.
Our first two deliverables were a User Requirements document and a simple proof-of-
concept. The User Requirements document allowed internal stakeholders to express
their opinions, needs, pain points, and so on, such that they could be synthesised into
a user-driven story. This was used as input for the second deliverable, a working
proof-of-concept.
The working proof-of-concept addressed feasibility questions around modern tools
for web site management and allowed the small project team to re-engage internal
stakeholders. Stakeholders were able to see and click on a working demo, building
2016 | 2016Nov21
10

confidence and providing a stepping stone for further problem discovery by all
parties.
2016 | 2016Nov21
10

Stakeholder expectations must be managed throughout the project.
Successful projects deliver a working service to their stakeholders. They must do
what is expected and stakeholders must be willing to use them. Unsuccessful projects
miss one or both of these conditions.
Stakeholders discuss business requirements too early. Business requirement
discussions lead to an overly specific solution-orientation that is not achievable early
in a project.
Stakeholders express concern (often informally) they are committing to something
they don’t fully understand in terms of the impact on their business, budget, or both.
Stakeholders focus more and more on process (how) rather than outcomes (what) as
their fears increase regarding project success.
What we did:
As the project evolved we expanded the notion of user engagement to a broader
community engagement.
We designated a small, skilled team of roadmap managers that met often. They were
ambassadors for the project amongst diverse communities including management,
11
2016 | 2016Nov21

end users, program managers, standards stewards, and operational groups.
We presented potential changes to stakeholders and listened to their feedback on
how those could affect their work.
The roadmap managers interacted with focused design and development teams to
incorporate environmental scans, best practices, standards, and appropriate
technology into priorities.
The roadmap managers were accountable for describing and prioritizing business
needs.
What we recommend:
Build a small team of experienced, multidisciplinary decision-makers who listen.
Listening and distilling are key skills.
Adjust the degree of stakeholder involvement to suit the degree of change to the
stakeholder.
Engage early with potentially challenging communities that may be threatened by or
adverse to change.
Do not overlook smaller communities that may have interesting insights or needs.
Government has a number of obligations that it must meet to deliver services that
involve data. A key obligation is to put in place appropriate business and
technological safeguards to protect Canadian’s data. We were mandated to follow the
Certification and Accreditation (C&A) process to demonstrate our solutions to these
responsibilities.
We engaged an Information Technology Security team early in this process. Together,
we devised a work plan that ensured the new service met the requirements of the
C&A process in a manner that would not delay the delivery of the service or put
undue pressure on the security team due to late engagement in the overall project.
2016 | 2016Nov21
11

Traditional project management systems (often modelled on construction projects)
tend to deliver usable components only at the very end of a long development cycle.
To have meaningful engagement, stakeholders must be able to judge tangible results
on a regular basis throughout development.
Demos to user and program stakeholders
During the early discovery and pre-launch phases of the project, we convened a
working group of program officers from within the Branch. We demonstrated our
progress and asked them for feedback using demos that provided a tangible
reference point for engagement. After the official launch, our engagement focus
shifted to a broader external audience. We were also able to leverage analytical data
from website usage logs.
Milestones are marked only by status reports rather than demos, proofs-of-
concept, pilots, or high-quality betas.
Waiting until demos are feature-complete before soliciting feedback.
Failing to prune or defer changes that do not align with a clear guiding principle or
12
2016 | 2016Nov21

strategic intent.
Failing to consider non-strategic changes that stakeholders are heavily invested in due
to tactical realities.
What we did:
We had regular stakeholder engagement meetings with demos of working code.
These purposely only showed a limited number of features, but they were real, and
they allowed the stakeholders to judge what constituted real improvement in their
eyes.
Our roadmap managers used the feedback from these demos to prune or defer
changes that failed to align with a strategic intent or that had significant pushback
from the stakeholders.
Our roadmap managers evaluated the feedback with particular attention paid to non-
strategic items that were obvious hot-buttons for the stakeholders.
We attempted to show in subsequent demos to the same stakeholders how their
input had influenced the project.
What we recommend:
Demo early, demo often, and respond to feedback.
Have the roadmap managers ask themselves such questions as: “Does this move us
toward open, web-based apps and services, or does it keep us in the closed,
proprietary world?”, “Would I be proud to share this as a best practice with my
colleagues, or am I glad it’s happening where no one can see it?”, “Does this change
contribute to the overall web and service development and operations community?”
Be aware that taking technical shortcuts (using hacks) to meet a demo schedule can
lead to compromises that may need to be fixed at a later date. The roadmap
managers need to weigh the potential cost of the fix against the benefit of the
confidence engendered by a smoother demo.
Strategic intent may not be explicit or well defined at the beginning of a project.
There may even be a tactical advantage to deferring an explicit definition. Make sure
the roadmap managers are prepared to discover and refine the strategic intent
themselves throughout the development process.
2016 | 2016Nov21
12

Do not attempt to deliver all of the solutions to all of the problems at the end of an
extended development effort. Rather, provide usable solutions in an incremental
fashion, such that stopping at any point still leaves you with measurable gains and
working systems.
A long development phase with no intermediate partial solutions. (Note that a partial
solution must be a working and stable deliverable - not just a demo or pilot.)
Large expenditures with no viable deliverables.
Expectations from stakeholders for a complete solution long before all problems have
been identified.
Reluctance of stakeholders to provide signoffs on partial deliverables, even when the
deliverables provide viable solutions.
What we did:
We managed the project in stages, where each stage comprised a viable business
deliverable. Although subsequent stages often relied on previous stages, each stage
worked as a deliverable unit, with no subsequent dependencies.
We learned not to ask the underlying technology to do something it was not designed
13
2016 | 2016Nov21

to do. (This lesson was sometimes learned by negative example.) Because
development was staged, such a lesson learned early on would benefit many
subsequent stages.
We learned not to attempt to fix a broken business process with a software solution.
Rather, the roadmap managers were able to engage with the stakeholders to fix the
processes at the business level.
By delivering often, we were able to keep expenditures in line with benefits and could
demonstrate to management the value being provided. This also made management
much more confident about signing off on future stages.
The roadmap managers were charged with managing stakeholder expectations.
Although many stakeholders initially expected a complete solution, they quickly
realized the benefit of having usable incremental improvements.
What we recommend:
Implement early and often. It is much easier to redirect efforts when feedback is
received in a timely manner.
Do not allow analysis cycles to become too long. The validation provided by a
working component outweighs the value of having a perfect design.
Know your tools. All software has best-case design patterns. Use them to your
advantage.
Allow the roadmap managers to balance technical capabilities against business
requirements. Do not allow business requirements to dictate technical solutions.
Similarly, do not let technical concerns affect business requirements.
During the first two years of operation of Buyandsell.gc.ca, we delivered a variety of
incremental improvements based on business drivers and stakeholder feedback. Each
of these delivered a new public-facing service and allowed the team to introduce or
refine processes and services. This continuous improvement allowed the delivery
team to learn by doing. Each new capability built confidence within the team and
demonstrated a commitment to listen and improve to the stakeholders. This
eventually led to the launch of buyandsell.gc.ca/tenders which became the official
Government Electronic Tenders Service in June 2013.
May 2010: Pre-Launch (gain early insight, exercise new web publishing
process)
Sept 2010: Public Launch (new high value content, namely: Canadian
Innovation Commercialization Program (now the Build in Canada Innovation
Program), For Government, For Business, Find Goods and Services)
Jan 2011: News Items capability (ability to publish short timely notices)
June 2011: Service Selector Tool for Professional Services
Sept 2011: Standing Offer/Supply Arrangement (data available as PWGSC’s
first Open Data set)
2016 | 2016Nov21
13

Oct 2011: Event Calendar (first ‘self publishing’ service)
Jan 2012: ‘Buy Green’ Micro-site
March 2012: Supply Manual & Standard Acquisitions Clauses and Conditions
June 2013: Government Electronic Tendering Services
2016 | 2016Nov21
13

Example:
The Accounts module is an excellent example of a self-contained, well-defined
component that interoperates with all of the other parts of the system through a
clean, modern interface. It has no knowledge of the business processes inside the
other services – it provides platform-level security enforcement, service and user on-
boarding, authentication and role-based authorization. In practical terms, this means
that even if the business rules for another module (such as the Tender Management
App) need to change, it does not affect Accounts. Similarly, Tender Management does
not have to know anything about Account management processes – it simply
consumes the service.
--------------------
Take advantage of modular delivery to allow separation of concerns. By managing
each module separately, you can use tools, products, and business knowledge
specific to the module to its greatest advantage.
14
2016 | 2016Nov21

Tightly coupled, complex communication between modules (keep it simple).
Lock-step dependencies between modules, such that if development on one stops,
development on all stop.
Large teams trying to manage too many components.
What we did:
We divided the project into self-contained modules such as Accounts, Tender
Management, Publishing, and the Supply Manual.
We ensured that each module had a clean, outward-facing interface that let other
modules interoperate with it without being dependent on any internal details.
We developed shared standards for all modules to simplify and standardize their
development and interoperation.
What we recommend:
Be sure to dedicate sufficient resources to standards development and enforcement.
Strive to keep the system architecture loosely coupled.
Swap team members from time-to-time to encourage cross-fertilization.
2016 | 2016Nov21
14

Integration, whether between modules or different systems, requires too much
knowledge of the other component’s internals. Interoperation through a clean
external interface allows modules to remain independent and autonomous.
Attempting to integrate with existing systems and other modules.
Tightly coupled, complex communication between modules (keep it simple).
Asking external systems or modules to change their interface.
Pushback from external system stewards concerned about changes they might have
to make.
What we did:
We began interoperating with external systems at an early stage. Testing for
interoperability was not left for the end of the project . It was executed incrementally.
Accommodated external systems by reverse engineering their data structures.
Built data adaptors and connectors to allow our modules to consume data from and
supply data to external (legacy) systems.
Built clean XML-based interfaces between modules.
Built standards-based (Atom) data publishing streams.
15
2016 | 2016Nov21

What we recommend:
Build simple, standards-based interfaces for system modules.
Explain to external data suppliers and consumers that the interface requires no
change to their processes or internals.
Encourage owners of external systems to start testing for interoperability at an early
stage.
2016 | 2016Nov21
15

Open technologies provide many significant advantages: they have no external
platform dependencies, they are easy to procure, they are broadly supported by large
user communities, and they do not create large sunk costs.
Attempting to choose a product with perfect fit. Requirements may change
(rendering the fit imperfect) and a good fit is almost always sufficient.
Using the very latest branch or version of a dynamic product. The so-called long term
support (LTS) versions are a superior choice for all but the most demanding
applications that require the latest features.
Procurement professionals or managers unfamiliar (and uncomfortable) with the
open source model and its benefits.
What we did:
We actively promoted the benefits of open-source solutions to our stakeholders,
particularly the absence of vendor lock in and the broad base of usage and support.
We did feature versus benefit analysis even for support system software (ticketing &
bug tracking system and documentation wiki).
We reviewed the level of community support and adoption of the products, using this
16
2016 | 2016Nov21

data as a proxy for the product utility and stability.
We sought guidance and advice from other users (both public and private) who had
used the products to deliver new services and systems.
What we recommend:
While it is important to match product features against your anticipated
requirements, remember and use the 80/20 rule. If the product matches 80% of your
known requirements, that is probably good enough.
Use part of the effort that would have been expended on a costly RFP exercise to
analyze several open technology options.
Engage with the product’s user community. They can provide a wealth of information
and a potential pool of resources.
The entire Buyandsell.gc.ca platform leverages widely adopted open-source software
and open standards. Open-source software is software that is created by a collective
of individuals (often backed by a major foundation or organization) and shared under
one of a number of common open licences. Open-source software makes evident the
fact that a number of problems are so well understood that it is possible to
collectively create software that codify the solutions. Previously challenging problems
have been elegantly solved with open-source software: operating systems, web
servers, networking software, content management systems, and many others. Even
companies that provide closed-source software recognize the value in open-source
solutions. For example, Apple now gives away its main computer operating systems
Mac OS and iOS.
Open-source technologies are particularly desirable in Government. Open-source
greatly simplifies the procurement process. Procurements are service-based, which is
less challenging and time consuming than proprietary software procurement. Open-
source software conforms to industry standards which prevent technology lock-in.
Enhancements and revitalization of open-source solutions is not tied to a single
vendor or technology. An active community of independent, collaborative developers
keeps systems up to date.
2016 | 2016Nov21
16

Buyandsell.gc.ca/tenders was a new service that incorporated functionality that had
been delivered by a long-standing third-party service to the Government of Canada.
This third-party service was well understood by the key stakeholders (business,
Government procurement officers). It had been adapted in small ways over many
years to meet special quirks (usually undocumented) in the procurement process.
As we explored the requirements and began implementation of
Buyandsell.gc.ca/tenders we failed to refer back to the existing provider’s
implementation. This was likely a result of a bias that the new solution would be more
effective and efficient. While the new service did achieve its goals, there were a
number of areas where we experienced issues post-launch that could have been
avoided by paying more attention to what was already implemented and widely
adopted.
--------------------
When replacing or improving a service, there is a tendency to start with a clean slate
without regard to the utility or functionality of existing systems and procedures.
17
2016 | 2016Nov21

Assuming that since part of a process is broken, all of it is broken.
Believing that stakeholders can accurately describe how they work, when, in fact,
they may neglect to mention many implicit assumptions that they use.
Let bad presentation blind you to the underlying utility of the information being
presented.
Focussing on one aspect of a process at the expense of another equally (or more)
important one.
What we did (wrong):
We focussed on the front end of the procurement process (gathering the data from
procurement professionals) and paid less attention to the back end (presenting the
data to the public).
We spent a lot of time engaged with demos to government procurement
professionals and relatively little with public data consumers.
We did not promote the benefits of the new cost-free (and registration free) tenders
system as aggressively as we should have.
What we recommend:
Engage with stakeholders at both ends of your processes with equal vigour.
Don’t ignore existing things that work well – even if you plan to replace them.
Be humble. Your improvements may actually make things worse.
2016 | 2016Nov21
17

Avoid two common pitfalls:
1) Using your development environment as a demo environment;
2) Taking shortcuts to make demos work. Make the demos even more useful by using
the collected feedback for the next demos.
No dedicated demo environment.
Demo systems that appear to function, but are actually little more than scripted
animations.
Attempting to mask any and all problems that a demo system may experience.
What we did:
We used three environments: 1) development; 2) staging/demo; 3) production. This
allowed development to continue even while demos were underway (where a series
of demos might take a few weeks) and it insulated the demos from development
changes.
We showed demos with warts and all which actually increased the confidence of the
audience in the validity of the demo and allowed us to show the direction
improvements were taking even if not complete.
18
2016 | 2016Nov21

What we recommend:
When you use demos to solicit feedback, make them real – otherwise the feedback
may be invalid and you may miss an opportunity to make small adjustments with a
big payback later in the project.
Do not avoid giving a demo with known flaws. Stakeholders will still see visible
improvements and have more realistic expectations of what can be done in a given
time frame.
Throughout the project we used working demonstrations to both engage
stakeholders as well as to facilitate communication and understanding amongst the
small project teams. Overall, this was an effective approach and it provided visible,
date-driven milestones to maintain project delivery momentum.
However, from time-to-time, particularly if the scheduled demonstration was to a
senior executive stakeholder group, the team took shortcuts in order to meet the
deadline. Typically, the shortcut was a faster, easier, but ultimately less sustainable
solution to the problem at hand. This often led to later implementation issues and
required re-work. In the industry this is often referred to as technical debt. At some
time the project will need to repay that debt in time and resources.
Earlier in the project, when computing resources were scarce, the demo effort could
block or slow down efforts for the main stream of the project deliverables.
Demonstrations are vital to solicit correctional feedback, but they do need to have
appropriate resources and planning to be used most effectively.
2016 | 2016Nov21
18

At some point, discovery and development will give way to operations. Developers
should strive to make the transition as frictionless as possible.
Imposing operational disciplines and change management too early in the discovery
and development phase where it will hinder rapid progress.
Attempting to retrofit operational reporting and management tools into completed
components.
Ignoring operational realities during development. Operations should be a vital,
contributing stakeholder.
Having QA focus solely on functional specs. QA should also consider operational
concerns and constraints.
What we did:
As part of the C&A process (see lesson 2) we created all of the required standard
operating procedures (SOPs) for an operational environment.
We used the SOPs to inform the efforts of developers, particularly in the areas of
audit logging, system security, and change management.
At a very early stage, we began using three environments – one of which was
19
2016 | 2016Nov21

production. This gave us early feedback on operational requirements.
What we recommend:
Engage the operations community at the earliest stages of the project.
Recognize and acknowledge the difference between development and operations
culture.
Ensure that operations have an opportunity to review early demos where they may
be able to suggest improvements or spot problems from an operational perspective.
One of the benefits of delivering smaller, modular components is that they can be
delivered into an operational environment with little or no disruption.
When first released in May 2010 as a pre-launch service, Buyandsell.gc.ca was a fairly
straightforward content management web solution. By September 2010, the team
had gained four months of operational experience in a relatively low pressure
environment.
After the official launch in Sept 2010, features were incrementally added - allowing
gradual and increasing opportunities to gain operational experience. By the time the
service was released as the full Government Electronic Tendering Services, the team
had over three years of operational experience to help manage the transition to this
new and very visible environment. As a measure of complexity, Buyandsell.gc.ca went
from a two virtual machines at the start to over thirty for the version released on
June 1st, 2013.
One significant challenge during the entire project was having limited financial
resources which meant we often did not have the optimal number of people to do
the work. Many individuals had to switch from design and build mode to operations
mode - sometimes more than once a day. This affected the individual’s efficiency, and
as a result, the delivery schedules.
2016 | 2016Nov21
19

20
2016 | 2016Nov21

21
2016 | 2016Nov21

22
2016 | 2016Nov21

23
2016 | 2016Nov21

20161121 goat2016 - ken mcmillan - bas lessons learned - all notes

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to 20161121 goat2016 - ken mcmillan - bas lessons learned - all notes (20)

Recently uploaded (20)

20161121 goat2016 - ken mcmillan - bas lessons learned - all notes