Todd Sedano

Software Engineering, Improv, Craftsmanship

Team Code Ownership

Team Code Ownership is “the ability for any developer on a team to change any of the team’s code.”

I’m presenting a paper on Team Code Ownership at EASE 2016 next week.

Read the paper on-line: https://goo.gl/vOeeMn Or download the pdf: https://goo.gl/LXPg36

Summary of Paper

  • My observations clearly indicate that team code ownership is a feeling that can be engendered, not a policy that the team adopts

  • Pivotal developers more acutely feel team code ownership when i) they understand the system context; ii) they have contributed to the code in question; iii) they perceive code quality as high; iv) they believe the product will satisfy user needs; and v) they perceive team cohesion as high.

  • The paper shows that diverse events and trends that can undermine sense of ownership

  • The paper replaces Beck’s definition of Collective Code Ownership (“anyone can change any piece of code in the system at any time”) by introducing the new term Team Code Ownership (“the ability for any developer on a team to change any of the team’s code”)

Theory of Sustainable Software Development Through Team Code Ownership

The theory describes how teams can continue to deliver value in spite of team disruptions. The theory is a collection of synergistic principles, policies, and practices encouraging a positive attitude towards team disruption, knowledge sharing and continuity, as well as caring about code quality. The Theory of Sustainable Software Development through Team Code Ownership is fully presented in the paper.

Principles

The theory underlying principles are as follows: Keep a Positive Attitude Toward Team Disruption by recognizing the value that new team members bring with their fresh perspectives and challenging team assumptions; Encourage Knowledge Sharing and Continuity by enabling the knowledge to spread from one developer to the next, eventually reaching the entire team; and Care about Code Quality by recognizing that a well cared for code base makes modifications easier.

Policies

Team Code Ownership empowers engineers to modify any part of the system under the team’s responsibility. When engineers agree that a section of the system needs to be changed, the team proactively and tacitly authorizes the change. Shared Schedule aligns the team’s work schedule to enable efficient daily rotation of the developers working on a track of feature development. Avoid Technical Debt enables a team to balance feature development with Continuous Refactoring. When a team is pressured to finish work by a deadline, it might be tempted to focus on feature delivery, stop refactoring, and hence take on technical debt. The team should prefer consistent software development by caretaking the code and leaving the code base in a state where any member of the team can pick up the next story for that part of the code base. This requires product and management support to avoid unnecessary thrashing of developers rushing incomplete work by taking on technical debt to deliver a feature early.

Removing Knowledge Silos Practices

By sharing knowledge throughout a team, any pair will have enough context to understand what needs to be done and know who to ask if they need more details. Continuous Pair Programming reinforces that there is no single owner for any line of written code. The team owns the code. As two developers collaborate, they generate shared context. That knowledge will spread around the team the following day via Overlapping Pair Rotation which explicitly rotates people off a track of work in order to cross-pollinate knowledge and avoid emergent knowledge silos and individual ownership. Knowledge Pollination is activities that promote knowledge sharing in unstructured ways between pairs and includes activities such as daily stand-ups, writing on whiteboards, overhearing a conversation, calling out an update to the team, or simply reaching out to others to ask questions as needed.

Caretaking the Code Practices

By caretaking the code, the team enables any pair to be able to work on any story in the backlog. Test Driven Development (TDD) / Behavior Driven Development (BDD) creates a safety net of tests that give any pair the courage to modify the system without the fear of breaking some other part of the system unexpectedly. The tests become a specification on how each component is used. Continuous Refactoring increases code quality by increasing code discoverability (knowing where to find the responsible code), increasing code readability, and increasing the simplicity of design. Live on Master enables the team to perform Continuous Refactoring as the code is easy to integrate. When teams have long running development branches, merge conflicts discourage Continuous Refactoring.

Performance Pair Programming

Performance pair programming is when one developer drives the entire implementation while the other watches. It is as if someone is solo programming with an audience. The performer prefers accomplishing a task more than taking advantage of the teaching opportunity.

When this occurs, there is no collaboration or discussion about how to proceed. Questions that the watcher asks are often ignored. Typically the Performer is unaware that their pair is not learning anything.

Since one of the most effective learning technique is “learn by doing,” the pair is missing a great learning opportunity.

The Origins of Lean Extreme Programming

Lean Extreme Programming is the combination of practices from Extreme Programming, Kanban software development, User centered design, and Lean Start-up.

Kent Beck’s Extreme Programming serves as the bedrock foundation for the practices at Pivotal. The company through Rob Mee experimented and piloted these ideas which formed the DNA for our software development practices. Over time these evolved. 1) The core technical practices remain the same. 2) Managing the backlog evolved from managing commitments for an iteration to just in time scheduling of work. By creating small stories, the need for story to task decomposition disappeared. Iteration planning meeting morphed into bi-directional communication about the immediate work to be done in order to create a shared context for the whole team. This removes unnecessary team task breakdown waste, mid-week checking in on programmers waste. It enables product to change the backlog whenever necessary by shifting stability from the iteration to the story. It enables developers to be present with the work and not focused into the future. 3) While Extreme Programming shifted the definition of features from engineers to product, XP did not specify how product determines these features (See below for more details.) XP enabled the developers to focus on the how and product to focus on the what.

When I first learned about Extreme Programming, it was described to me as “cowboy programming” and an excuse for developers to run amok with little discipline. I have been programming and teaching it since 2005 and can currently cannot imagine a saner way of developing software. I find it incredibly disciplined, applicable to many contexts, and I rely more heavily on it for challenging projects.

Lean Extreme Programming’s programming in the present comes from Kanban software’s idea of no iterations. The team looks to design and product to periodically communicate and discuss the next bit of work to be done. Typically this happens weekly or more frequently. Since the team is not focused on a fixed amount of work for a period of time, changes in priority do not necessarily interrupt team productivity. If design discovers a problem with a user interaction, a story to fix the issue can be added into the backlog and prioritized as the next most important thing to be done. Problems can arise if the team has no context on the work to be done, and performing an impromptu shared understanding meeting alleviates this problem. If too much trashing happens, the team needs to understand why there are daily fluxes in prioritization.

When I first heard about no iterations, I thought the idea was crazy. However, having experienced it myself, I appreciate the ability to show up at work and see what is the next important thing to get done.

Lean Extreme Programming’s validation of features by users comes from user-centered design and popularized by Lean Startup. The approach prioritizes the user’s needs over perceived business desires and suggests that conflicts between the two should result in examining the business desires in order to align with what the consumer wants. This idea also comes from Lean Thinking which defines delivering only what the user wants as value and delivering anything that the user does not want as waste.

Lean Software Development is not a direct influence on Lean Extreme Programming. Several of the ideas of lean thinking for reducing do show up. (Co-located teams, short feedback loops, anyone can stop software development by starting a team huddle.) Just as it is problematic to apply construction practices to software development, so to it is to apply manufacturing practices to software development. In Lean Thinking, most of the attention is on optimizing the production of identical components on an assembly line, not discussing how to optimize the design process of creating new products. While some software products are simply configured, at Pivotal, most it is rare for us to build the same component over and over again. Whenever that occurs, we design it once or refactor the code so that adding more is a trivial exercise. Since the work that we do is not repetitive like an assembly line, applying manufacturing technics to software development can be problematic.

“Renaming the Iteration Planning Meeting”

Summary

I suggest renaming the “Iteration” “Planning” Meeting (IPM) to accurately reflect Pivotal’s current goals and practices for that meeting.

Context

Over time, Pivotal has evolved its own form of Extreme Programming software development methodology by incorporating ideas from Lean Thinking from manufacturing and Lean Startup.

Some of our terminology needs refreshing to reflect the way we currently work. In particular, “Iteration Planning Meeting” does not accurately reflect what typically happens in that meeting.

In Extreme Programming (1999) and Extreme Programming 2nd Edition (2004) the development team commits to the stories to be accomplished during an iteration or weekly cycle just like a sprint in Scrum. The commitment can change as the week unfolds. Once Product identifies the stories for the week, programmers decompose the stories into tasks. After volunteering for a set of tasks, each programmer estimates only their tasks on Monday during an Iteration Planning Game (XP 1999) or Weekly Cycle meeting (XP 2004.)

In contrast, Pivotal follows a “continuous flow” where the work “flows” through people as people start the next needed story in the backlog. We limit work in progress to a story per pair of programmers. At Pivotal, we’ve customized the Extreme Programming process so that the developers accomplish the work that needs to get done at that moment in time. At any point in time, we can not predict who will be working on what work tomorrow, but we can say who is working on what work now. As that work is finished, we simply start the next unit of work to be done. We do not commit to which stories will be done in any given week, instead we follow a Lean Thinking’s just-in-time mechanisms for identifying what to work on next.

At Pivotal, stories and tasks are not owned by specific individuals. In Scrum and Extreme Programming, one person owns the task for the duration of the task. At Pivotal, many people might work on a single story. While the team notes who is working on each story right now, the team understands that these associations will change. After finishing a task, a development pair simply works on the next available story.

We’re familiar with the multi-day story: On Monday, Bailey and De begin working from the top of the backlog. On Tuesday, De and Kadeer continue on the story. On Wednesday, Kadeer and Lesley finish the story. For the participants, this can feel like a relay race. The entire team might even rotate through a complex story.

Pivotal prefers very small stories in order to deliver value quickly, remove engineering waste, shorten the feature feedback loop, and support iterative and incremental development. The smaller stories often do preclude the need to explicitly list out any tasks. It is rare for a pair of developers to need to list out the tasks associated with a story. Usually, the pair can effortlessly figure out what needs to get done.

We do not follow a Scrum Sprint, or an Extreme Programming Iteration or an Extreme Programming Weekly Cycle. We’ve successfully incorporated Lean into the way we prepare and execute our work.

The concept of “week” describes the team’s work rhythm

The team’s rhythm aligns with the natural rhythm of a work week. Retros are scheduled on Fridays at the end of the week. The meeting signifies closure for the week. There are practical reasons to schedule the meeting on a Friday. After a weekend off, the team forgets the details of the previous week. Just watch the Monday stand-up for confirmation as team members struggle to recall their stories from Friday.

The team schedules a “shared understanding” meeting once a week in order to 1) share context from product and design with the engineers. While the engineers are busy implementing stories, product and design are looking into the future work, specifying what needs to be done. This sharing of context helps anyone on the team pick up any story in the backlog. 2) The meeting provides feedback from enginering to product about the complexity and risk of the work. A story that seems simple to product might be expensive to impement. This emergent conversation allows product an opportunity to course correct to a simpler solution.

The meenting typically covers enough stories to make sure that we have enough work in the backlog. Teams typically schedule them on Mondays, Tuesdays, and Wednesdays since any day will do. (This is unlike an Extreme Programming Iteration where these need to be scheduled at the beginning of the week so developers know who is working on what.) If the backlog was extremely full, we could just cancel the meeting. Just because the team’s natural cadence aligns with the week does not mean we are doing Extreme Programming iterations.

The next time you find yourself using the word “iteration”, see if the word “weekly” would do instead. “What is our velocity this iteration?” becomes “What is our velocity this week?” and “How many iterations are left in this project?” becomes “How many weeks are left in this project?” The term iteration is a proxy for measuring time. Since we aren’t committing to work to be done in an iteration, our usage of the terms are interchangeable.

In communicating our process to potential clients and the outside world, I’d prefer to be accurate in describing our work flow as a Lean just-in-time system. Using the term “iteration” potentially miscommunicates that we are committing to specific work each week.

Updating our terminology at Pivotal

Since Pivotal has evolved its own form of Extreme Programming to incorporate Lean’s just-in-time practices, I suggest we use the accurate terminology in describing the way we work.

Consider our weekly meeting for creating shared mutual understanding and pointing the next stories in the backlog. Ironically, the meeting is called an “Iteration Planning Meeting” even though we do not commit to which stories we are doing next week and no explicit planning happens in the meeting. While we do point stories in the meeting, examining whether we are on track, re-prioritizing stories, and assigning work to individuals all happen outside the meeting. For us, planning happens at the product manager, client liaison, and client level as Pivotal discusses whether the project is on track and whether key features will be done by a certain date.

During our “Iteration” “Planning” Meeting, it is entirely possible that the stories we do discuss might happen two or even three weeks from now, not during the next “iteration.”

When I’m anchoring a project, my chief concern with our “Iteration” “Planning” Meeting is to make sure that our backlog has enough pointed stories for the next week. The team running out of work would be disastrous.

Finding a suitable replacement name for this meeting is challenging as many of us have different, sometimes conflicting expectations and goals for the meeting. Here is what I have found n interviewing 10 pivots and product managers from four offices.

Common goals for the meeting: 1) Create a shared understanding of the work that is to be done. (Where are we headed? What will we be building? Are we building the right system?) 2) Communicate to the PM risks and story complexity. We typically use estimation as a forcing function to have this conversation. 3) Estimate the work 4) Verify that each story has clear acceptance criteria

Goals 1 and 2 are frequently cited in my interviews. Some pivots argue that we should not be pointing and thus goal 3 is not a goal for them in the meeting. Goal 4 is infrequently mentioned.

We could call the meeting a “Pointing Meeting” (PM) which unfortunately has the same abbreviation as Product Manager (PM). In discussing the goals of the meeting with numerous Product Managers, Anchors, and Pivots, I personally prefer the term “Shared Understanding Meeting” as it reinforces the goal of getting everyone on the same page.

Updating Pivotal Tracker

Pivotal Tracker might minor updates to its messaging. Pivotal Tracker describes each week as an “iteration” and has charts that talk about “iteration velocity” (aka weekly velocity) “iteration start date” (aka week start date). Fixing this would be a simple string replacement. Fortunately, many of the charts simply reference “Date” instead of “Iterations.” The terminology seeps more into the Help documentation than the tool itself. Keeping the iteration terminology in the help documentation makes sense given that other companies use tracker, and they might have sprints or iterations. Since the weekly markers are not labeled as iterations, one could use tracker for quite some time without knowing that the tool reinforced the Extreme Programming iteration concept from 1999.

My own A-ha moment

In the movie, the Matrix, Keanu Reaves follows a discovery and evaluation process to determine if he is “The One.” While in a waiting room, he sees a boy bending spoons and wonders how that is possible. The secret is that “there is no spoon.” I had a similar revelation while working on a Pivotal Labs project. In the middle of the work week, I was reflecting on our rhythm of weekly retrospections and weekly pointing meetings and realized that there were no iterations. We did not commit to explicit work each week. Every day felt very much like the next day. We weren’t ramping up on new stories on Monday to wind them down on Friday. Instead, we were present with the features in the backlog. We weren’t planning for or committing to which stories would be done in each week. I mentioned this to my pairing partner and he said, “Of course there are iterations, each Monday we start a new iteration!” The anchor overheard this conversation and said, “Actually, since our project started on a Tuesday, I setup Pivotal Tracker to have iterations go Tuesday to Monday.” With three developers on one team, we had widely different opinions of whether we had iterations and when they started. This conversation reinforced my observation that we were not using iterations as prescribed by Extreme Programming.

Conclusion

I recommend we replace “Iteration Planning Meeting” with a phrase that accurately describes the meeting’s purpose and goals. I suggest “Shared Understanding Meeting” as a simple draft proposal which is intended to encourage discussion about its disadvantages or the purpose of generating better proposals.

Lean Extreme Programming

I’m still editing this blog post and it is subject to change. While what I want to say isn’t changing, the words that I use to say it are changing.

Lean Extreme Programming

Lean Extreme Programming (or Lean XP) is a hybrid of Lean and Extreme Programming methodologies. Lean Extreme Programming starts with all of the practices of Extreme Programming (XP) and replaces the concept of iterations with Lean’s just-in-time production philosophy manifested in pulled-based practices. This achieves a high quality, extremely readable, and well designed software with a faster ability to respond to change.

Lean XP continues to rely on all the technical practices of XP including pair-programming, test-driven development as well as the remaining process practices of XP including on-site customer.

Commitment level shifts from the iteration to story

While there is a commitment to the story in-flight, the developers with the product manager may realize that the story needs changing. While working on a story, the developers might discover a corner case not considered by the product manager, or find a confusing user interaction flow that requires the user experience to be re-worked. For these emergent and “internal” changes, the story needs updating. (reduces the waste from more design-up-front approaches. This is an advantage of the process.) For “external” changes where the product owner changes the feature, typically the product owner creates a new story that reflects the new insight into the feature. From this perspective, there is a commitment between the product owner and the team that the stories written reflect our current understanding of the system and will not be re-written by product.

By switching from an iteration based to a just-in-time workflow, the commitment level changes from the iteration to the story. In Extreme Programming, there is a weekly contract between the product owner and the development team that the work for a particular week is fixed. Anything can change about the project, as long as the changes occurred between iterations. With Lean Extreme Programming, the product owner is committing to the completion of any story in flight since it was at the top of the backlog when available developers pick it up and start working on it. In practice, product owners do not cancel in-flight stories. Stories might become blocked and be un-started. Theoretically it is possible that the direction of a project changes so drastically that you might stop working on a story, but in practice this is extremely rare.

Being in the moment

In this environment, developers learn to be flexible when showing up at work, just like actors performing improvisation without a script. Imagine a developer with expectations about their work for the day. At the beginning of the day, they pair up, and the pair may discover that the team’s needs will cause the developer’s expectations to be misaligned from reality. Developers learn to loosely hold any expectations and embrace what emerges in the day. Developers can be pleasantly surprised by what emerges. “Acceptance is the answer” to many of life’s problems. A developer can express their desires to the team. Examples include “I haven’t worked on this part of the code base recently” or “I haven’t paired with this person in awhile” or “I really enjoy working on UI stories.” However, if the build is broken, or the product manager needs help in accepting a story, then the reality of the moment will trump any of the developer’s plans.

Improvisors practice letting go of planning through exercises that reinforce handling of chaotic situations, rewarding spontaneity, and listening. In Lean XP, we learn to listen to our pairs, our tests, our code, our stories, our product owner, and our users.

“You Aren’t Going to Need It” (YAGNI) encapsulates part of this idea. Instead of worrying about future features, let’s focus on the current system and implementing the current feature. At its heart, its about being in the present with the code. By not committing to stories for each week, it shifts the conversation. Instead of discussing what was done last week, or what will be done next week, the team can focus on what needs to be done now.

Pros

This approach encourages flexibility in the developers. I don’t know what I’m working on today. If I do have plans, I need to be willing to let go.

This approach allows for quick re-distribution of the work. As developers finish a story, they will pick up the next important one. The PM can re-prioritize the backlog at any point and know that the most important story will be worked on as soon as a pair finishes their current work.

If everyone was committed to a task within a sprint, then explicitly coordination is required to determine which tasks should be postponed and dropped. Dependencies on those tasks could also be affected.

This simplifies the plan of work for the week Some engineering managers maintain intricate mappings of who is working on what and who is dependent upon what. If any one developer slips on their work, there is other work that is also delayed. Lean XP shifts the focus from developer dependency to story dependency. Stories that block other stories can be explicitly labeled. When the first story finishes, the second story can be unblocked and worked upon when it reaches the top of its backlog. There is no “chess” game of resource allocation and dependency management.

How is a SE kanban board lean? Instead of tracking about all of the requirements of a product, we can just track the stories in the next iteration or the features in the next release?

OS X Filesystem and Case Sensitive Bugs

Problem:

On one project, our test cases would be green locally, but then fail on CI due to filesystem case sensitivity differences between OS X (our development environment) and linux (our QA and production environments.). By default, OS X treats ~/project/myFile as equals to ~/projectMyFile. Thus if the code had a typo on an import, it might pass locally when it shouldn’t.

Solution:

I wanted to use the base pivotal OS X image and sprout wrap. The strategy is to create a case sensitive partition and symlink the workspace directory to it. Sprout wrap has issues with case sensitivity (pivotal_ide_prefs in particular), so we would run that from the user’s home directory, but run all project materials in the case sensitive directory.

Alternative solution: getting the pivotal OS X image on a USB key might allow us to configure the default partition’s setting on installation.

These steps could work with an existing machine, but I only tested on a clean install

Step 1) install the Pivotal Labs OS X image

Step 2) using Disk Utility, split the only partition in half. For the new partition select “Mac OS Extended (Case-sensitive, Journaled)”

Step 3) create a workspace directory in the new partition mkdir /Volumes/CASE_SENSITIVE/workspace

Step 4) symlink ~pivotal/workspace to the new partition’s workspace directory. ln -s /Volumes/CASE_SENSITIVE/workspace /Users/pivotal/workspace

Step 5) install sprout-wrap from ~, NOT from ~/workspace. I noticed that the pivotal_ide_prefs wouldn’t fully work with case sensitive on. (There may be other recipes that have issues too.)

Step 6) install your project code in ~/workspace

Note that everything in Applications et al. will be typical Mac Setup, but we were fine with that.

Estimated vs Actual Story Points

On my last project, I tried an experiment looking at estimation accuracy.

My results run counter the conventional wisdom of software engineering research and experience, but are consistent with my experience at Pivotal. Conventional wisdom says that engineers are optimistic and horrible at estimating work which is why some managers “double” estimates given to them.

On my last team, we tended to be cautious in estimating work and not overly optimistic about the risk and complexity of our work. This is similar to the “under commit, over deliver” adage.

Here is how we went about collecting the data 1) We limited our pointing scale to “Easy”, “Medium” and “Hard” — one of the pivots advocated this scale and I liked the simplicity of it. We mapped “Easy” to 1 story point, “Medium” to 3 story points and “Hard” to 8 story points. In the meeting we would hold up 1, 2, or 3 fingers, I would call out Easy/Medium/Hard and the PM would record the correct story points.

While I don’t have evidence for this, I felt that the IPMs were very efficient as there was less quibbling over minor point differences. We went with the majority point value. If a large number of people said Easy and a large number of people said Medium, we’d have a discussion. If most of the team said Easy and a few number of people said Hard, we’d have a discussion.

2) At the end of each story, the pair would assign a Pivotal Tracker label to reflect the actual point value. We used the labels “Actual Easy Points” “Actual Medium Points” “Actual Hard Points”, eventually we had to add a “Actual Zero Points”. I asked the pair not to look at the estimate on the story while recording the actual, but there could be some anchoring bias with the data.

3) I monitored tracker and reminded developers to put labels on stories they had finished but had not labeled.

Periodically, I would show results to the team. (After 1 month, 2 months, and end of project.)

  We were… (number of stories)
Estimate Conservative Accurate Optimistic
0 5 1
1 7 48 5
3 13 14 2
8 0 2
Estimate Conservative Accurate Optimistic
0 83.33% 16.67%
1 11.48% 48.28% 6.90%
3 44.83% 48.28% 6.90%
8 0.00% 100.00% 0.00%

Frustration Quote

I am frustrated when my expectations do not align with reality – Todd Sedano