Emergent team behavior: Mob Programming- 38 min read - Text Only
- A collaborative intermission
- Anti-collaboration in a remote world
- Process under my leadership
- Encouraging collaboration in a remote world
- How I encourage my team to collaborate
- My team's mob programming in practice
I was listening to Mob programming deep dive from The Changelog where Woody Zuill discussed his experience in evangelizing a teamwork style called "Mob Programming", also known as "Ensemble Programming" or "Whole Team Programming." While driving back from a July 4th celebration with friends, I reflected on how my team functions and how sometimes it does not function.
Five years ago, my workplace felt like a flat org. Everybody hired had some sort of specialty like iOS, android app, designer, full stack. I was hired as a full stack, but gravitated towards backend. At the time we hardly had much team work besides design reviews and hackathons around new features. Now that my workplace has grown up with project and product managers, QA, backend-specific developers, frontend-specific developers... It has been less collaboration than ever before.
Except on my team.
See, my team doesn't have a project or product manager. Well, we sort of did for a while, but they were so burdened by another team I hired and assembled (then handed off to my newly hired boss) that not once did they have a material impact on my team's objectives or priorities. It was on me to function as project, product, and people manager.
Remember that team I hired and handed off? There's only two out of six still left after the workforce reduction. Over the last sixteen months that team started eight new products.
Here's where I think things went wrong. We have many developers who do not know the domain. Every developer is immediately burdened with more than they can individually accomplish, assuming they knew the domain. When one assists another, they then feel behind on their tasks; they stress and cut corners to meet due dates; later they have to redo their work because it does not meet functional expectations.
Every developer has to badger others to review their code. Reviewing another developer's code puts the reviewer behind on their task.
This is what I saw first hand.
Nothing in this environment encouraged collaboration. Everything was handed from top down. Communication was atrocious, none of the new project and product managers knew the domain well enough to design their tasks with accuracy. None of the developers knew the domain well enough to criticize the tasks or to fill the gaps. It was like watching people pass paperwork through a bullet proof glass wall, only to speak briefly but never truly align and have a common understanding.
A new market opportunity opened up! One which I had my doubts on immediately.
The majority of the engineering org felt starved. Engineering needed to accomplish something and when leadership said re-prioritize to cater to this new market; they happily embraced the project.
Of course, leadership threw down some nearly arbitrary deadlines. We wanted to beat all our competition to it.
In our rush to beat our competition to the market (which we did), we were granted leniency on setting all other tasks and projects aside to focus on this.
Well, everybody but my team. We had to keep the lights on. You know, the whole site-reliability-engineering and business critical compliance. On top of that I taught my new boss the domain.
What I'm about to recount I only saw from a distance, I was not involved. After all, we have to stay PCI compliant.
A collaborative intermission
Fourteen engineers now had a unified mission.
I offered knowledge, my experience, code review from the side lines. Under my leadership, my team provided the scaffold for a new micro service to enable an integration with CI automation and deployment tooling.
These fourteen broke into smaller teams of two to four and worked through several related problems concurrently. What they did was amazing. In six weeks, I saw everybody helping each other; I saw code review treated positively; I saw people teaming up to understand documentation and proposing designs. These team members shared knowledge, they got to know each other, they got to know the domain, they got to share their specialties and enrich each other's skill-sets. And they had fun. I saw smiles on zoom every day I joined the stand-ups. These fourteen engineers were fulfilled by their work. And they delivered on time.
I did not see top-down decision and design by non technical crew. I did not see constant context switching in the engineering team. The Jira board was not saturated with story points and estimates.
This was an organizational and functional success. It was a morale boost.
But the impact had no long term benefit to the business. All this effort was a poor application of the business's resources.
Sales had the deals signed, legal had confidence in their agreements, and the engineers fulfilled all requirements to launch. But marketing went for a silent approach and then sprinkled on some push notifications to existing customers in the region.
With enough engineering done to launch that market, the organization resumed all prior projects. Every engineer had greater morale and personal investment in the team and what we were doing. Unfortunately, these engineers returned to an anti-collaborative atmosphere.
Here's what I saw come the second time around:
- Each engineer had their tasks maxed out; this provided a short term perceived productivity boost and decreased their long term effectiveness.
- Once more, engineers had to badger each another and the feeling of being a nuisance developed.
- Relatively new engineers were no longer connected with someone who knew the domain.
- Nearly every engineer started to silo their knowledge, earned through throwing themselves at the wall over and over. When they hit an issue they did not know who to ask.
- A year later, important projects are unable to launch due to a mismatch in reality and expectation in the foundational design.
Anti-collaboration in a remote world
I believe there's a logical timeline of how this workplace evolved towards anti-collaboration. It started with adding a remote workforce. We contracted with a company in Ukraine and kept them for the long run. I have only positive sentiment for those people, they are part of the team.
Our android engineer became a manager and our iOS engineer left. Effectively the team was unable to update the apps, and so contractors were brought in. They belonged to a company which we developed stronger relations with over the months.
As contractors they had full control of their build environment; they could develop the app against the production or staging server; and they were successful. At their own desire they would refactor and improve the application in ways that made future development smoother.
The trouble began when more were brought on.
Suddenly we had an isolated team without in-person technical support for a highly custom Java tomcat backend and an Angular JS (yes the old one) frontend. There was effectively no documentation. Every week a senior engineer had to remote control in to fix their build issues. It wasn't just missing imports here or there. IntelliJ would just break in non obvious ways.
It got so unpredictable that the solution was to completely wipe the repository and IDE configuration, then re-clone and re-import with an hour long process. They could do on their own time without being blocked for a full day to wait on an engineer from our end to assist.
Now that the remote team could operate mostly independent from the local team, we hit a new issue: communication. The remote team had capable people, but like new hires they did not have these essential things to function independently:
- They did not have the domain experience to work within our custom stack.
- They were not afforded the opportunity to refactor the codebase and own the build process.
- They had minimal understanding of the business domain and could not define acceptance criteria.
Ah, but the app developers seemed productive, isn't there a contradiction here? Nope, they could reference the web implementation which we developed locally ahead of the mobile applications.
When they also had to develop the web implementation, all remote developers lacked a reference. We developed with our business domain expertise as a reference, they did not have the domain expertise.
A process-based solution was pushed forward: rely on a Jira-centric development process. Locally we would prepare details in each ticket containing acceptance criteria, qa instructions, and so on. Backend work was separate from frontend work (they did not do full stack) and delegated to a pool of engineers that specialized in each. Over time tasks became more bite sized,
This team adopted agile processes, sprints, backlog grooming, all that stuff. Multiple local engineers were not interested. The local team collaborated less, instructions were given top down by new project managers without business domain experience. Sentiment towards those engineers from leadership and project management deteriorated. Their time was not tracked like their remote counterparts. They worked on things management did not prioritize. And so one by one, month by month, the engineers with the most domain knowledge left.
Process helped the remote team get a foothold. The price was our local talent.
A year after growing the remote contracting team, I saw them introducing more spring-oriented code styles to the code base, the productivity of that team seemed to jump once they had ways to introduce patterns and standards they were familiar with. In other words, they were finally making the codebase theirs and could own their work. Code quality went up, defects went down, engagement and ownership developed. By this point, they also developed a lot more domain expertise and could rely upon one another for guidance and understanding.
All those issues I listed above were finally waning with tenure and experience. They had finally demonstrated functional independence, domain expertise, and were no longer blocked every week.
Then the workforce reduction occurred and the contracting company was cut off one morning with immediate notice.
The process introduced is currently being expanded and refined instead of loosened and reevaluated.
Process under my leadership
Under my new boss (at the time) and at the request of my management peers, I used Jira more often than before. Project managers expressed concern that my team wasn't doing anything.
Over time these new project managers learned that my team is handling tasks from marketing, sales, the remote team, supporting local and remote engineers with blocking build issues, upgrading dependencies, handling vendor relationships, and the list goes on.
Blindly using Jira did not fix communication problems. Collaboration did. I'll get to that in the next section.
To quell the cross-team communication drama I began publishing biweekly "release notes".
- What we are currently engaged in
- Why our current engaged work is important
- Who is unblocked by our work
- What we plan to do in the future
- What we have identified as concerning or blocking in the future
After I prepared a draft, I reviewed with everyone on my team. Corrections and clarifications made and I would schedule its publication in Slack.
Suddenly everything became predictable in to leadership, project, and product managers. The drama melted away and the most recent workforce reduction impacted my team the least.
I invented my own process to solve a soft boring problem. Consequently the processes on interesting creative problems faded.
There are exceptions but they are tactfully applied.
Everybody works differently. Everybody has different experience levels.
One of the senior engineers works better when he can methodically work through problems. Just in time pivots and surprises are more stressful for them than for others. That stress leads to code that doesn't meet expectations which spirals into more stress.
At my recommendation, they now work through macro designs on a wiki, then they write up every micro task on another wiki page. When both are reviewed and corrected, each task is copied to Jira. Now that same engineer, which I'll call "Sam" below, will work on those micro tasks.
Unfortunately Sam does not collaborate as much as the rest of the team does, but he feels effective and more happy than before.
Here's where things get interesting!
I also have two part-time interns which are enrolled at a local university.
One of the interns had no prior work software experience. To help him get familiar with the code, our technology, and working in a team, I wrote Jira tickets.
I spent more time writing Jira tickets to help him solve the problem than it would have taken me to just do the work. It seemed so inefficient.
- Project A: One day of work for me took him two weeks, with three tickets. Each ticket referenced existing code, included sessions and support from me and the team.
- Project B: Five days of work for me took him four weeks, with twenty tickets. First ten tickets described what code to reference and had very clear criteria including unit tests. Last ten tickets were more freeform but maintained the same expectations on unit testing.
- Project C: Four weeks of work from Sam are now being done by both interns over eight weeks and is near completion. Tasks are similarly freeform like the latter half of Project B. Sam assists and mentors both interns.
- Future project D: The interns will design, review, task, and share tasks together.
The second intern joined in on project C, he came has software experience and started at the level he was fit for. The next step for both is to design and develop together.
I think the whole Agile Scrum stuff is only effective if the same team that executes the tasks author the tasks. Writing together and sharing the work together is collaborative. This is what Sam did for himself at first.
The next step down but still effective is active mentorship, which Sam is doing with the interns. Active mentorship is still collaborative for the individual contributors but not to the same extent above. While the interns are developing business and functional expertise, Sam is developing communication skills. Even active mentorship can be a growth opportunity for the mentor.
The least active approaches involve Ivory towers where designs and requirements are thrown onto engineers from above. This can be from a local team to a remote team, or from product managers down to engineers. The nature is the same: those who are executing creative work are not those who plan and design it. There is little buy in by the individual contributors and it fails to meet expectations. To compensate, more process is introduced to the point of micro management.
There's a better way. As ownership and experience increase, reduce micro-managing processes and encourage more collaboration in its place. These inefficient micro-managing processes are a crutch to get moving. Reduce and lift process away from creative work as the contributor becomes more effective.
Encouraging collaboration in a remote world
November 2020: my manager (this is before the new boss who came and got laid off) left and I became team lead of individual contributors under him. One of them, who I knew since before the pandemic, suggested a "water cooler" meeting. A casual time set aside to catch up on life. We could talk about work, but that was not the goal of getting everyone together. There was no agenda handed down from above. We could talk about whatever.
At first every one that attended scoped their topics to personal life. Their pets, projects. I talked about wood working, or cutting holes in my house to fish cables around.
But eventually the meeting just didn't end on time. Well, it wasn't a meeting in the traditional sense in business. It was a gathering. Anyway, the gathering stretched on throughout the day, sometimes with tens of minutes of silence, but sometimes there would be spurts of collaboration. Given the casual nature of this gathering, each participant felt more comfortable asking one another for their time.
"Hey could I get a code review?"
"I'm confused on something"
"What do you think about this?"
This is what happened in the open office environment we had before.
From this experience, I believe a requirement to collaboration is accessibility to one another.
If someone is overwhelmed with their own tasks, they will not be accessible to others. Micro-managing will make an individual contributor even less accessible.
If the team is not gathering, then others are not accessible to any team individual.
How I encourage my team to collaborate
This biweekly water cooler meeting lead to a new event. Every week on thursday, we have a "team day". A scheduled zoom call link is dropped into slack. Instead of asynchronous communication over slack during the day, everybody is encouraged to talk synchronously.
Further, sometimes we do screen shares with remote control or VS Code Live Share. We start sharing tasks with one another, instead of everybody having independent tasks.
Lastly, I stopped saying yes to other team requests on everything that came my way. I reduced the queue that my team members experienced. Most tasks now have at least two people focused on that effort. The others join in when things are interesting and float back to their things otherwise.
These changes lead to my team being better equipped to help the rest of the engineering organization in real time. Overall the team is happier, several new engineers say they feel better grounded in their work. Unlike the rest of the organization, there is actually knowledge sharing.
Woody Zuill mentions in the podcast Mob programming deep dive:
Every developer was on a team, but the only purpose of the teams was they were working on similar parts of the features; each team had their focus, but they never did anything as a team, and that really got me thinking about this. It’s like, we brought them all together - why don’t we accentuate our ability to work well together?
My team has figured out how to work well together.
My team's mob programming in practice
"Mob Programming" usually means more than two, but given my team size, it usually floats between two to four.
If you go look up Mob Programming you might see something like a "navigator" who decides what to do, a "driver" who does what the navigator does, and a crew of mostly silent people who upon request assist the navigator. And everyone rotates every 5-30 minutes.
We aren't following that strict format. I think we could benefit from some of its ideas, but that's a bit harder in a remote setting.
What was once a biweekly one-hour meeting became a once a week all day meeting. Now without any prompt from me, my team will spend 3-6 hours a day together on Zoom calls or Slack huddles. Unfortunately I am usually doing four to seven meetings a day so I have not gotten to participate outside of "team day."
In practice here is what I am seeing:
- Individuals have less domain ownership, even on the things they focus on.
- Others provide surface level code review; feedback given focuses on nits rather than the problem solved, as the reviewer does not understand the problem.
- More defects are found after code is deployed to production.
- Individuals report more stress, they feel swamped, they make mistakes under pressure.
- The team demonstrates greater domain ownership individually and as a whole.
- Less time is spent on code review, instead substantive review happened while it was being written and the reviewer also understands the problem.
- Less defects are found after the code is deployed to production without a QA team.
- Individuals report less stress and they feel comfortable, creative, engaged, and make less mistakes.
As a whole, I find that team programming promotes a healthier environment for all team members, the product is of greater quality, and as a whole per person the team is more effective. New hires are lifted, integrated, and mentored. Everybody feels like their work has greater impact and meaning to themselves and their peers. Collaboration is essential to a healthy team.
In case you are interested there's a book: Mob Programming: A Whole Team Approach, though I have not read it.