Emergent team behavior: Mob Programming

I was listening to Mob programming deep dive from The Changelog where Woody Zuill discussed his experience in evangelizing a teamwork style called "Mob Programming", also known as "Ensemble Programming" or "Whole Team Programming." While driving back from a July 4th celebration with friends, I reflected on how my team functions and how sometimes it does not function.

In Europe, "mob" also means to bully. Those who use "mob programming" as a term do not imply bullying, rather they refer to a group of people programming together.

Five years ago, my workplace felt like a flat org. Everybody hired had some sort of specialty like iOS, android app, designer, full stack. I was hired as a full stack, but gravitated towards backend. At the time we hardly had much team work besides design reviews and hackathons around new features. Now that my workplace has grown up with project and product managers, QA, backend-specific developers, frontend-specific developers... It has been less collaboration than ever before.

Except on my team.

See, my team doesn't have a project or product manager. Well, we sort of did for a while, but they were so burdened by another team I hired and assembled (then handed off to my newly hired boss) that not once did they have a material impact on my team's objectives or priorities. It was on me to function as project, product, and people manager.

I won't be holding back here. My opinions of my employer's performance are not praise.

I do think I've been blessed with the opportunity to learn the good and bad. And I am especially thankful that I have been in a position where I can change some things for the better.

Anti-collaboration

Remember that team I hired and handed off? There's only two out of six still left after the workforce reduction. Over the last sixteen months that team started eight new products.

It might have been seven, or nine, but it is clearly more than the number of people.

This kind of massively parallel project management was praised by the CEO in the all-hands meeting. He and the other top hats were giddy with all the progress their newly reorganized crew appeared to handle.

I'm listening to this, blinking in stunned silence for several moments. There's no way a team of 60% new people who don't know the domain are going to succeed. This is too imbalanced.

Today, as in literally the date on this post, twe of those projects are launching–half a year behind schedule.

That project manager did not survive the workforce reduction. Neither did my newly hired boss.

Here's where I think things went wrong. We have many developers who do not know the domain. Every developer is immediately burdened with more than they can individually accomplish, assuming they knew the domain. When one assists another, they then feel behind on their tasks; they stress and cut corners to meet due dates; later they have to redo their work because it does not meet functional expectations.

Every developer has to badger others to review their code. Reviewing another developer's code puts the reviewer behind on their task.

This is what I saw first hand.

Nothing in this environment encouraged collaboration. Everything was handed from top down. Communication was atrocious, none of the new project and product managers knew the domain well enough to design their tasks with accuracy. None of the developers knew the domain well enough to criticize the tasks or to fill the gaps. It was like watching people pass paperwork through a bullet proof glass wall, only to speak briefly but never truly align and have a common understanding.

Opportunity

A new market opportunity opened up! One which I had my doubts on immediately.

The majority of the engineering org felt starved. Engineering needed to accomplish something and when leadership said re-prioritize to cater to this new market; they happily embraced the project.

Of course, leadership threw down some nearly arbitrary deadlines. We wanted to beat all our competition to it.

I voiced my concern about the effort and return-on-investment (ROI) payout given the effort needed to facilitate this market.

Company leadership here: but what if this market opportunity opens up in other states too? Then the dev work is already done!

Really counting the eggs before they hatch. A year later, the ROI has payed off a tenth of the engineering that went into it. Not one additional market region has opened up.

In our rush to beat our competition to the market (which we did), we were granted leniency on setting all other tasks and projects aside to focus on this.

Well, everybody but my team. We had to keep the lights on. You know, the whole site-reliability-engineering and business critical compliance. On top of that I taught my new boss the domain.

Only my new boss (who isn't with us anymore) got taught the domain? Not the other new engineers? Yup. Did he teach the engineers under him? Nope.

What I'm about to recount I only saw from a distance, I was not involved. After all, we have to stay PCI compliant.

A collaborative intermission

Fourteen engineers now had a unified mission.

I offered knowledge, my experience, code review from the side lines. Under my leadership, my team provided the scaffold for a new micro service to enable an integration with CI automation and deployment tooling.

These fourteen broke into smaller teams of two to four and worked through several related problems concurrently. What they did was amazing. In six weeks, I saw everybody helping each other; I saw code review treated positively; I saw people teaming up to understand documentation and proposing designs. These team members shared knowledge, they got to know each other, they got to know the domain, they got to share their specialties and enrich each other's skill-sets. And they had fun. I saw smiles on zoom every day I joined the stand-ups. These fourteen engineers were fulfilled by their work. And they delivered on time.

I did not see top-down decision and design by non technical crew. I did not see constant context switching in the engineering team. The Jira board was not saturated with story points and estimates.

This was an organizational and functional success. It was a morale boost.

But the impact had no long term benefit to the business. All this effort was a poor application of the business's resources.

A few depressing realizations as a developer and manager

A few thing I've learned since becoming a manager...

It does not matter how good the technology is if the market isn't there to use it and the market successfully funds more development. This is regards to a business, personal projects are another matter.

It does not matter how much you A/B test and experiment on your user experience if you are not profitable and fail to grow the market. Otherwise you're scraping the bottom of the barrel

Profitability does not matter when you and your competition is backed by venture capitalists. Growth does.

Individual (not business) customers hardly give more money when given new features. They are not loyal when your goods and services are a commodity.

Oh, so that's why everything is locked behind Enterprise Plans.

However! You can get more sales with individual customers by reducing friction and cognitive load. Usually features increase cognitive load or friction.

But, I really like developing features...

Ah... Except new features have to be maintained forever. Leadership will refuse to let go even 0.5% of revenue backed by a legacy feature that holds you and your team back for months. And they won't believe you for years until it is too late.

Afterwards

Sales had the deals signed, legal had confidence in their agreements, and the engineers fulfilled all requirements to launch. But marketing went for a silent approach and then sprinkled on some push notifications to existing customers in the region.

I think this became a "If we build it, they will come" story. Hardly anyone new came. So the customer base did not grow.

Something about widening the funnel, right?

Yes, if you are adding a feature to enable a new customer base, you must market to that new customer base to build awareness. This usually involves an incentive: discounts, deals, free items, merchandising.

I love free things!

And when that promotion is over, you'll find something else.

You got me. I'm only loyal to my wallet.

With enough engineering done to launch that market, the organization resumed all prior projects. Every engineer had greater morale and personal investment in the team and what we were doing. Unfortunately, these engineers returned to an anti-collaborative atmosphere.

Here's what I saw come the second time around:

• Each engineer had their tasks maxed out; this provided a short term perceived productivity boost and decreased their long term effectiveness.
• Once more, engineers had to badger each another and the feeling of being a nuisance developed.
• Relatively new engineers were no longer connected with someone who knew the domain.
• Nearly every engineer started to silo their knowledge, earned through throwing themselves at the wall over and over. When they hit an issue they did not know who to ask.
• A year later, important projects are unable to launch due to a mismatch in reality and expectation in the foundational design.

I did assist the new engineers in learning the domain on a case by case basis. But they were verbally discouraged from requesting my time since I indicated I was burning out to my manager. Then I burned out anyway.

A year later, only two people remain with a thorough understanding of the domain. I am one of them, the other is the current director. The other five left in the last six months.

Anti-collaboration in a remote world

I believe there's a logical timeline of how this workplace evolved towards anti-collaboration. It started with adding a remote workforce. We contracted with a company in Ukraine and kept them for the long run. I have only positive sentiment for those people, they are part of the team.

Our android engineer became a manager and our iOS engineer left. Effectively the team was unable to update the apps, and so contractors were brought in. They belonged to a company which we developed stronger relations with over the months.

As contractors they had full control of their build environment; they could develop the app against the production or staging server; and they were successful. At their own desire they would refactor and improve the application in ways that made future development smoother.

The trouble began when more were brought on.

Suddenly we had an isolated team without in-person technical support for a highly custom Java tomcat backend and an Angular JS (yes the old one) frontend. There was effectively no documentation. Every week a senior engineer had to remote control in to fix their build issues. It wasn't just missing imports here or there. IntelliJ would just break in non obvious ways.

Some how, the fix at the time was to just use a specific older version of IntelliJ. I proposed this since I had not upgraded and my version did not have this issue every week like the rest of the local and remote team. It worked, but it was not going to be compatible with the next Mac OS release.

It got so unpredictable that the solution was to completely wipe the repository and IDE configuration, then re-clone and re-import with an hour long process. They could do on their own time without being blocked for a full day to wait on an engineer from our end to assist.

Now that the remote team could operate mostly independent from the local team, we hit a new issue: communication. The remote team had capable people, but like new hires they did not have these essential things to function independently:

• They did not have the domain experience to work within our custom stack.
• They were not afforded the opportunity to refactor the codebase and own the build process.
• They had minimal understanding of the business domain and could not define acceptance criteria.

Ah, but the app developers seemed productive, isn't there a contradiction here? Nope, they could reference the web implementation which we developed locally ahead of the mobile applications.

When they also had to develop the web implementation, all remote developers lacked a reference. We developed with our business domain expertise as a reference, they did not have the domain expertise.

A process-based solution was pushed forward: rely on a Jira-centric development process. Locally we would prepare details in each ticket containing acceptance criteria, qa instructions, and so on. Backend work was separate from frontend work (they did not do full stack) and delegated to a pool of engineers that specialized in each. Over time tasks became more bite sized,

This team adopted agile processes, sprints, backlog grooming, all that stuff. Multiple local engineers were not interested. The local team collaborated less, instructions were given top down by new project managers without business domain experience. Sentiment towards those engineers from leadership and project management deteriorated. Their time was not tracked like their remote counterparts. They worked on things management did not prioritize. And so one by one, month by month, the engineers with the most domain knowledge left.

With regards to process, check out The Remarkable Ways We Gain Insights where I reflect on my reading of that book. I've come to find process heavy environments stifle inspiration, fun, and engagement of software engineers.

Process helped the remote team get a foothold. The price was our local talent.

A year after growing the remote contracting team, I saw them introducing more spring-oriented code styles to the code base, the productivity of that team seemed to jump once they had ways to introduce patterns and standards they were familiar with. In other words, they were finally making the codebase theirs and could own their work. Code quality went up, defects went down, engagement and ownership developed. By this point, they also developed a lot more domain expertise and could rely upon one another for guidance and understanding.

All those issues I listed above were finally waning with tenure and experience. They had finally demonstrated functional independence, domain expertise, and were no longer blocked every week.

Then the workforce reduction occurred and the contracting company was cut off one morning with immediate notice.

The process introduced is currently being expanded and refined instead of loosened and reevaluated.

Process under my leadership

Under my new boss (at the time) and at the request of my management peers, I used Jira more often than before. Project managers expressed concern that my team wasn't doing anything.

I even had to fight to keep essential team members from being laid off. That was a very uncomfortable time where I had to really grow my communication skills.

Over time these new project managers learned that my team is handling tasks from marketing, sales, the remote team, supporting local and remote engineers with blocking build issues, upgrading dependencies, handling vendor relationships, and the list goes on.

Blindly using Jira did not fix communication problems. Collaboration did. I'll get to that in the next section.

To quell the cross-team communication drama I began publishing biweekly "release notes".

• What we are currently engaged in
• Why our current engaged work is important
• Who is unblocked by our work
• What we plan to do in the future
• What we have identified as concerning or blocking in the future

After I prepared a draft, I reviewed with everyone on my team. Corrections and clarifications made and I would schedule its publication in Slack.

Suddenly everything became predictable in to leadership, project, and product managers. The drama melted away and the most recent workforce reduction impacted my team the least.

I invented my own process to solve a soft boring problem. Consequently the processes on interesting creative problems faded.

The exceptions

There are exceptions but they are tactfully applied.

Everybody works differently. Everybody has different experience levels.

One of the senior engineers works better when he can methodically work through problems. Just in time pivots and surprises are more stressful for them than for others. That stress leads to code that doesn't meet expectations which spirals into more stress.

At my recommendation, they now work through macro designs on a wiki, then they write up every micro task on another wiki page. When both are reviewed and corrected, each task is copied to Jira. Now that same engineer, which I'll call "Sam" below, will work on those micro tasks.

Unfortunately Sam does not collaborate as much as the rest of the team does, but he feels effective and more happy than before.

Here's where things get interesting!

I also have two part-time interns which are enrolled at a local university.

One of the interns had no prior work software experience. To help him get familiar with the code, our technology, and working in a team, I wrote Jira tickets.

I spent more time writing Jira tickets to help him solve the problem than it would have taken me to just do the work. It seemed so inefficient.

1. Project A: One day of work for me took him two weeks, with three tickets. Each ticket referenced existing code, included sessions and support from me and the team.
2. Project B: Five days of work for me took him four weeks, with twenty tickets. First ten tickets described what code to reference and had very clear criteria including unit tests. Last ten tickets were more freeform but maintained the same expectations on unit testing.
3. Project C: Four weeks of work from Sam are now being done by both interns over eight weeks and is near completion. Tasks are similarly freeform like the latter half of Project B. Sam assists and mentors both interns.
4. Future project D: The interns will design, review, task, and share tasks together.

The second intern joined in on project C, he came has software experience and started at the level he was fit for. The next step for both is to design and develop together.

I think the whole Agile Scrum stuff is only effective if the same team that executes the tasks author the tasks. Writing together and sharing the work together is collaborative. This is what Sam did for himself at first.

The next step down but still effective is active mentorship, which Sam is doing with the interns. Active mentorship is still collaborative for the individual contributors but not to the same extent above. While the interns are developing business and functional expertise, Sam is developing communication skills. Even active mentorship can be a growth opportunity for the mentor.

The least active approaches involve Ivory towers where designs and requirements are thrown onto engineers from above. This can be from a local team to a remote team, or from product managers down to engineers. The nature is the same: those who are executing creative work are not those who plan and design it. There is little buy in by the individual contributors and it fails to meet expectations. To compensate, more process is introduced to the point of micro management.

There's a better way. As ownership and experience increase, reduce micro-managing processes and encourage more collaboration in its place. These inefficient micro-managing processes are a crutch to get moving. Reduce and lift process away from creative work as the contributor becomes more effective.

Encouraging collaboration in a remote world

November 2020: my manager (this is before the new boss who came and got laid off) left and I became team lead of individual contributors under him. One of them, who I knew since before the pandemic, suggested a "water cooler" meeting. A casual time set aside to catch up on life. We could talk about work, but that was not the goal of getting everyone together. There was no agenda handed down from above. We could talk about whatever.

At first every one that attended scoped their topics to personal life. Their pets, projects. I talked about wood working, or cutting holes in my house to fish cables around.

But eventually the meeting just didn't end on time. Well, it wasn't a meeting in the traditional sense in business. It was a gathering. Anyway, the gathering stretched on throughout the day, sometimes with tens of minutes of silence, but sometimes there would be spurts of collaboration. Given the casual nature of this gathering, each participant felt more comfortable asking one another for their time.

"Hey could I get a code review?"

"I'm confused on something"

"What do you think about this?"

This is what happened in the open office environment we had before.

Hold on, are you saying open offices are a good thing?

Well, not quite. I could barely focus and develop my ideas in that setting.

From this experience, I believe a requirement to collaboration is accessibility to one another.

If someone is overwhelmed with their own tasks, they will not be accessible to others. Micro-managing will make an individual contributor even less accessible.

If the team is not gathering, then others are not accessible to any team individual.

How I encourage my team to collaborate

This biweekly water cooler meeting lead to a new event. Every week on thursday, we have a "team day". A scheduled zoom call link is dropped into slack. Instead of asynchronous communication over slack during the day, everybody is encouraged to talk synchronously.

Further, sometimes we do screen shares with remote control or VS Code Live Share. We start sharing tasks with one another, instead of everybody having independent tasks.

Lastly, I stopped saying yes to other team requests on everything that came my way. I reduced the queue that my team members experienced. Most tasks now have at least two people focused on that effort. The others join in when things are interesting and float back to their things otherwise.

These changes lead to my team being better equipped to help the rest of the engineering organization in real time. Overall the team is happier, several new engineers say they feel better grounded in their work. Unlike the rest of the organization, there is actually knowledge sharing.

Woody Zuill mentions in the podcast Mob programming deep dive:

Every developer was on a team, but the only purpose of the teams was they were working on similar parts of the features; each team had their focus, but they never did anything as a team, and that really got me thinking about this. It’s like, we brought them all together - why don’t we accentuate our ability to work well together?

My team has figured out how to work well together.

My team's mob programming in practice

"Mob Programming" usually means more than two, but given my team size, it usually floats between two to four.

If you go look up Mob Programming you might see something like a "navigator" who decides what to do, a "driver" who does what the navigator does, and a crew of mostly silent people who upon request assist the navigator. And everyone rotates every 5-30 minutes.

We aren't following that strict format. I think we could benefit from some of its ideas, but that's a bit harder in a remote setting.

What was once a biweekly one-hour meeting became a once a week all day meeting. Now without any prompt from me, my team will spend 3-6 hours a day together on Zoom calls or Slack huddles. Unfortunately I am usually doing four to seven meetings a day so I have not gotten to participate outside of "team day."

In practice here is what I am seeing:

As a whole, I find that team programming promotes a healthier environment for all team members, the product is of greater quality, and as a whole per person the team is more effective. New hires are lifted, integrated, and mentored. Everybody feels like their work has greater impact and meaning to themselves and their peers. Collaboration is essential to a healthy team.

In case you are interested there's a book: Mob Programming: A Whole Team Approach, though I have not read it.

I just finished the final draft before meeting with the director. He didn't have any topics for our weekly meeting, but I definitely had these thoughts in mind.

Over July 4th weekend, all I could think about was.. project management. Exciting, right? He raised his brows and invited me to continue.

I doubted I could derail the whole process that had been instituted, so I minimized my suggestions. I described that I think we need to engage our developers in the task writing process, so that they feel ownership of the tasks being performed. That we need more collaboration and encourage peer or team programming (mob programming).

I brought up two case studies, so to say, two engineers that have worked well together and have lifted each other; also I described how the two interns worked together and the newest one (only a month of employment) was on equal footing now.

And I heavily emphasized that we should not permit an ivory tower style of task writing. After my condensed 10 minute descriptions of how I think things should change... It went something like this:

Wow, uh that's a lot to take in. Give me a moment to figure out how to respond.

I waited silently and watched him process it.

At first I thought you were going to say something like we have to throw everything out and start from scratch.

Instead, I'm glad that you're seeing a lot of the same things. I do welcome changes. It doesn't matter how much we have in flight at the same time if none of it gets delivered. If we're more effective pairing up or more, then that sounds good to me.

In fact, another example of pairing up that worked well. Project <name here> went smoothly, where <person> and <person> worked together.

I was glad to hear that he had seen similar improvements to released functioning projects when our people worked together.

However, product managers (not project managers, we don't have anymore) will still be deciding what we focus on.

I nodded and then proposed some immediate changes. Tomorrow my "team day" call will be open for everyone. We're going to see what happens when the whole engineering organization join the same zoom call, just to be accessible to one another.

Hopefully we instill more cooperation and collaboration with dynamic pairing or mob programming too.