Software development as a wicked problem

Prologue: A tale of two databases

Many years ago, I joined an organization in the throes of change. The company was in the process of replacing a venerable Lotus Notes–based system with a newer customer relationship management (CRM) product, and I was tasked with integrating data from the CRM with other syndicated and publicly available data sets. The requirements were complex, but the system design evolved through continual, often animated discussions between the development team and key business stakeholders in an environment characterized by openness and trust. The system was delivered on schedule, with minimal rework required.

Five years later, I was invited to participate in a regional project at the same company. The goal was to build a data warehouse for subsidiaries across Asia, driven by the corporate IT office located in Europe. Corporate’s interest in sponsoring this effort was to harmonize a data landscape that was to put it mildly messy, with each subsidiary taking a bespoke approach to data management to address local reporting needs. The subsidiaries thought their systems were just fine. They perceived the push for standardization as a corporate power play that would result in a loss of local autonomy over data. 

There would be no timely and amicable resolution this time. The discussion began to devolve into heated Skype exchanges between stakeholders, forcing regional IT to step in and call a meeting to resolve the conflict. This episode led me to a broader understanding of the social complexities that lie beneath software development projects, and a deeper appreciation for the context in which such projects unfold.

The wickedness within

A few weeks before the meeting, I’d stumbled upon a 1973 paper in the journal Policy Sciences by UC Berkeley academics Horst Rittel and Melvin Webber titled “Dilemmas in a general theory of planning.” In the paper, the authors define the concept of a “wicked problem”: a complex situation that’s perceived in distinct ways by different stakeholders, making it difficult to translate into a clear problem statement. 

Although Rittel and Webber were referring to problems related to town planning, the circumstances they described mapped cleanly to the data warehouse conundrum my team was facing. Consider the following 10 characteristics of wicked problems, taken verbatim from the paper, with my notes from that time: 

1. “There is no definitive formulation of a wicked problem.” The depiction of the conflict depends on who you ask. Is it a corporate power play to wrest decision-making autonomy from subsidiaries or a genuine attempt to standardize reporting across a global company?

2. “Wicked problems have no stopping rule.” Fundamentally, a data warehouse is never done; it continues to evolve as user requirements change. 

3. “Solutions to wicked problems are not true-or-false, but good-or-bad.” A database design is never “right” or “wrong” (true or false), but some designs are better than others. In our case, the design will depend on the approach we agree on corporate-focused or subsidiary-focused and one group’s definition of “good” will be the other group’s “bad.” 

4. “There is no immediate or ultimate test of a solution to a wicked problem.” Again, there’s no objectively “right” answer which approach qualifies as “better” depends on who you ask.

5. “Every solution to a wicked problem is a ‘one-shot operation’; because there is no opportunity to learn by trial-and-error, every attempt counts significantly.”  Building a data warehouse is an expensive endeavor. It might not be a “one-time” activity like building a bridge, but redesigning a data warehouse from scratch would be costly and implausible. In essence, we have one chance to get it right.

6. “Wicked problems do not have an enumerable (or an exhaustively describable) set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan.” There are, in principle, a huge number of possible database designs.

7. “Every wicked problem is essentially unique.” There are fundamental principles of data warehouse design, but each data warehouse is singular, reflecting the requirements, capabilities, and technology choices of the organization and, in our case, the politics at play.

8. “Every wicked problem can be considered a symptom of another problem.” Our problem is fundamentally one of organizational change, which is, in turn, a response to corporate management’s perception of business inefficiencies.

9. “The existence of a discrepancy representing a wicked problem can be explained in numerous ways. The choice of explanation determines the nature of the problem’s resolution.” In our case, the discrepancy is that we have two conflicting interpretations of the problem. One group sees the problem as one of data standardization across the enterprise, while the other perceives it as one of addressing subsidiaries’ local reporting requirements. Which viewpoint stakeholders hold is ultimately a political matter rather than a technical one.

10. “The planner has no right to be wrong.” The database designer will ultimately be held responsible for the consequences of any design decisions. These decisions will require the designer to (somehow) steer a narrow course between the opposing desires of subsidiaries and corporate in order to satisfy both parties. 

How many of these characteristics apply to large-scale software projects you’ve worked on? I’ll wager you’ve encountered at least a few. Software development is, itself, a wicked problem, requiring stakeholders to reconcile differing perceptions of what a product should do and how it should be built. We often assume the right project management methodology agile, scrum, kanban will solve the problem. However…