Introduction to Lightweight Software Development
In 1993, the Chrysler Motor Company set out to update its payroll system. Their goal was to consolidate various payroll applications into a single, comprehensive software tool, leveraging the advantages of newer object-oriented programming.
At the same time, another company across the country was developing a tool for object-oriented design, also based on Smalltalk, one of the pioneering object-oriented programming languages. Managers from both projects identified four significant challenges in their product development:
- Project requirements were not well understood before the work began.
- Users only knew what they wanted after seeing the software.
- Requirements almost always changed during development.
- Newer technologies made their strategies unpredictable.
To address these issues, the manager at Chrysler invented a method called extreme programming, while the manager at the other company developed the Scrum framework.
The Agile Manifesto
In 2001, these managers and developers came together to share their ideas and create a new approach to software development. They sought a method that was less burdened by excessive processes, documentation, and tools, which they referred to as “lightweight.” This collaboration led to the creation of the Agile Manifesto, which introduced a new way of thinking about software development.
Key Values of Agile
The Agile Manifesto was structured with four core values:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
These values were balanced against “anti-values” to emphasize what the Agile approach sought to avoid.
Implementing Agile Principles
For teams new to Agile, it can be tempting to adopt the new values while retaining old processes and tools. However, true productivity gains in Agile come from subtracting unnecessary activities. It’s essential to remember the anti-values to ensure the team is genuinely working in an Agile manner.
Just as the developers at Chrysler and the early Scrum teams innovated by sharing ideas and creating a lightweight methodology, modern Agile teams must find a balance between traditional practices and the challenges presented by newer technologies like artificial intelligence.
Scrum as an Empirical Process Control
Software development began to take shape in the mid-20th century, driven by space and business programs. These organizations approached software similarly to manufacturing: plan the work, then work the plan. This deterministic process involved a project manager defining everything before development, breaking the project into phases like analyzing requirements, planning, executing tasks, testing, and closing the project. This approach worked for decades for large teams delivering one big product.
The Shift to Object-Oriented Programming
In the 1990s, the advent of object-oriented programming languages allowed teams to create more software. However, managing this increased complexity became challenging. As a solution, teams started adopting lightweight development methods like Scrum. Instead of tackling large projects, smaller teams focused on smaller, complex products using an empirical approach.
Empirical Approach in Scrum
Unlike the deterministic approach, an empirical approach treats software development like a scientific experiment. Teams are free to try new things and learn from their mistakes without being constrained by rigid plans. Scrum, formalized in the late 1990s, embodies this empirical method. It views software development similarly to how a chemist experiments with new compounds—mixing and adjusting as needed.
The Structure of Scrum
Despite being a lightweight framework, Scrum has defined roles and practices. The latest Scrum Guide from 2020 outlines three roles and five events.
Roles in Scrum
- ScrumMaster: A servant leader who focuses on training and administration, ensuring the team follows Scrum practices. This role can involve activities ranging from training the team to ordering pizza for retrospectives.
- Product Owner: Ensures the team develops valuable products for the customer, often coordinating closely with the business to develop user stories.
- Developers: Everyone on the team who is not the Product Owner or ScrumMaster. In Scrum, anyone doing the work is considered a developer, although they may specialize in areas like testing, database development, or design.
Events in Scrum
- Sprint: The core development cycle.
- Sprint Planning: Setting goals and planning the sprint.
- Daily Scrum: A daily meeting to synchronize activities.
- Sprint Review: Reviewing the work done during the sprint.
- Sprint Retrospective: Reflecting on the sprint to improve future sprints.
Scrum teams work together like a team of scientists, mixing ideas, running experiments, and seeking customer feedback. This collaborative approach, inspired by early lightweight methods, was a significant influence on the Agile Manifesto. While Scrum is structured, it remains flexible enough to adapt and learn, embodying the essence of an empirical process.
The Rise of Lean Thinking
In the 1970s, Japanese car companies faced stiff competition from dominant American manufacturers. Japan had two major challenges: less financial capital compared to American companies and limited space for large inventories due to its island geography.
The Toyota Production System (TPS)
To address these challenges, key managers at Toyota developed the Toyota Production System (TPS). This system emphasized having a minimal flow of parts through production, pulling one car at a time through the system. Parts arrived from suppliers just in time for assembly, reducing the need for large inventories and significant investment in parts.
Lean Manufacturing
American companies observed this efficient process and labeled it lean manufacturing. Lean thinking focuses on eliminating waste from the production process and emphasizes respecting the workers, who are best positioned to identify wasteful practices.
Lean Thinking in Software Development
A few years after the Agile Manifesto, a consulting team applied lean manufacturing principles to software development. They saw that excessive processes bogged down software development and believed that lean principles could streamline it.
Key Concepts of Lean Software Development
- Developing Smaller Parts: Emphasizing the development of smaller, more manageable parts of the product.
- Eliminating Waste: Reducing wasteful byproducts such as software bugs, excessive task switching, and incomplete work.
Integration with Scrum
The creators of Scrum integrated key lean concepts into their framework, emphasizing:
- Self-Managed Teams: Pushing software development decisions down to the developers closest to the code, promoting bottom-up intelligence instead of top-down management.
- Transparency: Ensuring that people outside the team have insight into their work, a principle borrowed from lean thinking.
- Feedback: Actively seeking feedback to improve the product.
Introduction of Kanban
Inspired by lean thinking’s effectiveness, a consultant introduced Kanban, a lean approach using cards and columns on a board to manage workflow. Kanban is less structured than Scrum and focuses less on experimentation, making it suitable for teams with clear requirements and a desire to improve their work. It serves as a midpoint between deterministic processes and Scrum’s empirical approach.
Whether using Scrum or Kanban, lean thinking empowers development teams with more decision-making authority. This empowerment is particularly crucial for teams working with artificial intelligence, where adaptability and responsiveness are key.
The Agile Mindset
The Agile Manifesto begins with, “We are discovering improved methods for software development,” indicating that its creators were initially focused on software. However, much has changed since then. In 2003, lean software development emerged, linking lean thinking with the Agile community. Lean thinking, which started in manufacturing to optimize processes and eliminate waste, encouraged the Agile community to broaden their perspective. It wasn’t just about developing software anymore; it became a new way of working. Over time, Agile software development evolved into an Agile mindset.
In 2009, a book on product development flow further expanded the Agile mindset to the delivery of any type of product. Today, Agile approaches are applied in human resources, graphic design, and organizational management. The term “agility” has become synonymous with innovation, much like a one-word brand for business process improvement, akin to musicians like Adele or Drake. What began as software ideas has transformed into a high-tech, nimble way of thinking. However, it’s important to differentiate newer ideas under the Agile umbrella from its original intent. Many of these new ideas are excellent but should primarily be applied to software development teams.
Core Concepts for Artificial Intelligence
When applying Agile ideas to artificial intelligence, it’s crucial to focus on the core concepts from the early days of Agile. Scrum and lean thinking, which are closely related, offer the most value. Often, when people talk about an Agile mindset, they mean Scrum. Agile itself has no specific roles, meetings, or practices—only four values and 12 principles. Scrum, however, provided a lightweight framework to apply Agile beyond software development. It is described as an empirical process control for lean thinking, and the Scrum Guide states that Scrum is founded on empiricism and lean thinking. These principles are essential for Agile teams working with artificial intelligence, as the complexity of such products requires running experiments to stay on track. Additionally, organizations should respect team members’ ideas for process improvement, as they are closest to the product.
Agile Must Adapt to Artificial Intelligence
In the 1950s, AI innovation surged, marking the beginning of artificial neural networks and machine learning. More recently, we’ve witnessed the rise of generative AI.
It’s crucial to understand that AI and the Agile mindset are not equally important. AI is likely to be one of the most significant achievements in recent centuries, impacting almost every aspect of the world and transforming many jobs, while hopefully creating new ones. In contrast, the Agile mindset, although a great way to work, will have a much smaller long-term impact. Think of Agile as a fly on the back of the AI elephant.
The Need for Adaptation
Many people, including myself, have successfully applied the Agile mindset for years. However, it’s important not to equate the significance of Agile with that of AI. AI will be far more influential to more people than Agile. Therefore, the Agile mindset must adapt to AI, not the other way around. If Agile doesn’t integrate well with AI, organizations will seek alternatives. True believers in Agile should be open to evolving it to work effectively with AI.
Preparing for Change
A fundamental shift like this can be challenging, especially for those deeply committed to the existing framework. AI will profoundly disrupt Agile, necessitating its adaptation. Some changes will happen quickly, while others will unfold over time. Predicting the future is difficult, so we’ll examine current trends and speculate on their continuation, keeping in mind the many unknowns. The longer you work in technology, the more you’ll learn to embrace significant changes.
