There are many articles about this topic: your company/team is working Agile and you notice soon that you miss Software Architecture… or at least a little bit of software-design-common-sense if you don’t like big words.
I would like to share my view on the problem, and also what I think is a pragmatic solution to deal with this difficult Agile and Architecture combination.
If your team is Agile, you are probably used to Scrum or Kanban methodology. In these ways of working, one of the key rules is that you shouldn’t work on something if it can’t be mapped directly to business value.
Does the Architecture or even a little bit of Software Refactoring has business value? That is an important question. The answer depends on how good you are selling a software change or a new design. No matter how important and beneficial this change is according to you, you need to prove it. For example, is it a change that is going to save time in future developments?
If you pass the previous test (selling the technical task) and the Product Owner is convinced that this Architectural Change is good, then you will face the next challenge: can that task be completed within one Sprint? Can you split it into small chunks? If someone is really Agile-Purist then they may even ask you to write what you want to do in the format “As a [user/whatever] I want to [description of the goal] so [benefits of the goal]”.
Where are the Architects in Agile?
The Scrum methodology encourages architects to be part of the teams, so they should be working under the same Scrum rules. But this fact itself also causes problems, such as frustration in senior and creative people, and the fact that there is no technical ‘authority’ that can solve conflicts. I found a post that summarizes in a very good way the reasoning behind some technical people being against Agile, quoting it:
“[…] Like a failed communist state that equalizes by spreading poverty, Scrum in its purest form puts all of engineering at the same low level: not a clearly spelled-out one, but clearly below all the business people who are given full authority to decide what gets worked on. […]”
Even though I don’t fully agree with the post, I think it highlights some common mistakes made with Agile methodologies. In some companies, Agile is being used to fight fires, without thinking of mid/long-term vision. Senior people and architects have no special treatment: they are just team members. Some other companies managed to palliate this problem by having really good communication between the most Senior Engineers and the Product Owner, or even having Architects (the role) at the same hierarchy level.
An example of what can go wrong
Let’s imagine that you want to estimate a User Story in which you need to send an email. It’s the first time your application needs to do that. There are some developers in the team thinking that is a good opportunity to introduce the Spring Framework in your project, since your team could also benefit in the future from other Spring features, and in this case, you can use Spring integration with JavaMail.
The Product Owner may think that these devs are crazy: introducing Spring for such a small change! The PO knows that another team has some scripts in Python to send emails that they could integrate with the application, by running a command from your Java application (nasty, uh?). Check the numbers for this example: the estimation for the task using the scripts is a 5 (story points, bananas or whatever). Introducing Spring here would be a 13. These devs also say that maintenance of the scripts in the future is going to be a nightmare compared to the good support from Spring. But, how are you estimating future maintenance issues in Agile?
Eventually, since the PO doesn’t know about the future plans to integrate more emails or if that specific functionality will evolve, Python scripts are integrated.
However, if after one year that decision turns out to be a mistake and the application is calling hard-to-maintain scripts from everywhere: could it be changed? How much work would that involve? Who is going to change it? And When? And some high-level project managers may ask: who is responsible for that bad decision, and why it hasn’t been changed before?
The problem is that, at the time of the decision, nobody knew how many email features would have the application. Agile it’s business-oriented and, when done incorrectly, it’s also short-term focused.
Agile and Technical Debt
Wrongly-implemented Agile can be a Technical Debt generator where quick fixes prevail over any other solution. Sometimes it’s much easier to copy-paste something than extracting the code in common places and then reuse it. And sometimes there is no ‘technical authority’ to guard code quality, so who cares?
This could be a good summary of the problems I’ve seen so far:
- Agile has no room for Architecture, especially if there is no Project's vision (team of firefighters).
- Sometimes the only way to introduce software improvements or pay the technical debt is by associating these changes with a business feature that can really benefit from that, but then
- Technical improvements and/or refactoring usually make estimations higher, and then a discussion is needed to convince the Product Owner of the benefits, which is sometimes complicated since Agile can be a short-term trap.
If these Big Technical Changes don’t fit into Agile, so you need a plan.
Some pragmatic ideas
One of the first things you should do is to identify which parts of your legacy code are really critical and difficult to refactor. Since most probably it is a complex part of your code you could invest some effort in making it easy for everyone to understand it, so people can contribute to improving it while they are working on other issues.
Some ‘technical user stories’ that you can use are:
- Identify critical parts of your legacy code. Pay special attention to code that is really important, like the one that is constantly growing but has no documentation.
- Document these parts, even a basic diagram with the flow and technical overview is good enough. Ideally, you could also create a step-by-step guide to do the most common task in your team's daily work. It's really weird but sometimes you need to think about how to translate your current design into a really basic diagram to figure out how to improve it in the most efficient way.
- Write integration tests for these core parts. They will be useful to guard refactoring tasks. Avoid complex testing that needs lots of maintenance.
Another really important motivation booster that prevents frustration is making goals clear to everybody. Every member of a team should know where is the project going within three months, six months and one year. That way people can take good functional and technical decisions.
To achieve that, you may propose some tasks that should fit within one sprint:
- Work with business representatives to define short Main-Goal descriptions for these three milestones: 3, 6 and 12 months. This first draft should fit into a small presentation, four or five slides maximum (or a wiki page if you prefer).
- If needed, you can create a task per item that should be detailed a little bit further. But keep the description always telling how to map the general vision to the specifics.
- Identify which of these goals are more challenging (or complex) from a technical perspective given your current situation. This task has to be done in close communication with the Product Owners or Business Users. A good outcome of these discussions would be a Prioritization of the goals, based on the Return of Investment (how much value has this task compared with the effort that it's going to require).
Make a Plan
Once you complete the process of identifying and restraining legacy code and you get the Vision, then you should set up a Kanban board with the main technical improvements that the team wants to introduce. Once you have the topics, you will need to refine them so they can be split into small tasks. Some examples of vague topics that need to be detailed are ‘Migration to Java 9’, ‘Changing the UI to Angular’, ‘Make the build faster’, etc. You need to split those and make clear descriptions of how they map to the goals and how to achieve them (the required steps).
Personally, I don’t like spikes but in case the task is clearly related to research that needs to be done, it makes sense. Anyway, my recommendation would be that the result must always be a wiki page or in general something that can be used by others. Don’t do spikes without any result, it’s easy to get distracted from the main goal.
Don’t forget the main reasons behind that technical change, which will drive you to achieve your Vision goals. Most of the times they are system quality attributes: performance, robustness, readability, testability, etc. Depending on your project, some of them may be much more important than others. You just need to communicate across teams to decide what are the architectural changes that will make you achieve your goals faster.
The Architect role in Agile
With the introduction of Agile in many companies, Architects need to be more communicative than ever. They become a sort of Technical Coach. It’s not about making presentations or nice and complex UML diagrams, but more about being able to let stakeholders know that technology-related improvements are important. And, at the same time, keeping a good quality on the foundations, having a good test base, etc.
If you don’t have an Architect in your team, you can use Agile and the opportunities that it brings. It’s bottom-up, and that means that you can start proposing improvements for your team, creating small tasks and explaining why it’s so important to have a good technical vision. Don’t be afraid, just take a step forward.
What do you need to reach your goals and get a good combination of Software Architecture and Agile methodologies? Energy, patience, passion for what you do, and a plan.