Getting From Requirements to Product Architecture

From my experience I can tell you that it is not easy to create a successful architecture for a brand new system in an environment where you deal with rapidly-changing requirements and all sorts of constraints. The purpose of this post is to explain the steps I normally take in projects to evolve from requirements that are sometimes unclear to the first stable release of the product architecture.

This process can be difficult for the following reasons:

• An unclear view of the problem to be addressed. Sometimes the client has just a vague idea about what problem needs to be addressed. Maybe he cannot describe it exactly, or in some instances the problem is not stable enough and requires constant adaptation.
• Incomplete critical requirements. Based on the initial understanding of problems to be addressed, the set of requirements are carefully formulated. While it is normal to start with incomplete requirements, the real challenges appear when critical requirements are missing that are core requirements for the business. Sometimes the non-functional requirements like performance and scalability are overlooked because of a lack of understanding of the business environment. For this reason, the architect should stress these aspects from day one.
• Unsupportive development process. The development process has no planned activities for architecture, analysis, and design because of the belief that architecture will emerge as implementation progresses. Sometimes architecture may emerge up to a certain point, but resilient architectures will never emerge fully. They require a lot of thinking and preparation. Lack of support in the process for architecture building and monitoring leads to an architecture that is hard to scale and hard to maintain, which has negative consequences on the business.

To address these difficulties, here are some process-related considerations worth mentioning:

• Avoid anti-patterns like “analysis-paralysis” and “gold-plating.” Analysis-paralysis happens when there are many ideas and options on the table but there is a fear of making decisions because they might go wrong. Working in a time-boxed manner and reducing options to at least two viable solutions for further analysis is always necessary to avoid this situation. Gold plating is not preferred when there is too much detail provided in solution design beyond the limit of bringing real value from architectural perspective. Time-boxing helps here, i.e. knowing the distinction between an architectural decision and a non-architectural one. Architectural decisions have global and systemic impact and once these are made they are also very costly to change. It is said that if something can be refactored in an afternoon it is not an architectural decision. Another factor that goes against gold plating is that it ignores the details and has only two levels of system decomposition. The first level of decomposition is in subsystems and on a second level, decomposing each subsystem into components.
• Only requirements and risks drive the architecture. Apart from the cost/benefit analysis, every decision we make has to address a functional requirement, a non-functional requirement, or it has to address a risk. If it doesn’t address a requirement or a risk, we should not make it at this level. Instead, we should defer the decision to the design and implementation phases.
• Architectural elements should be obvious in code. Despite the fact that architectural design and code are at different levels of abstraction, they need to be in strict communication with each other. Subsystems and components need to have corresponding packages. Component dependencies need to have corresponding package dependencies and component responsibilities need to have corresponding refined interfaces.
• Architecture should be shared with all stakeholders. Making architectural decisions is always a trade-off between the simplest thing that could possibly work, stakeholders concerns, and the risks so we need to keep the stakeholders informed about any change in architecture that might affect their expectations from the overall system.
• Architecture should evolve with sprints. At the start of a sprint there is a period earmarked for analysis and design. This is the right time to assess the impact of new features to be implemented in system and also to assess an opportunity of getting rid of some of the technical debt. Normally if we need to add some new capabilities to a specific component it is advisable to verify beforehand if there is technical debt to be worked there, and only after that add something new.

Steps to build the architecture:

• Understand the problem to be solved. Sometimes we receive requirements or even tasks to implement without really understanding the context, i.e. why they are needed, what the problem is, or what was the thought process to come up with such requirements? Because of that situation there is a risk of not building the right system. Only knowing the entire business and technical context will give you the ability to have efficient requirements elicitation and further on to build a system which addresses the real needs.
• Understand the requirements. Review and refine the requirements to become specific, measurable, achievable, relevant and timely (SMART). Once put together, these must be prioritized, and one of the methods to prioritize it is by using the MoSCoW method. We need to extract the architecturally significant requirements as these requirements are the most valuable for the end-users and the main reason why the system is built. It comes as handy to classify the requirements in functional and nonfunctional terms, and the non-functional requirements to further classify them under runtime and non-runtime. Functional requirements are those that generate real functionalities for the end-users, while the non-functional requirements are the capabilities of the system.
• Understand constraints. We need to identify early all the business and the technical constraints. The business constraints are typically related to time, budget, resources, staffing, targeted customers, partners, legal aspects, and market. The technical constraints are related to hardware infrastructure, software licenses, limited access to data or code, legacy systems, developers experience on certain technologies. It is good to be aware that some of the constraints might affect and drive the architecture more than the requirements.
• Understand who are stakeholders and what are their concerns. A stakeholder is a person or a group of persons having an interest in the system being built. The usual stakeholders are the end-users, the developers, the testers, the operations team, the owners of the system/product, but there could be more in the list depending on the business context and the product lifecycle. User concerns are often around functionality and usability, testers are concerned about testability and configurability, the concerns of the operations team often hover around deployment, monitoring, configurability, the developers are more inclined towards robustness, dependencies, technology selection, and lastly, the owners are concerned about feasibility and total cost of ownership. We need to make sure those concerns are properly addressed in the architecture.
• Establish principles. A particular set of principles are very useful to introduce consistency, clarity, guidance and a common vision. These principles need to be realistic and accepted by the entire team. Having acceptance of the team will limit the technical disputes about how to approach a certain problem. Once in a while the principles need to be checked to see if they still apply to the current architecture or not. The principles should refer to very important aspects like data storage, communication style, APIs exposed or consumed, technology selection criteria, high level patterns or architectural references used.
• Establish goals. The goals are meant to align the architecture with the business goals in the long term. It is important that the goals do capture the expected lifespan of the system and respond to technological changes over that time such as new versions of middleware, new data storages, new clients and markets, but also respond to global trends like social networks, mobile devices, cloud environments or big data.
• Create the solution big picture. Start from the system as a big black box and establish the most important use cases refined from the architecturally significant requirements. Partition the system in subsystems corresponding to main business capabilities, and describe what the general responsibilities for each of those subsystems are so that they don’t overlap. Describe the high level business processes involving the subsystems. Refine the subsystems boundaries and responsibilities and business processes until they fit well together. Take each subsystem, partition it in components and assign each component a clear responsibility specified by required input, provided output, what should the component do, and with what other components need to collaborate. For this component design activity there is an old but useful technique: CRC – Component Responsibility Collaboration. Refine and adjust the components responsibilities and collaborations and the business processes until they fit well together. Review the use cases and validate their feasibility with components.
• Establish concepts. By working closely with the domain experts a set of key concepts have to be identified and modeled, based on that other key abstractions of the system can be derived. The domain model based on concepts and abstractions put together in a coherent manner with proper relationships is the conceptual foundation of the system for the entire system lifecycle. A good technique for this activity is Domain Driven Design which emphasizes the domain as a separate layer in the application and the partitioning in independent sub-contexts to match certain business capabilities.
• Select technologies. It is important to note that some of the technology choices are architectural, some are design related. The architectural technology choices should include communication style between components (REST/SOAP based, messaging based), common frameworks and mechanisms used across the system (MVC, Dependency Injection), persistence for each component (relational, file, non-relational), common tools (build, deployment, versioning control), other technologies for data processing (MapReduce). Once these choices are made, two or at the most three specific technologies for each choice should be selected and analyzed according to a set of criteria. These criteria include: financial impact (how much the company will have to pay for usage and maintenance), capabilities (how performant, scalable, available, secure it is, and how well the capabilities fit with the overall system), team impact (how much experience the team has with it, how long is the learning curve), maturity (how mature is it, how much support it has in the community), reliability (how robust and resilient it is, what known issues and limitations exist), interoperability (how easy is the integration process, what communication protocols are supported, how portable is on different platforms, how accurate and documented is the API)
• Create the architecture views. A pretty good architecture description can be created if we utilize the 4+ 1 Model View (Use case view, Logical view, Process view, Development view, and Deployment view). It is important that this description be a refinement of the solution big picture. The use case view is actually a formal representation of the significant requirements, the other 4 views are a representation of the components and inter-relationships seen from the perspective of different stakeholders such as end-users, developers, and the operations team.
• Realize use cases. This is a design activity where we need to have all the pieces put together to realize the defined use cases. This should be specified enough so that later on it could be used as guidance for implementation. Sequence diagrams are pretty well suited for this kind of a representation. First level of design should be at the component level, to show the entire stack of messages exchanged for each use case. Second level should be at object level representing the object lifecycle, the call stack, and the parameters exchanged.
• Prove architecture with code. Create the code skeleton in compliance with the implementation view and domain model, take one or two of the main use cases designed in the previous step and implement them end to end. This step proves that the system is feasible, provides the expected functionality, addresses some of the risks and reveals other risks. It is beneficial also as a learning exercise for the team. On the other side it may reveal some weaknesses requiring rethinking and adjustment of the foundations, design or technology selections, so all this process is iterative allowing us to go back and redo some of the previous steps.
• Formal review. The purpose of formal architectural reviews is to make sure the stakeholders concerns are captured and addressed in a balanced manner, identify and analyze the risks, and assess the architecture resilience to a set of change cases foreseen by the stakeholders. In case none of the stakeholders have important observations related to functional, non-functional requirements, risks and change cases, the architecture definition process is considered finalized. From this point on the architecture will have to evolve, the formal reviews need to be integrated in the development process to monitor the existing risks, identify new ones, assess compliance between business direction and architecture, assess compliance between architecture and code, do the necessary adjustments.