Scaling Engineering Decision making with Architecting Tenets

Author: Alex Handley, Platform Principal Architect, Jungle Scout

Jungle Scout’s heritage is rooted in moving fast to err on the side of delivering customer value and either scaling what works or decommissioning and then learning from the failures. With this in mind, the architecting tenets aim to support the scaling of architectural decision making.

To be clear, the definition of moving fast does not mean making architectural decisions that achieve functional requirements in the shortest way possible while sacrificing long term resilience. Great architectures enable simplicity and increase future velocity. Otherwise, we only have a perception of moving fast while actually moving slowly due to the re-architecting and refactoring we have to do each time we build something new.

So what is an Architecture Tenet?

An Architecture Tenet aims to take the opinion out of making decisions.
This means that when we debate architectural decisions, we can reference an agreed set of principles (tenets) instead of making decisions based on opinion.
As soon as you codify the principles that architects are using to design systems, it removes them as the bottleneck and allows you to empower teams to make decisions which is essential as you scale an engineering org.

Scaling decision making has been something that Jungle Scout’s engineering team has been focusing on over the last six months after we grew from around 50 engineers to nearly one hundred — What got you here won’t get you to the next level of growth.

The Tenets

These are our first set of Tenets; we aim to treat these much like code so they will continue to evolve with tenets being added and removed as we discover more and adopt how we architect systems.

Each tenet comes with an explanation and a question that the person can ask themself while making this decision.

1. Repeat yourself before following; don’t repeat yourself (DRY)

In a highly exploratory environment, prematurely following DRY leads to ill-informed design patterns so instead, wait until you see a pattern three times before following DRY.

2. Internal services should behave like external services

To protect internal services, we should architect them so they treat traffic like it was coming from an untrusted source. This should include implementing protections like fair usage policies to ensure that a rogue feature is never the cause of an outage on a critical shared service.
At Jungle Scout, we favour using AWS API Gateways in front of our services; one of the many benefits is that it removes the need to implement authentication and fair usage at the application level.

3. Experiment with a two-way door in mind

Architecting systems is one part asking good questions to the right audience and one part guesswork. So with this in mind, we need to make safe bets when architecting and we do this by experimenting using two-way doors.
A one-way door decision isn’t possible to reverse or would take too much time/money. An example would be migrating an entire SOAP API to GraphQL without first discussing it with your customers.
A two-way door is an approach or architectural decision that is easily reversible. An example of this would be creating a GraphQL implementation with just enough functionality to allow the team to test the approach and get customer feedback.

4. A safe bet is always to invest in the parts of the system that are hard to change later

Reduction of scope will always affect projects, but what do you cut? One mistake we’ve made in past projects is cutting this scope from things that are hard to change later.

Examples of things to invest in:

• Data layer/schema — The stateful layer is the most challenging to change if the feature gets traction.
• Public API contracts — Once a team has committed with other stakeholders to an API contract coordinating large changes will become tedious.
• Infrastructure as code — Redeploying a hand-rolled project in IaC after the fact is tedious and usually error-prone as parts of the configuration are often lost in migration.
• Monitoring — Shipping a new feature or service without any monitoring is like driving a car blindfolded. Essentially your customers become your monitoring tool, and they quickly become tired of reaching out when your deployment has broken something again.

5. The principle of least astonishment

The principle of least astonishment (or Least Surprise) suggests that a solution or approach would not surprise a reasonably knowledgeable person in the subject area when encountered for the first time (the audience may vary e.g. end-user, programmer, tester). In more practical terms, the principle aims to leverage the pre-existing knowledge of users to minimize their learning curve when using a module, so anything with a high unpredictability factor is a good candidate for re-design (Taken From this source).

This principle applies to every single aspect of development. For example:

• Naming services
• Nomenclature across API’s/Services to give a familiar feel Example: GSuite — every tool has a Make a Copy button under the file menu.
• Avoid introducing a new programming language unless there’s strong reasoning.
• Introducing new technologies. For Example: As a company, we favour using PostgreSQL as our relational DB of choice. If we were to deviate, we would need a good reason to introduce additional cognitive load when working on this system.

The counter to this principle is we don’t want to block innovation. We should always be balancing the least astonishment principle with innovation, so we continuously grow as a team.

6. Favour the path that provides independent value

When planning a large refactor or a new service with multiple phases, ensure that every phase provides independent value and doesn’t rely on a future action happening. The independent value could be a short demo or a feature with minimal functionality.

This is because startups are dynamic, work is often re-prioritized. We want to always ensure a piece of work can stand independently without immediate follow up work to improve or fortify it.

7. Buy vs Build — Standing on the shoulders of giants

We use AWS to stand on the shoulders of giants and take advantage of AWS’s work to remove toil from creating and maintaining resources.

When building solutions, it is better to choose native AWS products instead of software from other providers that are installed and managed by yourself. Example: install a PostgreSQL DB on EC2 VM versus using AWS RDS for PostgreSQL.
When you use a managed service like AWS RDS, AWS manages the underlying hardware, software patching and the setting of sane defaults. A great resource to refer to is AWS’s shared responsibility model, which helps to define where AWS’s and our ownership stops and starts.

The counter to this is to remember that not all AWS services are created equally; using newer service offerings comes with higher risks when compared to core offerings like S3 and RDS.

Summary

We hope that reading this article has inspired you to create some Architectural Tenets for your engineering team! The Tenets listed were copied and pasted from our internal engineering docs minus Jungle Scout technical jargon to try and make these as realistic as possible, so we hope you find some value.

Are you already using Tenets? If yes, we would love to hear more in the comments.

Contributors: @Lena Grygoryev @Vsevolod Geraskin