Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
With the economy still bleak, data and engineering teams that were already struggling to keep pace with business needs are now facing a potential hiring freeze or worse in 2023. That’s why there’s never been a more important time to maximize the productivity and impact of your teams, but the question is how to identify the big changes that move the needle in performance. Incremental adjustments that save a sliver of time here and there are not hard to come by, but what teams need are the breakthroughs that yield giant leaps in productivity and efficiency. Rather than a 10% improvement, it’s the 10x improvements that drive lasting gains.
As an engineer and CEO of a software company, I’m constantly looking for ways to boost the productivity of our teams. Our guiding principle is simple: Maximize the impact per hour invested for every individual. To do that, we must ensure that individuals maximize the time spent flexing their superpowers, minimize busy work and reduce dependencies on others that delay work or tie them up in meetings. While counterintuitive at first, we consistently find that collections of well-aligned, independently executing individuals produce a far greater impact than a large team swarming on a project. Put another way; the individual impact goes up when the number of people you depend on to deliver the result goes down.
I’ve developed an approach over the years that has allowed me to identify several opportunities for step-change advancements in productivity and efficiency, and I’d like to share it here, along with two concrete examples. I will also share an exercise we run with our teams to help identify and execute these opportunities.
First, here are my two main criteria for identifying the projects and tasks where we can effect changes that will have an outsized impact on the company:
● Big resource-consumers: Work that’s essential to the business but consumes a large amount of engineering and other resources is a top candidate for optimization.
● Big impact, easy win: Areas that will have a significant impact on company performance if we can do them faster and with higher frequency are also great candidates—even if they’re not consuming a ton of resources.
Here are two examples that illustrate each of those criteria and where we were able to see massive gains in productivity, quality of service, or both.
Tackling Big Resource-Consumers
At Ascend.io, we provide a platform for building and automating data pipelines, which requires us to build connectors into a huge variety of data sources. At one time, building a new connector required multiple engineers to spend two to three weeks writing more than 5,000 lines of code that spanned four microservices in three different languages. As we added more customers and needed to support more data sources, this quickly became unsustainable.
When we looked at this process, we saw two issues: Engineers were writing custom code for each connector in a way that would be better served by a common framework. Moreover, the process required engineers from multiple teams and disciplines to work together and coordinate their priorities. As soon as you get dependencies that require engineers to coordinate with others and wait on the outcomes of other people’s work, you’re wasting time.
We addressed this by designing a new development framework that exploits commonalities across the connectors we build. This now requires far less unique code and removes the need for multiple engineers to work together. In most cases, one person can build a new connector in days with 100 lines of code. This made us far more nimble and freed up engineering time to do other valuable work.
Achieving High Impact
Like most companies building software, we need to convey the capabilities of our product to new and existing users to ensure they receive maximum benefit from their use of Ascend. With new features and enhancements being released on a weekly basis, there is a lot to keep up with. This is high-impact work, however, and if we can do it in a vastly better way, it drives an outsized impact for Ascend and our customers.
For us, building these automated product tours typically required a product marketer or product manager to design the customer journey, and one or more engineers on a sustained basis to build the actions into the product. This again creates dependencies that slow progress and require significant manual coding.
The breakthrough for us was a tool called Intercom, which allows you to build sophisticated product tours without needing to write code. Our product team can now build more interesting and complex product tours for customers, and our engineers can stay focused on building new features instead of the code needed to showcase them. What’s more, they are able to design, test and deploy these tours independent of our product and code deployment cycle, further freeing them to iterate as fast as possible.
A Thought Experiment
So how do you actually find these opportunities for 10x improvement? I run the following thought experiment with our teams several times a year to uncover these opportunities:
Imagine your workload scales 10x overnight and you absolutely must meet this increased need, but without hiring more engineers or working more hours. How do you do it?
It’s amazing what this exercise can uncover—but there are ground rules and it can take time to see results. Coming up with these big ideas is a skill like any other, and most people are not great at it the first time they try it. Your first iteration may produce no options to pursue—some may have high-impact ideas but with even higher investment requirements, while others may reject the premise outright. That’s normal. So, don’t insist they sit in a room until they dream up that 10x breakthrough, because it will be demoralizing. Instead, plant the seeds by establishing the above framework and then let the ideas start to grow.
More likely, a few days later, they’ll be out for a hike or chatting with an old co-worker and they’ll suddenly have the germ of an idea. That may still not be the winning solution, but someone else will build on it, and then someone else and, finally, you iterate your way to a breakthrough. Pulling the team back together and embracing the “yes, and” approach can be both energizing and highly productive.
Brainstorming like this is fast and free, and you only need one great idea to have a material impact. We run this exercise a few times a year and have come up with several big ideas that were game-changers for the company. Besides the examples above, we also moved key parts of our software development from Scala and Go to Python a few years ago, which is a much more efficient language (as measured by developer time) and allows our engineers to build things much faster.
Software engineering is a complex beast that often spans many teams working on multiple processes in parallel. But that complexity is also your opportunity: Every complex system has room for optimization, especially at companies that moved fast in the beginning to build a product and find market fit. Now is the time to lift your gaze, look around and ask where significant gains can be made. They almost certainly exist, and a 10x improvement could be what allows you to not only survive the turbulent year ahead but emerge with a thriving team and business on the other side.
