The surprising result of combining engineering leadership, efficiency and open source technology
The path to higher ROI for software innovators?
Welcome to the first Tech Edge Insider article.
In this blog, I share my journey as a senior manager & principal engineer, noticing the blind spots and stumbling blocks that occur in tech-intensive companies, even those that occupy the top tier. I aim to shed light on practices that we as leaders often silently accept or overlook, and which hinder our progress and innovation. As a dedicated leader and engineer who puts efficiency first, my aim is to shape a culture that embraces change and drives us towards innovative and effective solutions.
This article is the first in a series focused on enhancing leadership and engineering efficiency. The articles all draw on my extensive technical experience and use of open source as a catalyst for change.
A view from the field
Join me in exploring how we can improve the performance, quality and expertise of our teams, drawing on my years of consulting with world-class organisations. These collaborations have always had the same overarching goal: to increase the return on engineering investment while ensuring the longevity of the solutions we build together. I will be open about the lessons, mistakes and triumphs. I've learned a lot along the way and I'm proud to have built a group of successful, forward-thinking and trusted technology departments.
Is this for you? Are you on a journey of adaptation and innovation in the evolving world of software technology? Then this blog will help you discover the keys to making your own way through this dynamic landscape. Whether you're looking to avoid pitfalls, build a consensus or optimise decision-making in your organisation, this blog promises to engage your curiosity and equip you for success.
Best intentions and lessons learned
A few years ago, my team and I set out to develop a web version of an embedded work scheduling application. Collaborating closely with key decision-makers and relying on their domain expertise, we faced the familiar challenge of tight deadlines, meticulously prepared backlogs, and noted non-functional requirements (NFRs). However, a critical oversight emerged - the lack of thorough refinement sessions to challenge requirements that were essentially copied verbatim from an existing solution. Surprisingly, neither I, as the team leader, nor anyone else raised the question of whether we were tailoring the implementation for a specific user group or a broader audience. My mistaken assumption that our audience would remain the same proved to be a costly mistake. While we successfully delivered the application, we failed to account for future extensions or personalization rules. Moreover, we've developed a bespoke time transformation library tailored solely to what we presumed was our most common audience. Consequently, during production usage and deployment worldwide, we encountered significant challenges and overhead. This lack of thorough research and reliance on our own initiative, rather than open source time libraries, exacerbated engineering difficulties for my team, ultimately resulting in accrued technical debt.
Take a moment to reflect on your own experiences, both successes and failures. Were your processes efficient? Was there effective technical leadership, or were you blindly chasing targets? Did you unnecessarily reinvent existing solutions already available on the market?
Chances are that you will see the validity of these questions and agree with them in part. Let's keep going with the intention of trying to see what's in our blind spot.
Taking a holistic view
Many of us, engineers and leaders, find it difficult to stay in touch with the big picture. The epics, stories, tasks and deadlines for which we're responsible often confine us to a relatively narrow focus. It's easy to lose sight of the critical dimensions within our activities.
This is something I had to learn after long periods of largely unnecessary stress. Fortunately, I found a way to keep my eye on the big picture while diving deep. I found that by integrating the three factors below, the quality of leadership, team results, and interactions at the organisational level improved significantly.
Engineering leadership. Engineering leaders get results by transferring engineering mindsets and ways of working to general leadership situations. They are also able to fully engage in detailed conversations with software teams. Practised engineering leaders have the ability to combine the big-picture vision of the business with a detailed technical roadmap for the product or service.
Engineering efficiency involves optimising your organisation's engineering capabilities. It typically starts with the ability to measure the alignment and contribution of engineering teams to your organisation's business outcomes. These measures can then be improved through agile methodologies, reducing technical debt, improving the developer experience, fostering innovation, promoting skills development and encouraging cross-functional collaboration.
Open source software is shaping enterprise software architectures. Open source software occupies an increasingly important place in our technology stacks. Open source software developers, individually and collectively, seek the best solutions to technology problems, making the software they create reliable and secure, as well as free. The virtuosity and vibrancy of the open source community gives it an edge and, because of their relentless efforts, the software gets better over time.
I've also found that it's not enough to focus on just one or two of these things. It's the integration of all three practices that greatly increases the likelihood of achieving the desired results. It's by no means a sure-fire recipe for success, but my trial-and-error process has shown me the interrelationship between these three areas of practice. It's this interrelationship that leads to the surprising result. Let's take a closer look.
What makes focussing on leadership, efficiency and open source worthwhile?
There's no shortage of advice for software innovators. So what makes this different?
What's different is that it's not a completely new methodology that requires you to spend years building collective behaviours before you see tangible results. It's an integration of familiar ideas, some of which you've probably already adopted. You may have already built behaviours on top of them. If so, you've definitely got a head start. Even if you're already familiar with these practices, the positive impact they bring is significant compared to the effort required. This is especially true when you consider the challenges of introducing completely new ways of doing things, which often meet with resistance and require considerable discussion within an organisation before they are accepted.
My job is to help you see where you can connect the dots across all three of the practice areas, so that your organisation can maximise the engineering ROI and longevity of your solutions.
The figure below illustrates the relationship between the three areas of practice and their outcomes:
Each time your organisation embraces an additional area of practice, the results jump to a higher order, generating stronger, wider impacts within your organisation and across the business.
This figure illustrates the jumps to a higher order result:
And here's a breakdown of what that concept looks like in practice:
You've probably noticed that each time you move to a higher order result, the effect is compounded because the higher order result works in combination with the lower order results. You can see it highlighted in the last row:
Now let’s understand the three individual areas of practice and the surprising things that happen when they are used together.
Engineering leadership: enabling teams to find their own success
This section deals with how engineers in leadership positions lead, rather than how to lead teams of engineers. It is equally useful for leaders of non-engineering or multidisciplinary teams as it is for those leading teams of engineers.
Organisations need effective leaders. The contribution of effective leadership to the achievement of organisational goals is significantly greater than even the most effective individual contributor.
In today's digital world, leaders with technical experience have a distinct advantage. Their understanding of technology enables them to make informed decisions about digital initiatives that directly impact the company's bottom line. Additionally, as a widely-read McKinsey report explains:
“The businesses that are achieving the greatest returns from their software investments are those willing to tackle entrenched cultural and structural barriers.”
Typically, engineering leaders have been at the forefront of decades of change, including the internet, the ubiquity of mobile devices, agile, the cloud, the transformation of business to digital business, and the move to decentralised architectures. In general, engineering leaders are well placed to tackle entrenched cultural and structural barriers within an innovation process. Additionally, The technical acumen inherent in an engineering leader’s execution of their role is an invaluable asset to the organisation they serve.
The increasing prevalence of leaders with strong technical backgrounds underscores the importance of technical acumen in leadership roles, a trend that is reflected in studies such as the Harvey Nash/KPMG CIO Survey 2020, which highlights the growing influence of technical leadership in various C-suites worldwide.
OK, enough about why engineering leadership matters in general. How do we make this work for us?
The agency and trust that engineering leaders often foster within teams makes it easier to:
Set the purpose and direction for the team
Connect with individual team members
Build confidence and psychological safety.
But it’s the qualities inherent in engineering leadership that make it engaging. It's these that unlock the potential to reach the organisational goals I mentioned earlier and empower teams to deliver tangible results. To illustrate, I'll share another insight from the project story I mentioned earlier.
Taking full responsibility for the failure was imperative.
You might say, "Don't you have a solid technical background?" I do, but to be called a technical leader is to think beyond your own expertise and consider broader perspectives. From a leadership perspective, this was a failure in this case. I failed to look beyond the immediate scope of my team's work and failed to ask the necessary questions. I mistakenly assumed that stakeholders had a technical understanding, simply because we were all part of a world-class technology company. This was not the case - they lacked the technical background and curiosity to understand the implications of the project before it started, and were focused solely on meeting annual targets.
Accountability, self-awareness and experience-based learning kept this blunder from becoming catastrophic. It was eventually fixed, and it was a worthwhile experience. I've discovered how to use critical thinking and trust my own experience when taking on new projects. Every problem in engineering is a marathon where every choice you make will affect how you finish. For this reason, research is necessary for every challenge.
What lessons can we learn to encourage engineering leaders to move forward in our organisation and enable teams to find their own success?
Lead with accountability: Authentic leaders empower their teams through delegation, holding them accountable for results while retaining responsibility for the outcomes.
Delegate for strategic focus: As a leader, striking a balance between involvement and delegation allows you to focus on setting the strategic direction for your team.
Leading through challenges: A leader's responsibility extends beyond successes, providing steady guidance and support through difficult times.
Alignment for success: When your personal goals are aligned with those of your team and organisation, everyone works together to achieve success.
Engineering efficiency: Approaches to get the best from your teams
Engineering efficiency is about getting the best out of your engineering teams. It's about working smarter and more strategically.
Here's the formula:
Alignment and impact
First, we need to ensure that our engineering efforts are aligned with overall business goals. This means understanding what success looks like for the business and making sure our engineering projects contribute directly to those goals.
Measure and improve
Once we know what we're trying to achieve, we need to be able to measure how well we're doing. This involves tracking metrics that reflect the impact of our engineering work on the business.
Agile advantage
Agile methodologies such as Scrum or Kanban can be powerful tools for improving efficiency. They encourage short, focused work cycles, continuous improvement and rapid adaptation to changing needs. From each of these, you can choose what suits your team and delivers the results you expect.
Taming technical debt
Technical debt, or accumulated compromises in code quality, can slow development and increase costs. Engineering Efficiency involves actively reducing technical debt to keep development smooth and efficient.
Happy developers, happy teams
The developer experience matters! When engineers have the tools, resources and environment they need to thrive, they're more productive and engaged. This improves overall team morale and efficiency.
Innovation engine
By fostering innovation and collaboration, we can create new solutions and solve problems more effectively. This requires a culture that encourages experimentation and knowledge sharing across teams.
Breaking down silos
Cross-functional collaboration is key. When engineers, designers, product managers and other teams work together seamlessly, it eliminates communication bottlenecks and streamlines the entire development process.
An emphasis on these factors helps us achieve engineering efficiency. This means that our engineers ask questions and experiment, rather than just completing tasks. The benefit is a significant acceleration in each of our development processes.
So what went wrong with the development lifecycle and the failure to achieve engineering efficiency in the work planning use case mentioned earlier? There were a number of factors involved, all of which ultimately boiled down to incompatible approaches to work.
Incompatible working methodologies
Like many traditional corporations, the organisation we collaborated with had entrenched processes and staunchly adhered to the waterfall methodology, which proved incompatible with modern software lifecycles. This led to issues.
Moreover, effecting wholesale organisational change was daunting. However, recognizing that blindly adhering to guidelines wasn't the solution, we optimised the team's workflow, pushing beyond prescribed boundaries to enhance effectiveness.
The events unfolded in a way you may recognise from some of your own projects:
The industry reality
As mentioned before, our client’s use of the waterfall methodology proved incompatible with modern software development lifecycles. The 2018 State of Agile Report (see below) highlights a widespread problem: many teams haven't adopted agile methodologies, hindering their overall effectiveness.
Bridging the gap
As a team leader with technical expertise, I acted as a bridge between the client's large structure and our team's agile approach.
Agile benefits
By adopting agile methodologies, we can improve the development experience, identify gaps more effectively and mitigate risk compared to the waterfall approach.
Our role
Leveraging our technical backgrounds, we optimised our team's workflow and demonstrated the benefits of agile practices. This ultimately influenced decision makers who then implemented agile methodologies across the organisation.
The State of Agile report I mentioned was produced by VersionOne. You can see the key findings below.
The report, which coincidentally coincides with the timeframe of the aforementioned work scheduling project, found that almost half of teams weren't using agile methods at all. While things have improved since then, it's clear that there's still room for improvement. This highlights why expertise from the software world is so valuable - it helps us refine development team processes and tackle challenges like this head on.
The work scheduling project, well, it wasn't exactly a string of successes. But what was the upside? It showed us the importance of questioning established processes. Having a technical background helped us tweak our team's workflow and get better results. By demonstrating the effectiveness of agile practices, we can influence decision makers to adopt a more agile approach across the organisation.
Open source investment: catalysing collaboration and innovation
The best development teams are like well-equipped builders. Efficiency and effectiveness are critical, but without the right tools, achieving them becomes a constant struggle. Imagine trying to build a house without a hammer, saw or level. It might be possible, but it would take forever, and you'd probably end up replicating those tools in some makeshift way, wasting even more time.
The good news for software development is that we have a treasure trove of open source tools at our disposal. This eliminates the need to reinvent the wheel and allows us to focus on what really matters: building great software efficiently.
The scenario I mentioned earlier reflects the importance of open source. My team was tasked with implementing software that required the invention of tools to achieve the end goal of a functional solution. Time transformations emerged as a central issue where I had failed to do proper research or heed warning signs. In our efforts, we developed an end-to-end library tailored to our specific needs, dismissing existing open source options as too cumbersome and generic for our use case. This oversight resulted in arduous maintenance and regional deployments requiring patches for each instance.
In retrospect, our biggest mistake was neglecting open source tools and skipping proof-of-concept evaluations. Relying solely on in-house development proved to be short-sighted. A more gradual approach, using open source and conducting POCs, would have identified bottlenecks and challenges early on. Taking smaller, defined steps helps to discover useful tools and methods more quickly and avoids wasting time reinventing the wheel.
The open source space offers countless end-to-end solutions, although customisation is often required to align with business needs. Red Hat's State of Enterprise Open Source 2022 report underscores the enduring relevance of open source in today's innovative and competitive marketplace, highlighting its value more than ever before.
The 2022 data shows that open source adoption was still below 50%. But just two years later, the trend is undeniable. Organisations are increasingly recognising the value of open source software (OSS), as the 2023 State of Open Source Report from Perforce and OSI confirms. We learned this lesson the hard way in 2018, when our resistance to OSS made things more complex. The shift towards OSS adoption is only accelerating.
Open source encourages collaboration and critical thinking among engineers on both sides. It requires a forward-looking approach and strong analysis to guide software development and meet the needs of the community. In contrast, commercially focused engineers may miss the bigger picture by focusing solely on getting the job done.
Joining the dots to improve engineering ROI and reduce time to market
I say that using this approach will consistently shift the odds in your favour.
Implementing even some of these principles may avert failure or substantially mitigate project risks. However, with the full complement of these elements in place, a clear path to success can be forged, accompanied by a substantial reduction in time to market.
When engineering efficiency, leadership, and open source investment converge, the result is a virtuous cycle of enablement, accelerated innovation and reduced time to market. Streamlined development processes, guided by visionary leadership, enable teams to capitalise on open-source technologies and collaborate seamlessly across organisational boundaries.
By leveraging open source, businesses mitigate risks, reduce costs, and gain access to cutting-edge solutions, thereby shortening the path from ideation to commercialization. Furthermore, the culture of openness and collaboration inherent in open source fosters agility and adaptability, enabling organisations to respond swiftly to changing market dynamics and emerging customer needs.
The synergy between effective engineering, strong leadership and the strategic use of open source is becoming increasingly important for technology companies. Many organisations strive to deliver effective services, but achieving this requires a multi-faceted approach. Does your current approach integrate all three?
Think of these elements not as complex building blocks, but as complementary ingredients in a recipe. You might be able to make something edible by focusing on just one, but the final dish wouldn't be as tasty or satisfying. When all three elements are strategically combined, the result is a robust and reliable solution. Expect success, not surprises.
Ways to join dots in your organisation
While there's no one-size-fits-all solution, here are some general tips to help you implement this approach, regardless of your specific situation.
Take a phased approach: Since you probably have an engineering background, start by focusing on improving your team's engineering efficiency. Once you're comfortable, explore integrating open source tools and contributing your skills to the community. Finally, consider how you can optimise your management style to best support this new approach. By taking things one step at a time, you can measure progress and make adjustments as needed.
Trust your experts, but check their advice: Seeking advice from others can be valuable, but don't blindly follow it. Some advice may even be contradictory. Ultimately, the responsibility lies with you. Evaluate all advice critically before implementing it, and trust your own expertise and intuition.
Analyse, discuss and experiment: There's no one-size-fits-all solution. Analyse your organisation's specific needs and discuss the proposed approach with your team. Run small tests or experiments to see if the changes are effective before full implementation.
Embrace innovation and ownership: Being an innovator means taking calculated risks. Don't be afraid to experiment and learn from your mistakes. It's better to apologise for a well-intentioned attempt that didn't work than to miss an opportunity altogether. If you succeed, your stakeholders are likely to want to share in that success.
Watch this space
Building a culture of innovation and tackling technical challenges with a holistic approach - that's what technical leadership teams strive for, right? And let's face it, embracing open source and leaving a digital footprint in this ever-changing tech world can feel a little overwhelming at times. But the results? They can be surprisingly rewarding compared to the effort we put in.
My own experience is a case in point. Looking back at a past project, I can see how a lack of focus on engineering efficiency, leadership and open source involvement made things more complex. The good news is that's the path most innovators take. We learn from our missteps, challenge assumptions, and continue to push our organisations towards a more innovative future.
In my next Tech Edge Insider article, we'll look at technical leadership. We'll see how it affects organisations, their people and customers.