Pia Nilsson and Mike Lewis discuss how the Backstage plugin system integrates various functionalities and demonstrate how Backstage can be enhanced and integrated further.

Pia Nilsson serves as the Director of Engineering at Spotify, and Mike Lewis holds the position of Staff Engineer at Spotify.

Software is revolutionizing the world. QCon London is dedicated to empowering software developers by promoting the dissemination of knowledge and innovative ideas within the developer community. QCon is a conference designed for technical team leads, architects, engineering directors, and project managers who are at the forefront of innovation within their teams.

Nilsson shares, “My name is Pia, and I am affiliated with Spotify. Here, I head the developer experience and have been leading Backstage since it was first introduced in 2017. I started my journey at Spotify in 2016. Before joining, I had acquired 14 years of engineering experience. I initially joined as an engineering manager for a team within the platform organization. The independent and innovative engineering culture at Spotify greatly impressed me, and I was eager to contribute to the leading global audio platform that Spotify represents.”

This is the reason for being on this stage, that I have never struggled so much in my entire life to add value quickly. Leading the Backstage team back then, which of course didn’t exist when I joined, is a healing journey for me, as well, because Backstage is trying to solve many of the challenges that I personally struggled with in the beginning.

My name is Mike Lewis. I am the tech lead for Backstage at Spotify, which means I get to work with the amazing team that we have at Spotify working on Backstage. I get to think about Backstage all day, which is so fun. I’ve been at Spotify about 5-and-a-half years now. When I joined Spotify, I was working in the premium mission, working on things related to spotify.com, and checkout, things like that. I was using Backstage every day and seeing the value that we get from Backstage at Spotify. When the opportunity came up to join the Backstage team and start working on it myself, I jumped at the chance.

We are here to speak to you about how we use our developer portal Backstage plugin architecture, to change the ways of working for our 3000 engineers. I think just that sentence is important, at least to me. It’s not all about technology, although that’s the heart of it. It’s technology in order to change the ways of working in a meaningful way.

Before we get into the plugin architecture, and why it matters so much to us, I think it’s important for you to know some little thing about what kind of challenges we were facing at Spotify, back in 2017, when we were really starting to talk about these problems. These are accurate clouds that I took from our ecosystem. Imagine them a little smaller back in 2017, but you can understand it was a similar challenge.

Joining Spotify as an engineer meant confronting massive scale, with every point representing a backend service component, and each line indicating a component interaction. This growing complexity was evident and problematic by 2017, affecting productivity substantially. One productivity metric tracked the number of days until a new engineer submitted their 10th pull request, which had escalated to 60 days—well beyond the target.

Conway’s Law suggests that systems mirror the organizational structures from which they emerge. Spotify boasted an autonomous engineering culture that fostered ownership and enthusiasm among team members. However, this autonomy led to challenges in 2017, as engineers were expected to deploy their codes and manage infrastructure independently. Teams developed data endpoints and backend services based on their discretions, resulting in significant inconsistencies. This variance made it tough to apply learnings universally and added to the struggle of dealing with decentralized documentation, diverse programming languages, and incompatible libraries.

Working within the platform organization, Mike and I aimed to overhaul this chaotic ecosystem to enhance productivity and satisfaction among engineers, thus accelerating business value creation. At that time, I led a small team responsible for managing the backend component catalog—a modest yet crucial system in addressing these challenges.

Only the backend engineers were truly passionate about the backend component catalog, which seemed to be a secret to everyone else. This small yet significant detail was the seed that grew into the concept we now recognize as the Backstage platform. We realized the unique challenges faced by not just backend engineers but all engineering disciplines when it came to handling complex, large-scale systems. Fortunately, backend engineers had some relief with their component catalog. This spurred the thought: why not extend this advantage to everyone?

Throughout different quarters, we conducted numerous engineering surveys to pinpoint key issues. Two major problems consistently emerged: significant obstacles to productivity and the difficulty of locating necessary information, along with the disruption caused by frequent context switching. These issues are deeply intertwined, as anyone familiar with these processes can attest. The routine of interrupting colleagues, pulling them into meetings, or messaging them creates a ripple effect, escalating the number of meetings and interruptions across various teams. This scenario encapsulates the essence of what we refer to as context switching.

Moreover, our platform organization acknowledged a third significant challenge. Reflecting on our past experiences, it was evident that our metrics were declining and new engineers were struggling to reach full productivity within their first 60 days. This inefficiency was largely due to a lack of standardized processes across the organization. Historically, Spotify had viewed standards as restrictive rather than beneficial. However, we in our team, feeling the strain of these issues, adopted and championed the motto “standards set you free.” We humorously envisioned ourselves proclaiming this message boldly at Spotify, arguing that without standardized processes in software ecosystems, complexities multiply. This lack of standardization leads to repetitive tasks like setting up CI/CD, building pipelines, and crafting templates from scratch across hundreds of teams, resulting in widespread duplicity and inefficiency.

This essentially sums up the core productivity challenges at Spotify and clarifies the objectives of a developer portal in engaging upper management. Our goal with such a platform is to enhance speed, manage scale effectively, and contain the chaos—issues that are palpable and relatable to many in the industry dealing with similar environments.

Here it is. This is what Backstage looks internally for us, what it is, a single pane of glass for your infrastructure, that’s what it is. It’s simply put, a homepage. Even simpler than that, it’s a catalog for your infrastructure, all of it. All of you I’m sure know, any system that some kind of platform organization, like what I’m representing, puts together and offers to our engineers internally, is only as useful as its adoption. It can be as beautiful as can be, but if three teams use it, it’s just very expensive.

How do we make sure that Backstage was used? That’s where infrastructure as code comes into play. The metadata on each component needs to be in the repositories. Hence, the ownership of that metadata needs to be transferred to the owning team of that component. I think that’s a little small, but important engineering best practice that I’m sure all of you know about. That is the basis for why I believe our catalog happened to actually stay relevant, stay useful to our engineering population. Then, of course, we want Backstage to be more than only components. We want to add more kinds of functionality, such as measuring standards across the fleet. There are so many, monitoring and CI/CD. In order to do that, that’s where the plugin ecosystem comes in.

I wanted to say, these two engineering best practices that I really think, if one tries to figure out like why was Backstage successful, really, for Spotify? Of course, we have been asking ourselves that question for very many years. I think it’s as simple as this, infrastructure as code. Then, for today’s talk, we’re going to focus on the second one, which is the distributed code ownership of the plugin architecture. Extensibility is the plugin architecture that enables this distributed code ownership.

That is key to distribute it, to decentralize the decision making to the team actually owning the expertise. Instead of having, in our example, my little team building the Backstage developer portal, trying to figure out how to build all of these new functionalities into the Backstage portal so that it would become meaningful for all of the 3000 engineers. It goes without saying that that can never happen, that will never scale. It’s like it’s a must to have a plugin architecture for us.

Now we’re going to deep dive into the plugin architecture structure, to give you a broad view of what it is.

Lewis: Extensibility is just so important for Backstage, and it’s important at Spotify, and it’s important elsewhere, too, because Backstage is open source. It’s used by thousands of companies around the world. It’s important in both of those contexts. I want to tell you a little bit about how extensibility works in Backstage and how it’s changed over the years, and what we’ve learned along the way. First, I think it’s important that we cover the high-level architecture of Backstage. How many folks are familiar with the full stack web architecture of building Node.js, React web apps, just JavaScript on the frontend? I’ll try and keep it fairly generic. At a high level, Backstage is a web based React frontend. There is a horizontally scalable backend written in Node.js. That backend is talking to some infrastructure, a database, potentially a cache.

Then the backend is also talking to some third-party APIs as well. Those lines of communication between those things are usually HTTP, although other things can be supported at a plugin level, too. There are some logos there to represent this tech stack. In that context, what’s a plugin? A plugin is just a bundle of TypeScript code that’s been compiled into JavaScript, published to npm for public packages, or even private for adopters that don’t want to share that package, and it’s just used for private internal use, there’s no need to publish it. It’s generally for use in either the frontend or the backend, although sometimes there’s isomorphic packages in the mix too. The standard is frontend or backend.

For those who are familiar with full stack web development, there’s maybe an argument to be made that we’ve done it at that point. React in the frontend and Express in the backend are already pretty extensible, or at least composable. You can render components from other places. You can bring middlewares into your Express app, and you’ve done it. You’ve extended your app with new backend functionality via the middlewares, and new frontend functionality with your React components.

Building extensions with only React and Express can be quite challenging. This approach lacks ready-to-use solutions that assist in the rapid development and implementation of plugins. Moreover, developers face numerous decisions concerning database management, logging, and other functionalities, resulting in increased cognitive load and decreased efficiency.

Furthermore, the inconsistency in the plugins developed within such an ecosystem could hinder user experience. Considering the plugin users, they encounter significant challenges as well, such as manually integrating various backend components within an Express application and managing dependencies. Similarly, on the frontend, users must correctly initialize React components and ensure they are adequately supplied with necessary dependencies. This manual integration demands substantial effort and time.

A more desirable plugin system would resemble a simple power plug—easy to connect and ready to function. Instead, the existing system requires intricate wiring, analogous to manually setting up an electrical connection, which is inefficient and cumbersome. Highlighting another critical point, fostering contributions in such a challenging environment can be discouraging. Since Backstage aims to be an open platform that thrives on community contributions, simplifying the process of plugin development is essential.

Encouragement can begin with what can be compared to a tool library, where in the real world, individuals have access to various tools without the need to own each one. Borrowing tools as needed for projects simplifies the process and eliminates the burden of choice and investment in tools, drawing a parallel to easier access and utilization of development tools in software creation. This approach could effectively lower barriers to contributions and enhance productivity in plugin development.

The comparison I’m drawing here involves a traditional tool library and our own Backstage tool library, which we’ve developed. It encompasses a bundle of essential services both in the back and front end, designed to enhance efficiency and productivity when developing your plugin. An interesting component of Backstage is the database management system. The setup allows plugin owners to connect to a database without needing to understand its origin, as it’s pre-configured by those implementing Backstage.

Moreover, configuration options include a shared database connection across multiple plugins or individual databases for each plugin. This system’s complexity is hidden, allowing developers straightforward access to use the database efficiently. This exemplifies the first approach we’re using to simplify the process for those creating or utilizing extensions.

Turning our focus to those adopting these tools, the earlier practices in Backstage required meticulous and manual integration of plugins with specific lines of code. This process, akin to complex electrical wiring, involved constant adjustments with updates. Over the past year, however, efforts have been aimed at developing a declarative integration approach. This innovation removes the need for manual code integration, allowing for the simple installation of plugins by adding a package through yarn, which is a popular package manager used in JavaScript environments.

This solution is currently in its alpha stage, particularly raw on both the frontend and backend aspects. It’s too soon for migrations as it’s a work in progress, but it’s showing promising developments and adding significant value to our environment. More details will be revealed in the upcoming demo.

Next, I’d like to delve into the mental architecture of Backstage, which extends beyond a mere interplay of core frameworks and ancillary plugins. The essence of Backstage, which I found intriguing since my involvement began, lies in its nested extensibility. This isn’t just about having a core framework which supports plugins. Rather, it’s about creating an ecosystem where plugins themselves can be extended through additional modules.

Consider the transformative impact of this. Imagine a base plugin that integrates universal features into Backstage. Attached to this base, there can be specific modules that link directly to unique upstream APIs. For instance, suppose there’s a core plugin drawing organizational data into the Backstage catalog. You could then have a module specifically designed to retrieve this data from an LDAP server. This model encourages users to craft modules that cater to their custom sources of organizational data, enhancing compatibility and flexibility. Open source contributions further enrich this ecosystem by providing modules for diverse integrations required by users.

A crucial aspect to discuss here is the ‘importable points of extension’, which I plan to demonstrate with some code shortly. To better understand this layered extensibility, imagine this scenario: At the foundational level is the core framework. For example, in Backstage’s structure, consider the HTTP router in the backend that manages incoming requests from the frontend. This system isn’t static but offers points for extension. In our model, a catalog plugin might extend this router with a bespoke middleware. This relationship exemplifies how plugins enhance the core framework’s functionalities without direct modifications by the end-users.

The base layer links to the middle tier of plugins where similar interactions occur between plugins and plugin modules. A specific plugin, such as the catalog, might present an extension point, like the catalog processing extension point which determines the handling of incoming entity data. Plugin modules are then able to extend these functionalities through the extension point. This nature of extending capabilities is layered, creating a powerful and flexible structure. As an illustration, I’ve created code which reflects these concepts, highlighting how extension points can be incorporated.

In the simplified code provided, we import the core services extension points from our primary framework in the upper code block. This involves retrieving the HTTP router and appending the plugin’s middleware to this router. Correspondingly, in the lower portion of the code, we are importing the catalog processing extension point from the catalog plugin and using it to introduce an entity provider that supplies the needed entity data.

Today, I will discuss a specific instance of extensibility in Backstage involving access control, a relatively fresh feature. Access control or authorization is fundamentally a product of decision-making and enforcement. Decision-making assesses if a user has the right to perform an action or access resources, while enforcement ensures that the system adheres to these decisions. These two components form the entirety of access control. In contemplating where to integrate extensibility, it is crucial to determine who holds responsibility for each component. In this scenario, it is argued that plugin owners are accountable for enforcement since Backstage’s extensive adaptability does not regulate how plugins govern their resources. The decision component, however, is managed by those who own and operate Backstage instances, as their needs regarding access control may vary considerably from one instance to another.

Some adopters face stringent regulatory constraints dictating user permissions for data visibility, while others operate in environments where transparency is paramount, allowing all users access to all information. Within this context, the Backstage framework serves as a cohesive system that integrates these diverse needs. It’s structured to ensure consistency in enforcement across every plugin, with each plugin owner required to implement this enforcement uniformly. However, decision-making authority regarding access control is entirely delegated to the adopters. A unique aspect of our approach involves a single extensible point where adopters can replace the default policy with custom code to determine access permissions dynamically or through a configurable interface that allows decision management on a case-by-case basis.

During a demonstration, the focus is on illustrating the procedure for integrating and configuring extensions within both the Backstage backend and frontend systems. Presently, the Backstage backend and frontend are operational locally, demonstrating a typical instance almost identical to what one would anticipate in a newly instantiated Backstage application, albeit with upgraded backend and frontend systems to simplify integration via a declarative approach. I will explore how to enhance this instance with additional functionalities and resources.

This standard instance showcases various resources accessible to different users like guests and registered members. Notably, no resources are currently defined within the system. Considering adding new resources, inspired by a planets poster, the goal is to include planetary data because the company holds responsibilities related to planet management. There’s an API already functioning which furnishes details about planets, including names and images, ready for integration into the Backstage catalog using an existing module that links these planetary entities to the system efficiently.

If I transition to the backend package within our Backstage setup and specify the package as catalog-backend-module-planets, there it is. We’ll remain in the terminal momentarily. You’ll notice that the backend has automatically restarted. Upon restart, it has identified a new backend module and you can observe the new log line stating “refreshing planet resources”. I am utilizing the built-in scheduling system of Backstage’s core services to execute this every 5 seconds. Consequently, the planet resources from the API are being refreshed every 5 seconds and stored in the catalog.

Switching back to the browser, let’s check this tab. By refreshing and scrutinizing the kind, you will now observe the resource kind, with all planets listed there. Clicking on one will reveal its details. Now, assuming we want to enhance the frontend as well. I will proceed similarly by switching to the terminal, selecting the app package, and incorporating the frontend planets module. It necessitates a restart of the frontend to implement these changes. It is evident that the backend continues to refresh those plugins for us.

Following this restart and after adding a specific configuration to manage the arrangement of entity cards on the display, everything should be configured. We might need to refresh to see the updated config effect. You will see a new card displaying the planet image. What I want to emphasize is that this integration required no coding alterations, just a configuration update in my Backstage instance. By simply integrating the modules, they seamlessly provided their functionalities to Backstage without any complex integration. Just connecting the plug to the socket. We are optimistic about its stability in the forthcoming year, allowing consistent benefits for those adopting Backstage.

What does this level of extensibility translate into? What advantages does it bring? Focusing on Spotify first, since all user-facing functions in Backstage, including fundamental ones like the catalog, are developed as plugins, this allows work to be parallelized. Teams can independently develop features without needing to synchronize or jointly work, enabling them to focus solely on their tasks. Moreover, this permits distributed ownership of these features. Distinct teams such as the cataloging and scaffolder teams can operate autonomously, and those crafting new functionalities on top can do so without any interdependence. This model of distributed ownership significantly empowers us to allocate our resources across various segments according to their significance within the system.

The other thing that we get from this is also consistency. Because everything is built on this Backstage foundation, expertise is transferable between plugins. If a person moves from team to team, they can easily contribute more quickly because it’s a Backage plugin still. What about outside Spotify? Firstly, all that same stuff. These things are true both inside and outside of Spotify. It’s still easy to parallelize both in open source and in other adopters’ instances of Backstage, those benefits for minimizing coordination and transferable expertise, distributed ownership, that’s all still true. The bonus that we get in the world outside of Spotify, is that the tech stack, the standards, the choices that we’ve made about how Backstage fits together at Spotify doesn’t have to be mirrored at other organizations in order for Backstage to be valuable.

They can pick the plugins that they want to use, or even build their own, to compose the perfect developer platform for their own needs. That’s very different from the one where we built a fixed Backstage at Spotify, and then tried to get everyone to use it, because in that situation, you have to convince everyone that they should work exactly like you work.

I want to pick out some key takeaways from this. These are technical takeaways that seem important to me when we’re thinking about how to build extensibility models into other software. The first is, when we’re reducing repetition in systems like these, we should be reducing it by persona, so thinking about who is writing what code. When I think about Backstage specifically, I think about moving code from adopter instances into plugins, and moving code from plugins into the framework. Because the framework you write once, plugins get written plugin by plugin. Adopters have the most instances, the most numerous in that group. The more we can push things up into plugins, and then into the framework, the more the overall repetition is reduced. The other thing is, use the framework that you’re building.

An extensibility solution where you have some core systems that aren’t built with the extensibility model, and then you’re trying to extend those things with a set of extensions that have different capabilities. It’s much harder to get that extensibility model right because you’re not leveraging it, especially in your core team. It’s just getting used separately by this separate group of people. Conversely, if you build your core systems with that extensibility model, you guarantee that extensibility model is powerful and fit for purpose. Nested extensibility is just such a powerful concept for us. I really think it can apply elsewhere too. Making sure that you can have extensions that are themselves extensible is so powerful for making sure that you’re enabling the maximum amount of flexibility in your system.

Nilsson concluded with insights on the ROI from implementing Backstage, detailing their initial performance metrics and current methods. Initially, Spotify tracked developer productivity by the time to reach the 10th pull request—a borrowed metric from Meta which lacked thoroughness. However, their evaluation methods have since been refined.

Currently at Spotify, where Backstage is fully integrated, usage is divided into frequent and less frequent user segments. The statistics mentioned concern the frequent half. Within this group, there is a noticeable increase in GitHub activity by 2.3 times, a reduction in cycle time by 17%, and a doubling in code changes. These results, among others, demonstrate substantial productivity gains. More comprehensive details are shared on their blog.

The enhanced engagement levels among frequent users of Backstage have also reportedly boosted their job satisfaction and loyalty to Spotify, with a 5% greater retention likelihood. This underscores the broader benefit of Backstage not only to Spotify’s engineering environment but also to their overall workplace fulfillment and stability. Nilsson emphasized the accessibility of Backstage as an open-source tool, encouraging its adoption and pointing out the commonality of scaling issues it addresses within the tech industry.

For those interested in further information or advances regarding Backstage, Spotify hosts biannual webinars showcasing new developments and features, reflecting both their commercial and open-source commitments. Attendees are encouraged to participate and discover more about Backstage at these events.

See more presentations with transcripts

Sep 05, 2024

Welcome to DediRock, your trusted partner in high-performance hosting solutions. At DediRock, we specialize in providing dedicated servers, VPS hosting, and cloud services tailored to meet the unique needs of businesses and individuals alike. Our mission is to deliver reliable, scalable, and secure hosting solutions that empower our clients to achieve their digital goals. With a commitment to exceptional customer support, cutting-edge technology, and robust infrastructure, DediRock stands out as a leader in the hosting industry. Join us and experience the difference that dedicated service and unwavering reliability can make for your online presence. Launch our website.