If you’re a Collibra user, you may have noticed in our recent releases that some areas look a little different — modern and even more user-friendly. That’s thanks to Arbor — Collibra’s newest design system.
Arbor is a React-based system that was built on a solid foundation of Material UI v5 components and has now evolved to encompass so much more. We’ve started using Arbor to refer singularly to our initiative to modernize our architecture and redefine how we design, build, manage, and release frontend applications at Collibra.
Its evolution is a story I’m excited to share.
Modernizing our frontend architecture
While we’ve always strived to provide the best-possible user experience with our products, our frontend tech stack has not always made delivering a great experience easy for our teams. There was a lot of legacy code to maintain, and the drain on developers’ time was only increasing as the component design system and frontend architecture aged.
By the time I joined Collibra in 2021, UX research and design work had already begun to replace the older Collibra Design System. But it was clear that we needed more than a redesign, we needed to reset our approach to frontend development. We needed to modernize it.
There was a lot of effort put into the UX design and research for the new component design system, but we couldn’t simply restart with a new design system on our existing frontend architecture, or we would once again begin incurring technical debt. We wanted an entirely new system that would make it easy for all Collibra frontend developers to create, use, share, and update features for our existing products and many new products to come.
I’m proud to say that’s what we’ve done with Arbor and the new frontend architecture.
Arbor is built on the best-of-the-best technologies for frontend development. The technology stack includes React, Material UI v5, TypeScript, JavaScript, Figma’s collaboration tools, Chromatic and Storybook for visual testing, Cypress and Jest for component and unit testing/iteration, Module Federation and a next generation build system using Nx which has first class monorepo support.
Building Arbor with the new frontend architecture and scaffolding took a small team of talented designers and engineers collaborating with many talented teams across the organization to create and implement a scalable system.
What we’ve built and how we built it is a great representation of Collibra’s culture and values in action. While we still have a way to go in replacing the old with the new, and getting to this point was no easy task, we are starting to see our efforts pay off. Moving to this new tech stack has tremendously increased developer productivity, reduced build time for our frontend developers, and reduced time to market to our customers. Here’s how we did it.
Creating the Central Frontend team
After the UX research and design were underway and had started taking shape under Design and Product Management leadership, our first step on the engineering front was to start building a solid team.
This team would be charged with not only refreshing the frontend user experience, but also transforming our approach to frontend development and refining how our developers worked. Finding the right people was a challenge because these were not straightforward frontend engineering roles. Their jobs would be part frontend development, part architecture evolution, and part evangelism and enablement.
Along with our product lead, we formed a team with a few enterprising Collibrians who we knew had the mindset to get us started in the right direction. We also hired a new Principal Engineer, Steven Rathbauer.
Steven’s modern mindset, stellar technical skills, can-do attitude, and the ability to think about Collibra’s short-term and long-term needs made him a perfect fit for this job and a great technical leader at Collibra. He hit the ground running and quickly became an amazing partner for me. Together we continued to grow the Central Frontend team, adding world-class engineers and leaders as the project took shape, including Product Manager Eileen Jones and Ian Forster, Senior Manager of Software Engineering on my team. In a few months’ time we had a fully formed, cohesive unit called the Central Frontend team functioning like a symphony.
These engineers and this team had a perfect mix of skills needed for the job, and they shared an ability to engineer at scale — thinking not just about what will fix a singular issue, but also about how the solution will apply broadly and over time. Engineering at scale and the willingness to identify best practices and promote them to others are critical traits for success on the Central Frontend team. These are skills we continue to look for when interviewing new team members.
Building a new design system and introducing a brand new architecture
Collibra started the new design system from the ground up by investing heavily in user research. We wanted to understand what our users were looking for, as well as research the latest best practices and UX design. A lot of investment went into the UX research and design phase of the component design system under the able guidance of the Design and Product team leadership.
With the research and initial design in place, the Central Frontend team started making deep inroads into implementing the design system using modern technologies. We leveraged Material UI v5 and React as our base, and we relied on great collaboration tools like Figma for design workflows and Chromatic Storybook for UI visualizations, documentation, reviews, and testing.
Work to evolve the architecture, tooling, and core developer libraries ran parallel to theming the Material UI v5 components as well as building custom component compositions. We wanted to move away from an aging frontend monolith. We used cross-functional design thinking exercises to come up with a modern micro-frontend architecture that would let us scale, implement, and incrementally roll this out across the organization as we replaced the old system with the new.
We bootstrapped a new frontend architecture for the rest of the organization to use, including the technical stack, the design system, the component libraries, the repositories, and the build system with all the scaffolding and checks in place. Then we formed a task force with Collibrians from various scrum teams across the organization to work on the first implementation of what we called the shell.
Collibra’s frontend development technical stack and architecture
The end result of this first phase was:
- The first delivery of a Material UI v5-based design library using React and Typescript as core technologies
- A brand new frontend architecture with the concept of easily manageable and maintainable modules in a monorepo with Nx as our next generation build system under it all.
- A scaffolding with common core APIs, common core libraries, and common look and feel via Material UI v5 and the Arbor design system
- A focus on testing, with automation at its core and high thresholds — supported by advanced reporting and feedback tooling
- The repository structure and the build system with all the scaffolding and checks in place using Figma’s collaboration tools, Chromatic Storybook, and Cypress and Jest for testing and iteration
- A bootstrap/quickstart guide for any developer, including workspace generators, linting/code style/IDE starters, naming conventions, library structure recommendations, and best practices for them to get a head start in frontend development
- A common shell with a common top navigation bar built on the above technical stack
- A group of developers from the task force who were equipped with practical knowledge and best practices to share with their respective scrums
Now, when a new developer joins or when a new team wants to get started, they can use the Arbor design system, common tools and best practices as a foundation to write their new features or new components. It gives them a huge head start and saves a significant amount on initial investment. Developers get to work on the latest cutting-edge technologies and can deliver their work in a faster, more streamlined, and more consistent manner.
Evolving the architecture
We felt confident in our approach as we spun up the new architecture and successfully executed the first set of deliverables, but we did not stop there. We continued looking for ways to improve further — more so when we knew that multiple other teams would soon sign up to participate.
Inevitably, challenges arose.
One such challenge we identified early on was the way the monorepo was organized. It was not optimal for multiple teams to co-develop their applications independently, and we knew this would be important for driving adoption across Collibra and for scaling with new teams and products in the future.
Steven rose to the challenge and came up with a solution using Module Federation. Module Federation is an architecture based on the concept of a host and remotes, where the shell is considered the host application and other applications (feature areas) that use the top navigation are considered remote applications. Implementing this and several other improvements, including Nx affected and caching features, resulted in an architecture where each application can develop, test, implement, and iterate relatively independently.
These changes reduced the overall build time by 60-70%.
The evolution of the Central Frontend team
Today, with Arbor’s implementation well underway, the Central Frontend team’s role has shifted. In addition to providing the support for new components, they now act as consultants to other teams and frontend developers who are building with Arbor.
We intentionally built the Central Frontend team as a distributed team with members in multiple time zones. This has helped with evangelizing Arbor, establishing best practices, and connecting the Central Frontend team with other frontend developers across Collibra.
Over the last year, the team has grown but remains fairly small with fewer than 10 highly productive engineers.
The first word that comes to mind when I think of the Central Frontend team is “Superstars.” They are a wonderful group of open, humble people who are catalysts for change and force multipliers. Though they primarily work asynchronously, they have gelled like a harmonious band and established themselves as a helpful, centralized resource for other teams.
As team leaders, Steven, Ian, and Eileen continue to help me drive the efforts to align our product roadmap with our new frontend architecture.
Contributing to new Collibra products
A year of hard work building Arbor has already paid dividends to our users.
Recently, we introduced new features to the Collibra Data Intelligence Cloud, including Collibra Data Marketplace, Collibra Usage Analytics, and Collibra Homepage. These features were the first implementations of Arbor.
All these new additions were developed as an application in partnership with the Central Frontend team. We are now moving very quickly to adopt Arbor within multiple other features, and we are developing several exciting new areas to release in 2023.
As a former senior leader and technical lead on the high-profile Master Data Management initiative and other cloud-native product initiatives at IBM, I have experienced my fair share of amazing teams and projects. Building the Central Frontend team at Collibra and helping bring Arbor to life has been another career highlight!
