Gunjan Titiya is the Engineering Manager of the Messaging and Eventing team within our Platform Engineering organization. The team’s work will touch nearly every aspect of Collibra’s product, so we asked her to share what her team is working on and how she approaches engineering leadership.
What does the Messaging and Eventing team do?
Our team is responsible for the messaging and eventing tools engineers throughout the organization use to build their product. The work is two-fold. We have to build and maintain the infrastructure, and we have to empower engineers throughout the organization to use it.
First, the infrastructure. Our tech stack uses Apache Pulsar as the foundation, which, of course, means the team works with Kubernetes, as well as Grafana, Prometheus, OpenTelemetry, AWS, and GCP. Like app developers use iOS to build apps, Collibra engineers use our messaging and eventing platform to develop real-time data streaming within products across Collibra’s portfolio.
I like to say we focus on ease of maintenance for our team and ease of use for our users, who are other Collibra engineers.
We don’t expect every engineer at Collibra to understand and feel comfortable with Apache Pulsar or the other tools in the stack — we need them to focus on their own area of expertise! We’ve built the platform to be as intuitive as possible for engineers who need to use it, and we’ve developed enablement resources for additional support. This is the other aspect of our work.
We monitor usage-based data to understand which teams are using the platform for which products and in what capacity. This way we can allocate the right resources to the right products and plan out the future of the platform. I like to say we focus on ease of maintenance for our team and ease of use for our users, who are other Collibra engineers.
Why do we use Apache Pulsar for messaging and eventing?
I joined the organization a year ago to build the team, select the best messaging and eventing solution for Collibra, and implement it.
The process for selecting the stack was a great way for my team and me to learn the Collibra way. We had to maintain the legacy system while deciding what to do going forward. We worked together to do a discovery stage where we talked to users, then we explored different technologies, created proofs of concept, and finally made a selection. We tested our decision by partnering with a team working on a brand new product. It was great because we didn’t have to worry about any legacy code.
We compared Kafka, RabbitMQ, and Pulsar as main technology for the event streaming platform. Some of our key requirements were:
Multi-tenancy support
- We needed a solution that could enhance security by efficiently supporting multiple products on a single platform while ensuring data isolation for different customers or applications.
- We wanted to simplify resource allocation and usage-based monitoring for all products to save infrastructure cost.
Elastic scalability
- Our ideal solution would decouple storage and compute units to provide elastic scalability of each layer.
- We also wanted a solution that could evenly distribute the load. With Pulsar, no one broker handles more than 20% of traffic at peak usage.
Geo-replication and data durability
- We aimed to enhance fault tolerance by replicating data across multiple regions.
- We wanted a solution with built-in tiered storage. Puslar, for example, stores live data on SSDs while archival, colder data goes on HDDs or S3 buckets — saving additional costs.
While all technologies have pros and cons, Apache Pulsar provides solutions for each of these key features of our enterprise-grade SaaS platform vision, making it the right strategic choice.
How do you view your role as a leader of the team?
Because I was an engineer myself, I try to be the manager I liked having when I was in that role. I put a lot of trust in my team, and I give them a lot of power and autonomy to make decisions. They are the experts, and they are closer to the code on a daily basis. I view their role as coming up with the best possible solution from a technical standpoint, and my role is to help evaluate those solutions through a strategic business lens. I make sure we’re factoring in various constraints and requirements while setting ourselves and our customers up for long-term success.
On a more individual basis with each member of my team, I really enjoy helping people realize their full potential. For example, in a previous management role, I had a person on my team who was hesitant about asking for a promotion because he felt he didn’t have the required skills. I encouraged him to take on a side project and coached him on how to socialize his work. He really opened up after that and continues to thrive.
As much as I love the technical aspects of engineering, I find it even more rewarding to help someone develop their strengths and watch them succeed as a result.
What do you look for in a team member?
Beyond meeting the requirements for the role, I try not to just check boxes of specific skills. Instead, I look for people who have the potential to grow with us. Even if a candidate doesn’t have experience with the exact technology we are using, I look for experience that will translate well to what we are doing and where we are going. I seek out people who are open to learning and who have demonstrated that they can adapt their skill sets to new technologies. Technology is constantly changing, so flexibility and openness are really important to me.
When it comes to working with the team, I also look for people who can approach conflict in a positive, constructive way. We work as a team, and it’s important to bring on new team members who will respect and value what each of us brings to the table.
Why did you decide to join Collibra?
I was a fan of Collibra as a customer before I decided to join. I was working in data engineering for a large bank that used Collibra for data management. My first impression of Collibra was that it was a great product that made a big impact on our organization. In addition to that, every person I interacted with from the Engineering and Support teams was easy to talk to. It seemed like a great culture.
When I decided I was ready for a new challenge, Collibra was on my radar. I applied to a few different positions before finding the right fit. Throughout the recruiting process, each new Collibrian I encountered had the same open and kind demeanor I experienced as a customer.
I’m glad to say I continue to feel that One Collibra spirit one year into my Collibra journey.
What data intelligence trends make you excited for the future?
AI governance is a major opportunity for Collibra, and the Messaging and Eventing team has a huge role to play in Collibra’s success.
It seems like everyone is jumping in and using generative AI tools. But the reality is that no matter how good your model is, if your data is bad, the outputs will be bad. We’re solving the problem of real-time data streaming, which makes real-time data maintenance possible. This means the data used to train AI models — Collibra’s own models and our customers’ — will be the best possible quality at any given moment.
I am really proud of my team for getting Collibra to this point, and I’m so excited for what is to come.
