Why Sansan Data Hub Development Excites Me

Hi, my name is Masanori Ikeda, a Software Engineer specializing in backend and Salesforce development, and currently serving as a Team Leader in Sansan Data Hub group.
In this post, I would like to share why Sansan Data Hub development excites me.

Data is Becoming More And More Complex

These days, we often hear phrases like "AI is incredibly powerful! It's going to take your job!"
Okay, I get it.
AI can be highly effective, but only if the food (data) is delicious (clean).
However, keeping data clean is not easy task.
During my time developing and operating CRM systems at enterprise companies, I frequently encountered data issues.
For example, users might create duplicate records or update ones with incorrect information.
I often had to write programs to correct these records.

What is Sansan Data Hub?

Our Sansan Data Hub package for Salesforce was awarded "AppExchange Partner of the Year 2020", which has increased its awareness in the Japanese market.
However, Sansan Data Hub is relatively unknown in the global market "yet", so I have to explain about it.

In one word, Sansan Data Hub is a "magic box making data valuable".
Let me give you an example: imagine you are a salesperson who has just created a lead record.

Additionally, you have to go to a party, so you just input only a few fields.
Don't worry! Sansan Data Hub will identify the record and enrich it with valuable information while you're at the party.

Based on added valuable information by Sansan Data Hub, you can utilize it for various business purposes.
(This is just a sample. Sansan Data Hub can add more information than just the fields shown in the image.)

Technical challenges in Sansan Data Hub development

I've explained about Sansan Data Hub, and I believe you understand its value.
However, there are various technical challenges to overcome in order to realize the value of Sansan Data Hub.

I want to dive into the inside of the 'magic box'.

To make data valuable, Sansan Data Hub must retrieve and identify massive sets of records from external systems, which presents various challenges (by the way, we handle 8 billion transactions per month).

The difficulties in Sansan Data Hub development are as follows:

Complex Data Flows:

Managing vast amounts of data from multiple sources, such as Salesforce, requires robust and scalable architectures. Ensuring that all data pipelines run smoothly without bottlenecks or loss of data integrity is a constant challenge, especially as business needs grow.

Integration with Any External Systems:

There are many kinds of IT systems in our world, and any of them can be target for the Data Hub solution. These systems often vary in architecture, data formats, and protocols, which can complicate data retrieval and normalization. To handle the variety, it is essential to ensure flexible configuration; however, this presents challenges in data modeling. Also, it is necessary to keep stability and development agility alongside this flexibility.

Data Volume:

Handling over 8 billion transactions per month presents significant challenges in terms of performance and scalability. Designing systems that can process, store, and retrieve this massive volume of data efficiently while maintaining low latency and high availability is a daunting task. To address these challenges, we utilize a variety of databases, including both relational and NoSQL databases, depending on the situation. As the data volume continues to grow, optimizing our data architecture to support this scale is critical.

Technical challenges are exciting, aren't they? If you are a software engineer.
While it may be difficult to tackle these issues, overcoming them elevates us to the next level.