(If you are not familiar with us yet, you can read more at our previous post)
In the era of modern software development, there is one hypothesis that is always true: ”Data is king”.
Data helps us get a better understanding of our customers, employees, decisions, analyses, insights, predictions, revenue, and more.
At Jeeng, we believe that each decision we make should be based on data. In fact, making data-informed decisions is one of our official company values, as we believe data belongs to our entire company and not just to our engineering and data science teams.
As part of our journey to support our customers better, and make sure we provide them as well as our internal staff with better data-informed insights, we have decided to adjust our platform to handle big data.
The process of changing an architecture to support this is long and complicated. We started by analyzing our internal requirements and our customers’ current needs, plus the future requirements we can predict. After deep analysis, we started choosing the right tools for each task: Kafka as an event streaming platform, Big Query as a data warehouse, and Spark as a large-scale data processing tool.
Choosing each tool was a meticulous job; we needed to do a thorough background check to see if they met our needs in different aspects, such as: effectiveness, pricing, maintenance, support, documentation, etc.
After choosing the tools and performing a POC for each of them, we started implementation. This was the tough part, as we needed to make sure we could still deliver on our requirements and keep the platform working with minimal effect on our customers.
In order to do this we implemented each section piecemeal, so we needed to make sure our system was loosely coupled and each tool could act as a standalone component.
As our journey continues, we will keep you updated on the process we are going through and the positive effects it will have! We are very excited about what’s to come.