From legacy data stack to modern data stack

Astrafy's role was to design and implement from scratch a new data stack for Watchfinder on Google Cloud and with leading data technologies. Astrafy also took care of knowledge transfer and training sessions once the solution was implemented.

Watchfinder had a traditional data stack where there was a long time to insights from data generation to exploiting this data in various downstream applications. A significant amount of time was spent on infrastructure and fine-tuning applications instead of working on the data and getting valuable insights from it. Data quality was also often problematic and there was no automation whatsoever.

The solution is a greenfield Modern Data Stack project that involved different parts. We started with infrastructure and security foundations in order to build the data stack on solid foundations. Then we deployed the data applications on a GKE cluster and used BigQuery with dbt as the center for the transformations. Automation was a key part of the solution, with use of Terraform and a DataOps tool.

Results were really transformative for Watchfinder and could be summarized as follows:

• Reduced time to insights by 80%

• Full visibility on data quality with test automation

• Onboarding of new engineers in a matter of hours and convenience of dev environment to iterate quickly

• 100% automation of infrastructure via Terraform

• IAM bindings with "least privileged access" and "segregation of duty"

“This new data stack is a real game-changer for us and we are now ready to get the most out of our data. We have transitioned from a paradigm where we were firefighting data issues and debugging data applications to a paradigm where we just focus on transforming data the way we want and everything else is automated. Shoutout to Astrafy for making this project a success!"

Aphonso Rafael Data & Analytics Manager @ Watchfinder

“This new data stack is a real game-changer for us and we are now ready to get the most out of our data. We have transitioned from a paradigm where we were firefighting data issues and debugging data applications to a paradigm where we just focus on transforming data the way we want and everything else is automated. Shoutout to Astrafy for making this project a success!"

Aphonso Rafael Data & Analytics Manager @ Watchfinder