Data Engineering
Ingestion and Transformation of ESG Data Using Dataflow
How Astrafy refactored Wequity's ESG data pipeline from Kedro on a VM to Google Cloud Dataflow, cutting costs by 35% and increasing speed by 70%.
Wequity is a tech company that uses AI to automate ESG compliance and corporate reporting. The platform helps organizations quickly process and fill out complex sustainability, security, and vendor questionnaires by automatically extracting data from internal documents.
Website
Industry
Software & Technology
Location
Belgium
Stack
Google Cloud Dataflow · Apache Beam · Terraform · Cloud Scheduler · BigQuery
Astrafy refactored a solution (from an existing one in Kedro) to transform tweets and newspaper articles using Google Cloud Dataflow. The Dataflow pipeline included many steps ranging from simple Python transformations to more complex NLP operations.
The challenge
Scaling limitations of a VM-based Kedro pipeline
Before the project, the customer was running a pipeline on a VM using the Kedro framework. It had various drawbacks: scaling issues, lack of proper monitoring, and the need to maintain the infrastructure.
The solution
Refactoring the pipeline with Cloud Dataflow and Terraform
The refactoring of the pipeline using Cloud Dataflow solved all the aforementioned issues and made it very simple to add or amend stages in the Dataflow pipeline. We deployed the solution using Terraform for all the infrastructure components and Python Apache Beam for all the code related to the Dataflow pipeline. Cloud Scheduler runs the pipeline on a daily basis to start the pipeline.
Results
Faster, cheaper, and fully observable data pipeline
Results of using Dataflow compared to Kedro framework on a VM:
70% speed increase
35% cost reduction
Full observability on each stage of the pipeline with detailed logs and monitoring
Optimization of BigQuery output tables with partitioning and clustering
Fast iteration on new development due to well-structured codebase provided by Astrafy
Latest case studies
