Blog

Marketing teams don't need another dashboard. They need spend tied to revenue and answers in the tools they already use. Here's the architecture.

Every org assessment hits the same wall. The console shows one resource at a time. BigQuery answers the joins. Here's how to bridge them.

An opinionated guide to Data Modeling

Looker is becoming the semantic brain for AI through the Model Context Protocol (MCP). By connecting LLMs to the semantic layer instead of raw databases, MCP ensures governed, accurate queries, turning AI from a SQL guesser into a reliable interface for exploring trusted business data.

How Semantic Data Graphs move logic into the database, and why our organizations might not be ready for it.

In the AI era, the BI department is no longer a reporting function. It is the integrity layer of the business.

67% of AI projects fail to become a real business asset. The fix isn't a better model: it's a better strategy for production. Here's the MLOps path.

Is open source the future? I can't say for sure. What does seem clear, though, is that it definitely had us rethinking technology. With tools like ChatGPT, access to information has been democratized. Answers are now just a prompt away, it's just a matter of asking.

A practical guide to connecting Airbyte's community edition with Airflow without authentication, using Python operators and built-in error handling.

Outlining best practices for creating secure and scalable data infrastructures on Google Cloud Platform (GCP). It emphasizes Infrastructure as Code (IaC) using Terraform and GitOps for automated deployments, robust security measures including IAM, VPC Service Controls, and encryption,

Enhancing Looker dashboards with advanced period-over-period (PoP) tooltips is simplified using LookML and Liquid templating. Users can create dynamic Month-over-Month (MoM) percentage change measures utilizing Looker's period_over_period functionality.

Looker introduces a new Period-over-Period (PoP) measure type, simplifying time-based comparisons directly in LookML. It offers clean syntax, consistent metrics, but requires a date dimension in visualizations and has static comparison parameters, limiting dynamic user interactions.

In marketing, making informed marketing decisions requires an integrated view of data from multiple sources - Google Ads, Google Analytics, Google Search Console, and various custom systems.

Comparing multiple periods in Looker visualizations can be challenging as Looker does not offer a native way to achieve this. This article demonstrates a flexible, easy-to-use period comparison solution that has been successfully adopted by many clients.

With the rise of big data and AI, the Chief Data Officer (CDO) role has evolved from compliance-focused to a strategic leadership position. This article explores the multifaceted role of the ideal CDO, examining their responsibilities, organizational structures, and critical skills needed to thrive.

A step-by-step guide for developing LookML code using dev or prod tables on a single Looker instance, solving the challenge of dynamically switching between dev and prod BigQuery tables without native Looker support.

High-quality data ensures that analytics and reporting are trustworthy, enabling decision-makers to act confidently. This article covers systematic data quality checks across structural integrity, cross-system consistency, business logic, anomaly detection, and timeliness.

dbt 1.9 introduces microbatch incremental models as a core new feature, enabling processing of smaller data chunks for faster refreshes and reduced strain on data warehouses. This article also covers other enhancements included in the release.

Conversational analytics in Looker via Gemini shows promise but still has significant limitations. AI must be seen as a co-pilot, not an auto-pilot - strong semantic layers and well-built data models remain essential for accurate AI-generated insights.

By building and testing dashboards in a dev mode, teams can ensure the accuracy of the data being displayed, verify the functionality of filters and visualizations, and fine-tune the user experience before publishing them in production.

Airflow Datasets revolutionize data pipeline triggering by enabling dynamic task execution based on data availability, rather than predefined schedules. This simplifies dependency management, enhances workflow flexibility, and reduces latency. A practical example is integrating Google Cloud Pub/Sub

Open-source data tools are powerful but challenging to deploy without strong DevOps skills. The article presents a Kubernetes-based accelerator that simplifies this process, helping companies efficiently build a robust data stack with enterprise features, focusing on strategic tool selection.

Airflow summit is just around the corner (September 10-12), and the timing was, therefore perfect for a new important release of Airflow. This minor release, "2.10" does not disappoint!

If not managed properly, Kubernetes can sometimes lead to unexpected expenses. This article aims to provide practical tips for optimizing Kubernetes costs on Google Cloud Platform (GCP). We will focus on strategies to reduce the number of machines needed, which in turn reduces costs associated ...

Discover how Astrafy's GA4 Accelerator transforms raw Google Analytics data into deep, actionable insights using advanced dbt models and Looker dashboards, empowering businesses to make informed decisions and boost performance.

As LLMs and RAG systems reshape traditional search engine dynamics, understanding and enhancing your website's SEO becomes critical. We will you on integrating Google Search Console and Google Analytics 4 data using a dbt pipeline to optimize SEO. Staying proactive ensures your website remains ...

In today's data-driven world, businesses rely heavily on powerful analytics tools to make informed decisions. Looker, Looker Studio, and Looker Studio Pro are three such tools that offer robust data, but what are their main differences?

Google's Gemini LLM is used at Astrafy to enhance Google Analytics data transformation. By leveraging BigQuery's ML capabilities, they improve website traffic categorization, overcoming traditional ML limitations. Gemini's advantages include minimal data dependency, adaptability, and efficiency, ...

Introducing a custom alert system integrated into Google Cloud for specific error and anomaly detection. It uses BigQuery log sinks and views to detect issues, with Cloud Functions running queries every 15 minutes. Alerts are sent to Google Security Command Center and Slack, enhancing security ...

Data quality issues cost businesses billions annually, with a $3.1 trillion impact on the US economy. Elementary, a dbt-native data observability platform, helps detect data anomalies to ensure robust and reliable dbt projects and data pipelines. The article explores how Elementary integrates...

Apache Airflow is a powerful, free tool designed to manage complex workflows and data pipelines. It uses directed acyclic graphs (DAGs) to ensure tasks run in the correct order without looping back. By writing workflows in Python, Airflow offers benefits like version control, easy collaboration, ...

Is a Google Cloud Champion Title in Your Reach? Alex's One-Year Journey Shows You How!

Every four months, dbt releases a new version of its dbt core and dbt adapters. In those first days of May, dbt 1.8.0 was released and the latest features do not disappoint. In this article, we will…

Automate the orchestration of SQL pipelines in Google Cloud Platform using Dataform and Terraform.

Explore the crucial role of code refactoring in data engineering, focusing on dbt Core code optimization.

Understand Data Quality issues in dynamic data ecosystems, emphasizing the need for Data Contracts.

Understand Cloud logging, apply FinOps practices, and reduce your spending in Google Cloud.

Implement VPC Service Controls early in Google Cloud projects like BigQuery and Vertex AI.

Learn how dbt incremental works and swap partition columns cheaply without the need for full refreshes.

Understand the implications of running dbt in different environments focusing on Google Cloud — Running dbt in production has become one of the commonly asked questions among dbt practitioners.

Apply FinOps principles on Google Cloud to manage costs and optimize investment, including cost monitoring, anomaly detection, and budgeting. Automated solutions like daily and monthly billing reports, dashboards, and alerts help maintain cost efficiency and control.

Streamline Looker's period-over-period analysis using our GitLab toolkit to quickly integrate and visualize data, reducing setup to a swift, under-10-minute process

Google Cloud Workstations standardize computer setups across teams, enhancing operational efficiency and allowing new engineers to onboard quickly. The system offers ready-to-use tools, Terraform for easy setup, AI assistance, security, automatic Git repository integration, and cost-saving features.

Transitioning to GA4, a simplified session/pages granularity model is proposed for efficient data analysis, accommodating multiple dimensions and measures while being mindful of storage and cost implications.

Tooltips in dashboards offer additional context and promote interactivity without cluttering the view, and can be personalized with colors and symbols for clarity. Adding them in Looker involves calculating and displaying period-over-period percentage changes to enhance data understanding

The article presents a guide on securing dbt documentation and Elementary UI with Google Cloud tools, focusing on easy access and robust data management for data teams.

Blog post explores the idea of partitioning BigQuery tables on clustered columns using dbt's insert_overwrite strategy. It suggests partitioning only on specific cluster columns for efficiency and discusses a workaround using UNION ALL for cost-effective clustering and partitioning control

Node Selector, Affinity, and Taint/Toleration in Kubernetes optimize pod placement for performance and security, essential for managing expanding clusters efficiently.

Data Mesh promotes separate dbt projects for data product manageability and interoperability, using packages.yml and dependencies.yml for model imports. Dbt-loom bridges functionality gaps for dbt-core users, enhancing collaborative data management in alignment with Data Mesh principles.

Slack commands, integrated with Google Cloud Functions, automate notifications and task workflows, including security checks and admin interventions. This powerful combination enhances productivity and ensures security in managing internal operations.

Images enhance data dashboards by improving engagement with visuals, providing context, and supporting branding. Looker allows image integration through URLs or Base64 encoding, with options for customization and clickable logos.

Transitioning from Universal Analytics to Google Analytics 4 introduces significant changes in analytics for e-commerce, including new definitions for sessions, users, and other key metrics. Businesses must adapt to these changes to accurately analyze user behavior and refine online strategies.

Google Cloud Workload Identity secures cloud resources by linking external identities to service accounts, removing the need for static credentials. It uses identity pools and providers, and integrates with GitLab CI via OIDC, enhancing security and simplifying access in cloud deployments.

Astrafy automates Airflow deployments on GCP using Terraform for efficient environment replication and testing at scale, offering cost savings, customization, and better control compared to managed solutions like Cloud Composer.

Astrafy leverages budget alerts, expense reports, and a data mesh approach to manage and monitor unexpected cloud costs on Google Cloud, ensuring efficient and transparent cloud financial management.

AGI transitions from fiction to reality with advancements in AI, highlighted by Astrafy's development of Astragy, an autonomous and versatile AI prototype, marking progress towards achieving AGI's transformative potential.

The article discusses using dbt and Data Vault 2.0 for managing Astrafy's sales and marketing data, focusing on practical implementation with technologies like Google Cloud and BigQuery for efficient, scalable data architecture.

Data Vault 2.0, advancing with dbt integration, offers agile and scalable data management solutions, detailed in a two-part series highlighting its application for efficient, data-driven decisions.

Astrafy's Astrabot, leveraging advanced RAG architecture, provides businesses with a customizable and private AI chatbot solution, adapting to diverse needs while ensuring data privacy.

Workload Identity Federation ensures secure authentication in Google Cloud with short-lived credentials, eliminating the need for long-lived service account keys and enhancing security and management efficiency when integrated with Terraform Cloud.

Automating Google Cloud deployments with Terraform simplifies setting up HTTPS load balancers and managing GKE clusters, enabling efficient application deployment and scaling with modules for streamlined infrastructure management.

Monitoring BigQuery activity through a structured data product approach, using dbt and BI dashboards, enables detailed insights into usage and costs, while also facilitating broader applications such as machine learning and automated alerts for comprehensive data management and optimization.

The AI Code Reviewer enhances efficiency by automating standard code reviews, allowing focus on critical issues, though it faces challenges with large diffs and security.

Part 3 of implementing data contracts with dbt, Google Cloud, and Great Expectations covers deploying the solution, monitoring data quality, and integrating alerts to ensure reliable data pipelines.

The Data Contracts Management Layer enables collaboration between Data Producers and Consumers on schema agreements, using a git repository and Cloud Storage for versioning and secure deployment of data contracts, ensuring data integrity and quality.

The Data Contracts Management Layer facilitates collaboration between Data Producers and Consumers through a git repository and Cloud Storage, enabling version control and secure deployment of data contracts to ensure data quality and integrity.

DuckDB enhances dbt project efficiency with its fast, in-memory analytics and simple setup, significantly reducing query execution times. It integrates well with dbt for rapid development cycles, but is suited for local development due to its early stage and limitations with large datasets.

Automated Airflow data quality implementation at Astrafy focuses on prevention, with Great Expectations as a core tool for ensuring data integrity. The approach emphasizes collaboration, effective tool selection, and shared responsibility across teams for maintaining high data quality standards.

The article outlines a method for deploying Pytorch Hugging Face models to Google Cloud using custom containers, covering steps from model preparation and cloud setup to automated deployment and testing for batch predictions.

The article outlines an automated system for sending Google Cloud expense reports to Slack, using dbt, Airflow, Pub/Sub, Cloud Functions, and Terraform, to enhance financial management and operational efficiency.

Open source dbt packages enable efficient and modular data code reuse, enhancing productivity and adherence to best practices in dbt projects. They provide a community-driven foundation for robust, well-documented, and tested data workflows

Infracost enhances FinOps by offering Terraform cloud cost estimates, integrated into CI pipelines for better management of cloud expenses and adherence to FinOps principles, ensuring efficient cloud financial management.

Astrafy sets up a cost-efficient GKE cluster to host open-source Modern Data Stack applications, leveraging Google Kubernetes Engine for flexibility, Istio for networking, and comprehensive security and access management, while detailing benefits like cost savings and operational simplicity.

The dbt Analytics Engineering Certification exam is challenging and demands extensive dbt practice and study, offering a credible validation of one's dbt skills.

This article presents an Airflow DAG factory for auto-generating workflow files from YAML configs, with a future focus on testing and version control.

This article explains the benefits of using Terraform provider aliases for managing infrastructure with multiple credentials, emphasizing the best practices of least privilege and infrastructure segmentation to enhance security and efficiency.

Part 3 of the 'dbt on Google Cloud' series explores advanced orchestration, DataOps strategies, documentation, and monitoring, emphasizing the need for meticulous project management and the integral role of Cloud Composer in streamlining data workflows.

Part 2 of the "dbt at scale on Google Cloud" series explores advanced topics like infrastructure optimization, dbt version control, style guides, macros, data quality assurance, and governance, emphasizing secure and compliant data management.

Efficient data engineering on Google Cloud with dbt streamlines data ingestion, transformation, orchestration, and distribution, leveraging open-source tools and Google Cloud services for an integrated data management solution.