Introduction

Every day, unnoticed security issues or errors could be silently draining your business’s resources or worse exposing your business to outside attacks (excessive IAM bindings, service account key leakage, etc.). What if you could have a tailored system that alerts you of them, so you can take action before they cause harm?

This article introduces an alert system integrated into Google Cloud designed to address your company’s specific error/anomaly detection needs. While current error/anomaly notification solutions exist, this system stands out by offering the capability to receive alerts for customized issues unique to your operations, providing a higher level of personalization and relevance.

How does it work?

The following diagram depicts the overall solution:

Flowchart of a custom alert system: "Ingestion" with an "Audit logs sink" leading to "Transformation" with "Anomalies views." Cloud Functions dispatch findings every 15 minutes via a Cloud Scheduler, leading to "Distribution" with alerts sent to Slack and Google Cloud Security Command Center.

Hereafter are the different steps involved in this solution:

  1. Creation of a log sink of the audit logs in BigQuery with partitioning; this sink logs all activities occurring within your Google Cloud organization.

  2. Definition of a series of views that are based on tailored queries to detect specific anomalies related to your business use cases.

  3. Every 15 minutes, a Cloud Function triggered by a Cloud Scheduler runs those queries from step 2 and sends the results to the Google Security Command Center.

  4. Users can access the alerts in a structured and organized manner via the Google Security Command Center UI. Additionally, it is possible to connect the Google Security Command Center to Slack for personalized notifications, allowing users to receive timely alerts directly in Slack.

The advantage of this solution is that it allows for complete customization of what should be considered an alert for your business, providing security departments with the flexibility to tailor the system to their specific needs and criteria.

This solution can be thought of as a data product around your logging data with the raw data being the audit log sink and the anomaly views being the transformations applied to this raw data. The activation of this data is then carried out by the Clodu Function and end users can consume the result of this data via Google Security Command Center.

Custom alerts

Custom alerts are based on BigQuery views which contain specific logic to retrieve an error/security anomaly from the audit logs.

An interesting anomaly detection query might look like the following:

Screenshot of a SQL query for Google Cloud audit logs. The query selects various fields like timestamps, caller IP, service name, and resource labels, and checks if flow logs or log configuration are disabled in Google Cloud subnetwork activities, filtering results where these conditions are met.

The query above will detect the deactivation of VPC flow logs. Those logs can be deactivated but it’s of major importance that those are not deactivated for unknown reasons and having this detection mechanism in place will make your security team proactive to investigate further.

Any kind of query (from simple to complex) can be defined and some examples of interesting alerts are:

  • IAM Policy granted on User Outside of Customer Approved Domain List

  • Impersonation of a service account by a user

  • Abnormal amount of data egress outside of your Google Cloud organization

  • Alert on Super Admin (Org./Owner) Login

  • …

More examples can be found in this github repository.

All those queries are run via Cloud Function every 15 minutes and all the rows returned by those queries are logged as findings in Google Security Command Center.

Google Security Command Center

This product is the backbone of the security offering on Google Cloud and acts as a single pane of glass for all your security topics within Google Cloud. The list of capabilities of this product is beyond the scope of this article and here we will on the “findings” section.

Generic Findings

A finding is a record of a security issue that the Security Command Center services create when they detect security a issue. Some of the actions that you can perform on the findings page include the following:

  • Query findings

  • Inspect findings

  • Mute findings

  • Add security marks to findings

Findings are listed in the Findings query results panel of the Findings page. You can click on a finding to see the details of the finding, as well as its full JSON format.

By default, Google Cloud will log default security findings such as “SSL not enforced”, “Public IP address”, “Open RDP port”, etc. All those findings can then be explored and the level of information you get is very detailed and even contains the next steps to mitigate those findings. Hereafter is a summary of a finding related to SSL not being enforced on an SQL instance:

Screenshot of a Google Cloud Security Command Center alert for "SSL not enforced." The alert is active with high severity, detailing an SQL instance with unencrypted communications. It includes detection details, affected resource information, and next steps for remediation, such as enabling SSL and verifying instance status. Related compliance standards are listed.

Custom Findings

A great feature is that you can push custom findings and those will appear in the UI next to the findings detected by Google Cloud. This is what is being done by the solution presented in this article where Google Cloud is pushing all the results from the BigQuery views to Google Security Command Center using the “Node.js 22” client. The code is pretty simple with first the creation of the finding object and then the POST of this object to the Security Command Center endpoint. All the attributes encountered in the UI can be specified while creating the finding object as you can see in the function hereafter:

Screenshot of a JavaScript function "createFindingObject." The function processes a source name and labels to generate a finding object with attributes like state, resource name, category, and event time. It excludes certain keys and formats values into a sourceProperties object, returning a structured finding object for logging security issues.

Once the findings are exported to the Security Command Center, the user can easily explore those within the UI in the same way as the other findings set by Google Cloud. Let’s look at an example with one finding that we defined via a BigQuery view to detect IAM roles being added. In the UI we can easily filter on that finding:

Screenshot of the "Findings" section in Google Cloud Security Command Center. It shows a query result for active, non-muted findings related to "IAM role add." The results include categories, unspecified severity, and event times. Quick filters on the left allow filtering by state and category, such as non-org IAM members and public bucket ACLs.

By clicking on any of those findings we can get exhaustive details about the specific finding and see what role has been added to which principal and who was responsible for that action. This makes the work of your security team much easier.

Screenshot of an "IAM role add" finding in Google Cloud Security Command Center. The "Source Properties" tab lists details such as action (ADD), caller IP, log name, member, method name (SetIamPolicy), principal email, project ID, receive timestamp, role (iam.serviceAccountKeyAdmin), and type (project). Sensitive information is redacted.

Compliance Needs

Security nowadays is not just about making sure that your organization is bulletproof against internal and external threats. It is also making sure that your organization complies with the different laws and regulations in place in the different countries you operate. Logs retentions via BigQuery sinks of audit logs and then centralizing findings within Google Cloud Security Command Center are definitely very helpful tools to have in your toolkit in your journey to IT compliance.

Google Cloud Command Security Center actually comes with a “Compliance” subproduct that facilitates the work of your security & compliance team by identifying the missing compliance parts of different renowned regulatory and industry mandates.

Conclusion

The customized alerting and logging solution outlined in this article provides a robust and adaptable approach to monitoring security events and anomalies within Google Cloud environments. By leveraging the power of BigQuery, Cloud Functions, and the Google Security Command Center, businesses can achieve a higher level of visibility and control over their security posture.

This system not only enhances proactive detection of potential security threats but also integrates seamlessly with existing workflows, such as Slack notifications, ensuring that relevant alerts reach your team in real-time. The ability to define custom queries tailored to your specific needs ensures that the alerts you receive are highly relevant, helping your security team to act swiftly and effectively.

In conclusion, adopting this advanced logging and alerting solution will not only fortify your defense mechanisms against potential threats but also streamline your security operations, ensuring that your business remains resilient in the face of evolving cyber threats.

Thank you

If you enjoyed reading this article, stay tuned as we regularly publish technical articles on Google Cloud and how to secure it at best. Follow Astrafy on LinkedIn, Medium, and Youtube to be notified of the next article.

If you are looking for support on Modern Data Stack or Google Cloud solutions, feel free to reach out to us at sales@astrafy.io.