bala.ramamurthy

BigQuery Job Monitoring & ML-based Query Classification This solution builds a passive BigQuery job monitoring and ML-based classification system . It continuously collects BigQuery job metadata, enriches it with reservation capacity context, derives meaningful features, and trains a BigQuery ML model to identify good vs. problematic jobs . No production queries are modified No jobs are killed or throttled 100% metadata-driven and read-only Fully automated using scheduled queries This solution consist of two Phases, What Phase 1 Does Collects BigQuery job execution metadata on a schedule Captures slot reservation capacity information Derives job-level features such as runtime, slot usage, SQL patterns, and time attributes Applies rule-based classification to label jobs as good or problematic Trains a BigQuery ML model to learn query behavior patterns Retrains the model daily to stay accurate as workloads evolve Key Components bq_job_history – raw BigQuery job execution ...

Security Considerations for Multi-Cluster Cloud Architecture (HA EKS with Databases) Running a highly available multi-cluster EKS architecture brings powerful benefits—zero downtime, disaster recovery, and global scalability. But it also multiplies your security challenges. Securing a single EKS cluster is already complex. Add multiple clusters across regions, databases with sensitive data, and cross-cluster communication, and the attack surface grows significantly. One misconfigured security group or exposed secret can compromise your entire infrastructure. This guide covers essential security considerations for multi-cluster architectures: network isolation, encryption, IAM management, secrets handling, and incident response. We'll focus on practical measures that protect your infrastructure without sacrificing performance or availability. Let's build a secure, highly available system. 1. Network Security & Isolation VPC Architecture Separate VPCs per cluster or use share...