Written by Albert Friedman
on 

Edge AI Without the Trade‑offs: A Practical Multi‑Cloud Pattern Using Azure Arc Edge RAG and Google Distributed Cloud

Hybrid and multi-cloud used to be about “where to run VMs.” In 2025, it’s increasingly about “where to run AI inference and retrieval.” Two announcements this year make that clear: Microsoft’s Edge RAG (Retrieval-Augmented Generation) arrived in public preview as an Azure Arc extension, designed to keep data and models local on Azure Local (formerly Azure Stack HCI). And Google made Gemini available on-premises through Google Distributed Cloud (GDC), including an air‑gapped option. Together they mark a turning point for edge AI architectures that need cloud-scale tooling without giving up control, privacy, or latency. (learn.microsoft.com)

This article is a hands-on blueprint for teams who want to design a pragmatic, multi-cloud edge AI pattern—mixing Azure Arc Edge RAG with GDC Gemini—without getting lost in vendor hype.

Why this matters now

The bottom line: you can now keep sensitive data at the edge, run modern LLMs on-prem, and still use cloud-native operations across providers.

The reference pattern in a nutshell

Here’s a simple, repeatable pattern that works across U.S. enterprise constraints like latency SLAs, data residency, and vendor diversity:

This gives you two self-sufficient edge stacks (one Microsoft‑aligned, one Google‑aligned) that can cooperate when needed, without a single point of failure in any one cloud.

What changed in 2025

For architecture decisions, the preview/GA details matter: today, Edge RAG supports AKS on Azure Local; it isn’t a “run anywhere” extension for any Arc‑connected cluster. GDC provides managed Gemini endpoints and scales from single servers to racks, connected or air‑gapped. (learn.microsoft.com)

A step‑by‑step way to pilot

The fastest path to value is to deploy each stack where it’s strongest, then integrate.

1) Prepare Azure Local and Arc

2) Deploy Edge RAG locally

3) Stand up Google Distributed Cloud with Gemini

4) Keep data local; control egress

For compliance and cost, pin your data plane on‑prem and scope egress tightly. As a simple example for the Arc cluster:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-external-egress
  namespace: arc-rag
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    # Allow only your BYOM endpoint or internal services
    - namespaceSelector: { matchLabels: { name: default } }
    - ipBlock: { cidr: 10.0.0.0/8 }  # internal ranges

If you must call a cloud endpoint (e.g., Azure AI Foundry or a GDC connector), enumerate only those FQDNs and ports via egress gateways or explicit allow‑lists, and prefer private links where available. For cross-cloud connectivity, leverage managed offerings; Google highlights performance benefits with Cross‑Cloud Network over public internet paths. (cloud.google.com)

A few gotchas to plan for

Minimal “hello RAG at the edge” workflow (Azure side)

1) Connect your cluster to Arc and verify:

az connectedk8s connect --name edge-west --resource-group rg-edge
az connectedk8s proxy -n edge-west -g rg-edge
kubectl get ns

(learn.microsoft.com)

2) Prep AKS Arc node pools per guidance (CPU‑only or CPU+GPU):

# Example (PowerShell) from MS docs for CPU pool
$cpuPoolName="<CPU Pool Name>"
$k8scluster="<AKS Arc Cluster>"
$rg="<Resource Group>"
$cpuVmSku="Standard_D8s_v3"
$cpuNodeCount=6
az aksarc nodepool add --name $cpuPoolName --cluster-name $k8scluster -g $rg --node-count $cpuNodeCount --node-vm-size $cpuVmSku

(learn.microsoft.com)

3) Deploy Edge RAG and choose BYOM or a Microsoft‑provided model. After deployment, confirm the extension shows up as microsoft.arc.rag. (learn.microsoft.com)

4) If using BYOM, set up an OpenAI‑compatible endpoint (Azure AI Foundry, Foundry Local, Ollama) and plug its URL+API key into Edge RAG’s configuration. Microsoft’s doc provides exact formats (e.g., Azure AI Foundry chat completions URL). (learn.microsoft.com)

5) Configure MetalLB, certs/trust, and DNS so the local RAG UI/API are reachable within your site and through your SSO boundary (Entra). (learn.microsoft.com)

Minimal “hello Gemini on-prem” workflow (Google side)

1) Pick GDC mode:

2) Provision the managed Gemini endpoint on GDC and confirm NVIDIA hardware support if you need higher throughput. Confidential computing options are available. (cloud.google.com)

3) Optionally pilot Agentspace search (preview) for out‑of‑box, permission‑aware enterprise search over on‑prem data. (cloud.google.com)

4) Wire up cross‑cloud traffic only where necessary (e.g., for comparisons, observability export, or failover). Consider private or managed network options for predictable latency and cost. (cloud.google.com)

Operating the pattern: a short checklist

When to prefer which stack

In many enterprises, you’ll do both—matching each site’s constraints and skill sets.

Final thoughts

Edge AI in 2025 isn’t “cloud versus on‑prem.” It’s “cloud‑operated, locally executed”—with vendor choice per site. Microsoft’s Edge RAG gives you a managed, on‑prem RAG pipeline anchored in Azure Arc and Azure Local. Google’s GDC brings Gemini behind your firewall with options for connected or fully air‑gapped operations. With careful egress controls, identity, and consistent operations, you can deploy edge AI where it belongs—next to the data and users—without sacrificing enterprise guardrails or multi‑cloud flexibility. (learn.microsoft.com)

If you’re starting this month:

Small, well‑scoped wins will snowball—and you’ll avoid the lock‑in trap while delivering measurable value fast. (azure.microsoft.com)

Resources to bookmark:

This is an edge AI pattern you can actually run—and expand—today.