🤖 Azure AI Foundry: GenAI Architecture, Security & Governance
Core-level guide to design, team isolation, quota management, private network controls, filtering policies, and continuous assessment in Azure AI Foundry.
01. Hub vs Project: Team Isolation Pattern
Azure AI Foundry's hierarchical model separates critical infrastructure from the experimentation and development layer using the Hub & Projects scheme.
Infrastructure & Security
The Hub (Parent Resource)
It is managed exclusively by the Platform / DevOps team. Centralize all the expensive and complex components to configure:
- Networking: Managed VNet, Private Endpoints and Ingress/Egress rules.
- Support Resources: Fixed connections to Key Vault, Storage Accounts and AI Search.
- Global policies: Shared connections and security settings.
Workspace / Sandbox
The Project (Child Resource)
It is assigned to development or Data Science teams. Automatically inherits all security from the Hub without allowing its alteration:
- Autonomous space to develop prompt flows, create deployments and perform evaluations.
- RBAC limited to the Project level (engineers cannot touch the Hub).
- Transparent consumption of shared connections.
Recommended Isolation Pattern: Deploy one Hub per environment/security domain (ex: hub-prod-eastus) controlled by Platform, and one Project per team or use case (ex: proj-chatbot-finance). Data scientists gain full autonomy in their project without compromising network infrastructure or global credentials.
02. Deployment Options: Serverless, Managed Compute or Azure OpenAI?
The choice of execution model determines the cost structure, latency, and operational responsibility.
| Deployment Model | Collection Mechanism | Main Use Case | Advantages & Disadvantages |
|---|---|---|---|
| Serverless API | Pay-as-you-go (for 1K tokens processed). | Catalog models (Llama 3, Mistral, Cohere) with variable or non-predictable traffic. | Pro: Zero infra management. Con: Less control over the underlying GPU. |
| Managed Compute | Fixed payment per VM/GPU hour (even idle). | Models that only offer IaaS/VM format or loads with constant throughput. | Pro: Full control of the environment and dedicated instances. Con: Compute operational management. |
| Azure OpenAI (Standard) | Pay-as-you-go based on token consumption. | General applications consuming GPT-4o or Embeddings without strict SLA guarantee. | Pro: No initial fixed cost. Con: Possible throttling (HTTP 429) on global peaks. |
| Azure OpenAI (PTU) | Provisioned Throughput Units (fixed reserve). | Critical production with high demand that demands predictable latency and zero throttling. | Pro: Guaranteed performance. Con: Financial commitment and high initial cost. |
03. TPM Quota Management and Contention between Teams
Azure OpenAI quota is allocated at the Subscription and Region level expressed in Tokens Per Minute (TPM). When multiple projects compete for the same share, bottlenecks and HTTP 429 (Too Many Requests) errors occur.
💡 Quota Governance Strategy:
- Explicit Allocation: Distribute TPM quota per deployment based on business priority.
- PTU Reservation: Isolate critical production loads using PTUs to avoid shared quota contention.
- AI Gateway (Azure API Management): Place an API Gateway in front of AI services to apply rate-limiting per consumer, balance requests across multiple regions/accounts, and record exact consumption metrics per team.
- Physical Isolation: If contention persists, migrate the computers to independent subscriptions, since the quota is delimited by subscription-region.
04. Securing the Hub Network & Outbound Rules
In enterprise and regulated environments, public access to the AI Foundry Hub must be disabled completely (publicNetworkAccess: Disabled).
Private Inbound Access
Private Endpoints
Developer and application traffic must enter through ExpressRoute or private VPN connected to the VNet. Dedicated Private Endpoints are created to:
• The AI Foundry Hub
• Azure Storage Account
• Azure Key Vault
• Azure AI Search / Azure OpenAI
Egress & Managed VNet
Outbound Rules Restricted
The Hub's managed VNet can be configured in "Allow only approved outbound" mode. In this mode, it is imperative to authorize the explicit FQDNs required by the Model Catalog and Serverless APIs to communicate with Microsoft dependencies.
Operational Cost: Blocking everything requires maintaining an active whitelist of endpoints to avoid failures in downloading model weights.
05. Specific RBAC Roles & Passwordless Connections
Security at the control and data plane requires discarding the use of shared API Keys in favor of granular RBAC roles and Managed Identities.
RBAC Role Matrix in AI Foundry
| RBAC Role | Assignment Level | Grant/Restrict Permissions |
|---|---|---|
| Azure AI Developer | Project | Allows: Develop prompt flows, create deployments and consume connections. Restricts: You cannot create or modify Hubs, networks, or RBAC. |
| Azure AI Inference Deployment Operator | Project / Hub | Allows: Manage and update only model inference deployments. |
| Cognitive Services OpenAI User | AI Resource | Allows application identities (Service Principals/Managed Identities) to consume inference (Data Plane). |
| Contributor / Owner | Hub / Subscription | Reserved exclusively for the Platform/DevOps team to manage infrastructure. |
Credential Management in Connections
Connections are reusable references to external resources (AI Search, Storage, Azure OpenAI). They are stored encrypted in the Key Vault linked to the Hub. The recommended architecture completely eliminates secrets using Connections based on Entra ID and Managed Identity, avoiding manual rotation cycles.
06. Content Safety & Custom Filters
Azure AI Content Safety analyzes inputs (prompts) and outputs (responses) classifying content into 4 categories: Hate, Violence, Sexual Content and Self-Harm.
Content Filter Governance: By default, deployments block Medium and High severity content. It is possible to implement Prompt Shields (to detect Jailbreak attacks) and filters against protected material. The power to modify or relax these filters should be restricted via RBAC to the Responsible AI or Platform team, preventing developers from disabling corporate risk controls.
07. LLM Assessment & Continuous Observability
GenAI application validation is not limited to initial deployment; requires a continuous cycle of quantitative and qualitative evaluation.
Development / CI/CD Phase
Pre-Release Evaluations
They are executed using a "Judge Model" (LLM-as-a-Judge) against test datasets integrated into the CI/CD pipeline as quality gate:
- Quality: Groundedness, Relevance, Coherence and Fluency.
- Risk: Evaluation of vulnerabilities to Jailbreaks and harmful content.
Production Phase
Continuous Monitoring & Tracing
Distributed tracing via direct export to Application Insights:
- Capture of prompts, tools invoked, tokens used and end-to-end latency.
- Continuous evaluation on real traffic samples to detect degradation or hallucinations in production.
08. Catalog Model Governance with Azure Policy
To prevent unauthorized use of untested or expensive models, organizations apply declarative governance at the infrastructure level.
💡 Control Mechanism with Azure Policy:
Using built-in Azure Policy policies assigned at the Management Group level, you restrict the deployment of catalog models by collection, provider, or explicit whitelist. By using the deny effect, the platform automatically rejects any unauthorized deployment attempts, ensuring that only models approved by the Center of Excellence (AI CoE) are operable.