Decision owner
AI platform engineer or data engineer
The named role remains accountable for the consequential decision.
Technology & Software
Designed for AI platform engineer and data engineer
Continuously detect new or changed Azure Blob Storage objects, parse and chunk them, generate embeddings, persist chunks and vectors in Azure Cosmos DB, and let platform engineers compare vector, full-text, hybrid and semantic-reranked retrieval over the same corpus while honoring source ACLs, versions and ingestion status.
Contract TEC-04 · primary aggregate SourceDocumentIngestion · mode loading
Decision brief
This brief starts with the decision, evidence, and accountable owner. The guided workflow then shows how the recommendation was produced.
The evidence and control points that shape this recommendation.
Decision owner
AI platform engineer or data engineer
The named role remains accountable for the consequential decision.
Governed evidence
6 record types
Structured records and cited text ground the recommendation.
Workflow
5 controlled stages
Each transition records its input, output, status, and audit context.
Human controls
1 approval gate
Approval gates prevent the copilot from making consequential decisions.
A 94-page compliance handbook is replaced in the governed-docs container, producing a new blob version.
Continuously detect new or changed Azure Blob Storage objects, parse and chunk them, generate embeddings, persist chunks and vectors in Azure Cosmos DB, and let platform engineers compare vector, full-text, hybrid and semantic-reranked retrieval over the same corpus while honoring source ACLs, versions and ingestion status.
promotedRetrievalConfig gives AI platform engineer or data engineer the evidence needed to begin the final governed business outcome, with the decision trail retained for review.
Step 1 of 5
| Field | Value |
|---|
| Field | Value |
|---|
The output appears here immediately. More technical evidence remains available below.
The demonstration keeps deterministic controls, AI assistance, and approval separate.
Source systems enforce identity, workflow order, policy filters, and record updates.
Retrieval and generation assemble relevant evidence into a cited recommendation or draft.
AI platform engineer or data engineer verifies material claims and approves the consequential action.
Run the stages in order before using the Copilot. Every stage explains what happens, why it matters, the evidence it uses, and the complete API request and response.
Automated stage
What happens: Event Grid blob-created and blob-deleted subscription performs blob change detection using blobChangeEvent and produces SourceDocument. The transition is deterministic and does not require autonomous model approval.
Why it matters: SourceDocument gives Ingestion function with Document Intelligence and Azure OpenAI embeddings the evidence needed to begin Parse, chunk and embed, with the decision trail retained for review.
Loading relevant sample records.
Ready to run.
POST /api/domain/workflow/blob-change-detection
Run this API to see its formatted outcome.
| Field | Value |
|---|
| Field | Value |
|---|
Automated stage
What happens: Ingestion function with Document Intelligence and Azure OpenAI embeddings performs parse, chunk and embed using SourceDocument and produces ContentChunk. The transition is deterministic and does not require autonomous model approval.
Why it matters: ContentChunk gives Change-feed ingestion coordinator the evidence needed to begin Version cutover and stale chunk retirement, with the decision trail retained for review.
Loading relevant sample records.
Ready to run.
POST /api/domain/workflow/parse-chunk-and-embed
Run this API to see its formatted outcome.
| Field | Value |
|---|
| Field | Value |
|---|
Automated stage
What happens: Change-feed ingestion coordinator performs version cutover and stale chunk retirement using IngestionRun and produces currentVersionChunkSet. The transition is deterministic and does not require autonomous model approval.
Why it matters: currentVersionChunkSet gives AI platform engineer the evidence needed to begin Retrieval mode comparison, with the decision trail retained for review.
Loading relevant sample records.
Ready to run.
POST /api/domain/workflow/version-cutover-and-stale-chunk-retirement
Run this API to see its formatted outcome.
| Field | Value |
|---|
| Field | Value |
|---|
Automated stage
What happens: AI platform engineer performs retrieval mode comparison using RetrievalExperiment and produces RetrievalResultSet. The transition is deterministic and does not require autonomous model approval.
Why it matters: RetrievalResultSet gives AI platform engineer or data engineer the evidence needed to begin Engineer promotion of a retrieval configuration, with the decision trail retained for review.
Loading relevant sample records.
Ready to run.
POST /api/domain/workflow/retrieval-mode-comparison
Run this API to see its formatted outcome.
| Field | Value |
|---|
| Field | Value |
|---|
Human approval required
What happens: AI platform engineer or data engineer performs engineer promotion of a retrieval configuration using RetrievalResultSet and produces promotedRetrievalConfig. A named human approver must review the evidence before the transition completes.
Why it matters: promotedRetrievalConfig gives AI platform engineer or data engineer the evidence needed to begin the final governed business outcome, with the decision trail retained for review.
Loading relevant sample records.
Ready to run.
POST /api/domain/workflow/engineer-promotion-of-a-retrieval-configuration
Run this API to see its formatted outcome.
| Field | Value |
|---|
| Field | Value |
|---|
Run a stage to see its explanation, evidence, full request, and full response.
Ask a question and receive an answer grounded in retrieved SourceDocumentIngestion evidence.
Select a starter question or enter your own.
Chunk, vectorize, and store a document through the same engine that Event Grid and Azure Data Factory invoke in Azure mode.
Loaded live from /api/config, /api/domain/model, and
/api/domain/sample.
sourceDocumentIngestions/tenantId + /sourceDocumentIdA 94-page compliance handbook is replaced in the governed-docs container, producing a new blob version. Event Grid fires, the ingestion function parses it with Document Intelligence, produces 418 chunks, embeds them with the text-embedding-3-large deployment, bulk-writes them under the new blob version id, and flips the current-version flag in one step so no query sees a mixed corpus. An engineer then asks a retention-period question through all four modes: pure vector returns the right section at rank 4, full-text at rank 1 because the query used the exact statutory phrase, hybrid at rank 1 with better supporting context, and semantic reranking matches hybrid at rank 1 while adding 380 milliseconds. The engineer promotes hybrid without reranking for this corpus and records the decision with the measured latency and request units.
| Container | Role | Hierarchical partition key | Document types |
|---|---|---|---|
sourceDocumentIngestions |
Primary | /tenantId + /sourceDocumentId |
SourceDocument, IngestionRun, ContentChunk, RetrievalExperiment, RetrievalResultSet, GroundedAnswer |
| Type | Container | Partition key | Identifier pattern | Embedding fields |
|---|---|---|---|---|
SourceDocument |
sourceDocumentIngestions |
/tenantId + /sourceDocumentId |
source::{sourceDocumentId} |
None |
IngestionRun |
sourceDocumentIngestions |
/tenantId + /sourceDocumentId |
source::{sourceDocumentId}::run::{blobVersionId}::{runId} |
None |
ContentChunk |
sourceDocumentIngestions |
/tenantId + /sourceDocumentId |
source::{sourceDocumentId}::chunk::{blobVersionId}::{chunkOrdinal} |
contentVector |
RetrievalExperiment |
sourceDocumentIngestions |
/tenantId + /sourceDocumentId |
source::{sourceDocumentId}::experiment::{experimentId} |
queryVector |
RetrievalResultSet |
sourceDocumentIngestions |
/tenantId + /sourceDocumentId |
source::{sourceDocumentId}::experiment::{experimentId}::result::{retrievalMode} |
None |
GroundedAnswer |
sourceDocumentIngestions |
/tenantId + /sourceDocumentId |
source::{sourceDocumentId}::answer::{answerId} |
None |
Rank current-version chunks by vector distance alone as the semantic baseline in the comparison. Application-side partition-key corpus retrieval: the ingested corpus spans sourceDocumentId partitions by design, so the scan is bounded by the tenantId prefix, the current-version flag, the caller's ACL principals and a topK limit.
sourceDocumentIngestions
partition-key-fan-out
SELECT * FROM c WHERE c.tenantId = @tenantId AND c.type = 'ContentChunk' AND c.isCurrentVersion = true AND ARRAY_CONTAINS(c.aclPrincipalIds, @callerPrincipalId) ORDER BY VectorDistance(c.contentVector, @queryVector)
Rank the same chunk set by lexical BM25 scoring to expose where keyword matching beats embeddings. Application-side partition-key corpus retrieval: bounded by the tenantId prefix, the current-version flag, the caller's ACL principals and a topK limit over the full-text index.
sourceDocumentIngestions
partition-key-fan-out
SELECT * FROM c WHERE c.tenantId = @tenantId AND c.type = 'ContentChunk' AND c.isCurrentVersion = true AND ARRAY_CONTAINS(c.aclPrincipalIds, @callerPrincipalId) AND FullTextContainsAny(c.chunkText, @queryTerms) ORDER BY RANK FullTextScore(c.chunkText, @queryTerms)
Fuse vector and full-text rankings with reciprocal rank fusion for the hybrid arm of the experiment. Application-side partition-key corpus retrieval: bounded by the tenantId prefix, the current-version flag, the caller's ACL principals and a topK limit fused by reciprocal rank fusion.
sourceDocumentIngestions
partition-key-fan-out
SELECT * FROM c WHERE c.tenantId = @tenantId AND c.type = 'ContentChunk' AND c.isCurrentVersion = true AND ARRAY_CONTAINS(c.aclPrincipalIds, @callerPrincipalId) ORDER BY RANK RRF(VectorDistance(c.contentVector, @queryVector), FullTextScore(c.chunkText, @queryTerms))
Show blobs whose newest version is not yet searchable so staleness can be reported honestly in answers. Application-side partition-key fan-out control-plane query: bounded by the tenantId prefix, the non-terminal ingestion states and the freshness SLA cutoff.
sourceDocumentIngestions
partition-key-fan-out
SELECT * FROM c WHERE c.tenantId = @tenantId AND c.type = 'SourceDocument' AND c.ingestionStatus IN ('pending','parsing','embedding','failed') AND c.lastIngestedAtUtc < @freshnessSlaUtc ORDER BY c.lastIngestedAtUtc ASC