Technology & Software

Continuous Blob Knowledge Ingestion and Retrieval

Designed for AI platform engineer and data engineer

Continuously detect new or changed Azure Blob Storage objects, parse and chunk them, generate embeddings, persist chunks and vectors in Azure Cosmos DB, and let platform engineers compare vector, full-text, hybrid and semantic-reranked retrieval over the same corpus while honoring source ACLs, versions and ingestion status.

Contract TEC-04 · primary aggregate SourceDocumentIngestion · mode loading

Decision brief

Engineer promotion of a retrieval configuration: evidence ready for review

This brief starts with the decision, evidence, and accountable owner. The guided workflow then shows how the recommendation was produced.

Decision signals

The evidence and control points that shape this recommendation.

Decision owner

AI platform engineer or data engineer

The named role remains accountable for the consequential decision.

Governed evidence

6 record types

Structured records and cited text ground the recommendation.

Workflow

5 controlled stages

Each transition records its input, output, status, and audit context.

Human controls

1 approval gate

Approval gates prevent the copilot from making consequential decisions.

Business problem

A 94-page compliance handbook is replaced in the governed-docs container, producing a new blob version.

Decision to make

Continuously detect new or changed Azure Blob Storage objects, parse and chunk them, generate embeddings, persist chunks and vectors in Azure Cosmos DB, and let platform engineers compare vector, full-text, hybrid and semantic-reranked retrieval over the same corpus while honoring source ACLs, versions and ingestion status.

Outcome to show

promotedRetrievalConfig gives AI platform engineer or data engineer the evidence needed to begin the final governed business outcome, with the decision trail retained for review.

  1. 1Blob change detection
  2. 2Parse, chunk and embed
  3. 3Version cutover and stale chunk retirement
  4. 4Retrieval mode comparison
  5. 5Engineer promotion of a retrieval configuration

Step 1 of 5

Ready: Start with the business question, then use the single button above.

The output appears here immediately. More technical evidence remains available below.

System logic, GenAI, and human accountability

The demonstration keeps deterministic controls, AI assistance, and approval separate.

System logic

Source systems enforce identity, workflow order, policy filters, and record updates.

GenAI contribution

Retrieval and generation assemble relevant evidence into a cited recommendation or draft.

Human accountability

AI platform engineer or data engineer verifies material claims and approves the consequential action.

Workflow stages

Run the stages in order before using the Copilot. Every stage explains what happens, why it matters, the evidence it uses, and the complete API request and response.

Seeded record
Loading
Document type
Loading
Container
Loading
Partition key
Loading
Current stage
Loading

API demonstration guidance

Run a stage to see its explanation, evidence, full request, and full response.

Grounded chat

Ask a question and receive an answer grounded in retrieved SourceDocumentIngestion evidence.

Answer and evidence

Select a starter question or enter your own.

Evidence

Continuous ingestion console

Chunk, vectorize, and store a document through the same engine that Event Grid and Azure Data Factory invoke in Azure mode.

Retrieval mode comparison

Domain implementation workspace

Loaded live from /api/config, /api/domain/model, and /api/domain/sample.

Domain objective
Continuously detect new or changed Azure Blob Storage objects, parse and chunk them, generate embeddings, persist chunks and vectors in Azure Cosmos DB, and let platform engineers compare vector, full-text, hybrid and semantic-reranked retrieval over the same corpus while honoring source ACLs, versions and ingestion status.
Primary aggregate
SourceDocumentIngestion
Primary container
sourceDocumentIngestions
Hierarchical key
/tenantId + /sourceDocumentId
Schema version
1

Sample scenario

A 94-page compliance handbook is replaced in the governed-docs container, producing a new blob version. Event Grid fires, the ingestion function parses it with Document Intelligence, produces 418 chunks, embeds them with the text-embedding-3-large deployment, bulk-writes them under the new blob version id, and flips the current-version flag in one step so no query sees a mixed corpus. An engineer then asks a retention-period question through all four modes: pure vector returns the right section at rank 4, full-text at rank 1 because the query used the exact statutory phrase, hybrid at rank 1 with better supporting context, and semantic reranking matches hybrid at rank 1 while adding 380 milliseconds. The engineer promotes hybrid without reranking for this corpus and records the decision with the measured latency and request units.

Physical Cosmos DB containers

Every container provisioned for this accelerator with its hierarchical partition key.
Container Role Hierarchical partition key Document types
sourceDocumentIngestions Primary /tenantId + /sourceDocumentId SourceDocument, IngestionRun, ContentChunk, RetrievalExperiment, RetrievalResultSet, GroundedAnswer

Document types

Contract document types, their owning container, and their identifier pattern.
Type Container Partition key Identifier pattern Embedding fields
SourceDocument sourceDocumentIngestions /tenantId + /sourceDocumentId source::{sourceDocumentId} None
IngestionRun sourceDocumentIngestions /tenantId + /sourceDocumentId source::{sourceDocumentId}::run::{blobVersionId}::{runId} None
ContentChunk sourceDocumentIngestions /tenantId + /sourceDocumentId source::{sourceDocumentId}::chunk::{blobVersionId}::{chunkOrdinal} contentVector
RetrievalExperiment sourceDocumentIngestions /tenantId + /sourceDocumentId source::{sourceDocumentId}::experiment::{experimentId} queryVector
RetrievalResultSet sourceDocumentIngestions /tenantId + /sourceDocumentId source::{sourceDocumentId}::experiment::{experimentId}::result::{retrievalMode} None
GroundedAnswer sourceDocumentIngestions /tenantId + /sourceDocumentId source::{sourceDocumentId}::answer::{answerId} None

Query patterns