Chunking & Vectorization

After content is ingested, Dataworkz processes it through a chunking and vectorization pipeline before it can be searched. This step transforms raw text into a set of vector embeddings stored in the vector database — the form the retrieval engine actually searches.

Chunking

Chunking divides documents into smaller segments before embedding. Chunk quality directly affects retrieval quality: chunks that are too large dilute relevance; chunks that are too small lose surrounding context.

Dataworkz supports the following chunking strategies, configurable in the Chunk & Vectorize step of the RAG wizard:

Strategy	How it works	Best for
Fixed size	Splits at a set character or token count	Uniform, predictable content
Paragraph / sentence	Splits at natural language boundaries	Prose documents, articles
Semantic	Splits at topic shifts detected by the model	Heterogeneous or long-form documents
Custom delimiter	Splits on a character or pattern you define	Structured formats (CSV, logs)

Key parameters:

• Chunk size — target length of each chunk (in tokens or characters)

• Chunk overlap — number of tokens shared between adjacent chunks, preserving context at boundaries

Vectorization

Each chunk is passed through the configured embedding model (see Embedding Models) to produce a vector. The vector and its source chunk text are stored together in the vector database, along with any metadata extracted during ingestion (file name, URL, document date, custom fields).

Dataworkz handles the full chunking and vectorization pipeline automatically — there is no manual step required between ingestion and retrieval.

Vector Storage

Vectors are stored in the vector database configured for the workspace:

• Default — MongoDB (provided for all accounts)

• Paid tiers — Pinecone, OpenSearch, Weaviate, and other supported vector databases

The vector database is optimized for sub-second similarity search across millions of chunks. When a query arrives, the retrieval engine finds the stored vectors closest to the query vector and returns the corresponding chunks — see Retrieval.