# Unstructured and multimodal data Data that is not rows and columns, such as documents, images, audio and video, stored in the lake alongside tables and made usable by extracting text, structure and metadata from it. > [!NOTE] > > **Before reading** > > [Object storage and the S3 API]() ยท [Catalog and lineage]() ## Why it matters Most of an organisation's information is unstructured: contracts, manuals, emails, support chats, call recordings, product photos. Generative AI made this content directly useful, for search, question answering and extraction. Without the same management as tables, it becomes a second, ungoverned data lake: files nobody can find, with personal data nobody tracks. ## How it works The files themselves live in object storage, like any other data. What turns a folder of files into a dataset is a file catalog table: one row per file, with its path, type, size, checksum, source, owner, timestamps, classification and access rules. The table is an ordinary lakehouse table, queryable, versioned and governed through the catalog. Searching for "all manuals for the X200, in German, updated this year" is a query on it. Processing pipelines extract usable content from the files, and write the results as more tables. - Parsing pulls text and structure from documents: headings, paragraphs, tables, page numbers. PDFs are the hardest case, because their layout does not describe their structure. - OCR, optical character recognition, reads text from scanned pages and images. - Transcription turns audio and video into text with timestamps and speakers. - Vision and multimodal models describe images, read labels, or classify content. - Extraction turns text into fields, such as invoice numbers and amounts from invoices. Each step records which model or tool version produced it, so outputs can be traced and rebuilt when a better parser arrives. Chunks of the extracted text then feed embedding and retrieval. ``` flowchart LR F[(Files in object storage:
PDFs, images, audio)] --> MT[(File catalog table:
path, type, owner,
classification)] MT --> P[Processing:
parse, OCR,
transcribe, describe] P --> X[(Extracted tables:
text, pages, fields,
with tool version)] X --> CH[Chunks and
embeddings] X --> AN[Analytics and
extraction uses] ``` File formats are evolving for this. Parquet suits tables of small values, but is inefficient for large binary objects and for random access to individual rows, which training and retrieval need. Formats such as Lance store large objects, such as images, together with their metadata and embeddings, with fast random access and versioning; some teams use them as a multimodal lakehouse format alongside Iceberg tables. Others keep files in object storage and only references and extracted data in tables. Governance applies as for tables. Files are classified, often by scanning their extracted text for personal data, since a scanned ID card or a recorded call is personal data however it is stored. Access rules apply to the files, their extracted text and their chunks alike. Retention and erasure must reach every derived copy: the file, its text, its chunks and its embeddings. ## Key terms - Unstructured data Content without a fixed tabular structure, such as documents, images and audio. - Multimodal data Data combining several types, such as images with text. - File catalog table A table with one row per file, describing and governing it. - Parsing Extracting text and structure from documents. - OCR Reading text from images of documents. - Transcription Converting speech to text. - Lance An open columnar format designed for multimodal data with fast random access. ## Example The shop's product content: 40,000 product photos and 6,000 PDF manuals in several languages. - Files sit under s3://lake/content/; the table shop.content\_files has one row per file with product\_id, kind (photo or manual), language, updated\_at, checksum and audience. - A nightly job parses new or changed manuals into shop.manual\_pages (text per page, with section headings) and records the parser version. - A vision model describes each photo; the descriptions feed product search and accessibility text. - When a manual is withdrawn, its row is marked deleted; the pipeline removes its pages, chunks and embeddings. ## Pitfalls - Files without a file catalog table. Nobody can find, govern or delete them reliably. - Ignoring personal data in files. Scans, recordings and attachments are often the most sensitive data a company holds. - Extraction outputs without tool versions. They cannot be compared or rebuilt when tools improve. - Deleting the file but not its derivatives. Text, chunks and embeddings outlive the source. - One parser for everything. Tables, forms and scanned pages need different tools; measure extraction quality. ## Learn more - [Lance documentation]() - [Gradient Flow, Multimodal lakehouses]() ## Related [Embeddings, vector search and RAG]() [PII and data classification]() [Privacy lifecycle]() [Training data management]()