Topic 2 of 5
Data file formats
A lot of data doesn't live in a database but in files: CSV exports, JSON documents, images, or huge data lake tables. The chosen format decides who can read them, how much space they take, and how fast they're processed. The exam asks you to recognize each format and its use case.
Contents
File storage
Storing data in files is a core element of any computing system. Files can live on a computer's disk or a USB drive, but in most organizations important data is stored centrally in shared storage. Increasingly, that central location is in the cloud, which offers cost-effective, secure, and reliable storage for large volumes.
How a format is chosen
The file format depends on three factors:
Type of data
Structured, semi-structured, or unstructured.
Who uses it
The applications and services that need to read, write, and process the data.
Readable or optimized
Whether a person needs to read it, or whether space and processing speed matter more.
Delimited text (CSV)
Plain-text data with field delimiters and row terminators. The most common is CSV (comma-separated values): fields are separated by commas and each row ends with a new line. The first line can include the field names.
FirstName,LastName,Email
Joe,Jones,joe@litware.com
Samir,Nadoy,samir@northwind.com- Other variants: TSV (tab-separated), space-delimited, and fixed-width (each field takes a fixed number of characters).
- When to use it: structured data that many applications and services need to open, in a human-readable format.
JSON
JavaScript Object Notation uses a hierarchical document schema to define entities (objects) with multiple attributes. Each attribute can itself be an object or a collection of objects, so it works for both structured and semi-structured data.
{ ... }
Objects in braces
[ ... ]
Collections in square brackets
"name": value
Pairs separated by commas
{
"customers": [
{ "firstName": "Joe", "lastName": "Jones",
"contact": [ { "type": "email", "address": "joe@litware.com" } ] },
{ "firstName": "Samir", "lastName": "Nadoy",
"contact": [ { "type": "email", "address": "samir@northwind.com" } ] }
]
}XML
Extensible Markup Language is a human-readable format that was popular in the 1990s and 2000s. It's largely been superseded by the less verbose JSON, but some systems still use it. It defines elements and attributes with tags in angle brackets.
<Customers>
<Customer name="Joe" lastName="Jones">
<ContactDetails>
<Contact type="email" address="joe@litware.com"/>
</ContactDetails>
</Customer>
</Customers>BLOB (binary objects)
Ultimately, all files are stored as binary (ones and zeros). In human-readable formats, those bytes map to printable characters through an encoding like ASCII or Unicode. Other formats, especially for unstructured data, store data as raw binary that an application must interpret and render: images, video, audio, and application documents.
These files are called BLOBs (Binary Large Objects).
Optimized formats: Parquet and Avro
Human-readable formats are convenient, but they aren't optimized for space or processing. Over time, specialized formats emerged that enable compression, indexing, and efficient storage and processing.
Parquet · columnar
- An Apache project, widely used in data lakes and lakehouses.
- The file is organized in row groups; the data for each column is stored together.
- It includes metadata describing the rows in each chunk, so apps can locate data quickly and read only the columns they need.
- Efficient with nested data types and with compression and encoding schemes.
Avro · row-based
- Also an Apache project.
- Each file has a header in JSON that describes the structure.
- The data is stored as binary, in blocks of records; the app uses the header to parse it.
- Good for compressing data and minimizing storage and network bandwidth.
Columnar vs. row-based
An analytical query usually reads a few columns across millions of rows (for example, summing the amount of every sale). A columnar format like Parquet reads only those columns. A row-based format like Avro keeps each full record together, which fits when whole records are processed.
Delta Lake
Delta Lake is an open-source storage format that builds on Parquet by adding a transaction log. That enables:
ACID transactions
Reliable writes over files
Versioning
A history of data changes
Reliable updates
On files stored in a data lake
It's the default format in Azure Databricks and the standard format for lakehouse tables in Microsoft Fabric (D4).
Comparing formats
| Format | Readable | Data type | Typical use |
|---|---|---|---|
| CSV / TSV | Yes | Structured | Exchange between applications |
| JSON | Yes | Structured and semi-structured | APIs, flexible documents |
| XML | Yes | Semi-structured | Legacy systems |
| BLOB | No | Unstructured | Images, audio, video |
| Parquet | No | Structured (columnar) | Analytics in data lakes |
| Avro | No (JSON header) | Structured (row-based) | Compression and transport |
| Delta Lake | No | Tables on Parquet | Lakehouses with ACID |
Exam scenarios
| Situation | Format | Why |
|---|---|---|
| Export a table so many different apps can open it and a person can read it | CSV (delimited text) | Human-readable and compatible with almost everything. |
| Store customer profiles with fields that vary | JSON | Hierarchical, flexible schema. |
| Integrate with an older system that exchanges tagged data | XML | Legacy format, still in use. |
| Store photos, video, and audio | BLOB | Binary data with no structure. |
| Analyze a few columns of a huge table in a data lake | Parquet | Columnar: reads only the columns needed. |
| Minimize storage and bandwidth when moving whole records | Avro | Row-based, compact, with a JSON header. |
| Update data lake files with ACID transactions and versions | Delta Lake | Parquet plus a transaction log. |
Common exam traps
❓ Does Parquet store data by rows?
No. Parquet is columnar. Avro is the row-based format.
❓ What format is an Avro file header in?
JSON. The data itself is stored as binary.
❓ Is Delta Lake a separate format that replaces Parquet?
No. It builds on Parquet and adds a transaction log for ACID and versioning.
❓ Is XML the most widely used format for semi-structured data today?
No. The less verbose JSON has largely superseded it; XML remains in older systems.
❓ Which format is best for people and many apps to read tabular data?
Delimited text like CSV: human-readable and widely compatible.
❓ What does BLOB stand for?
Binary Large Object: binary data such as images, audio, or video.
Did you get this topic?
Test what you just learned
An analytics team queries only three columns of a table with billions of rows stored in a data lake. Which format reduces reading the most?
Ready to test "Data file formats"?
Original questions, each with a full explanation.
Start a mock exam