Digest Data

To digest data is to turn a large, detailed dataset into a smaller, meaningful summary you can act on: parse the raw input, keep the fields that matter, aggregate them, and reduce the result to a few numbers or a compact table. It is what a log file, a sales export or a database dump goes through before it becomes a report.

The work usually follows five steps - parse, extract, aggregate, summarize, reduce. You can stop at any of them, depending on how small the answer needs to be.

1. Parse - turn raw data into fields

Raw data is rarely ready to count. Break it into records and columns first:

• CSV / TSV: split on the delimiter, honour quotes and escaped characters (RFC 4180).
• Log files: match each line with a pattern, e.g. a timestamp, an IP and a status code. In Python: re.match(r'(\S+) (\S+) (\d{3})', line).
• JSON / JSONL: one object per line; read the keys you need (or use jq).

If the source is a database, parsing is free - the engine already has typed columns.

2. Extract - keep only what matters

Dropping columns and rows early is the cheapest way to make everything else faster. Select the handful of columns you will group or measure, and filter to the period or segment you care about, e.g. SELECT category, amount FROM sales WHERE sold_at >= '2026-01-01';.

3. Aggregate - group and roll up

Aggregation turns many rows into a few. Group by one or more keys and compute counts and totals:

• SQL: SELECT category, COUNT(*) n, SUM(amount) total FROM sales GROUP BY category ORDER BY total DESC;
• Python / pandas: df.groupby('category').agg(n=('amount','size'), total=('amount','sum'))
• Pivot: put one key on rows and another on columns to compare groups side by side (pivot_table, a crosstab, or a pivot in Excel).

4. Summarize - statistics and top-N

A summary condenses further into descriptive statistics: count, average, median, minimum, maximum and percentiles. In pandas, df.describe() prints all of them at once; in SQL, MIN(), MAX(), AVG() and window functions give the same. Top-N lists (ORDER BY ... LIMIT 10) are often the most useful summary of all - which 10 products, IPs or customers dominate.

5. Reduce - dedupe, sample, split

When the dataset itself is the deliverable, shrink it physically:

• Deduplicate: SELECT DISTINCT, or sort -u on a text file.
• Sample: keep every Nth row to prototype on a large set.
• Split and compress: break a multi-GB file into manageable parts and gzip them - BigTextFileSplitter splits huge CSV/TXT by row count or size without loading the whole file into memory.

Do it with Withdata

• DataFileConverter - convert and reshape CSV, Excel, JSON, XML and SQL files.
• BigTextFileSplitter - cut a huge file into smaller, digestible ones.
• File to DB - load a CSV or log into a database so you can aggregate it with SQL.
• DB to File - export the aggregated result back to CSV or Excel.
• Free online converter - quick CSV, JSON and SQL conversions in the browser.

FAQ

What does it mean to digest data?
Turning a large dataset into a smaller, meaningful summary: parse, keep the fields that matter, aggregate and reduce.

How do I summarize a CSV file?
Load it into a spreadsheet, pandas (df.groupby(...).agg(...), df.describe()) or SQL, then use GROUP BY with COUNT, SUM, AVG, MIN and MAX.

How do I reduce a very large file?
Filter rows and columns first, remove duplicates, then split into smaller parts and compress them.