Python scripts for CSV cleanup

1) Treat script cleanups as pipelines

A clean pipeline should have an explicit order. The most stable order is parse, normalize schema, type-check, then write. Keep this explicit in code and documentation.

• Read with explicit encoding and delimiter strategy.
• Drop malformed rows to a dead-letter file if your process requires strict schema.
• Normalize null-like values and whitespace consistently in one pass.

2) Use idempotent transforms

If a script run twice changes output each time, debugging becomes expensive. Idempotent operations make reruns safe and reduce confusion between script and UI steps.

• Log each action and file hash for every step.
• Avoid random ordering in dedupe operations.
• Preserve original values in a change log for audits.

3) Validate before and after

Script output should be checked with measurable rules: row counts, null density, duplicate rate, and schema drift. Use spot checks in a browser tool for contextual validation (header names, sample rows, edge cases).

For recurring imports, include an explicit step for how to remove null rows from csv in the scripted workflow.

4) Blend with local-first QA

Use Python for large transformations and bulk workloads, then pass a sample through local cleanup UI before publication. It is faster and catches presentation issues (like delimiters and quoting) that automated scripts may overlook.