Качество данных: практики и инструменты

Data quality определяет trustworthiness всей downstream analytics. Bad data → wrong decisions.

Dimensions качества

Accuracy: соответствие reality. Completeness: no missing values где нельзя. Consistency: одинаковые значения в разных системах. Timeliness: свежесть. Uniqueness: no unintended duplicates. Validity: соответствие правилам (email format). Все измеряемы.

Testing

Schema tests: типы, nullability. Value tests: ranges, uniqueness, referential integrity. Distribution tests: aggregate statistics vs baseline. Freshness tests: data updated recently. Volume tests: количество rows в норме. dbt tests, Great Expectations, Soda.

Data contracts

Formal agreement между producer и consumer. Schema versioning. SLA на data quality metrics. Breaking changes с notice. Testing до deploy производителем. Tools: Buz, custom implementations на Apache Avro/Protobuf schemas.

Operational

Circuit breakers: pause pipelines при data quality issues. Alerting: quality metrics в SIEM. Data catalog для discoverability. Ownership: каждый dataset имеет владельца. Regular audits. Investment: 20% engineering time на data quality — industry benchmark.