DataOps

A collaborative data management practice focused on improving the communication, integration and automation of data flows between data professionals and data consumers.

Implementation

• Establish progress and performance measurements at every stage of the data flow. Where possible, benchmark data-flow cycle times.

• Define rules for an abstracted semantic layer. Ensure everyone is “speaking the same language” and agrees upon what the data (and metadata) is and is not.

• Validate with the “eyeball test”: Include continuous-improvement -oriented human feedback loops. Consumers must be able to trust the data, and that can only come with incremental validation.

• Automate as many stages of the data flow as possible including BI, data science, and analytics.

• Using benchmarked performance information, identify bottlenecks and then optimize for them.

• Establish governance discipline, with a particular focus on two-way data control, data ownership, transparency, and comprehensive data lineage tracking through the entire workflow.

• Design process for growth and extensibility. The data flow model must be designed to accommodate volume and variety of data. Ensure enabling technologies are priced affordably to scale with that enterprise data growth.”