Conventionally, a central data team manages the data infrastructure for organizations. This team would maintain the data warehouse, own data transformation, and ensure compliance with the regulatory requirements. There are problems with this approach.
There’s too much data for a single team to own. All data related issues get routed to this central team. This single skeleton team becomes the “go to team†if the right prices don’t ring at the Point of Sales (POS) or data issues impact the predictions made by Data Scientists.
Data Engineers may be working with data all the time, but they are not expected to understand the business insights. For example, a metric number_ of_ passengers may mean a number of paying customers or number of souls (including flight crew & pets) onboard depending on the business function that’s interested in the insights.
There’s a better approach to overcome this bottleneck. Shared collaboration, more participation from cross-functional teams, and a democratized approach to data access. The change in the approach and outcomes may include some of the following.
1) A central data team empowers analysts to participate in the data development process instead of owning all aspects of data management.
2) Anyone (other than data engineers) who knows SQL should be allowed to contribute to the data transformation process.
3) Compliance can be achieved through precisely sufficient documentation, data testing, and version control.
4) In the shared collaboration approach, the data quality testing gets easier and more comprehensive.
5) With full transparency on data lineage and changes to the analytics code base, the need to maintain Data Warehouses becomes minimal and more efficient.
JetBlue has been able to turn around the state of data management to be more trustworthy by taking a shared collaborative approach and investing in emerging tools.
Rebuilding a new data pipelining infrastructure by scraping the legacy system is much easier said than done. The impediments may include major pushback from stakeholders and can be overcome by “Eat the elephant one bite at a time†approaches such as Piloting, Proof of Concept or Minimum Viable Product (MVP) or One-business-domain- at-a-time.