Many business organisations use Big Data generated from various sources such as transaction systems, media, and streaming data from the Internet of Things.
In this post, I introduce the lifecycle management of the Big Data process at a high level and with simplified language gleaning from methods I used in my data solutions. The key roles in this process are data architects, technical data specialists, data analysts, and data scientists.
Big Data architects and specialists start solutions by understanding the lifecycle. They engage in all phases of the lifecycle. The roles and responsibilities may differ in different stages. However, they need to be on top of the life cycle management end to end.
Based on my experience, I introduce 12 distinct phases in the overall data lifecycle management, which can also apply to Big Data. I combined some relevant activities in a single phase to make it concise and easily understandable.
These phases may be implemented under different names in various data solution teams. There is no universal systematic approach to the Big Data lifecycle as the field is still evolving. For guiding purposes, I propose the following distinct phases in this area:
- Phase 1: Foundations
- Phase 2: Acquirement
- Phase 3: Preparation
- Phase 4: Input and Access
- Phase 5: Processing
- Phase 6: Output and Interpretation
- Phase 7: Storage
- Phase 8: Integration
- Phase 9: Analytics
- Phase 10: Consumption
- Phase 11: Retention, Backup, and Archival
- Phase 12: Destruction
A Simplified Guide To Big Data Lifecycle Management
Dr Mehmet Yildiz 