The history of data in the enterprise

- Before we talk about how to become a data engineer, let's talk about how we got here. You've probably heard phrases like the world has been flooded with data in the past. But let's be concrete about this. By 2025, global data company Statista projects that global data creation will grow to 180 zettabytes. For context, a zettabyte is a thousand exabytes, an exabyte is a thousand petabytes, a petabyte is a thousand terabytes, and a terabyte is a thousand gigabytes. That's a lot of data. This proliferation of data has provided a rich resource for businesses looking to gain insights, optimize operations, and improve customer experiences. However, it can be cumbersome and difficult to consume this data effectively. Seeing this companies deployed their software engineers against this problem. Let's be clear, though, the idea of engineers focused on data is not a new concept. Enterprises have long relied on data to gain insights into their operations, optimized processes, and drive decision making and software engineers were at the forefront of this innovation. The challenge came when these engineers realized the unique aspects of dealing with modern data, both from a scale and volume perspective. They quickly established best practices for handling this data and distributed these best practices. These engineers became the first data engineers. Despite their work, though, many organizations are only beginning their data journey. A recent survey conducted by NewVantage Partners found that 97% of participating organizations are investing in data initiatives, but just 19.3% indicate that they've established a data focused culture. That means there's still a lot of work to be done using data efficiently and effectively. The adoption of modern data platforms, such as data lakes and data cloud warehouses, has made it easier to analyze, manage, and access large amounts of data across the organization. This has led to an increase in data literacy across different business functions with data being used to drive decisions in areas such as finance, marketing, and operations. This increase in data volume has also led to a new iteration of enterprise business intelligence processes. Decisions are now backed by concrete data that has been analyzed through BI tools and data analysts. That data in turn has been sourced from systems, cleaned, loaded, and mined for value. The modern enterprise is heavily reliant on data to drive informed decision making. As this trend continues, the volume and types of data sourced will continue to increase. Enterprises must ensure that their workforce has the right skillset to thrive in this environment and this represents your opportunity as a data engineer.