Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
What can data scientists learn from DevOps? A lot, actually. By applying DevOps principles to the world of big data, you get DataOps, which streamlines data analytics.
If you’re reading this site, you’re probably already familiar with DevOps. The DevOps philosophy emphasizes continuous collaboration between software developers (the “dev” people) and system admins (the operations, or “ops,” folks)—as well as everyone in between, such as software testers.
It’s possible to apply similar ideas to the world of big data and data analytics. There, as in the world of software development and delivery, different teams traditionally have worked in isolation from one another. Those teams include storage admins, who keep data storage systems running; data analysts, who write and execute programs for interpreting large volumes of data; data quality specialists, who ensure data consistency; and data security teams, which protect data from attackers.
As DevOps professionals know, keeping different teams isolated from one another is not efficient. It limits collaboration and maximizes confusion when problems occur.
The data world, therefore, has much to gain by ending this isolation between teams and adopting a DataOps workflow. This will keep data operations flowing smoothly and provide continuous insight into the data that organizations collect.
DataOps is not a brand new concept. People like Andy Palmer have been writing about it for a little while now. But the term and the idea have yet to gain nearly as wide a following as DevOps. (There is not yet even a Wikipedia entry for DataOps—and you know an idea is still in its infancy when no one has taken the time to write about it on Wikipedia.)
Making DevOps Bigger
Meanwhile, from the DevOps perspective, the growth of interest in DataOps is significant because it exemplifies the broad applicability of DevOps principles. So far, the DevOps conversation has focused on the relatively narrow context of software development and delivery. But it can extend much further, too. DataOps is a start.
So, if you’re a data analyst, storage admin, data security professional or someone who works with data in any other capacity, it will be worth your while to borrow some ideas from your DevOps brethren. And if you work in software development, testing or delivery, it may be worth your time to educate your data friends on DevOps principles. It will make their workflows more efficient, while introducing DevOps to a wider audience.
