Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
DataOps seeks to eliminate existing barriers between people, technology, tools and data
It’s no secret that COVID-19 has put the economy under enormous strain and future economic prospects are uncertain. A smart way to give your organization a leg up against competitors is DataOps. Adopting a DataOps culture makes the best use of your data—it helps to optimize the data powering your business because it successfully monetizes and systematizes your approach to working with data. It will help to get your new products to market faster and also accelerates business agility.
For example, a DataOps approach can help a data scientist who has spent weeks developing a new machine learning model that is complete but not ready to go into production. She will need to set up data stream monitoring and other basic requirements before she can put it in place.
This type of challenge is routinely faced by DevOps teams and the best practices they implement help avoid potential setbacks to quickly put their hard work and innovation to use. Applying the same development practices to the deployment of data-intensive services and applications significantly speeds up project completion times.
Why DataOps?
DataOps puts the principles and powerful DevOps practices such as automation, continuous delivery and quick feedback cycles to work in a data environment. Its strategic focus is to bring people closer to data, by delivering proven tools that deliver data to people with big-picture business knowledge. DataOps also removes complicated engineering and IT processes that stand between data and those with business knowledge to give projects an important and meaningful business context that greatly improves outcomes.
The goals of a DataOps approach are to eliminate the friction that’s holding business stakeholders, developers and data scientists back and to increase project productivity and business outcome success. Automating and streamlining processes help people to quickly and easily move from ideas to development to production without lag time.
DataOps also helps take concepts into production by fitting data projects into familiar, standard CI/CD development workflows and plugging them into any existing framework to deploy applications.
How to Put DataOps to Work With GitOps
GitOps is becoming a core component of DataOps practices and puts them into action. It allows data engineers to define data processing applications and pipelines as configuration and manage them through Git and mature CI/CD pipelines. This brings with it higher levels of standardization, security, governance and, as a result, an accelerated rate of delivery. Adopting GitOps in the data management world can pose problems for some organizations working with powerful and fast-changing technologies such as Apache Kafka and Kubernetes, but the automated processes it brings turbo-charge delivery time.
Cloud migration is another way in which a GitOps approach makes a difference for organizations that want to take advantage of managed services in Microsoft Azure, AWS or Managed Streaming for Apache Kafka (MSK). GitOps helps simplify migration of Kafka clusters used for streaming to the cloud service of an organization’s choice—simply define the desired state, step back and GitOps takes over.
GitOps’ pull-based approach to deployment, in which an operator watches Git and applies the desired state, creates several benefits. One benefit is not needing to poke holes in production environments to provide push-based actions from software such as Jenkins, which has numerous common vulnerabilities and exposures (CVEs). With a GitOps pull-based approach, attack surfaces are reduced, enabling solution migration across technology and clouds.
A Better Process Results in Better and Faster Outcome
Real-time business is here to stay and there is huge pressure on businesses to change fast. As the saying goes, “You snooze, you lose.” With the right tools and strategy backed by a good communication plan, DataOps will work to your advantage, bringing the benefits of a DevOps culture.
Faster time to market via increased productivity is one advantage of a DataOps approach. Putting people closer to data, together with the right tools, helps you empower business users and developers so they can add new features and capabilities to any deliverable, and deploy and monitor it quickly.
When working at a faster clip, don’t overlook the importance of governance and security. Build governance into the DataOps process to ensure deployment of the correct applications in the right way. And, maintain business continuity without disruption to existing infrastructure and processes.
DataOps seeks to eliminate existing barriers between people, technology, tools and data. By bringing business and technology imperatives closer together, you gain invaluable context and a better ability to focus on driving strategic business outcomes that matter.
