Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
A few years back companies used innovation and digital transformation mostly to differentiate themselves and to stay competitive. The drastic growth in digital and cloud computing over the last couple of years have changed this mindset.
Today, organizations have to be innovative and leverage the latest technologies just to sustain and stay in business. Enterprises that implement online retail business, online banking and several other online services aren’t considering those channels simply as another route to increase their revenue—they have realized that online services are fast becoming their primary revenue channel. According to Forrester’s Report, “The Digital Business Imperative,” data analyzed just for three months showed that 84 percent of U.S. banking customers had used online banking for their transactions and 43 percent of the customers had used a mobile phone for these activities.
Organizations are working at a very fast pace reviewing and analyzing their processes to seize opportunities for digital transformation. It is important to understand that to realize digital transformation, companies may need to completely re-engineer their current processes to make use of technologies such as the internet of things (IoT), big data analytics and artificial intelligence instead of doing patchwork on existing processes to adapt to digital technologies. It is also important for senior IT executives to consider digital initiatives in tandem with their cloud strategy instead of treating them in isolation. There is absolutely no doubt that the scale, speed and massive infrastructure requirements such as compute power, storage and other associated platforms to support these technologies are best provided by the cloud. The question to ask is, “Are we going to develop and host the infrastructure we need to support digital, or are we going to leverage the constantly improving robust infrastructure and services that cloud providers are offering?”
Organizations that plan for massive growth and transformation are usually the ones that invest considerably in technology to support innovative ideas. Advancements in the digital space help fuel innovation, but without a solid cloud strategy accompanied by agile development processes, ideas are likely to remain simply as ideas on paper and will take forever to get materialized into products or solutions that provide value. In today’s context, digital and cloud are almost inseparable due to the robust infrastructure, services and tools available on the cloud to support digital initiatives. All leading cloud providers offer competing solutions and services to help organizations move forward with their digital initiatives at a rapid pace.
In my previous article, “Transforming with the Cloud: If Not Now, When?” I touched upon some of the key areas that organizations should be considering with regards to transforming their businesses with the cloud. I have attempted to structure this article in a similar fashion, with the focus being on the role played by cloud to support some of the key digital technologies such as artificial intelligence, big data and analytics and IoT. The idea is to provide a very high-level overview of the digital landscape and how the leading cloud providers are helping enterprises with their digital initiatives.
Artificial Intelligence
The digital revolution, which centered around mass production of computers and communication devices, has changed the way businesses have operated over the last several decades, resulting in constant improvements ranging from generating new ideas for products and services to innovative product designs to improving the customer experience. At present, the world is going through another, possibly even stronger wave of revolution. It is the use of artificial intelligence to perform complex cognitive tasks to solve business problems in ways that previously were either highly complicated or extremely resource-intensive. Most organizations deal with business propositions that yield small to medium value and require a high volume of human work, such as reviewing large numbers of documents such as RFPs to understand requirements and estimate costs versus returns to determine whether to proceed with those proposals. AI is proving to be the best alternative to handle cases such as these that are currently being dealt by human beings, but the volume of work involved challenges businesses on the feasibility of continuing those without AI.
AI systems try to mimic the human brain, which uses patterns to generate perceptions, and logic to drive the structured approach of analyzing a situation from a rational perspective. AI systems process large volumes of data that could come in from various sources such as sensors, online applications, textual data from social media and the like. AI processes the data using perception to analyze patterns and incorporates machine learning to utilize structured evaluation methods and rational decision-making, not only to extract pieces of meaningful information, but also to assemble such pieces of information to make valuable decisions.
Cloud plays a significant role in enhancing the power of applications that incorporate AI. Almost all major players in the cloud business have developed AI services that use powerful cognitive engines to process structured data such as relational databases and unstructured data from NoSQL databases, sensors and other technologies that get uploaded to the cloud. The robust pattern-matching algorithms and the powerful logic component built into these cognitive engines are highly sophisticated. Data and compute power are the two most critical requirements to make these engines effective. The engines predict more accurately with larger datasets. AI applications such as image recognition, video analytics, natural language processing and speech recognition leverage machine learning using highly sophisticated neural networks that perform detection and prediction from large volumes of data. Parallel processing with the use of graphics processing units (GPUs) make these data processing and computations faster. Building and implementing such robust GPU-based parallel processing engines on-premises would be a very expensive and resource-intensive initiative. Cloud addresses this problem by providing APIs to access several machine learning services such as video analytics, speech recognition, process automation, vision detection and natural language processing. Behind these APIs are complex infrastructures that combine the power of clusters of GPU-based compute engines, neural networks and data lakes.
Amazon’s AWS Marketplace offers machine learning and AI software solutions from several major software vendors to help AWS customers increase the efficiency and productivity of their businesses. AWS’ artificial intelligence solutions offer several pretrained services that can be utilized by developers to add intelligence to their applications via simple API calls. Advanced deep learning functionalities are provided by Amazon Lex, which chatbots could incorporate for automatic speech recognition and natural language processing abilities. Similarly, AWS provides Amazon Polly to build applications that can convert text into speech delivered in natural sounding male or female voices. Amazon Rekognition, meanwhile, uses technology to analyze billions of images, so applications can be developed to search images, detect and compare faces, objects and scenes.
Google Cloud also offers competing products and services to support their customers’ AI initiatives. Google Cloud Natural Language provides powerful machine learning models that analyze text to provide meaningful insight such as customer sentiments about a product via conversations on social media. Google provides Cloud Speech-to-Text, a service that understands about 120 languages and converts audio to text. Similarly, Cloud Text-to-Speech can convert text to natural-sounding speech in 30 different voices and in several languages. Google also provides products such as Cloud AutoML, a suite of machine learning products, and Cloud Machine Learning (ML), a managed service that can be leveraged by developers and data scientists to build powerful machine learning models.
Microsoft Azure provides the infrastructure, tools and services to build smart applications with data that can live in the cloud, on-premises or on edge devices. Several products and tools are available via marketplace, and Azure provides sophisticated AI services to integrate the tools to develop intelligent business applications. Powerful chatbots could be developed by integrating Azure Bot Service, language understanding and external systems such as Office 365 and Dynamics CRM to streamline and automate common activities. Another example is an automated failure detection system for assembly lines built based on Azure’s image classification system using a convolutional neural network to help improve the identification of faulty electronic components. Azure also provides Visual Studio Code Tools to build, test and deploy AI solutions. The powerful AI Toolkit for IoT Edge from Azure enables IoT devices with AI that runs locally through pre-built deep learning models. Last but not least, Azure’s Machine Learning Studio helps develop and manage predictive-analytics solutions.
Big Data and Analytics
With the growth of internet, cloud and social media, there has also been an exponential growth of data across the world. According to statistics on big data generation in the last five years, the average volume of data created in the world every day is about 2.3 trillion Gigabytes. When the nature of the data was more structured and organized, companies relied on data warehouses and BI applications to help make important data-driven business decisions. Traditional data warehouses were built based on relational databases that could be queried using SQL; the data could be extracted, transformed and loaded from one or more data sources via ETL jobs that ran up to several times a day. This overall approach has proved to be ineffective when it comes to handling and modifying continuous streams of real-time data flowing in from multiple dissimilar sources such as social media, IoT, the public web and relational databases. Big data analytics, which helps examine large structured and unstructured data sets, has become a major enabler for enterprises to make critical business decisions by providing insight and knowledge with data mining, predictive analytics and forecasting. As big data processing evolved, the concept of data lakes came into existence. A data lake is a centralized repository that allows the storing of structured and unstructured data as is and enables the use of a variety of tools and approaches to answer business questions.
The ability of the cloud to scale vertically and horizontally makes it the ideal platform for big data hosting and analytics. With vertical scaling, it is possible to increase the capacity of a server by adding resources as needed by the applications. Horizontal scaling allows expanding hardware as the processing requirements increase. Hadoop, which leads the big data revolution, is designed as a distributed system so that it can take advantage of scaling. Parallel processing is an important part of its design so that the system can process several independent small tasks such as serving data stores and file systems, processing streaming data and handling queries in tandem. Cloud-based systems offer high bandwidth, enormous amounts of memory and scalable processing power to help big data applications with improved real-time processing and analysis of streaming data. Cloud undoubtedly is an obvious choice for applications running large workloads and storing enormous volumes of data. Cloud providers offer highly scalable database services coupled with tools and services to support information-management, business intelligence and analytics.
All major cloud service providers offer competitive and comprehensive tools and approaches to process big data at the volume and velocity that your business demands, while at the same time being conscious to reduce the associated operational costs. To perform faster and efficient transfer of petabytes of data to the cloud, AWS provides solutions that employ different techniques such as network multiplexing, reduced storage footprint via data deduplication and compression and advanced data replication. AWS offers services including Amazon Elastic Compute Cloud (EC2) and Simple Storage Service (S3) to provide the elasticity needed for your data lake along with your choice of database products, ETL and reporting tools via AMS Marketplace solutions. Similarly, with Microsoft Azure HDInsight, solutions can be developed using open-source frameworks such as Hadoop, Spark and Hive to analyze streaming or historical data. To process large volumes of data, HDInsight offers a broad range of cost-effective solutions for ETL and data warehousing. Azure HDInsight offers specific cluster types such as Apache Hadoop, Apache Spark, R Server and Apache HBase, with cluster customization capabilities such as adding components, utilities and languages.
Internet of Things
IoT is the concept of connecting devices such as security sensors, surveillance cameras, smartphones, smart watches and other wearables and even household appliances such as washing machines, refrigerators and several others that have the ability to communicate and transfer data over the network without direct human interaction. This disruption in technology has not only empowered individuals and families with greater control of their household systems and appliances, but enabled organizations with data that provide greater insight into critical areas of business interest, opening the doors for innovative products, solutions and new business opportunities.
The rate of adoption of IoT by businesses is growing exponentially as the number of connected devices continues to increase. As per Gartner’s recent study, businesses could already be using as many as 3.1 billion IoT devices today, and by 2020 this number is likely to rise to about 7.6 billion. According to an article in Forbes, IoT will have an impact in all industries, from manufacturing to logistics to health care. We are not too far from the days where nearly everything in the world will be connected. IoT is at present playing a dominant role in optimizing production, managing supply chains, tracking assets, making financial decisions and improving the customer experience. Digital transformation of the fitness industry has been remarkable. Wearable devices with built-in sensors can constantly collect data on physical activities such as distance travelled, calories burnt and sleep patterns to provide detailed analyses and insight for continuous health care. IoT applications are designed to use data from connected devices, and the sophisticated tools available in the cloud let you visualize, explore and build complex analytics. In complex IoT applications that use several devices, it is important to understand the state of the devices and to frequently communicate to the application components that leverage those devices. It is also essential to ensure secure identity and access between devices and applications.
As with any revolution in technology, IoT also has its associated challenges. With more devices being connected to the internet, the volume of data generated is also immense, which in turn puts significant pressure on the internet. So there obviously is a need to support infrastructure that can transmit and store this data more efficiently. Due to the vast number of connected devices, and with that number constantly increasing, there is a push toward making these “edge” devices intelligent to perform some degree of processing and then send the results to the servers rather than sending massive amounts of data to central servers to perform all the processing. For example, surveillance cameras in the past would send all the videos to the central recording device, which in turn would record only when it detects motion. Consider the scenario where hundreds of high-definition surveillance cameras are sending video feeds constantly to the central server—the impact this could have on the network is unimaginable. It should also be noted that as the physical distance between the devices and the server increases, network transmission latency increases. With edge computing, the video cameras are smart enough to sense motion and they send videos to the cloud-based central recording system only whenever they detect motion. This drastically reduces the volume of data that is transmitted over the network and makes it a more efficient system. Cloud and IoT complement each other—connected devices generate huge amounts of data and cloud provides the infrastructure to store, process and analyze the data. It is also interesting to note that IoT is an important data source for big data analytics.
Cloud IoT Core service, built by Google Cloud Platform (GCP) partners, offers connectivity to devices by providing the necessary hardware, software and solutions. These services include secure connection of devices (via identity and authentication) over industry standard protocols including MQTT and HTTP. Cloud IoT Core also allows remote controlling of devices from the cloud by maintaining a logical configuration of the devices. Since this IoT service is serverless, there is no need for upfront software installation and the horizontal scaling of the GCP can instantly be leveraged when required. A host of tools such as Dataflow, BigQuery, Bigtable and Data Studio are available to help with data analytics and insights.
Microsoft provides services for IoT via the Azure IoT Hub. Features offered by the Azure IoT Hub include ensuring bi-directional communication with billions of IoT devices, establishing device-specific access control, secure and scalable provisioning of devices and integration between cloud and edge to distribute intelligence across devices. Azure IoT Edge provides features for optimized performance of the edge and cloud such as ensuring the lowest latency between edge devices and the cloud. IoT Edge helps control IoT costs by reducing the volume of data transferred from the edge devices to the cloud. It leverages services such as Azure Stream Analytics to process data at the edge and transmit only the bare minimum data that is needed by the cloud for further analysis.
Amazon provides a fleet of services such as AWS IoT Core, AWS IoT Device Management, AWS IoT Analytics and several others to support IoT initiatives. For easy deployment, programming and management, Amazon offers the FreeRTOS lightweight operating system for small edge devices. The AWS Greengrass software further enhances the power of edge devices by offering local compute, secure messaging and data caching. AWS IoT Analytics, a fully managed service, provides all the capabilities required to run complex analytics on IoT data to obtain insights and to build machine learning use cases.
Taking Everything Into Account
Technology is certainly a major driver in any business today and companies that struggle to integrate technology effectively will eventually find it difficult to sustain or do well in their marketplace. Connected devices, social media and massive volumes of several forms of structured and unstructured data feeds have paved the way to further leverage technology and transform businesses. This “digital transformation,” which includes use of artificial intelligence, big data analytics and IoT, is fast becoming a key requirement for organizations to be innovative and competitive. Digital has opened doors for better analytics and decision-making, leading enterprises to explore, analyze and obtain new insights and ideas on growing their businesses.
With cloud reaching a high level of maturity with regards to services, tools and security, it has become the ideal platform for digital technologies. All leading cloud providers have comprehensive and competitive solutions, tools and associated services to address even the most complex digital transformation initiatives for their customers. Furthermore, the horizontal and vertical scaling of infrastructure offered by the cloud makes it highly suitable for the compute requirements demanded by digital technologies.
Escaping from or ignoring the digital revolution will be disastrous to organizations as well as individuals. Humans and businesses are generating trillions of gigabytes of data every day. Whether we have realized it or not, digital is transforming our lives. It is important for organizations to get on board soon and devise a digital strategy that works together with their cloud initiatives and strategy.
