Upcoming Big Data Database Management Practices DBAs Need to be Prepared for

We have to start with painting the bigger picture first. In another article a couple of years ago, we have pointed out that more than half of the big data workloads may be running on cloud. As per the latest statistics available, this scenario is already running out as the surveys show more than about 45% of the respondents run at least a part of their big data workloads on the cloud.

This cloud impact on database administration is now also redefining all basic structural assumptions about architecting and managing data. At on-premise databases, data management was primarily all about sizing it right enough to be optimally utilized, but haven’t had more capacity to trigger any audits. For big data, the need was to bring computational abilities to data as the network overhead of moving all the extended volume of data, which was particularly rational to consider.

Entering cloud had revolutionized data management capabilities in terms of commodity infrastructure, faster networks, cheapening storage, and most importantly limitless scalability. In the case of database vendors, it was the other side of the drawing board with the ability to separate computer from storage and handle both effectively.

Adding more combustive fuel to the fire, the best value from cloud-based database deployment is achieved through the latest concept of DBaaS (Database-as-a-Service), in which all the upgrades, patches, backups, provisioning, and failovers all handled by the cloud-based database service providers itself, but not by an internal DBA.

Machine learning enabling self-driven databases

Another major breakthrough in cloud-based big data administration was the application of ML (machine learning) to make the DBaaS self-running. Over a year ago, Oracle set the doors open for this approach with the Autonomous Data Warehouse 18c, following which the Autonomous Transaction Database 18c was also released after six months. However, Oracle now only offers autonomous database in the public cloud, where it directly control the environment, not through DBAs.

Applying machine learning capabilities to the database is an ideal approach to effective database management for many reasons. At the first point, database operations of big data tend to generate huge volumes of log data. Database operations in managed cloud data services also resist any drifts and finally, ML helps automate legworks like configuring databases for various load patterns, optimization of queries, etc., which ultimately adds value to overall database administration.

There is no surprise that the advancement in autonomous databases management created some fear to the DBAs in terms of their job security. One of the major topics of discussion as we can see online is about the question of Autonomous database management vs. DBA. However, RemoteDBA.com experts point out the fact that unless the employers who deployed DBAs are so stupid, their jobs are secured. DBAs will remain as an inevitable part of strategic decision making when it comes to tasks from choosing a database to designing schema, defining the data sources, setting database policies, as well as ensuring the database security. So, DBA remains essential in end-to-end administration of data.

For the rest of 2019, we can surely expect that more will be coming up from the cloud-based database providers with Oracle already leading the suite. Deploying machine learning to run databases may turn out to be a standard in all DBaaS offerings, and we can also expect a few providers are trying to differentiate some of the key concepts of the new Oracle-approach to the database and apply those to their on-premise database deployments for more efficacy.

Serverless computing becoming another standard

We may also expect serverless computing become another fast-growing standard in big data administration. Introduced with AWS Lambda, the serverless concept is aimed to simplify the app development approach by eliminating any need for servers for scaling. As the idea seems to be bright and solving many existing issues, it may also become widespread in DBaaS cloud services.

In such a scenario, the database administrators may specify the lower and upper thresholds for the databases, which can auto-scale according to the need of the situation. Some existing examples of this include the DynamoDB by Amazon, in which a serverless approach is the core of the design. In Amazon Aurora, serverless got introduced recently as an add-on for those applications in which spikes are hard to predict. At the latest, Google Cloud Firestore also comes serverless, and MongoDB too introduced Stitch serverless for Atlas cloud services.

However, as of now, serverless is not ideal for all types of use cases. Say, for example, if your load is steady and predictable, then it is always ideal to reserve capacity to be more economical and manageable. However, the increasing demand from app developers may make serverless a desirable choice in cloud-based database operations in 2019 and beyond.

Distributed databases

Another big thing which is actualized with cloud-based big data management is distributed databases. In 2019 itself, we may see that the distributed databases making writes on par with the reads. The concept of distributed databases didn't start with the cloud. We can see some early examples also like Clustrix which was then acquired by MariaDB, NuoDB, Aerospike along with the new NoSQL databases like Couchbase MongoDB, and Cassandra, etc.

Lately, we can see that MongoDB has become big break among the NoSQL databases offering distributed database approach, largely accounting for its developer-friendly approach. However, on the other hand, the cloud puts forth some unfair advantages also for the distributed databases. Looking from this point of view, cloud eliminated the need for setting up data centres and secondly much of the data like logs, IoT data, product catalogues, etc., already lived in the cloud. Most importantly, the cloud also adds some adverse architectural advantages as the providers of the cloud also may natively engineer an automated replication, automated scaling, and smart storage to their platforms.

Big data means big responsibility

To conclude, these are some of the expected changes in cloud-based big data administration sector in the future, which the database administrators need to be prepared. There is no doubt that big data is enabling a big leap to businesses in terms of analytics and data-driven insights, but handling this needed to be well-informed and well-administered to reap the best possible results out of it.

Tell us your story

Would you like to write for nichemarket just like Trudy has? Find out how to submit a guest post and when you're ready, you can contact us.

Are you looking to promote your business?

South African businesses and freelancers can create your free business listing on nichemarket. The more information you provide about your business, the easier it will be for your customers to find you online. 

Registering with nichemarket is easy; all you will need to do is head over to our sign up form and follow the instructions. If you require a more detailed guide on how to create your profile or your listing, then we highly recommend you check out the following articles.

• How to create a nichemarketer profile
• How to create a nichemarket business listing

Recommended reading

If you enjoyed this post and have time to spare why not check out these related posts and dive deeper down the rabbit hole that is data analysis.

• Google Analytics Launches Sample Dataset for BigQuery
• Why people analytics is the number 1 game changer in Tech
• How To Set Up Data Blending With Google Data Studios
• The Definitive Guide To All Things GDPR

Tags: guest post , big data