Hadoop WebApp Automation

 Abstract :

Today is an era of Technology and with the increase of technology the amount of data it produces increases every second even now as we are talking. The rate at which the data is growing is not square or cube or logarithmic but exponential. This arises a new problem of Big-Data. The definition of big data is data that contains greater variety, arriving in increasing volumes and with more velocity. Put simply, big data is larger, more complex data sets, especially from new data sources. These data sets are so voluminous that traditional data processing software just can’t manage them and moreover today all that matters is the speed with which the data is accessed as no one can afford a huge amount of latency. This project offers our users a web portal from where they can manage their Big-Data problems in the most efficient manner. Whereas the security of the user is concerned we only need their IP address so that we can establish a remote connection to solve their issues. It not only provides a platform for the users concerned with Big-Data but even to those who are new to the Linux world/community with some commands and their syntax.

Purpose of this Project :

• This project is a Database management app, which will manage your database in the most efficient manner and will organize it in a way easier to access. 
• Managing Data-Bases can be a tedious task as one can’t determine the exact size of stripping the files into smaller sizes so that they can be stored more efficiently or the number of copies to be generated so it remains safe in the future even if any node fails. 
• In this project we have also considered the newbies that come to the Linux platform and want to master it but still face difficulties with the right commands to use, So they all can use our second option of the Linux Commands and can master their Linux commands on the terminal provided to them.

Problem Definition :

This project offers our users a web portal from where they can manage their Big-Data problems in the most efficient manner. Whereas the security of the user is concerned we only need their IP address so that we can establish a remote connection to solve their issues. It not only provides a platform for the user’s concerned with Big-Data but even to those who are new to the Linux world/community with some commands and their syntax

Implemented Screenshot of The Project :

 

    

                                        Landing Page of our WebApp

Starting Data node services

Starting Name node services

 

Basic Linux Commands that were Implemented

Conclusion

Conclusion:

                    

                    In nutshell, managing our Big-Data can be a very complex and tedious task. So, we have provided a Web-App that will manage our Big-Data, using Hadoop. Now configuring Hadoop is a lengthy and complex process, so we have automated it using Ansible. Furthermore, we have also provided some Linux commands so that the user can configure their Linux machines ad can use them as their management systems. Our system accepts the input as IP-address and number of salve nodes and creates a cluster for the user.