Hadoop WebApp Automation

 Abstract :

Today is an era of Technology and with the increase of technology the amount of data it produces increases every second even now as we are talking. The rate at which the data is growing is not square or cube or logarithmic but exponential. This arises a new problem of Big-Data. The definition of big data is data that contains greater variety, arriving in increasing volumes and with more velocity. Put simply, big data is larger, more complex data sets, especially from new data sources. These data sets are so voluminous that traditional data processing software just can’t manage them and moreover today all that matters is the speed with which the data is accessed as no one can afford a huge amount of latency. This project offers our users a web portal from where they can manage their Big-Data problems in the most efficient manner. Whereas the security of the user is concerned we only need their IP address so that we can establish a remote connection to solve their issues. It not only provides a platform for the users concerned with Big-Data but even to those who are new to the Linux world/community with some commands and their syntax.

Purpose of this Project :

• This project is a Database management app, which will manage your database in the most efficient manner and will organize it in a way easier to access. 
• Managing Data-Bases can be a tedious task as one can’t determine the exact size of stripping the files into smaller sizes so that they can be stored more efficiently or the number of copies to be generated so it remains safe in the future even if any node fails. 
• In this project we have also considered the newbies that come to the Linux platform and want to master it but still face difficulties with the right commands to use, So they all can use our second option of the Linux Commands and can master their Linux commands on the terminal provided to them.

Problem Definition :

This project offers our users a web portal from where they can manage their Big-Data problems in the most efficient manner. Whereas the security of the user is concerned we only need their IP address so that we can establish a remote connection to solve their issues. It not only provides a platform for the user’s concerned with Big-Data but even to those who are new to the Linux world/community with some commands and their syntax

Implemented Screenshot of The Project :

 

    

                                        Landing Page of our WebApp

Starting Data node services

Starting Name node services

 

Basic Linux Commands that were Implemented

Conclusion

Conclusion:

                    

                    In nutshell, managing our Big-Data can be a very complex and tedious task. So, we have provided a Web-App that will manage our Big-Data, using Hadoop. Now configuring Hadoop is a lengthy and complex process, so we have automated it using Ansible. Furthermore, we have also provided some Linux commands so that the user can configure their Linux machines ad can use them as their management systems. Our system accepts the input as IP-address and number of salve nodes and creates a cluster for the user.

Object Recognition using CNN model

 Task Description 💬

📌 In this task :
👉Create a model that will detect a car in a live stream or video and recognize characters on the number plate of the car 
👉Secondly, it will use the characters and fetch the owner's information using its own API.
👉Create a Web portal where all this information will be displayed (using html,CSS, and js)

Step 1: Create an ML model detecting vehicle number plate

Step 2: Display the screen showing the WebApp taking the input car number from the user

Step 3: Display the final output showing the details of the car

Thank you very much for your valuable time reading this article...

Kubernetes Integration with Python-CGI

 What is Kubernetes?

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.

The name Kubernetes originates from Greek, meaning helmsman or pilot. K8s as an abbreviation results from counting the eight letters between the "K" and the "s". Google open-sourced the Kubernetes project in 2014. Kubernetes combines over 15 years of Goggle's experience running production workloads at scale with best-of-breed ideas and practices from the community.

What is Python?

Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. Python's simple, easy-to-learn syntax emphasizes readability and therefore reduces the cost of program maintenance. Python supports modules and packages, which encourages program modularity and code reuse.

What is CGI?

The common gateway interface (CGI) is a standard way for a Web server to pass a Web user’s request to an application program and to receive data back to forward to the user or in laymen terms it allows to use a web page such as HTML in front-end and programming language like python, java in the back-end part.

What is Python CGI?

Python CGI stands for Common Gateway Interface. An HTTP server invokes a Python CGI script so it can process user input that a user may submit through an HTML <FORM> or <ISINDEX> element. Such a script usually lives in the server's special CGI-bin directory. Python CGI module handles situations and helps debug scripts.

How to run a CGI program in Python?

Name your script with a .cgi file extension ( example.cgi ).

Configure Apache to recognize and allow the .py file extension as a CGI script.

<< Provide Your own YAML file. Get access to all commands.

• $kubectl get nodes
• $kubectl get deployments
• $kubectl get pods
• $kubectl get ns

<< Commands in Web UI

Codes of CGI files

Thank you for your valuable time reading my article...

Use Case of JavaScript

 Task 7.2 :

>>Write a blog explaining the ‘use case’ of ‘JavaScript’ in any of your favorite industries.

According to the reports, There were meaningful differences across industries in how and why people use JavaScript.

• tech: 45%

• finance: 7%

• advertising and marketing: 5%

• education: 5%

• entertainment: 5%

• business support and logistics: 4%

• healthcare: 4%

• retail: 3%

• government: 2%

• manufacturing: 2%

JavaScript developers look to React Native framework for this type of development.
As an open-source framework, React Native can be used for building cross-platform native apps. When used in combination with JavaScript, React Native applications are indistinguishable from apps built using other languages such as Swift, Java, and Objective-C.

What is JavaScript used for?

JavaScript is mainly used for web-based applications and web browsers. But JavaScript is also used beyond the Web in software, servers, and embedded hardware controls. Here are some basic things JavaScript is used for:

1. Web Applications

As day-by-day there is a continuous improvement in the browsers, so JavaScript gained popularity for making robust web applications. We can understand it by taking the example of Google Maps. In Maps user just requires to click and drag the mouse; the details are visible just by a click. There is a use of JavaScript behind these concepts.

2. Web Development

JavaScript is commonly used for creating web pages. It allows us to add dynamic behavior to the webpage and add special effects to the webpage. On websites, it is mainly used for validation purposes. JavaScript helps us to execute complex actions and also enables the interaction of websites with visitors. Using JavaScript, it is also possible to load the content in a document without reloading the webpage.

3. Mobile Applications

Now a day’s mobile devices are broadly used for accessing the internet. Using JavaScript, we can also build an application for non-web contexts. The features and uses of JavaScript make it a powerful tool for creating mobile applications. React Native is the widely used JavaScript framework for creating mobile applications. Using React Native, we can build mobile applications for different operating systems. We do not require writing different codes for the iOS and Android operating systems. We only need to write it once and run it on different platforms.

4. Game

JavaScript is also used for creating games. It has various libraries and frameworks for creating a game. The game can either be 2D or 3D. Some JavaScript game engines such as PhysicsJS, Pixi.js help us to create a web game. We can also use the WebGL (web graphics library), which is the JavaScript API to render 2D and 3D images on browsers.

5. Presentations

JavaScript also helps us to create presentations as a website. The libraries, such as RevealJs, and BespokeJs, can be used to create a web-based slide deck. They are easier to use, so we can easily make something amazing in a short time.

The Reveal.js is used to create interactive and beautiful slide decks with the help of HTML. These presentations work great with mobile devices and tablets. It also supports all of the CSS color formats. The BespokeJS includes animated bullet lists, responsive scaling, and a wide variety of features.

6. Server Applications

A large number of web applications have a server-side to them. JavaScript is used to generate content and handle HTTP requests. JavaScript can also run on servers through Node.js. The Node.js provides an environment containing the necessary tools required for JavaScript to run on servers.

7. Web Servers

A web server can be created by using Node.js. Node.js is event-driven and not waits for the response of the previous call. The servers created using Node.js are fast and don’t use buffering and transfer chunks of data. The HTTP module can be used to create the server by using the createServer() method. This method executes when someone tries to access port 8080. As a response, the HTTP server should display HTML and should be included in the HTTP header.

</> LinkedIn :

LinkedIn relies on NodeJS for its mobile site. A few years back, LinkedIn used Rails for its mobile site. As with other large Rails applications, it was slow, monolithic, and it scaled poorly.

LinkedIn switched over to NodeJS to solve its scaling problems. Node’s asynchronous capabilities allowed the LinkedIn mobile site to perform more quickly than before while using fewer resources. Node also made data sharing and building APIs easier for LinkedIn developers.
After switching to node.js the Linkedin development team realized the advantages.

1. Much better performance and lower memory overhead than other tested options, running up to 20x faster in some scenarios

1. Programmers could leverage their JavaScript skills.

1. Frontend and backend mobile teams could be combined into a single unit.

1. Servers were cut to 3 from 30. Enough headroom remains to handle 10x current levels of resource utilization.

1. The development could focus more on application development than firefighting.

JavaScript is Everywhere!

Python CGI with Docker (Task 7)

 In this project, I have integrated python with Docker !!

Python is one of the most popular languages nowadays and Docker is used for containerization.

What is python CGI?

A CGI script is invoked by an HTTP server, usually to process user input submitted through an HTML <FORM> or <ISINDEX> element.

Most often, CGI scripts live in the server’s special cgi-bin directory. The HTTP server places all sorts of information about the request (such as the client’s hostname, the requested URL, the query string, and lots of other goodies) in the script’s shell environment, executes the script, and sends the script’s output back to the client.

For now info visit :

https://docs.python.org/3/library/cgi.html

Whats is docker?

Docker is a set of platform as a service (PaaS) products that uses OS-level virtualization to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries, and configuration files; they can communicate with each other through well-defined channels. All containers are run by a single operating system kernel and therefore use fewer resources than virtual machines.

Now starting with my project,

Step 1:For this firstly I installed docker engine on my red hat 8 using command

# yum install docker -ce --nobest

Step 2: Then I started docker services using

#systemctl start docker

Step 3: Then I made one directory for my project using #mkdir doc_project and in this directory, I made my one file try.py which will have my code related to python CGI which is made using html, CSS, and python 3.

Step 4: In the same directory I will make one more file named Dockerfile which is used to make my own docker image and the configuration inside this file will be like shown in the figure below :

In this we have installed one apache server and python 3 . Then copied my try.py file into the root directory of apache server for cgi related fields root directory is /var/www/cgi-bin/ and for other purposes root directory of apache will be /var/www/html/.

So after I copied this file I have given executable permissions to this file using chmod command

And then as I normal os, we can start httpd services by using #systemctl start httpd but this will not work inside Docker so for starting httpd services i have used the last command.

Step 5:Then I will build my image using this Dockerfile and the command used for this will be

# docker build -t any_name

For e.g., :

Step 6: After making this image i will run this using command :

#docker run -i -d -t -p any_free_port:port_of_httpd --name any_name image_name

For e.g.,:

Step 7: Now our container is ready to work on. Now I will put my IP address of base os with the port I have given on any browser which is connected to same network

Face Recognizer

Hello Fellas!!!

This post a task given by Mr. Vimal Daga sir during my summer internship program.  

Task Description 📄

❄️ Create a program that perform below mentioned task upon recognizing a particular face.

📌 When it recognize your face then -
👉 It send mail to your mail id by writing this is face of your_name.
👉 Second it send whatsapp message to your friend, it can be anything.

📌 When it recognize second face, it can be your friend or family members face.
👉 Create EC2 instance in the AWS using CLI.
👉 Create 5GB EBS volume and attach it to the instance.

Create Face Dataset :

First we need to create face dataset. The data set will contain 100 image sample of a face.

Importing Face Dataset:

Now we need to import the face dataset that we have created.

Email Function:

Here we will define a function for sending email. The email will contain the information of the face owner that is recognized by our module. 

WhatsApp Message Function:

Now we will define a function that will automatically send a message to a number containing the details of the face owner.

AWS EC2 Launch Function:

Here we will create a function to create a EC2 instance and attach a 5GB EBS volume to it. 

Final Code:

Now we will create a program that will run the above created functions. This is the final step.

So finally, we have successfully created a python program that detects our face using opencv and automatically sends a WhatsApp message, an email and launches a AWS EC2 instance and attach a 5GB EBS volume to the instance.

Thank you for your valuable time reading this article. Do comment as your review matters a lot... Until then see you next time.

ARTH TASK - 10

💥 Task Description →

👉 Write an Ansible PlayBook that does the
following operations in the managed nodes:

• Configure Docker

• Start and enable Docker services

• Pull the httpd server image from the Docker Hub

• Run the docker container and expose it to the public

• Copy the html code in /var/www/html directory and start the web server

    Firstly, I had installed Ansible on the top of Rhel 8 OS. Considering

that you know the installation of ansible I am directly jumping to the

use-case part that we have to perform.

    So, you can see the inventory file which is named as ip.txt, in this file

I had provided the target node IP so that we can connect to the

target node we have with us. Also, you can see the configuration file

of ansible in which I had given the path of the inventory file I had and

some other default values. The location of configuration file is inside

the /etc/ansible directory.

    You can take the help of ansible-doc file to see the syntax and

keywords used for the particular things you want to configure and

then you can apply those syntax accordingly in your yml file.

Refer to the figure given below :--

    On the left side, we have our Managed Node on which we have to

make a directory and then create a .yml format file in which we have

to write the ansible code. On the right side, we have the Target Node

on which we have to apply all the features that has been asked.

Command which is used to compile or run the yaml file is

#ansible-playbook -v <name of yml file>

Where, v is used for the verbose.

    Coming on to the task part we have to firstly install the Docker

software in our operating system.

To install the docker module we need to configure yum first, as you

can see in the code below that I had automated the yum

configuration under yum_repository.

    

    Then I had used the command module to install the docker-ce in

Target Node.

    In this part I had configured the docker services in the Target Node

with the help of ansible playbook.

    On the left side of the picture you can see the code that I had used in

docker.yml file. This code has been written with the help of ansible-

doc file.

    In this part I had installed the Python SDK of Docker and also I had

pulled a httpd image from the DockerHub.

    In this image as you can see I had created a workspace or the

directory named as wsp and then in next part I had created the

web.html file given the input of the content inside it.

    In this image as you can see I had setup the httpd webserver for docker container, exposed the ports and given the volumes. On the left side you side the compilation of yml file.

    Now in the Target Node by the inspect command you can see the IP

of the docker webserver IP

    Going on to the firefox just copy the webserver IP and then the

webserver file name.

    You will get the desired output or the content that you had given in

the web.html file.

Thank you learners for your precious time reading this post. Hope you like the integration of Ansible and Docker.