24X7 Site Monitoring

image

We are an open consumer internet product and customers access our website/mobile site/Android & iOS Apps round the clock 365 days an year. To make sure we take care of customers, we have the responsibility of making sure no new code change affects the site in ways it is not intended and under various load states based on traffic.

This necessitates sites like us to have a continuous monitoring system from an end user perspective. As an organization we have invested in some 3rd party services that takes care of this need. But we felt the checks weren't sufficient or satisfying some of the needs.

Hence we built our own capability of monitoring core functions of our site for all platforms. Our Automation testing suite was the backbone for this implementation along with Jenkins.

This write-up gives you an overview of how we went about the objective. Hope our journey benefits you.

We started by evaluating the different tools that are available. The below table sums up our understanding:

Tool Evaluation:

Jenkins Travis CI Teamcity GitLab CI
Open source Paid tool Free for open source projects only Free for trial period of 30 days, then paid
300+ Plugins No specific Plugins Less Plugins No Plugins
Can be hosted in any Platform Can be hosted only on GitHub Can be hosted only on Git and BitBucket Can be hosted only on GitLab SCM
Easy to upgrade Upgrading is easy but its a manual process Upgrading is long and manual process Need Backup to upgrade
Programming language skills not needed Programming language skills not needed Complete Java skills need Programming language skills not needede

So based on the comparison Jenkins fit our bill as:
a. We can configure eclipse with Jenkins within the workspace itself
b. We can run multiple builds dynamically in Jenkins where as we cannot do it with other CI tools.
c. Where as TeamCity is not open source and Travis does not have jobs as in Jenkins.
d. We can use 'Jobs' to customize alerts for each respective module.
e. Can run multiple builds in parallel.
f. Provides wide range of plugins. 23+ plugins.
g. Supports all OS that we wanted: Windows, Linux, Mac OS X, and Unix operating system.

There are few limitations though but not big show stoppers:
a. Not port flexible as it runs in particular host only. eg: localhost:8080.
b. Merging the scheduled jobs in not available.
c. Configuration is bit tough for beginners in jenkins.
d. Error messages / details are not very clear about what went wrong with the failure.
e. Frequent login failures.

Jenkins Setup Process
Download Jenkins

Download Jenkins from it’s official website. By default, the latest release and the Long-Term support release will be available for download. The past releases are also available for download. Click the Long-Term Support Release tab in the download section.

Click the link “Older but stable version” to download the Jenkins war file.

Starting Jenkins
Open the command prompt. From the command prompt, browse to the directory where the jenkins.war file is present. Run the following command D:\>Java –jar Jenkins.war

After the command is run, various tasks will run, one of which is the extraction of the war file which is done by an embedded web server called winstone.
D:\>Java –jar Jenkins.war
Running from: D:\jenkins.war
Webroot: $user.home/ .jenkins
Sep 29, 2015 4:10:46 PM winstone.Logger logInternal
INFO: Beginning extraction from war file

Once the processing is complete without major errors, the following line will come in the output of the command prompt. INFO: Jenkins is fully up and running

Accessing Jenkins
Once Jenkins is up and running, one can access Jenkins from the link − http://localhost:8080
This link will bring up the Jenkins dashboard.

Unlocking Jenkins

When you first access a new Jenkins instance, you are asked to unlock it using an automatically-generated password.

1. Browse to http://localhost:8080 (or whichever port you configured for Jenkins when installing it) and wait until the Unlock Jenkins page appears.
2. On the Unlock Jenkins page, paste this password into the Administrator password field and click Continue.

Customizing Jenkins with plugins

After unlocking Jenkins, the Customize Jenkins page appears. Here you can install any number of useful plugins as part of your initial setup.

Click one of the two options shown:

Install suggested plugins - to install the recommended set of plugins, which are based on most common use cases.
Select plugins to install - to choose which set of plugins to initially install. When you first access the plugin selection page, the suggested plugins are selected by default.

If you are not sure what plugins you need, choose Install suggested plugins. You can install (or remove) additional Jenkins plugins at a later point in time via the Manage Jenkins > Manage Plugins page in Jenkins.
The setup wizard shows the progression of Jenkins being configured and your chosen set of Jenkins plugins being installed. This process may take a few minutes.

Creating the first Admin User
Finally, after customizing Jenkins with plugins, Jenkins asks you to create your first administrator user.
1. When the Create First Admin User page appears, specify the details for your administrator user in the respective fields and click Save and Finish.
2. When the Jenkins is ready, click Start using Jenkins.
3. Notes:
This page may indicate Jenkins is almost ready! instead and if so, click Restart.
If the page does not automatically refresh after a minute, use your web browser to refresh the page manually.
4. If required, log in to Jenkins with the credentials of the user you just created and you are ready to start using Jenkins!

Manage Jenkins
Once the installation and administrator creation completed, follow the below steps,

1. Login with your Jenkins account.
2. Click on “Manage Jenkins’.
3. Now select ‘Global Tool Configuration’.
4. Find ‘JDK’ and click on ‘JDK Installation’, add the path of Java and save it.
5. Go back to “Manage Jenkins’ and select ‘Configure System’.
6. Find “Extended E-mail Notification” and fill the necessary fields as mentioned below,
A. Enter “smtp.gmail.com” in SMTP server field.
B. Now click on ‘Advanced’ option and check “Use SMTP Authentication”.
C. Enter the username, password
D. Check on “Use SSL” and enter the “SMTP” port as 465.
E. Also enter the “Default Recipients” & “Reply To List” (eg:yyyyyy@ymail.com,zzzzzzzz@zmail.com ) and save it.
7. Repeat the above mentioned steps for “Extended E-mail Notification”, exclude point number “E”.

Now we will see how to create a project and execute it.

Step 1 − Go to the Jenkins dashboard and Click on New Item
Step 2 − In the next screen, enter the Item name, in this case we have named it ‘Helloworld’. Choose the ‘Freestyle project option’
Step 3 − In General, → Advanced → check 'Use custom workspace' and enter path of your workspace.(eg: C:\Users\Admin\workspace\Jenkins)
Step 4 − In Build, Select "Executable Window batch command" from drop down "Add build step" and give your batch file(eg: run.bat)
Step 5 − Post-build Actions, Select "Editable Email Notification" from drop down "Add post-build action".
Step 6 − Project Recipient List(xxxxxxxxxx@xmail.com), Attachments(Reports/Login_6.0.html,)
Step 7 − Click on Advanced setting, In Triggers, need to select the success and failure cases, both should be mentioned as recipient list.
Step 8 − Save it, once entered all the necessary details.

How to generate batch file for reporting:
Step 1 − In Eclipse, Open your Project and right click from SRC folder → Create new file → enter name and save it with .bat(eg: run.bat).
Step 2 − Open the bat file and enter the following comment along with your testng.xml file,
Comment: java -cp bin;lib/* org.testng.TestNG testng.xml

Reports:
1. From the drop down menu, change the value of Default Content Type to HTML (text/html).
2. Update the value of Default Subject to [$BUILD_STATUS] - $PROJECT_NAME - Build # $BUILD_NUMBER ($BUILD_ID).
We prefer to have the build status first (like a tag).
3. In the box named Default Content enter the following block of data:
4. $PROJECT_NAME - Build # $BUILD_NUMBER - $BUILD_STATUS.

Check console output to view full results.
If you cannot connect to the build server, check the attached logs.

Following is the last 100 lines of the log.
--LOG-BEGIN--

${BUILD_LOG, maxLines=100, escapeHtml=true}

--LOG-END--

The email you will receive after a successful execution will be similar to the one below:
Subject: [Successful] - banana - Build # 77 (77)
Body:
Press the Apply button to save the changes.
You will notice that in the email we mention the following:
If you cannot connect to the build server, check the attached logs.
Select your project from the main screen and then click on Configure on the left column.
Scroll down to Post-build Actions section.
From the Add post-build action drop down list select Editable Email Notification.
Set the value of Attach Build Log drop down to Compress and Attach Build Log and then hit the Apply button.

System Requirements

JDK JDK 1.5 or above
Memory 4GB
Disk Space No minimum requirement. Note that since all builds will be stored on the Jenkins machines, it has to be ensured that sufficient disk space is available for build storage.
Operating System Version Windows, Ubuntu/Debian, Red Hat/Fedora/CentOS, Mac OS X, openSUSE, FReeBSD, OpenBSD, Gentoo.
Java Container The WAR file can be run in any container that supports Servlet 2.4/JSP 2.0 or later.(An example is Tomcat 5).

Now we have a schedule mechanism for our monitoring.

What we achieved
Continuous monitoring for our Login, Registration and Payment pages in Android App, PWA and Desktop Alerting the teams via Email, Call and SMS when the job fails in Jenkins Realtime failure reasons and data can be collected

We integrated ‘Pager Duty’ plugin to get different types of alerts like Call, SMS and Email when the application gets Success / Failure.

The entire setup runs continuously one platform at a time 24X7. This mechanism gives us a good hold on checking the site availability and we can immediately in times of need.

Thanks for reading this far. We will continue to publish our work on how we setup pager duty, appium studio setup, protactor in the near future.