How to Monitor and Fix the Server Load

Featured

August 5, 2019 in Tips and Tricks

How to Monitor and Fix the Server Load

What is Server Load?

Website owners and users will be familiar with the computing term “Load”. In Unix computing, the system load is a measure of the amount of computational work that a computer system performs. The load average represents the average system load over a period of time. It conventionally appears in the form of three numbers which represent the system load during the last one-, five-, and fifteen-minute periods.

What are the causes of Server Load?

The server load can be caused by the excessive usage of the system resources. Excessive usage of the CPU or Memory or Disk Space can lead to the server load.

How to check the Server Load?

There are many ways to check the server load. The commonly used command to view the server load is ‘top‘ but ‘top‘ command displays the processes running dynamically. It needs resources a bit more when comparing to other methods. So normally, when a server is under high load, the usage of ‘top‘ command is not encouraged. We can view the server loads by using the following commands ‘w‘ and ‘uptime‘.

server load

        Viewing Server Load using ‘w’ and ‘uptime’ commands

server load

        Monitoring Server Load with the ‘top’ command

These are the ways to check the Server Load. As we can see, while using the top command, it will give a dynamic output which means the output screen will be updated within certain time intervals and so that we will be able to see the load and processes with the resource usage instantly.

In the output of the ‘top‘ command, on the line that says “Cpu(s)”, check the “%id” section which shows the percentage of which your CPUs are idle. The higher the number the better the situation. A 99% idle CPU is not doing much of anything, and a 1% idle CPU is heavily tasked. We can sort the output by processes that are currently consuming the most CPU by hitting ‘p’.

We can use the ‘top c‘ command also to view the current CPU usage.

Check the Memory Usage

The next step is checking the RAM. Run the following command to view the current memory usage.

server load

        Using ‘free -m’ command to monitor the Physical Memory usage

Check the Current disk I/O usage

We can check the current disk usage by running the following command.

# iostat -x 1 10

Please keep in mind that this will not work on OpenVZ/Virtuozzo containers.

This will print the disk usage statistics 10 times, every 1 seconds. Check the %util column.

Check the Process List

You can view the process list with the help of the following command.

# ps auxf

server load

This will help to identify which process is consuming more resources.

These are the methods used to identify the Server Load by monitoring the system resources. Once the Server Load is identified, we can troubleshoot it easily by taking appropriate actions.

If you need any further assistance please contact our support department.

August 5, 2019 in Tips and Tricks

Check the System Load on Linux

Uptime

First let’s check the load on your server using the uptime command.

$ uptime
15:16:45 up 41 days,  2:35,  2 users,  load average: 0.01, 3.01, 2.70

The example shows the output from uptime. When the command was run at 15:16:45, the server had been up for 41 days 2 hours and 35 minutes, there were two users logged on and the load averages were 0.01, 3.01 and 2.70.

The load average represents the work being done by the system. The three numbers show the load averages for the last minute, 5 minutes and 15 minutes, respectively. A load average of 1 reflects the full workload of a single processor on the system. A load of 2 on a system with two CPUs means that those CPUs were working at maximum. On a system with four CPUs, that 2 reflects a workload using about half of the available processing power.

Under normal circumstances you’ll want the load average to be 70% of your total CPU usage or less. You can check the number of CPUs available to your instance by running the following command:

grep processor /proc/cpuinfo | wc -l

If the load averages show that the load is increasing and is above the number of CPUs, you should look into the cause of the high load. A load above the number of CPUs means that the system is having to queue up processor requests, which reduces performance.

top

The top command displays real time information regarding the server’s resource usage. The first few lines will give you a summary of the resource utilisation on your system and you can sort the list of processes by CPU (P) or memory (M) use which allows you to quickly see where your server is receiving the biggest demands on its resources.

vmstat

The amount of memory a system has is one of the most common restraining factors. The swap is an area of the hard drive where data is moved to free up physical memory (RAM) for a process to use (not all servers have swap space configured). A system using its swap area does not necessarily mean it is low on memory, but if most of your system’s swap is being consumed it could indicate that your server is trying to do more than its available memory permits.

If swap space is configured and you suspect your server is running out of standard memory, you can use vmstat to show how much swapping is occurring.

The following example displays a system’s virtual memory statistics 10 times at 5 second intervals.

$ vmstat 5 10
procs -----------memory-------------  ---swap---- -----io---- --system--  -----cpu------
r  b    swpd    free   buff  cache       si    so    bi    bo   in    cs us sy  id wa st
2  3   138096   3904    140    584    14829 33632 16684 33633 1353 11562 64  9  16 11  0
7  4   156592   3800    132    492    53570 48618 54264 48622 2762 8148  20  5  50 24  0
2  7   258552   3040    128    668    37910 39822 39766 39826 2763 10861 43  8  23 26  0
7  4   261608   4312    116    776    41696 30854 42171 30855 2771 8631  26  6  33 35  0
3  4   259316   3824    112    576    44616 32316 44697 32320 2746 7087  15  4  52 28  0
0  5   257000   3376    116    880    25895 19847 26622 19848 1673 2877   1  2  68 29  0
4  4   263240   3384    124   1424    30018 21066 30497 21070 1732 4559   9  3  64 24  0
2 14   264656   5740    120   1812    27685 19030 28029 19031 2194 5345  11  3  51 35  0
6  3   338044   3764    132   1568    20087 31066 20470 31070 1798 9847  46  6  33 14  0
2  4   325904   4048    108   1172    13251 7190  13812  7190 1193 8838  38  5  24 32  0

In the example the free column shows that the amount of free memory is only around 4MB.

The si and so columns show the amount of data being transferred between the system’s swap and its memory. In this example you can see there is lots of data being moved. so is the amount of data being moved to the swap to free up memory. si is the amount of data being pulled from the swap back in to memory. This happens when the data is required for a process to run.

When a server is constantly swapping into and out of memory it is an indication that the load it is being placed under is too great for the resources it has available. top can be used to help identify the processes that are consuming the most resources.

Summary

If top and vmstat indicate that the server is using all its resources you need to look at optimising your current set up; this can include running any processing jobs outside peak hours, killing any processes no longer required and reconfiguring processes so they require less resources. You may also want to consider increasing the size of your server to better match your requirements.

July 20, 2019 in News, Tips and Tricks

Separate Registration of GST in each state of India, for each business.

India is a sovereign country with many powers for states. Unlike many other countries, GST in India is implemented as duel tax system. Means, two types of GSTs are implemented in India. As you know, there are IGST, CGST and SGST. Each taxable transaction of goods and services in India attracts duel GST – CGST and IGST or SGST. So the main reason for duel GST is to account the share of state governments. Ie, SGST (State Goods and Service Tax) and the share of IGST (Integrated Goods and Service Tax).

So, an account of GST Tax payers is to be accounted separately. At present government insists separate GST registration in each state, if a person operates his business. For example, if XYZ company has business transaction of goods or services in Maharashtra, West Bengal and Tamil Nadu, he has to take registration of GST in all the three states. Means a separate GST registration number has to be obtained in each state, although the Permanent Account Number, PAN is same.


As per sub section 1 of section 19 of GST Law explains that every person who is liable to take a Registration will have to get registered separately for each of the States where he has a business operation and is liable to pay GST

July 2, 2019 in Tips and Tricks

setup curl cron aws

Here I am going to explain the simple steps to write your own Cron Jobs on AWS EC2 Server.

a. First, you have to Log in to your AWS EC2 instance

b. Run the below command
    $ crontab -e

When you run Crontab command for the first time you will get some of the existed editors options to open

  1.   /bin/ed
  2.   /bin/nano
  3.   /usr/bin/vim.basic
  4.   /usr/bin/vim.tiny

Choose 1-4?

Select option 2 and enter.
Now you will get an editor to add Cron Jobs.

c. Add your every file paths/function paths which you want to schedule.
    Note that you should specify single file path per line

      For example:
         0 10 * * * php  /var/www/public_html/api/src/cronjob/notification.php
                                               OR
         0 10 * * * /bin/php  /var/www/public_html/api/src/cronjob/notification.php
Here I am mentioning about PHP file path which sends notifications to the users on every day 10:00am.

  • First term is Cron expression i.e 0 10 * * *

               Cron Expression Examples:
                * * * * *      // Run every minute. ex: for 5 mins => */5 * * * *
                0 * * * *      // Run on the hour, every hour ex: every 2hrs => * */2 * * *
                0 23 * * *     // Run at 11 p.m. every day
                0 0,12 * * 6   // Run at midnight and midday every Saturday
                30 2 */2 * *   // Run every second day at 2.30 a.m.

              5 stars are explained below:
                1st star:      Minute (ranges from 0-59)
                2nd star:     Hour  (ranges from 0-23)
                3rd star:      Day (ranges from 1-31)
                4th star:      Month(ranges from 1-12)
                5th star:      Day-of-week (0-7. 0 & 7 is Sun. 1-Mon, 2-Tue…etc)

  • Second term is PHP command path. I.e php or /bin/php
  • Third term is PHP file Path i.e /var/www/public_html/api/src/cronjob/notification.php

Command path will change depends on the programming language.

d. Once you enter your Cron Job Commands you have to save it.
   To save type Cntrl+x and Select ‘Y’ and Enter.

e. To check whether your Cron Jobs is saved or not, run the below command.
    $crontab -l

Now you can see your Cron Job file which has all your Cron Jobs.

For testing, you can schedule a task for a minute and check.

Send an email with the output every time the cron job runs. 

You can have cron send an email every time it runs a command which produces output. If you do not want an email to be sent for an individual cron job, you can redirect the command’s output to /dev/null. For example: mycommand >/dev/null 2>&1

June 25, 2019 in Laravel, Tips and Tricks

www to https redirect laravel | laravel htaccess redirect to https

 What about just using .htaccess file to achieve https redirect? This should be placed in project root (not in public folder). Your server needs to be configured to point at project root directory.
<IfModule mod_rewrite.c>  
RewriteEngine On
 # Force SSL    
RewriteCond %{HTTPS} !=on    
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]    
# Remove public folder form URL    
RewriteRule ^(.*)$ public/$1 [L] 
</IfModule>
I use this for laravel 5.4 (latest version as of writing this answer) but it should continue to work for feature versions even if laravel change or removes some functionality.

June 18, 2019 in MySql, Php, Tips and Tricks

Problem with phpMyAdmin and PHP 7.2: “Warning in ./libraries/sql.lib.php#613 count(): Parameter must be an array or an object that implements Countable”

It’s possible that when you installed phpMyAdmin, the version in the repository (phpMyAdmin v4.6.6) was not fully compatible with PHP 7.2. There is a newer version available on the official website (v4.8 as of writing), which fixes these compatibility issues with PHP 7.2. 

 sql.lib.php
This error is caused by a line of code in /usr/share/phpmyadmin/libraries/sql.lib.php.
If you don’t want to wait for the repositories to update with the latest version, it is strongly recommended that you manually upgrade to the latest version of phpMyAdmin yourself.
Alternatively, you can make a change to sql.lib.php to temporarily fix the error.
Firstly, backup sql.lib.php before editing.
sudo cp /usr/share/phpmyadmin/libraries/sql.lib.php /usr/share/phpmyadmin/libraries/sql.lib.php.bak
Edit sql.lib.php in nano.
sudo nano /usr/share/phpmyadmin/libraries/sql.lib.php
Press CTRL + W and search for (count($analyzed_sql_results['select_expr'] == 1)
Replace it with ((count($analyzed_sql_results['select_expr']) == 1)
Save file and exit. (Press CTRL + X, press Y and then press ENTER)
Import/Export issues
If you are also getting an error Warning in ./libraries/plugin_interface.lib.php#551 under import and export tabs:
Backup plugin_interface.lib.php
sudo cp /usr/share/phpmyadmin/libraries/plugin_interface.lib.php /usr/share/phpmyadmin/libraries/plugin_interface.lib.php.bak
Edit plugin_interface.lib.php
sudo nano /usr/share/phpmyadmin/libraries/plugin_interface.lib.php
Press CTRL + W and search for if (! is_null($options) && count($options) > 0) {
If not found, try search for if ($options != null && count($options) > 0)
Replace with if (! is_null($options) && count((array)$options) > 0) {
Save file and exit. (Press CTRL + X, press Y and then press ENTER)

June 17, 2019 in Tips and Tricks, Ubuntu

Create Swap space in ubuntu 18.04 aws

 Your problem is related to the probable fact of having only 1gb of RAM and not having memory swap, so let's create and enable it to take some time off at the time of composer execution.
Folow this steps:
1) Creating 1gb memory swapfile:
$ sudo fallocate -l 1G /swapfile 
$ sudo dd if=/dev/zero of=/swapfile bs=1024 count=1048576 
This command created the /swapfile file with 1GB in size.
2) Configuring file to swap:
$ sudo chmod 600 /swapfile 
$ sudo mkswap /swapfile 
3) Enable swap:
$ sudo swapon /swapfile 
To see result you can run free -m or see on htop
Example: $ sudo free -m
              total       usada       livre    compart.  buff/cache  disponível Mem.:          7664        1052        3436          56        3175        6259 Swap:          1024           0        1024 
Example: $ sudo swapon --show
NAME      TYPE SIZE USED PRIO /swapfile file   1G   0B   -2

June 15, 2019 in Tips and Tricks

http to https www redirect

1. Redirect All Web Traffic

If you have existing code in your .htaccess, add the following:

RewriteEngine On 
RewriteCond %{SERVER_PORT} 80 
RewriteRule ^(.*)$ https://www.yourdomain.com/$1 [R,L]

2. Redirect Only a Specific Domain

For redirecting a specific domain to use HTTPS, add the following:

RewriteEngine On 
RewriteCond %{HTTP_HOST} ^yourdomain\.com [NC]
RewriteCond %{SERVER_PORT} 80 
RewriteRule ^(.*)$ https://www.yourdomain.com/$1 [R,L]

3. Redirect Only a Specific Folder

Redirecting to HTTPS in a particular folder, add the following:

RewriteEngine On 
RewriteCond %{SERVER_PORT} 80 
RewriteCond %{REQUEST_URI} folder 
RewriteRule ^(.*)$ https://www.yourdomain.com/folder/$1 [R,L]

Note: Replace “yourdomain” with your actual domain name wherever required. Also, in case of the folder, replace /folder with the real folder name.

June 11, 2019 in Learning, MySql, Tips and Tricks

which language is best for socket programming

It really depends on what you mean by network programming. So I’ll run through a couple possibilities. Commentators: Don’t fret if I forgot “your language.” The question is so open-ended. It’s like asking ‘What kinds of vehicles are good for getting me from New York City to Boston?’

  • You want to write a program that participates in XML-RPC or similar.
    • C# or Java
  • You are network programming in the context of a web page
    • JavaScript
  • You need some machines to talk to each other in some other existing protocol, and you need it done quickly.
    • Python or Perl
  • You need to write a distributed program that needs to be up and running all the time, not even resting for upgrades
    • Erlang
  • You need to write a distributed program that is very fast and does a bunch of things at once
  • You need to write a new network driver, for a new network protocol you thought up yourself
    • C, C++
  • You just want to network some TCP sockets together

Soa Technology

June 11, 2019 in Learning, MySql, Tips and Tricks

5 Best Programming Languages to Learn for Cyber Security

1. C and C++

C and C++ are critical low-level programming languages that you need to know as a cyber security professional.

These languages provide access to low-level IT infrastructure such as RAM and system processes, which if not well protected, hackers can easily exploit.

The C programming language is the backbone of most operating systems. It is a lean, flexible, and efficient language that can be used to complete a wide range of tasks such as cryptography, image processing, and socket networking.

Essentially, C++ is usually regarded as C’s big brother — which has been concocted with crack, meth, and steroids and mixed without any favors. C++ is a fantastic language that is largely based on C’s source code.

There are several cyber security programs created using C++. For example, Nmap, the network mapper tool, is created using C++.

Here is a marvelous quote from Bjarne Stroustrup, the creator of C++:

“C makes it easy to shoot yourself in the foot; C++ makes it harder, but when you do, it blows your whole leg off.”

As a cyber security expert, if you are experienced at using C/C++ programming languages, you’ll know how to respond to attacks targeting lower level operations within your computing environment.

2. Python

Python is a high-level programming language that is increasingly becoming popular among cyber experts.

It’s gaining traction mainly because of its adherence to code readability, clear and simple syntax, and availability of an extensive number of libraries.

So, whatever task you want to do, you can always complete it easily with Python.

For example, you can use the language to send TCP-packets to machines, perform malware analysis, and create intrusion detection systems with minimal reliance on third-party tools.

However, unlike C/C++, Python is not low-level; therefore, it may not provide enough visibility to hardware resources.

Learning Python for cyber security will give you an edge in your career. You’ll be equipped with programming skills that can assist you to identify vulnerabilities and discover how to fix them.

No programming knowledge required

3. JavaScript

JavaScript is a high-level programming language, which is often referred as the “lingua franca” of the web.

JavaScript is a core technology that powers the Internet. Primarily, it’s the language that adds interactivity to web pages.

Although JavaScript was initially implemented only on the client-side in web browsers, it’s now possible to use the language in other types of host infrastructure, such as server-side in databases and offline applications like PDF programs.

Therefore, because of its extensive usage, learning JavaScript can make you go one step ahead of the hackers.

You’ll understand the concepts of how websites and other applications work and the best designs to employ to ward off malicious users.

For example, cross-site scripting is a JavaScript-based attack that involves an attacker implanting malicious code in a web application.

If you are experienced in using the JavaScript programming language, you can prevent such types of attacks from occurring.

4. PHP

PHP is a server-side programming language for developing websites. Because most websites are created using PHP, learning the language will enable you to know how to fend off intruders.

For example, DDoS (Denial-of-service) attacks usually attempt to make web applications unavailable to intended users.

With PHP programming knowledge, coupled with skills in other technologies like JavaScript, you can implement robust solutions to secure web applications.

5. SQL

SQL (Structured Query Language) is mostly used in managing data stored in databases.

Because of the current explosion of data storage systems, SQL is widely used for maintaining and retrieving data.

Similarly, hackers are increasingly orchestrating the language for damaging or exfiltrating the stored data.

For example, SQL injection attacks involve exploiting SQL vulnerabilities to steal or modify data kept in databases.

Therefore, having a good understanding of the SQL language is critical in your cyber security career.