December 27, 2019 in Tips and Tricks, Ubuntu

apport ubuntu

What is this all about?

Debugging program crashes without any automated tools has been pretty time consuming and hard for both developers and users. Many program crashes remain unreported or unfixed because:

• Many crashes are not easily reproducible.
• End users do not know how to prepare a report that is really useful for developers, like building a package with debug symbols, operating gdb, etc.
• A considerable part of bug triage is spent with collecting relevant information about the crash itself, package versions, hardware architecture, operating system version, etc.
• There is no easy frontend which allow users to submit detailed problem reports.
• Existing solutions like bug-buddy or krash are specific to a particular desktop environment, are nontrivial to adapt to the needs of a distribution developer, do not work for crashes of background servers (like a database or an email server), and do not integrate well with existing debug packages that a distribution might provide.

Apport is a system which:

• intercepts crashes right when they happen the first time,
• gathers potentially useful information about the crash and the OS environment,
• can be automatically invoked for unhandled exceptions in other programming languages (e. g. in Ubuntu this is done for Python),
• can be automatically invoked for other problems that can be automatically detected (e. g. Ubuntu automatically detects and reports package installation/upgrade failures from update-manager),
• presents a UI that informs the user about the crash and instructs them on how to proceed,
• and is able to file non-crash bug reports about software, so that developers still get information about package versions, OS version etc.

We are sure that this will lead to a much better level of quality assurance in the future.

If you want to make crash reports of your software even more useful when being reported through apport, please see /DeveloperHowTo.

What does it look like for users?

The user side of apport is designed to be extremely simple and as unannoying as possible.

If any process in the system dies due to a signal that is commonly referred to as a ‘crash’ (segmentation violation, bus error, floating point exception, etc.), or e. g. a packaged Python application raises an uncaught exception, the apport backend is automatically invoked. It produces an initial crash report in a file in /var/crash/ (the file name is composed from the name of the crashed executable and the user id). If the crashed process belongs to the user who is currently logged in, or it belongs to a system process and the user is an administrator, apport informs the user about the crash and offers to report the problem:

You can click on “Show Details…” to see what data it collected:

If the user leaves the “Send error report” checkbox enabled, Apport uploads the collected information to the bug tracking system. After that it opens the packages’ bug filing page with a sensible default bug title and leaves the rest of bug filing process to the web UI.

Why is apport disabled by default?

Apport is not enabled by default in stable releases, even if it is installed. The automatic crash interception component of apport is disabled by default in stable releases for a number of reasons:

1. Apport collects potentially sensitive data, such as core dumps, stack traces, and log files. They can contain passwords, credit card numbers, serial numbers, and other private material.This is mitigated by the fact that it presents you what will be sent to the bug tracker, and that all crash report bugs are private by default, limited to the Ubuntu bug triaging team. We can reasonably expect developers and technically savvy users, who run the development release, to be aware of this and judge whether it is appropriate to file a crash report. But we shouldn’t assume that every Ubuntu user of stable releases is able to do so. In 12.04 and up this is transparently handled by whoopsie, see ErrorTracker.
2. During the development release we already collect thousands of crash reports, much more than we can ever fix. Continuing to collect those for stable releases is not really useful, since
   • The most important crashes have already been discovered in the development release.
   • The less important ones are not suitable for getting fixed in stable releases (see https://wiki.ubuntu.com/StableReleaseUpdates
   • Asking users to send crash reports to us is insincere, since we can’t possibly answer and deal with all of them.
3. Data collection from apport takes a nontrivial amount of CPU and I/O resources, which slow down the computer and don’t allow you to restart the crashed program for several seconds.

Note apport does not trap SIGABRT signals. If you are getting such a signal, then please see DebuggingProgramCrash.

How to enable apport

Apport itself is running at all times because it collects crash data for whoopsie (see ErrorTracker). However, the crash interception component is still disabled. To enable it permanently, do:

sudo nano /etc/apport/crashdb.conf

… and add a hash symbol # in the beginning of the following line:

        'problem_types': ['Bug', 'Package'],

To disable crash reporting just remove the hash symbol.

December 24, 2019 in Tips and Tricks, Ubuntu

check storage space in ubuntu | how to check hard disk size in ubuntu terminal

show total of disk space

$ df -h --total

which will show it in one line as

• (1) total space
• (2) total space used
• (3) total space still available
• (4) percentage of drive used.

All of this in gigabytes.

I mapped this to a shell command dspace and when I type that into terminal it instantly shows me my disk space usage.

$ df -h --total | grep total

lists the size of each partition

$ lsblk

1. Checking File System Disk Space

The “df” command displays the information of file system device names, disk blocks, total disk space used, available disk space, percentage of usage and mount points on a file system.

ubuntu@www:~$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
udev              491680       0    491680   0% /dev
tmpfs             100764     780     99984   1% /run
/dev/xvda1     101583780 7599572  93967824   8% /
tmpfs             503804       0    503804   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             503804       0    503804   0% /sys/fs/cgroup
/dev/loop0         91264   91264         0 100% /snap/core/8268
/dev/loop1         18432   18432         0 100% /snap/amazon-ssm-agent/1480
/dev/loop2         18432   18432         0 100% /snap/amazon-ssm-agent/1455
/dev/loop3         91264   91264         0 100% /snap/core/8213
tmpfs             100760       0    100760   0% /run/user/1000

2. Display Disk Space in Human Readable Format

As you see in the first example, the output is difficult to read or understand. By default df command shows disk space information in bytes which is difficult to understand for humans. We can easily understand if the results are in megabytes and gigabytes.

Don’t worry, the good df command has an option to display information in human readable format like in MB and GB. You just need to add -h flag to see.

ubuntu@www:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            481M     0  481M   0% /dev
tmpfs            99M  780K   98M   1% /run
/dev/xvda1       97G  7.3G   90G   8% /
tmpfs           492M     0  492M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           492M     0  492M   0% /sys/fs/cgroup
/dev/loop0       90M   90M     0 100% /snap/core/8268
/dev/loop1       18M   18M     0 100% /snap/amazon-ssm-agent/1480
/dev/loop2       18M   18M     0 100% /snap/amazon-ssm-agent/1455
/dev/loop3       90M   90M     0 100% /snap/core/8213
tmpfs            99M     0   99M   0% /run/user/1000

3. Display information of all file systems

If you want to see all file systems including which has zero block sizes, pass parameter -a or -all along with df command. The output would be similar to this.

ubuntu@www:~$ df -all
Filesystem     1K-blocks    Used Available Use% Mounted on
sysfs                  0       0         0    - /sys
proc                   0       0         0    - /proc
udev              491680       0    491680   0% /dev
devpts                 0       0         0    - /dev/pts
tmpfs             100764     780     99984   1% /run
/dev/xvda1     101583780 7517872  94049524   8% /
securityfs             0       0         0    - /sys/kernel/security
tmpfs             503804       0    503804   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             503804       0    503804   0% /sys/fs/cgroup
cgroup                 0       0         0    - /sys/fs/cgroup/unified
cgroup                 0       0         0    - /sys/fs/cgroup/systemd
pstore                 0       0         0    - /sys/fs/pstore
cgroup                 0       0         0    - /sys/fs/cgroup/devices
cgroup                 0       0         0    - /sys/fs/cgroup/cpu,cpuacct
cgroup                 0       0         0    - /sys/fs/cgroup/rdma
cgroup                 0       0         0    - /sys/fs/cgroup/perf_event
cgroup                 0       0         0    - /sys/fs/cgroup/pids
cgroup                 0       0         0    - /sys/fs/cgroup/hugetlb
cgroup                 0       0         0    - /sys/fs/cgroup/net_cls,net_prio
cgroup                 0       0         0    - /sys/fs/cgroup/blkio
cgroup                 0       0         0    - /sys/fs/cgroup/memory
cgroup                 0       0         0    - /sys/fs/cgroup/freezer
cgroup                 0       0         0    - /sys/fs/cgroup/cpuset
systemd-1              -       -         -    - /proc/sys/fs/binfmt_misc
hugetlbfs              0       0         0    - /dev/hugepages
mqueue                 0       0         0    - /dev/mqueue
debugfs                0       0         0    - /sys/kernel/debug
fusectl                0       0         0    - /sys/fs/fuse/connections
configfs               0       0         0    - /sys/kernel/config
/dev/loop0         91264   91264         0 100% /snap/core/8268
/dev/loop1         18432   18432         0 100% /snap/amazon-ssm-agent/1480
/dev/loop2         18432   18432         0 100% /snap/amazon-ssm-agent/1455
/dev/loop3         91264   91264         0 100% /snap/core/8213
lxcfs                  0       0         0    - /var/lib/lxcfs
binfmt_misc            0       0         0    - /proc/sys/fs/binfmt_misc
tmpfs             100760       0    100760   0% /run/user/1000

4. Display File System details in Bytes

To display all file system information and usage in 1 K blocks , pass the option ‘-k‘  as follows.

ubuntu@www:~$ df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
udev              491680       0    491680   0% /dev
tmpfs             100764     784     99980   1% /run
/dev/xvda1     101583780 7600328  93967068   8% /
tmpfs             503804       0    503804   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             503804       0    503804   0% /sys/fs/cgroup
/dev/loop0         91264   91264         0 100% /snap/core/8268
/dev/loop1         18432   18432         0 100% /snap/amazon-ssm-agent/1480
/dev/loop2         18432   18432         0 100% /snap/amazon-ssm-agent/1455
/dev/loop3         91264   91264         0 100% /snap/core/8213
tmpfs             100760       0    100760   0% /run/user/1000

5. Display File System Information in MB

To display all file system information and usage in MB or megabytes  , pass the option ‘-m‘.

ubuntu@www:~$ df -m
Filesystem     1M-blocks  Used Available Use% Mounted on
udev                 481     0       481   0% /dev
tmpfs                 99     1        98   1% /run
/dev/xvda1         99203  7343     91845   8% /
tmpfs                492     0       492   0% /dev/shm
tmpfs                  5     0         5   0% /run/lock
tmpfs                492     0       492   0% /sys/fs/cgroup
/dev/loop0            90    90         0 100% /snap/core/8268
/dev/loop1            18    18         0 100% /snap/amazon-ssm-agent/1480
/dev/loop2            18    18         0 100% /snap/amazon-ssm-agent/1455
/dev/loop3            90    90         0 100% /snap/core/8213
tmpfs                 99     0        99   0% /run/user/1000

6. Display File System Information in GB

To display all file system details and usage in GB or gigabytes  , pass the option ‘-h‘.

ubuntu@www:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            481M     0  481M   0% /dev
tmpfs            99M  780K   98M   1% /run
/dev/xvda1       97G  7.3G   90G   8% /
tmpfs           492M     0  492M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           492M     0  492M   0% /sys/fs/cgroup
/dev/loop0       90M   90M     0 100% /snap/core/8268
/dev/loop1       18M   18M     0 100% /snap/amazon-ssm-agent/1480
/dev/loop2       18M   18M     0 100% /snap/amazon-ssm-agent/1455
/dev/loop3       90M   90M     0 100% /snap/core/8213
tmpfs            99M     0   99M   0% /run/user/1000

7. Display /home file system information

You can see only /home file system device infomation by executing below df command.

ubuntu@www:~$ df -h /home
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       97G  7.3G   90G   8% /

8. Display File System Type in Linux

To see the type of each file system pass the -T option. It gives output with file system types column. You could see the type of each file system listed such as ext4, ext3, tempfs, devtmpfs and etc.

ubuntu@www:~$ df -T
Filesystem     Type     1K-blocks    Used Available Use% Mounted on
udev           devtmpfs    491680       0    491680   0% /dev
tmpfs          tmpfs       100764     780     99984   1% /run
/dev/xvda1     ext4     101583780 7602896  93964500   8% /
tmpfs          tmpfs       503804       0    503804   0% /dev/shm
tmpfs          tmpfs         5120       0      5120   0% /run/lock
tmpfs          tmpfs       503804       0    503804   0% /sys/fs/cgroup
/dev/loop0     squashfs     91264   91264         0 100% /snap/core/8268
/dev/loop1     squashfs     18432   18432         0 100% /snap/amazon-ssm-agent/1480
/dev/loop2     squashfs     18432   18432         0 100% /snap/amazon-ssm-agent/1455
/dev/loop3     squashfs     91264   91264         0 100% /snap/core/8213
tmpfs          tmpfs       100760       0    100760   0% /run/user/1000

9. Include or Exclude only certain File System Types.

If you want to see only ext4 file systems, use df command with option -t

If you want to see all file system types except ext4, then pass -x option and type ( ext3, ext4) to exclude from the output.

 df -t tmpfs
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs             100764   780     99984   1% /run
tmpfs             503804     0    503804   0% /dev/shm
tmpfs               5120     0      5120   0% /run/lock
tmpfs             503804     0    503804   0% /sys/fs/cgroup
tmpfs             100760     0    100760   0% /run/user/1000

10. Display df Command Options and Help

You can see all the available options with df command by typing below command.

$ df --help
Usage: df [OPTION]... [FILE]...
Show information about the file system on which each FILE resides,
or all file systems by default.

Mandatory arguments to long options are mandatory for short options too.
  -a, --all             include pseudo, duplicate, inaccessible file systems
  -B, --block-size=SIZE  scale sizes by SIZE before printing them; e.g.,
                           '-BM' prints sizes in units of 1,048,576 bytes;
                           see SIZE format below
  -h, --human-readable  print sizes in powers of 1024 (e.g., 1023M)
  -H, --si              print sizes in powers of 1000 (e.g., 1.1G)
  -i, --inodes          list inode information instead of block usage
  -k                    like --block-size=1K
  -l, --local           limit listing to local file systems
      --no-sync         do not invoke sync before getting usage info (default)
      --output[=FIELD_LIST]  use the output format defined by FIELD_LIST,
                               or print all fields if FIELD_LIST is omitted.
  -P, --portability     use the POSIX output format
      --sync            invoke sync before getting usage info
      --total           elide all entries insignificant to available space,
                          and produce a grand total
  -t, --type=TYPE       limit listing to file systems of type TYPE
  -T, --print-type      print file system type
  -x, --exclude-type=TYPE   limit listing to file systems not of type TYPE
  -v                    (ignored)
      --help     display this help and exit
      --version  output version information and exit

Display values are in units of the first available SIZE from --block-size,
and the DF_BLOCK_SIZE, BLOCK_SIZE and BLOCKSIZE environment variables.
Otherwise, units default to 1024 bytes (or 512 if POSIXLY_CORRECT is set).

The SIZE argument is an integer and optional unit (example: 10K is 10*1024).
Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers of 1000).

FIELD_LIST is a comma-separated list of columns to be included.  Valid
field names are: 'source', 'fstype', 'itotal', 'iused', 'iavail', 'ipcent',
'size', 'used', 'avail', 'pcent', 'file' and 'target' (see info page).

GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Report df translation bugs to <http://translationproject.org/team/>
Full documentation at: <http://www.gnu.org/software/coreutils/df>
or available locally via: info '(coreutils) df invocation'

December 20, 2019 in Tips and Tricks, Ubuntu

How to Setup Varnish HTTP Cache on an Ubuntu

Varnish Cache is a web application accelerator that can be used as a proxy to your Apache web server. The open-source software sits in front of your web server to serve web traffic very fast. If you are running multiple servers, Varnish Cache can also be used as a load balancer.

Varnish works by caching regularly requested web content on the system memory, and this ensures faster information retrieval if the same information is asked for several times.

$ sudo apt-get install varnish

By default, Apache listens on port 80 for HTTP traffic. We need to make some changes here. Instead of the default settings, Varnish will instead listen on port 80  and forward all traffic to Apache web server which we will configure to listen on port 8080.

$ sudo nano /etc/apache2/ports.conf

Listen 8080
<IfModule ssl_module>
        Listen 443
</IfModule>
<IfModule mod_gnutls.c>
        Listen 443
</IfModule>

Press CTRL + X, Y and hit Enter to save the file once you make the changes.
Next, edit the default Apache Virtual Host to listen to port 8080 too:

$ sudo service apache2 restart

 Configure Varnish HTTP Cache to listen on port 80

Next we will configure Varnish to listen on port 80 and forward all requests to our Apache web server.

We can do this by editing Varnish configuration file ‘/etc/default/varnish’

$ sudo nano  /etc/default/varnish

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256m"

Then, press CTRL + X, Y and hit Enter to save the file.

Next, check the file ‘/etc/varnish/default.vcl’ using a nano text editor. You should see the below content and this means Varnish will forward http traffic to port 8080:

$ sudo nano /etc/varnish/default.vcl

File contents:

# Default backend definition. Set this to point to your content server.
backend default {
.host = "127.0.0.1";
.port = "8080";
}

We also need to edit the port on the file ‘/lib/systemd/system/varnish.service’ file. To do so, type the command below:

$ sudo nano /lib/systemd/system/varnish.service

Change the default port from 6081 to 80 as shown below

[Unit]
Description=Varnish HTTP accelerator
Documentation=https://www.varnish-cache.org/docs/4.1/ man:varnishd
[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :80 -T localhost:6082 -f$
ExecReload=/usr/share/varnish/varnishreload
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
[Install]
WantedBy=multi-user.target

Then, press CTRL + X, Y and hit Enter to save the file.

Restart Apache, Varnish, and the Systemd Daemon

$ sudo systemctl restart apache2
$ sudo systemctl daemon-reload
$ sudo systemctl restart varnish

If the setup was successful, Varnish will now be the default HTTP Listener on port 80.

Testing the Setup

You can now try visiting your server one more time on a web browser:

http://public_ip_adress
or
http://example.com

The server traffic should now be handled by Varnish HTTP Cache software and forwarded to Apache.

To make sure that Varnish is working, use the curl command for verification purposes:

$ curl -I server_ip_address

You should get an output similar to the below text. If you see the line ‘Via: 1.1 varnish (Varnish/5.2)’, then Varnish is working like expected.

HTTP/1.1 200 OK
Date: Thu, 05 Jul 2018 20:56:11 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Fri, 29 Jun 2018 07:19:34 GMT
Vary: Accept-Encoding
Content-Type: text/html
X-Varnish: 7
Age: 0
Via: 1.1 varnish (Varnish/5.2)
ETag: W/"2aa6-56fc2ab77545d-gzip"
Accept-Ranges: bytes
Connection: keep-alive

December 11, 2019 in Tips and Tricks, Ubuntu

How To Set Up/Enable a Firewall with UFW on Ubuntu

UFW, or Uncomplicated Firewall, is an interface to iptables

Prerequisites

To follow this tutorial, you will need:

UFW is installed by default on Ubuntu. If it has been uninstalled for some reason, you can install it with sudo apt install ufw.

$ sudo nano /etc/default/ufw

Then make sure the value of IPV6 is yes.

$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing

$ sudo ufw allow ssh

$ sudo ufw allow 22

$ sudo ufw enable

$ sudo ufw allow 80

$ sudo ufw allow 443

$ sudo ufw allow 6000:6007/tcp
$ sudo ufw allow 6000:6007/udp

$ sudo ufw allow from 203.0.113.4

$ sudo ufw allow from 203.0.113.4 to any port 22

$ sudo ufw allow from 203.0.113.0/24

$ sudo ufw allow from 203.0.113.0/24 to any port 22

$ sudo ufw deny http

$ sudo ufw deny from 203.0.113.4

$ sudo ufw status numbered

$ sudo ufw delete 2

$ sudo ufw delete allow http

$ sudo ufw delete allow 80

$ sudo ufw status verbose

$ sudo ufw disable

$ sudo ufw reset