Rate limit SSH access using IPTables/Firewalld to counter Brute Force attacks.

Hello Guys,

The Public IP ranges of Cloud Platforms like Amazon AWS, Oracle OCI, Microsoft Azure are published in Public. If you have an instance with open SSH rules lets say, Security Group on AWS, Security List on Oracle OCI, I am very sure that you have seen SSH break in attempts like below.

Sep 30 12:29:47 host sshd[9731]: Invalid user admin from 222.173.30.222 port 3644
Sep 30 12:29:47 host sshd[9731]: input_userauth_request: invalid user admin [preauth]
Sep 30 12:29:48 host sshd[9731]: Connection closed by 222.173.30.222 port 3644 [preauth]

Sep 30 12:36:22 host sshd[9779]: Invalid user ftpuser from 202.28.33.166 port 58268
Sep 30 12:36:22 host sshd[9779]: input_userauth_request: invalid user ftpuser [preauth]
Sep 30 12:36:22 host sshd[9779]: Received disconnect from 202.28.33.166 port 58268:11: Normal Shutdown, Thank you for playing [preauth]

NB: I am leaving those IPs from my logs here as they were causing troubles!

this clearly tells us that you need to reconsider the security settings on your cloud network. Ideally, sensitive ports like SSH (TCP/22) should be opened up only to trusted IPs or IP Networks. However, this might not be possible if the customer has users accessing the instance from various location. 

At times these attacks can become so severe, it slows down your instance and could end up crashed due to kernel Out of Memory Killer (OOM Killer) or because of CPU starving.

Setting up rate limiting rules using IPTables/Firewall on TCP/22 can help in reducing the effect of brute force attacks. But I should remind you that rate limiting is not the solution, you should consider accessing the instance via a bastion host for safe guarding the data and ensuring security best practices. 

Let’s check out how to make use of IPTables, Firewalld to set rate-limit rules to reduce brute force attacks.

Modern operating systems flavours like OEL7, RHEL7, CentOS7 uses Firewalld. Older OS flavours use IPTables. Let’s check out how to set rate-limit rules using both.

 

Using Firewalld:

The below sets rate limiting rules temporarily:

# firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m tcp --dport 22 -m recent --update --seconds 180 --hitcount 3 --rttl --name SSH --rsource -m comment --comment "SSH Brute-force" -j LOG --log-prefix "SSH_brute_force"

# firewall-cmd --direct --add-rule ipv4 filter INPUT 1 -p tcp -m tcp --dport 22 -m recent --update --seconds 180 --hitcount 3 --rttl --name SSH --rsource -m comment --comment "SSH Brute-force" -j DROP

# firewall-cmd --direct --add-rule ipv4 filter INPUT 2 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name SSH --rsource -m comment --comment "SSH Brute-force" -j ACCEPT

# firewall-cmd --zone=public --remove-service=ssh

Once the above rules are set, do test the rules yourself. Try logging in to the instance from a new session and make sure you are able to SSH in. Please do not close the existing session to revert the changes in case you get locked out. If the above rules are set properly, with in 3 minutes you will be able to SSH in to the instance 3 times and the 4th attempt will be blocked.

Logging:

The rules logs the blocked SSH attempt in "/var/log/messages" as follows:

# grep SSH_brute_force /var/log/messages  | tail -5

Aug 27 13:30:10 test kernel: SSH_brute_forceIN=ens3 OUT= MAC=02:00:17:00::00:17:85:9a:40:08:00 SRC=130.XX.XX.8 DST=10.0.0.35 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=20404 DF PROTO=TCP SPT=37076 DPT=22 WINDOW=26880 RES=0x00 SYN URGP=0

Aug 27 13:30:11 test kernel: SSH_brute_forceIN=ens3 OUT= MAC=02:00:17:00::00:17:85:9a:40:08:00 SRC=130.XX.XX.8 DST=10.0.0.35 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=20405 DF PROTO=TCP SPT=37076 DPT=22 WINDOW=26880 RES=0x00 SYN URGP=0

Aug 27 13:30:13 test kernel: SSH_brute_forceIN=ens3 OUT= MAC=02:00:17:00::00:17:85:9a:40:08:00 SRC=130.XX.XX.8 DST=10.0.0.35 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=20406 DF PROTO=TCP SPT=37076 DPT=22 WINDOW=26880 RES=0x00 SYN URGP=0

Aug 27 13:30:17 test kernel: SSH_brute_forceIN=ens3 OUT= MAC=02:00:17:00::00:17:85:9a:40:08:00 SRC=130.XX.XX.8 DST=10.0.0.35 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=20407 DF PROTO=TCP SPT=37076 DPT=22 WINDOW=26880 RES=0x00 SYN URGP=0

Aug 27 13:30:25 test kernel: SSH_brute_forceIN=ens3 OUT= MAC=02:00:17:00::00:17:85:9a:40:08:00 SRC=130.XX.XX.8 DST=10.0.0.35 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=20408 DF PROTO=TCP SPT=37076 DPT=22 WINDOW=26880 RES=0x00 SYN URGP=0

You may make the rules permanent once you have completed testing:

# firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p tcp -m tcp --dport 22 -m recent --update --seconds 180 --hitcount 3 --rttl --name SSH --rsource -m comment --comment "SSH Brute-force" -j LOG --log-prefix "SSH_brute_force: "

# firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 1 -p tcp -m tcp --dport 22 -m recent --update --seconds 180 --hitcount 3 --rttl --name SSH --rsource -m comment --comment "SSH Brute-force" -j DROP

# firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 2 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name SSH --rsource -m comment --comment "SSH Brute-force" -j ACCEPT

You may also remove the SSH service from the public zone.

# firewall-cmd --permanent --zone=public --remove-service=ssh

Using IPTables:

Older OEL/RHEL/CentOS uses IPTables instead on Firewalld. Here is the iptables rules for SSH rate limiting.

# iptables -N LOGDROP

# iptables -A LOGDROP -j LOG

# iptables -A LOGDROP -j DROP

# iptables -I INPUT 4 -p tcp --dport 22 -i ens3 -m state --state NEW -m recent --set

# iptables -I INPUT 5 -p tcp --dport 22 -i ens3 -m state --state NEW -m recent  --update --seconds 180 --hitcount 3 -j LOGDROP --log-prefix="SSH Brute-force: "

Logging:

The rules log the blocked SSH attempt in "/var/log/messages" as follows:

# grep "DPT=22" /var/log/messages | tail -5

Aug 27 13:54:14 test kernel: IN=ens3 OUT= MAC=02:00:17:00:74:b2:00:00:17:85:9a:40:08:00 SRC=160.XX.XX.133 DST=10.0.0.35 LEN=48 TOS=0x00 PREC=0x00 TTL=58 ID=0 DF PROTO=TCP SPT=53224 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0

Aug 27 13:54:16 test kernel: IN=ens3 OUT= MAC=02:00:17:00:74:b2:00:00:17:85:9a:40:08:00 SRC=160.XX.XX.133 DST=10.0.0.35 LEN=48 TOS=0x00 PREC=0x00 TTL=58 ID=0 DF PROTO=TCP SPT=53224 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0

Aug 27 14:28:49 test kernel: IN=ens3 OUT= MAC=02:00:17:00:74:b2:00:00:17:85:9a:40:08:00 SRC=130.XX.XX.8 DST=10.0.0.35 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=62756 DF PROTO=TCP SPT=37091 DPT=22 WINDOW=26880 RES=0x00 SYN URGP=0

Aug 27 14:28:50 test kernel: IN=ens3 OUT= MAC=02:00:17:00:74:b2:00:00:17:85:9a:40:08:00 SRC=130.XX.XX.8 DST=10.0.0.35 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=62757 DF PROTO=TCP SPT=37091 DPT=22 WINDOW=26880 RES=0x00 SYN URGP=0

Aug 27 14:28:52 test kernel: IN=ens3 OUT= MAC=02:00:17:00:74:b2:00:00:17:85:9a:40:08:00 SRC=130.XX.X.8 DST=10.0.0.35 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=62758 DF PROTO=TCP SPT=37091 DPT=22 WINDOW=26880 RES=0x00 SYN URGP=0

Test the rules: Try logging in to the instance from a new session and make sure you are able to SSH in. Please do not close the existing session to revert the changes in case you get locked out. If the above rules are set properly, with in 3 minutes you will be able to SSH in to the instance 3 times and the 4th attempt will be blocked.

Once you have tested the rules properly, you may delete the previous SSH allowed rule.

You may also save the IPTables rules to make it permanent.

# iptables-save > /etc/sysconfig/iptables

Automating the rate-limits using 3^rd party tools:

  You may also use tools like Fail2Ban or SSHGuard to automatically block offenders.

Hope this helps! Let me know if you have any questions.

Bash interpreting a Command

By default bash is the default shell in Linux. Lets see how bash shell interprets the command we type in.

It not as simple as checking the binaries in directories specified in the $PATH variable. It flows through the following steps:

1. Redirection
2. Aliases
3. Expansion
4. Shell Function
5. Shell Built-in
6. Hash table
7. PATH variable

Lets go through the steps :

1. Redirection.

This comes into picture when you use redirection : ">" and ">>".

You can see that a file list.txt is automatically created before the "ls" command is executed with redirection ">". How does this happened ? Redirection is done and the o/p file is created before the command "ls" is executed.  

2. Alias.

Alias is actually shortcuts or easy to remember alternatives of a command.

Here we have defined alias "ls=cat", now when we type "ls list.txt" , ls is replaced with cat and contents of list.txt file is displayed.

Just FYI, we can make the shell ignore alias by preceding "\" with the command.

 

3. Expansion : 

Expansion stands for command substitution, variable and parameter expansion, tilde expansion, brace expansion, arithmetic expansion, word splitting, and path name expansion. 

 

Like this ~, *,? etc will be replaced accodingly.

4. Functions defined in the Shell

Shell also allows to define functions like other programming languages. After defining functions you can execute it just like any other commands.

As you can see, here I have shown you the order of interpretation. Alias has precedence over Functions!. When we first executed "ls" the alias /bin/ls is executed. Then ignored alias with "\" and function is executed. Then we deleted the alias "unalias" command. Now since there is no alias, text candidate to run is function.

5. Shell Builtin

The bash code contains some commands as well. These commands gets precedence over other binaries. The type command can be used to identify who the command is interpreted. 

Lets play a bit with this :

[jk@rhel1 ~]$ type echo
echo is a shell builtin

[jk@rhel1 ~]$ which echo
/bin/echo

[jk@rhel1 ~]$ type ls
ls is a function
ls ()
{
    echo "I am a Function"
}

[jk@rhel1 ~]$ alias ls="echo GNU/Linux"
[jk@rhel1 ~]$ type ls
ls is aliased to `echo GNU/Linux'

[jk@rhel1 ~]$ ls
GNU/Linux

[jk@rhel1 ~]$ unalias ls
[jk@rhel1 ~]$ ls
I am a Function

[jk@rhel1 ~]$ unset ls
[jk@rhel1 ~]$ type ls
ls is hashed (/bin/ls)
[jk@rhel1 ~]$

I shall explain why "hashed" is displayed in the last command.

6. Hash Table 

The concept of hash table is similar to cashing in Linux, shell store the full path of all the executed commands to speed things up. 

Lets see how hast table works. We have started a new shell and executed "ls", "cat" and "echo" commands. Then we checked the hash table entries. The hash table, as already mentioned, contains two columns, first column contains the cache hits and the 2nd column contains the full path of the command. Also note that the shell built in command "echo" is not listed in the hash table. Remember the purpose the hash table is to speed up the execution. 

You can see that when the command "ls" is executed the second time, the hits value is increased.

7. Path Variable

After all the above steps are over or not available, shell searches for the given command in the directories specified in the PATH variable. 

[jk@rhel1 ~]$ echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/jk/bin
[jk@rhel1 ~]$

Disk Metadata : Superblock, Directory and Inodes

Metadata

Filesystem blocks are user for 2 purposes : To store User data and Metadata

• User data - stores actual data contained in files
• Metadata - stores file system structural information such as superblock, inodes, directories

Metadata describes the structure of the file system. Most common metadata structure are superblock, inode and directories.

Superblocks

Every FS has a superblock which contains info about filesystems such as :

• File system type
• Size
• Status
• Information about other metadata structures
• • For filesystems with 1k blocksizes, a backup superblock can be found at block 8193
  • For filesystems with 2k blocksizes, at block 16384
  • For 4k blocksizes, at block 32768.

List backup superblocks:

# dumpe2fs /dev/hda3 | grep -i superblock

If Superblock  is corrupted, restore with backup :

# e2fsck -f -b 8193 /dev/sda3

 

Inode

 

 An inode is a data structure on a Linux Unix FS which stores stores basic information about a regular file, directory, or other file system objects.  

Recover deleted files used by any process in Linux from RAM memory.

Every thing in Linux is a File. A file in turn is actually a pointer to inode which contain the actual data on the disk, permissions, ownership. Now what happens when a file is deleted ? Only the link is removed by not the inode or the actual data. if a process is using the file, or if the file is open , the inode is not released for overwriting util the process is done with the file. Such files will remain in the server memory (RAM). 

Lets discuss how to recover such deleted files which is being used by a process.

Lets do it with an example.

Create a test file.
# touch testfile.txt

Echo some random data on it.
# cat /dev/random > testfile.txt

Open the file using some command like below.
# less  testfile.txt

# ps -ef | grep -i less

less 4607 root  4r  REG 254,4   21  
           8880214 /root/testing.txt (deleted)

All the open files remain in the memory and hence in the /proc filesystem. The important columns in the above output are the second one, which gives you the PID of the process that has the file open (4607), and the fourth one, which gives you the file descriptor (4). Now, we go look in /proc, where there will still be a reference to the inode, from which you can copy the file back.

# ls -l /proc/4607/fd/4
lr-x------ 1 root root 64 Apr  7 03:19 
             /proc/4607/fd/4 -> /root/testing.txt (deleted)

To recover the deleted file in memory, just copy as below.
 #cp /proc/4607/fd/4 testing.txt.bk


Hurray you got your file back. Just make sure not to use "-a" switch while copying the file as this will copy the broken softlink.

JK

 

Sendmail Error : NOQUEUE: SYSERR(root): No local mailer defined NOQUEUE: SYSERR(root): QueueDirectory (Q) option must be set

Error using m4 macro while building sendmail.cf from sendmail.mc on RHEL 6 / CentOS 6 flavors?

[root@cent1]# m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

m4:/etc/mail/sendmail.mc:10: cannot open `/usr/share/sendmail-cf/m4/cf.m4': No such file or directory

[root@cent1]#

The above error is because the package sendmail-cf is not installed.

You will have the below error messages on the /var/log/maillog file also. 

sendmail[3215]: gethostbyaddr(10.0.3.15) failed: 2

sendmail[3215]: NOQUEUE: SYSERR(root): No local mailer defined

sendmail[3215]: NOQUEUE: SYSERR(root): QueueDirectory (Q) option must be set

The above error is because the sendmail.cf is not configured properly. 

Fix is to install the package sendmail-cf, build sendmail.cf from sendmail.mc and restart sendmail service.

Install the sendmail-cf

# yum install -y sendmail-cf

Comment the entry which makes the sendmail listens only to loopback address as below by adding "dnl" on /etc/mail/sendmail.mc

# vi /etc/mail/sendmail.mc

dnl # DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl

# m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

# service sendmail restart

Check if sendmail is started properly.

[root@cent1 ~]# netstat -tulnp |  grep -i :25

tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      3748/sendmail

[root@cent1 ~]#

Hope this helps.

Jk

Change Default MTA in RHEL 6 / Cent OS 6

By default Postfix is set as the default mail transfer agent in Redhat Enterprise Linux 6 and Cent OS 6 flavors.

However you can change the default Mail transfer Agent in RHEL 6 using the below commands.

* Install Sendmail if not yet installed.

# yum install sendmail -y

Change the default MTA.

# alternatives --config mta

There are 2 programs which provide 'mta'.
  Selection    Command
-----------------------------------------------
 + 1           /usr/sbin/sendmail.postfix
*  2           /usr/sbin/sendmail.sendmail
Enter to keep the current selection[+], or type selection number: 2
#

Congrats, now your default MTS is changed from postfix to Sendmail.

Regards,
Jk

How to configure Veritas Volume Manager VxVM 6.0.2

The disks should be discovered by OS before it can be added to Veritas Volume Manager.


1. List the disks detected by VxVM.

# vxdisk -o alldgs list


Eg :

[root@rhel61 ~]# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
sda          auto:none       -            -            online invalid
sdb          auto:none       -            -            online invalid
sdc          auto:cdsdisk    mydg01       mydg         online
sdd          auto:cdsdisk    mydg02       mydg         online
[root@rhel61 ~]#

sda and sbd are not configured / initialized for VxVM; where as sdc and sdd are initialized and added to a disk group.


2. Initialize disks.

vxdisksetup -i [attributes]

# vxdisksetup -i sdc
# vxdisksetup -i emc0_dd1

Initializing a disk will create Privae and Public regions in the disks, Meta data is stored in private region and public reagion is available for general purpose data usage. 32 MB is reserved for private region.