How to view httpd.conf file or any config files in Linux without coments

Hello,

Have you ever wondered how to view the Linux configurations files without those comments? Well those comments are indeed helpful, but think about configuration files such as httpd.conf and squid.conf files. These files have good amount of commented lines.

The issue with the httpd.conf file in particular is that not all commented lines starts with #. Some commented lines start after a tab.

Example:

# Further relax access to the default document root:

    #
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews

You could remove those tabbed comments as well using "sed" or using "egrep" with proper Regex.


# egrep -v "^$|*#" /etc/httpd/conf/httpd.conf

Or

# sed '/ *#/d; /^ *$/d' /etc/httpd/conf/httpd.conf


Hope this helps.

Regards,
Jay

Automatically encrypt Ephemeral volumes on AWS EC2 instances

Encrypting Data at rest if mandatory requirement compliance regulations such as PCI DSS and HIPAA. EBS, S3 has an option to encrypt the data stored in it. However, Instance Store volumes which provides temporary block level storage does have an option to encrypt the data stored in it. Customers need to use configure encryption using tools like dm-crypt.

Lets see how to automate encrypting the ephemeral volumes for Instance Store volumes on EC2 instances. I have written a user-data script which takes care of all the encryption setups.

The script will do the below to make it seamless for the customer:

- When you launch an instance or change the instance type, the script automatically detects the Ephemeral disks available to the instance type, setup encryption and make them available.

- Automatically installs the required packages for encryption if its not installed.

- The script also takes care of encryption in case of instance reboot or stop/start.

- The user-data script automatically take care of Stop/Start, Reboots and instance re-size.

- After stop/start you will lose all the data and configuration on the ephemeral volumes. This is expected, the user-data script will automatically detect this and setup encryption  on the new disks and mounts it.

- When the instance is rebooted, the data persists. On a reboot we just need to decrypt the filesystem and mount it.

- The user data script will encrypt the volume and create a filesystem if its not already encrypted (These actions will be taken if you launch a fresh instance with this user-data or when you do a stop & start of your instance).

Now, if the volumes are already encrypted (this will happen on an instance reboot), the script will simply initialize the encrypted volume and mount it on the respective directory.

- The script will encrypt all the ephemeral volumes, create EXT4 filesystem and mount them under /encrypted_X (X being the last letter of the device name of the NVMe device name).
You may change the filesystem type, mount point as per your requirement, but you will need to update the logic to map volumes to correct mount point on the script.

- The encryption key/passphrase used is encrypted using AWS KMS and kept in a file in your S3 bucket. This is a one time setup and you could use the same encrypted passphrase file for your future installations. Instance need to be attached with an IAM instance profile with permission to download files from the S3 bucket.









Here are the step by step instructions:
1. Create an S3 Bucket for storing encrypted password file (or use an existing bucket in your account).

2. Create a IAM policy with the below policy document, this policy is going to be used for instance profile to allow it download the password file from s3 bucket:

Sign in to the AWS Management Console and navigate to the IAM console: https://console.aws.amazon.com/iam/home?region=us-east-1#/home
In the navigation pane, choose Policies, choose Create Policy, select Create Your Own Policy, name and describe the policy, and paste the following policy. Choose Create Policy.


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1478729875000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::/LuksInternalStorageKey"
            ]
        }
    ]
}

 

---> Replace with the one you created/selected in step 1.



3. Create an EC2 role with the above policy.

In the IAM console, choose Roles, and then choose Create New Role.
In Step 1: Role Name, type your role name, and choose Next Step.
In Step 2: Select Role Type, choose Amazon EC2 and choose Next Step.
In Step 3: Established Trust, choose Next Step.
In Step 4: Attach Policy, choose the policy you created in Step 1

4. Create a KMS Key and add usage permission for the role you created above from the KMS console: https://console.aws.amazon.com/iam/home?region=us-east-1#/encryptionKeys/us-east-1
Step 1: Click ‘Create key’.
Step 2: Provide an Alias and Description.
Step 3: Add IAM role you created above as Key user under - ‘Define Key Usage Permissions’.

5. Create a secret password, encrypt it with KMS and store it in S3.

# aws --region us-east-1 kms encrypt --key-id 'alias/EncFSForEC2InternalStorageKey' --plaintext "ThisIs-a-SecretPassword" --query CiphertextBlob --output text | base64 --decode > LuksInternalStorageKey

--> Replace 'alias/EncFSForEC2InternalStorageKey' with the id of the KMS key you created above. The IAM user as which you run the above command should have permission to access the KMS key as well.
—> You should also replace the string “ThisIs-a-SecretPassword" with some strong passphrase.

6. Upload the encrypted key file to S3.

# aws s3 cp LuksInternalStorageKey s3:///LuksInternalStorageKey

7. Add the below userdata script to your instance at launch time. You can add/modify the user-data on a stopped instance as well.

I3, F1 instance type provides NVMe based instance store volumes and those disks are detected as /dev/nvmeXn1, X being the NVMe device number. Other instance types provides HDD/SSD based instance store volumes which gets detected as /dev/sdX. The user-data script detect the device names from EC2 meta-data in case of HDD/SSD and using nvme tool in case of NVMe based disks. I have written separate user data script for these volume types.

Use - user-data_NVMe_InstanceStore.txt for I3, F1 instances with NVMe disks.
Use - user-data_InstanceStore.txt for other instance types.

8. Once the instance is launched, you need to update the cloud-init configuration to make sure the user data runs every time the instance boots.

On Amazon Linux AMI based instances:
# sed -i 's/scripts-user/[scripts-user, always]/' /etc/cloud/cloud.cfg.d/00_defaults.cfg

On RHEL/CentOS instances:
# sed -i 's/scripts-user/[scripts-user, always]/' /etc/cloud/cloud.cfg


9. On Amazon Linux AMI instances, you might need disable crypt module on boot time to make sure the encrypted filesystems are not detected on the early boot stages:
# sed -i 's/#omit_dracutmodules+=""/omit_dracutmodules+="crypt"/' /etc/dracut.conf
# dracut --force
 

User Data Scripts:

-- user-data_InstanceStore.txt

#!/bin/bash

REGION=
S3_Bucket=

# Install required packages if not installed
[ $(which unzip) ] || yum -y install unzip
[ $(which aws) ] || "$(/usr/bin/curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" ; /usr/bin/unzip -o awscli-bundle.zip; ./awscli-bundle/install -b /usr/bin/aws ; rm -rf /awscli-bundle*)"
[ $(which cryptsetup) ] || yum install -y cryptsetup

# Get the encrypted password file from s3
/usr/bin/aws s3 cp s3://${S3_Bucket}/LuksInternalStorageKey .
# Decrypt and store the passphrase in a variable
LuksClearTextKey=$(/usr/bin/aws --region {REGION} kms decrypt --ciphertext-blob fileb://LuksInternalStorageKey --output text --query Plaintext | base64 --decode)

for ephemeral in $(curl -s http://169.254.169.254/latest/meta-data/block-device-mapping/ |grep ephemeral)
do
DEV=$(curl -s http://169.254.169.254/latest/meta-data/block-device-mapping/$ephemeral |sed 's/s/\/dev\/xv/g')
DEV_1=`echo "${DEV: -1}"`

[ "$(/bin/mount | grep -i ${DEV})" ] && /bin/umount ${DEV}

TYPE=`/usr/bin/file -sL ${DEV} | awk '{print $2}'`

if [ $TYPE == "LUKS" ]
then
    # Open and initialize the encryped volume
    /bin/echo "$LuksClearTextKey" | /sbin/cryptsetup luksOpen ${DEV} encfs_${DEV_1}
    # Check and create mount point if not exists
    [ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
    # Mount the filsystem
    /bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
else
     # Encrypt the volume sub
     /bin/echo "$LuksClearTextKey" | cryptsetup -y luksFormat ${DEV}
     # Open and initialize the encryped volume
    /bin/echo "$LuksClearTextKey" | cryptsetup luksOpen ${DEV} encfs_${DEV_1}
    # create a filesystem on the encrypted volume, mount it on the required directory
    /sbin/mkfs.ext4 /dev/mapper/encfs_${DEV_1}
    [ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
    /bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
fi
done
# Unset the passphrase variable and remove the encrypted password file.
unset LuksInternalStorageKey
rm LuksInternalStorageKey

-- user-data_NVMe_InstanceStore.txt

#!/bin/bash

#set -x

REGION=
S3_Bucket=

# Install required packages if not installed
[ $(which unzip) ] || yum install -y zip unzip
[ $(which python) ] || yum install -y python
[ $(which aws) ] || "$(/usr/bin/curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" ; /usr/bin/unzip -o awscli-bundle.zip; ./awscli-bundle/install -b /usr/bin/aws ; rm -rf /awscli-bundle*)"
[ "$(which cryptsetup)" ] || yum install -y cryptsetup
[ "$(which nvme)" ] || yum install -y nvme-cli

# Get the encrypted password file from s3
/usr/bin/aws s3 cp s3://${S3_Bucket}/LuksInternalStorageKey .
# Decrypt and store the passphrase in a variable
LuksClearTextKey=$(/usr/bin/aws --region ${REGION} kms decrypt --ciphertext-blob fileb://LuksInternalStorageKey --output text --query Plaintext | base64 --decode)

for ephemeral in $(nvme list | grep dev | awk {'print $1'})
do
DEV_1=$(echo "${ephemeral:9:1}")

[ "$(/bin/mount | grep -i ${ephemeral})" ] && /bin/umount ${ephemeral}

TYPE=`/usr/bin/file -sL ${ephemeral} | awk '{print $2}'`

if [ $TYPE == "LUKS" ]
then
    # Open and initialize the encryped volume
    /bin/echo "$LuksClearTextKey" | /sbin/cryptsetup luksOpen ${ephemeral} encfs_${DEV_1}
    # Check and create mount point if not exists
    [ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
    # Mount the filsystem
    /bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
else
     # Encrypt the volume sub
     /bin/echo "$LuksClearTextKey" | cryptsetup -y luksFormat ${ephemeral}
     # Open and initialize the encryped volume
    /bin/echo "$LuksClearTextKey" | cryptsetup luksOpen ${ephemeral} encfs_${DEV_1}
    # create a filesystem on the encrypted volume, mount it on the required directory
    /sbin/mkfs.ext4 /dev/mapper/encfs_${DEV_1}
    [ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
    /bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
fi
done
# Unset the passphrase variable and remove the encrypted password file.
unset LuksInternalStorageKey
rm LuksInternalStorageKey

Regards,
Jay

How to perform Instance Actions (Instance Stop/Start) using REST API on Oracle Cloud Infrastructure Compute Instances

Hello,

Using REST API to interact with Oracle Cloud Infrastructure is highly desirable for programmatic access to OCI resources from your applications.

For performing instance actions such as STOP, START, RESET and others as mentioned in OCI API documentation [1], you need to pass the parameters as follows rather than via json file. Somehow, I am unable to pass the parameters using json file.

You could create an empty json file as follows and use it on the REST API Call.

 

Load oci-curl function:

# . ./oci-curl

 Create an empty JSON file as we need to use POST method:

# touch empty.json

Syntax:

#oci-curl iaas.us-ashburn-1.oraclecloud.com post ./empty.json "/20160918/instances/?instanceId=&action=STOP"

Refer OCI REST API documentation [2] for more details how to generate signature and authorization strings for accessing API.

 

A copy of the oci-curl script is provided below. You need to update the following parameters in the script though:

local tenancyId="ocid1.tenancy.oc1..aaaaaaaaoumXXXnhkrc72dn6wjs27gq";

local authUserId="ocid1.user.oc1..aaaaaaaa6dmk7XXXhfshe6cacmpt2tjmq";

local keyFingerprint="b3:8f:55:XX:XX:XX:XX:86:a9:be:d4";

local privateKeyPath="/Users/muyself/.oci/oci_api_key.pem";

I am passing the parameters to the rest endpoint after "?". Multiple parameters can be passed using "&".

In the above example, I am passing the 2 required parameters "instanceId" and "action" after ? Separated using &.


Example:

#oci-curl iaas.us-ashburn-1.oraclecloud.com post ./empty.json "/20160918/instances/ocid1.instance.oc1.iad.abuwcljtttj6sgpah77XXXXXXXXXwde5fhdy6apfu5mk7svvgigxq?instanceId=ocid1.instance.oc1.iad.abuwcljtttj6sgpah77XXXXXXXXXwsvvgigxq&action=STOP"

 

Here is the oci-curl script:

# Version: 1.0.2
# Usage:
# oci-curl <host> <method> [file-to-send-as-body] <request-target> [extra-curl-args]
#
# ex:
# oci-curl iaas.us-ashburn-1.oraclecloud.com get "/20160918/instances?compartmentId=some-compartment-ocid"
# oci-curl iaas.us-ashburn-1.oraclecloud.com post ./request.json "/20160918/vcns"
# oci-curl iaas.eu-frankfurt-1.oraclecloud.com post ./empty.json "/20160918/instances/ocid1.instance.oc1.eu-frankfurt-1.antheljt2nxxxxxud53gpggoryh2k?instanceId&action=START"

function oci-curl {
    # TODO: update these values to your own
        local tenancyId="ocid1.tenancy.oc1..aaaaaaaaouminldguXXXX6snhkrc72dn6wjs27gq";
        local authUserId="ocid1.user.oc1..aaaaaaaadmkXXXXXXX2imyg23khy4vuuv6a5aytjmq";
        local keyFingerprint="b3:8f:55:5e:xx:xx:xx:xx4:86:a9:be:d4";
        local privateKeyPath="/Users/myself/.oci/oci_api_key.pem";

    local alg=rsa-sha256
    local sigVersion="1"
    local now="$(LC_ALL=C \date -u "+%a, %d %h %Y %H:%M:%S GMT")"
    local host=$1
    local method=$2
    local extra_args
    local keyId="$tenancyId/$authUserId/$keyFingerprint"

    case $method in

        "get" | "GET")
            local target=$3
            extra_args=("${@: 4}")
            local curl_method="GET";
            local request_method="get";
            ;;

        "delete" | "DELETE")
            local target=$3
            extra_args=("${@: 4}")
            local curl_method="DELETE";
            local request_method="delete";
            ;;

        "head" | "HEAD")
            local target=$3
            extra_args=("--head" "${@: 4}")
            local curl_method="HEAD";
            local request_method="head";
            ;;

        "post" | "POST")
            local body=$3
            local target=$4
            extra_args=("${@: 5}")
            local curl_method="POST";
            local request_method="post";
            local content_sha256="$(openssl dgst -binary -sha256 < $body | openssl enc -e -base64)";
            local content_type="application/json";
            local content_length="$(wc -c < $body | xargs)";
            ;;

        "put" | "PUT")
            local body=$3
            local target=$4
            extra_args=("${@: 5}")
            local curl_method="PUT"
            local request_method="put"
            local content_sha256="$(openssl dgst -binary -sha256 < $body | openssl enc -e -base64)";
            local content_type="application/json";
            local content_length="$(wc -c < $body | xargs)";
            ;;

        *) echo "invalid method"; return;;
    esac

    # This line will url encode all special characters in the request target except "/", "?", "=", and "&", since those characters are used
    # in the request target to indicate path and query string structure. If you need to encode any of "/", "?", "=", or "&", such as when
    # used as part of a path value or query string key or value, you will need to do that yourself in the request target you pass in.
    local escaped_target="$(echo $( rawurlencode "$target" ))"

    local request_target="(request-target): $request_method $escaped_target"
    local date_header="date: $now"
    local host_header="host: $host"
    local content_sha256_header="x-content-sha256: $content_sha256"
    local content_type_header="content-type: $content_type"
    local content_length_header="content-length: $content_length"
    local signing_string="$request_target\n$date_header\n$host_header"
    local headers="(request-target) date host"
    local curl_header_args
    curl_header_args=(-H "$date_header")
    local body_arg
    body_arg=()

    if [ "$curl_method" = "PUT" -o "$curl_method" = "POST" ]; then
        signing_string="$signing_string\n$content_sha256_header\n$content_type_header\n$content_length_header"
        headers=$headers" x-content-sha256 content-type content-length"
        curl_header_args=("${curl_header_args[@]}" -H "$content_sha256_header" -H "$content_type_header" -H "$content_length_header")
        body_arg=(--data-binary @${body})
    fi

    local sig=$(printf '%b' "$signing_string" | \
                openssl dgst -sha256 -sign $privateKeyPath | \
                openssl enc -e -base64 | tr -d '\n')

    curl "${extra_args[@]}" "${body_arg[@]}" -X $curl_method -sS https://${host}${escaped_target} "${curl_header_args[@]}" \
        -H "Authorization: Signature version=\"$sigVersion\",keyId=\"$keyId\",algorithm=\"$alg\",headers=\"${headers}\",signature=\"$sig\""
}

# url encode all special characters except "/", "?", "=", and "&"
function rawurlencode {
  local string="${1}"
  local strlen=${#string}
  local encoded=""
  local pos c o

  for (( pos=0 ; pos<strlen ; pos++ )); do
     c=${string:$pos:1}
     case "$c" in
        [-_.~a-zA-Z0-9] | "/" | "?" | "=" | "&" ) o="${c}" ;;
        * )               printf -v o '%%%02x' "'$c"
     esac
     encoded+="${o}"
  done

  echo "${encoded}"
}

 


You may copy the oci-curl script and load it to the shell. After that you should be able to use REST API methods.    

 

This is an example of how to use REST API.

 

[1] Instance Action API reference: https://docs.cloud.oracle.com/iaas/api/#/en/iaas/20160918/Instance/InstanceAction
[2] OCI REST API usage: https://docs.cloud.oracle.com/iaas/Content/API/Concepts/usingapi.htm
[3] https://docs.cloud.oracle.com/iaas/Content/API/Concepts/signingrequests.htm#Bash


Regards,
Jay.

Set up Secure VNC server in RHEL /OEL / CentOS using SSH tunnel

Hello Guys,

Its been long time since I posted something here. I am trying to make time to write something and make this blog more active.

Lets see how to setup secure VNC server using SSH tunnel in RHEL/OEL/CentOS based Oracle Cloud OCI instances.

As you know, VNC protocol is unencrypted. Even though the log in process has some encryption, it is possible to sniff VNC traffic and collect sensitive infomration. You can fully secure a VNC session by tunnelling it via a SSH tunnel. Another advantage of tunnelling VNC via SSH is that you do not need to open VNC ports – TCP 590X on your Subnet’s Security List. The existing rule for SSH traffic will do fine.

I expect you have desktop environment installed on the instance. By default, VNC is configured to use Xterm as the terminal emulator and twm as the window manager for the X Window System.

RHEL 7/ OEL 7/ CentOS 7 OCI Instances

1.     Install VNC server.

$ sudo yum -y install tigervnc-server pixman pixman-devel libXfont

2.     Setup vnc password for your user.

We are setting up VNC for the default user ‘opc’, if you want to set it up for another user, just change the steps accordingly.

Log in to the user opc and set up VNC password

$ su – opc

$ vncpasswd

3.     Add a VNC Service configuration file.

The VNC daemon configuration file is available in systemd directory below:

$ ls /lib/systemd/system | grep -i vncserver

vncserver@.service

Copy and setup the VNC configuration file.

Backup the configuration file:

$ sudo cp /lib/systemd/system/vncserver@.service  /etc/systemd/system/vncserver@:1.service

     Update/create configuration file as follows:

$ sudo cat /etc/systemd/system/vncserver@\:1.service | egrep -v "^#"

[Unit]

Description=Remote desktop service (VNC)

After=syslog.target network.target

[Service]

Type=forking

ExecStartPre=/bin/sh -c '/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :'

ExecStart=/usr/sbin/runuser -l opc -c "/usr/bin/vncserver %i -geometry 1280x1024 -localhost"

PIDFile=/home/opc/.vnc/%H%i.pid

ExecStop=/bin/sh -c '/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :'

[Install]

WantedBy=multi-user.target

Do remember to change the user name if you are setting VNC for a different user, here we are setting it up for user opc. Also, please note that the parameter “-localhost” makes VNC server to listen on loopback interface and accept connection from a tunnel only.

Make sure your VNC Xstartup file has the below contents:

$ cat /home/opc/.vnc/xstartup
#!/bin/sh
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
#exec /etc/X11/xinit/xinitrc
/bin/gnome-session &

Set permission:
$ chmod 755 /home/opc/.vnc/xstartup

You must reload the systemd system initialization program after setting up the VNC server.

$ systemctl daemon-reload

4.     Start VNC server.

You can now start your VNC server using:

$ sudo systemctl start vncserver@:1 

RHEL 6/ OEL 6/ CentOS 6 OCI Instances

The VNC configuration is different in previous versions on RHEL based systems. You may follow the below steps to set up a secure VNC server.

1.     Install VNC server packages

$ sudo yum install -y vnc-server xorg-x11-fonts-Type1

2.     Update VNC server parameters in /etc/sysconfig/vncservers as follows:

$ cat /etc/sysconfig/vncservers

      VNCSERVERS="1:opc"

      VNCSERVERARGS[1]="-geometry 640x480 -localhost"

3.     Set VNC password

Log in to the user opc and set up VNC password

$ su – opc

$ vncpasswd

4.     Start VNC server

# sudo service vncserver start

Setup your clients to connect to the VNC server.

We have set up the VNC server to be available only via a secure tunnel. As such, we need to create an SSH tunnel from the client to the server before you can access the VNC session.

1.     Setup SSH Tunnel on your client machine

On Mac OS/Linux based clients, you may set up an SSH tunnel as follows:

$ ssh -i /path/to/key -L 5901:localhost:5901 -N -f opc@IP

      On Windows clients, you may set up SSH tunnelling using Putty.

Start putty and under Connection -> SSH -> Tunnels add:

Source port: 5901
Destination: localhost:5901

Then click “Add” to create port forwarding.

And connect to your server at its IP address and port 22 via PuTTY.

 

 Don't forget to click "Add" after updating the forwarded port and destination.


  You may access your instance using VNC using any VNC client using localhost:5901 now.

-->



Hope this helps. Let me know if you have any questions.


Jay
 
-->
-->