Mohammed Wasay

Category: Linux

  • Going back to the basics….moving out of Amazon Drive!

    Going back to the basics….moving out of Amazon Drive!

    As of June 8, 2017, it was announced that when when users try to sign up for Amazon Drive they will not be able to select an unlimited cloud storage option. Instead they can choose either 100 GB for $11.99 per year, or 1 TB for $59.99, with up to 30 TB available for an additional $59.99 per TB. (The prior pricing was unlimited everything for $59.99.) My data came up to about 5TB, which according to their new pricing structure would cost me $300+ (Data is always growing!!).

    That is quite costly for just 5TB of storage when I can buy two 8TB drives and have it locally in a RAID configuration or a mirrored set. I shopped around with other popular cloud providers but each and every one of them have some sort of limitations. I decided to purchase two 8TB drives and maintain it locally. 

    I found very little help on Google when searching for ways to move out of Amazon Drive with ease. I found a lot of cool little utilities but none were able to do a clean and consistent sync copy/move. It most cases the application would either hang, or incomplete the job.

    I tried a lot of tools to get a synced local copy but the process seemed harder and harder. I tried a lot of freeware and shareware utilities as well as those offered by Amazon. I am just listing my personal experiences here so that I can save time for those whole have a similar situation.

    Tools I tried:

    Amazon Drive Desktop Sync

    •  Horrible transfer speeds +
    • Buggy Software
    • Startup & Resuming files would delay download significantly.

    SymLink (MacOS/ Linux)

    • Somewhat works but metadata is lost.

    NetDrive

    • Mounts the Amazon Cloud Drive and a Network Drive
    • Constant disconnects + too many app updates
    • Application hangs with large files
    • Service needed to be restarted multiple times to connect with Amazon

    Cloudberry Explorer

    • Quirks around Admin Mode.
    • Ghostfiles (0kb) leftover.
    • Acts like an FTP Client but missing a lot features

    rClone (Banned)

    AllwaySync

    • The Oneway transfer feature is nice but it was taking a long time between files
    • This might have worked if my filebase was a whole lot smaller but failed for larger jobs.

    Expandrive

    • Similar to NetDrive but a whole lot stable, but would fail on larger files.

    Odrive

    • Horrible interface. Didn’t work most of time.

    & a few more applications…. that didn’t work out!

    Syncovery

    Syncovery was the  winner in my case. This tools was the best in speed and got me an exact copy out from Amazon Cloud drive. It supports resuming! It is available on all platforms. It has a nice layout and can run as a scheduled job!

    It took Syncovery literally 2 days to get all of my data downloaded. I was simply amazed at how efficient this tool was working. It maintained a consistent speed. Didn’t lose any metadeta. I ran a file check and all of them checked out 100%.

    The trial version worked in my case and I am considering getting the Pro version. It excelled where all other failed. It wasn’t a resource hog and did the job in the first go! Thank you Syncovery!

     

    Couple of lessons learned in getting success with all my data downloaded.

    1. Metadata is important especially when dealing with older files. Try not of lose it, as once it is lost there is no going back.
    2. Don’t copy to the same path as the original. Use an external drive and copy it there.
    3. If dealing with a lot of smaller files break them into chunks or batches to avoid application hang
    4. Apart from Syncovery, there were some utilities that might delete the files from Amazon and put them in Trash. Make sure you look there if you notice files missing file. It is most certainly there. I personally didn’t have this issue but some people have reported this with other utilities.
    5. Share your experiences to help out others.

    Conclusion

    I am in no way promoting a product from Syncovery, but based on my personal experience I found it to be the easiest to move the amount of data I had from Amazon down to my local server.  I am going to sway away from the public cloud space for a while at-least for my personal stuff. Based on the pricing, limitation of file size and types, and amount of data I have, I am still searching for good cloud store. I am evaluating ownCloud for now. If I ever goto a public cloud storage solution again, I am going to try my exit exercise/ strategy prior to bulk upload.

    Another strategy people are recommending is hosting all the files in a VM on AWS/ Google/ Azure. My issue there is access cost. If my access is within the VM I am good, but any data I am pulling or accessing out of the VM – I am paying for it!

  • How to Configure Time Server (NTP Server) on Centos 7?

    How to Configure Time Server (NTP Server) on Centos 7?

    Prerequisites:

    VM/ Physical – It is recommended to have the time server as a physical server, but VMs should be okay depending on your setup and requirement. My virtual lab environment is VMware based so here’s the guide for a VM. Once the CentOS Minimal is installed on the VM make sure to install open-vm-tools.

    yum install open-vm-tools

    If you are not conformable with Vi Editor please use install nano for editing.

    yum install nano

    Installing the time server:

    1. First we need to install NTPd . to do this open command line : 
      [root@mytimesrv01 ~]# yum -y install ntp
    2. Then open NTP main configuration file for editing:[su_note note_color=”#fafae8″]If you are planning on hosting your time servers externally like time01.yourdomain.com, time02.yourdomain.com with ports 123 open on the firewall. Each of the time server will be talking to 0.us.pool.ntp.org, 0.us.pool.ntp.org, 0.us.pool.ntp.org, 0.us.pool.ntp.org. Later on you can probably add your time servers to the NTP.org by joining the pool. For my example, I am not using the centos default pool, but i am using the U.S. zone. For internal servers/workstations you can point it to the local ip/dns name for the local time servers.[/su_note] 
      [root@mytimesrv01 ~]# vi /etc/ntp.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).

#Your time servers go here:
server 0.us.pool.ntp.org iburst
server 1.us.pool.ntp.org iburst
server 2.us.pool.ntp.org iburst
server 3.us.pool.ntp.org iburst

#Default OOB time servers
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats

# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor

logfile /var/log/ntp.log
    3. Then start and enable ntp server: 
      [root@mytimesrv01 ~]# systemctl start ntpd
[root@mytimesrv01 ~]# systemctl enable ntpd
    4. You need to allow ntp service on firewall(NTP service uses UDP port 123). So run next command: 
      [root@mytimesrv01 ~]# firewall-cmd –add-service=ntp –permanent
success
[root@mytimesrv01 ~]# firewall-cmd –reload
success
    5. Verify you ntp service with next command: 
      [root@mytimesrv01 ~]# ntpq -p

    Your output should be similar to:

  • How to restart management agents on ESX or ESXi host

    How to restart management agents on ESX or ESXi host

    If you are not unable to connect ESXi server to vCenter, or when you cannot connect to ESXi server from VI client it may be necessary to restart the management agents on ESX or ESXi host.

    To restart the management agents on ESXi 6.x

    This applies to ESX4/5.x/6.x

    For the restart of the management agents (mgmt-vmware and vmware-vpxa) do the following:

    Log in to SSH or Local console as root.
    Run these commands:

    /etc/init.d/hostd restart
/etc/init.d/vpxa restart

    Or also (alternative way)
    To reset the management network on a specific VMkernel interface, by default vmk0, run the command:

    esxcli network ip interface set -e false -i vmk0; esxcli network ip interface set -e true -i vmk0

    [su_note note_color=”#fafae8″]Note: Using a semicolon (;) between the two commands ensures the VMkernel interface is disabled and then re-enabled in succession. If the management interface is not running on vmk0, change the above command according to the VMkernel interface used.[/su_note]

    to restart all management agents on the host, run the command:

    services.sh restart

    To restart the Management agents on ESXi Server – via the console:

    1.) Connect to the console of your ESX Server and press F2
    2.) Login as root and when using the Up/Down arrows navigate to Restart Management Agents.
    3.) Press Enter and press F11 to restart the services.
    4.) When the service has been restarted, press Enter. Then you can press Esc to logout of the system.

    Screen should be similar to:

    To restart the management agents on ESXi 4.x and 5.x:

    From Local Console or SSH:
    1. Log in to SSH or Local console as root.
    2. Run this command:
    /sbin/services.sh restart

    You can also check:  Service mgmt-vmware restart may not restart hostd (1005566).

    To restart the management agents on ESX Server 3.x, ESX 4.x:

    1. Login to your ESX Server as root from SSH session or directly from the console.
    2. Type service mgmt-vmware restart  and press Enter
      Make sure that automatic Startup/Shutdown of virtual machines is disabled before running this command otherwise you might reboot the virtual machines. See more at 103312
    3. Type service vmware-vpxa restart and press Enter.
    4. Type logout and press Enter to disconnect from the ESX Server.

    Successful output :

    Stopping vmware-vpxa: [ OK ]
Starting vmware-vpxa: [ OK ]

    This may also server as a solution for the error “Unable to access file since it is locked. An error occurred while consolidating disks: One or more disks are busy.”

  • Linux: CentOS 6 kernel panic+0xa7/0x16f fix

    Linux: CentOS 6 kernel panic+0xa7/0x16f fix

    So ran into a view issue for a Linux Host today. Kernel Panic !!!

    There could be a number of reasons that you might get this error. The three common reasons may be:

    1. There was a change to selinux config – (that might be an error in the config file)
    2. There are disk errors
    3. The kernel is corrupted

    None of the kernels in the GRUB boot menu will boot, you get a kernel panic:

    1-kernel-panic

    Kernel panic – not syncing: Attempted to kill init!
    Pid: 1, comm: init Not tainted 2.6.32-504.3.3.el6.x86_64 #1
    panic+0xa7/0x16f
    do_exit+0x862/0x870
    fput+0x25/0x30
    do_group_exit+0x58/0xd0
    sys_exit_group+0x17/0x20
    system_call_fastpath+0x16/0x1b

    Booting in single user mode doesn’t work either.

    Here’s how to fix this:

              1. Reboot, and go in the GRUB menu. You have 3 seconds to strike the arrow keys before it will automatically boot the default kernel.2-grub-menu
              2. Select the first line, the default kernel, and press the E key on the keyboard to edit the parameters. You will then see the following.
                3-grub-edit
              3. Use the arrow keys to select the 2nd line, that starts with kernel. Press the E key to change this line, use the arrow keys to go to the end and type a space followed by enforcing=0
                4-grub-edit
              4. Press enter to conform and then press B to boot the system.
                It should boot up fine now.

                5. Choose your solution:

                [su_tabs vertical=”yes”]
                [su_tab title=”SELINUX Config File”]

                Disabled for the SELINUXTYPE variable, that’s wrong it has to be for the SELINUX variable. In the screenshot below I show you the correct settings in the config file to disable SELINUX.

                5-selinux-disabled

                [/su_tab]
                [su_tab title=”Disk Errors”]

                fsck -a /dev/sda1

                [/su_tab]

                [su_tab title=”Rebuild Kernel”]

                yum remove kernel
                yum-complete-transaction
                yum update

                [/su_tab]
                [/su_tabs]

              5. Reboot.

  • “Host SMBus controller not enabled!”

    CentOS guest instances in VMware sometimes come up with the boot error message:

    piix4_smbus 0000:00:007.3: Host SMBus controller not enabled!

    This error is being caused because VMware doesn’t actually provide that level interface for CPU access, but Ubuntu try to load the kernel module anyway.

    How to fix it:

    sudo nano /etc/modprobe.d/blacklist.conf

    add the line:

    blacklist i2c-piix4

    Reboot.

    NOTE: for older versions use blacklist i2c_piix4 instead.
    NOTE: it works both in VMWare Fusion 5 and 6, and Ubuntu LTS 12.04 and 14.04

  • How to Start and Enable Firewalld on CentOS 7

    It is highly recommended that you have a firewall protecting your server.

    Pre-Flight Check

    • These instructions are intended specifically for enabling and starting firewalld CentOS 7.
    • I’ll be working from a Liquid Web Self Managed CentOS 7 server, and I’ll be logged in as root.

    Check the Status of Firewalld

    To check the status of firewalld, run the following command as root:

    systemctl status firewalld

    Enable Firewalld

    To enable firewalld, run the following command as root:

    systemctl enable firewalld

    Start Firewalld

    To start firewalld, run the following command as root:

    systemctl start firewalld

  • How to check status, stop, and disable firewall on CentOS 7

    Firewalld is a complete firewall solution that has been made available by default on all CentOS 7 servers, including Liquid Web Core Managed CentOS 7, and Liquid Web Self Managed CentOS 7. On occasion, perhaps for testing, disabling or stopping firewalld may be necessary. Follow the instructions below to disable firewalld and stop firewalld.

    It is highly recommended that you have another firewall protecting your network or server before, or immediately after, disabling firewalld.

    Pre-Flight Check

    • These instructions are intended specifically for stopping and disabling firewalld CentOS 7.
    • I’ll be working from a Liquid Web Self Managed CentOS 7 server, and I’ll be logged in as root.

    Check the Status of Firewalld

    And finally, to check the status of firewalld, run the following command as root:

    systemctl status firewalld

    Stop Firewalld

    To stop firewalld, run the following command as root:

    systemctl stop firewalld

    Disable Firewalld

    To disable firewalld, run the following command as root:

    systemctl disable firewalld

  • How to Install VMware Tools on RHEL 7/CentOS 7

    VMware Tools is one of important components for virtual machine (VM) in order get excellent performance. It is a group of utilities that help to enhances the overall performance of the virtual machine’s guest operating system (OS) and improves management of the VM. Without the VMware Tools, guest OS performance will lacks some of the important functionality. Below steps shows how to install the VMware Tools on RHEL 7, CentOS 7 and Oracle Linux 7.

    How to Install VMware Tools ?

    Option 1 (Recommended by VMware) – VMware support of open-vm-tools

    open-vm-tools is an open source implementation of VMware Tools from third parties and contains utilities that enhances virtualization management, administration and functions of the virtual machine in VMware environments. The ultimate goal is to enable the operating system vendors and/or communities and virtual appliance vendors to bundle VMware Tools into their product releases.

    # yum install open-vm-tools

    Option 2 :

    1. Install prerequisites :

    # yum install perl gcc make kernel-headers kernel-devel -y

    2. Attach the vmware tools sofware from vSphere client.

    3. Mount the vmware tools package into /mnt :

    # <strong>mount /dev/cdrom /mnt</strong>
mount: /dev/sr0 is write-protected, mounting read-only

    4. Copy vmware tool packagae into /tmp :

    # cd /mnt

    # ls
manifest.txt     VMwareTools-9.4.0-1280544.tar.gz  vmware-tools-upgrader-64
run_upgrader.sh  vmware-tools-upgrader-32
# <strong>cp -p VMwareTools-9.4.0-1280544.tar.gz /tmp
</strong>

    5. Go to /tmp directory and extract the vmware tools package :

    # cd /tmp
# <strong>tar xzvf VMwareTools-9.4.0-1280544.tar.gz
</strong>

    6. Go to extracted folder, vmware-tools-distrib :

    # <strong>cd vmware-tools-distrib</strong>
# ls
bin  doc  etc  FILES  INSTALL  installer  lib  vmware-install.pl

    7. Run vmware-install.pl to start installation :

    # <strong>./vmware-install.pl</strong>

    Example:

    # ./vmware-install.pl
Creating a new VMware Tools installer database using the tar4 format.

Installing VMware Tools.

In which directory do you want to install the binary files?
[/usr/bin]

What is the directory that contains the init directories (rc0.d/ to rc6.d/)?
[/etc/rc.d]

What is the directory that contains the init scripts?
[/etc/rc.d/init.d]

In which directory do you want to install the daemon files?
[/usr/sbin]

In which directory do you want to install the library files?
[/usr/lib/vmware-tools]

The path "/usr/lib/vmware-tools" does not exist currently. This program is
going to create it, including needed parent directories. Is this what you want?
[yes]

In which directory do you want to install the documentation files?
[/usr/share/doc/vmware-tools]

The path "/usr/share/doc/vmware-tools" does not exist currently. This program
is going to create it, including needed parent directories. Is this what you
want? [yes]

The installation of VMware Tools 9.4.0 build-1280544 for Linux completed
successfully. You can decide to remove this software from your system at any
time by invoking the following command: "/usr/bin/vmware-uninstall-tools.pl".

Before running VMware Tools for the first time, you need to configure it by
invoking the following command: "/usr/bin/vmware-config-tools.pl". Do you want
this program to invoke the command for you now? [yes]

Initializing...


Making sure services for VMware Tools are stopped.

Stopping vmware-tools (via systemctl):                     [  OK  ]


The module vmci has already been installed on this system by another installer
or package and will not be modified by this installer.

The module vsock has already been installed on this system by another installer
or package and will not be modified by this installer.

The module vmxnet3 has already been installed on this system by another
installer or package and will not be modified by this installer.

The module pvscsi has already been installed on this system by another
installer or package and will not be modified by this installer.

The module vmmemctl has already been installed on this system by another
installer or package and will not be modified by this installer.

The VMware Host-Guest Filesystem allows for shared folders between the host OS
and the guest OS in a Fusion or Workstation virtual environment.  Do you wish
to enable this feature? [no]

The vmxnet driver is no longer supported on kernels 3.3 and greater. Please
upgrade to a newer virtual NIC. (e.g., vmxnet3 or e1000e)

The vmblock enables dragging or copying files between host and guest in a
Fusion or Workstation virtual environment.  Do you wish to enable this feature?
[no]

VMware automatic kernel modules enables automatic building and installation of
VMware kernel modules at boot that are not already present. This feature can be

enabled/disabled by re-running vmware-config-tools.pl.

Would you like to enable VMware automatic kernel modules?
[no]

No X install found.

Creating a new initrd boot image for the kernel.
Starting vmware-tools (via systemctl):                     [  OK  ]
The configuration of VMware Tools 9.4.0 build-1280544 for Linux for this
running kernel completed successfully.

You must restart your X session before any mouse or graphics changes take
effect.

You can now run VMware Tools by invoking "/usr/bin/vmware-toolbox-cmd" from the
command line.

To enable advanced X features (e.g., guest resolution fit, drag and drop, and
file and text copy/paste), you will need to do one (or more) of the following:
1. Manually start /usr/bin/vmware-user
2. Log out and log back into your desktop session; and,
3. Restart your X session.

Enjoy,

--the VMware team

    8. Once successfully installed, make sure you umount (not unmount- missing the n) back the /mnt :

    # umount /mnt

  • How to configure static ip address on CentOS 7

    In this small tutorial i will explain how to configure static ip address on CentOS 7 minimal.
    First, need to edit the set up for the ethernet. Let’s start with editing “/etc/sysconfig/network-scripts/ifcfg-enp0s3″ file:

    vi /etc/sysconfig/network-scripts/ifcfg-enp0s3
    HWADDR=08:00:27:6C:FF:91
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPADDR=192.168.0.88
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp0s3
UUID=ea68db6e-461e-427d-b9a8-bfcf6e1a4fc6
ONBOOT=yes

    Save and exit.

    Now, configure default getaway:

    vi /etc/sysconfig/network

    And add following line:

    NETWORKING=yes
HOSTNAME=centos7
GATEWAY=10.1.1.253

    Configure DNS Server

    vi /etc/resolv.conf

    add following line:

    nameserver 8.8.8.8
nameserver 8.8.4.4

    Now restart your network or rebooting system

    /etc/init.d/network restart

    Check your network:

    ip addr
    1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:6c:ff:91 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.88/24 brd 192.168.0.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe6c:ff91/64 scope link 
       valid_lft forever preferred_lft forever

  • How to setup network after RHEL/CentOS 7 minimal installation

    After installing RHEL/CentOS 7 minimal, You may not able to connect network in that machine. This will happen because Ethernet interfaces are not enabled by default.
    This guide explain you to setup network on RHEL/CentOS 7.

    Setup network on CentOS 7 minimal

    First, type “nmcli d” command in your terminal for quick list ethernet card installed on your machine:

    CentOS_7-network-setup

    “nmcli d” command output

    Type “nmtui” command in your terminal to open Network manager. After opening Network manager chose “Edit connection” and press Enter (Use TAB button for choosing options).

    CentOS_7-Network-manager-screen

    CentOS_7 Network manager screen

    Now choose you network interfaces and click “Edit”

    Edit-your-network-interfaces

     

    Edit your network interfaces

    DHCP configuration

    Choose “Automatic” in IPv4 CONFIGURATION and check Automatically connect check box and press OK and quit from Network manager.

    Set-ip-adress-using-DHCP

    Set ip adress using DHCP

    Reset network services:

    service network restart

    Now your server will get IP Address from DHCP .

    CentOS-7-check-ip-address

    CentOS 7 check ip address.