Home RAC Related Posts RAC 12.1 Installation on OEL6.5 [VirtualBox]
Oracle Database 12c Release 1 RAC On Oracle Enterprise Linux 6.5 Using VirtualBox
This article describes the detailed step by step installation of Oracle RAC 12cR1 on Oracle Linux 6.5 using Virtual Box. In each seperate section, a different configuration subject is handled. For example in one section the network configuration is done whereas in another the shared disks are configured. I tried to prepare a detailed fool-proof documentation that covers all the subjects required to install a RAC server without refering other documents and directing the user to other websites for some steps. We even install our own DNS server required for the scan listeners. Enjoy!

PRELIMINARY INFORMATION
Required Software
We'll need the following software and tools for this tutorial. Beware of the fact that, you can just download and install any Oracle product without entering any activation keys. But they are not free! You must NOT use any Oracle products for any purpose without a licence. I am usually preparing documents using the real installations after changing the names and configuration details. When I use my own laptop for the installations, I immediately delete them after the installation is successful. The good news is that; there is a free version for the Oracle Database which is fully functional: Oracle Express Edition (XE). You can freely use XE for database curriculum and instruction.

Oracle Enterprise Linux 6.5
Oracle Database Grid Infrastructure 12.1.0.2.0
Oracle Database 12.1.0.2.0
Oracle Virtual Box
Putty
Xming


The Architecture
After all the steps are applied, the architecture would be something like:


The Network Architecture
Interface Network IP Full Name Name Entry
eth0 public 192.168.10.101 rac01.testserver.com rac01 Hosts
eth0 public 192.168.10.102 rac02.testserver.com rac02 Hosts
eth0 virtual 192.168.10.111 rac01-vip.testserver.com rac01-vip Hosts
eth0 virtual 192.168.10.112 rac02-vip.testserver.com rac02-vip Hosts
eth0 scan 192.168.10.121 rac-scan.testserver.com rac-scan DNS
eth0 scan 192.168.10.122 rac-scan.testserver.com rac-scan DNS
eth0 scan 192.168.10.123 rac-scan.testserver.com rac-scan DNS
eth1 private 192.168.20.101 rac01-priv.testserver.com rac01-priv Hosts
eth1 private 192.168.20.102 rac02-priv.testserver.com rac02-priv Hosts
eth0 dns 192.168.10.100 dns.testserver.com dns Hosts
DNS SERVER INSTALLATION
This section is about basic DNS configuration necessary to use the Single Client Access Name (SCAN). I am not a DNS nor Linux exper so the configuration here might not be the more appropriate one, and you may require to read some other sources for only DNS.

Start the Installation








































The following packages should be installed:
Base System > Base
Base System > Compatibility libraries
Base System > Hardware monitoring utilities
Base System > Large Systems Performance
Base System > Network file system client
Base System > Performance Tools
Base System > Perl Support
Servers > Server Platform
Servers > System administration tools
Servers > Network Infrastructure Server > bind* (everything related with bind)
Desktops > Desktop
Desktops > Desktop Platform
Desktops > Fonts
Desktops > General Purpose Desktop
Desktops > Graphical Administration Tools
Desktops > Input Methods
Desktops > X Window System
Applications > Internet Browser
Development > Additional Development
Development > Development Tools





















Network Configuration of the DNS Server









Edit the /etc/hosts file
[root@dns ~]> cat /etc/hosts

127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
192.168.10.100  dns.testserver.com      dns
Stop & Disable Firewall
[root]> service iptables stop
[root]> chkconfig iptables off
Disable SELINUX
[root]> cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
Edit /etc/named.conf
[root]> cat /etc/named.conf

zone "." in {
	type hint;
	file "/dev/null";
};
zone "testserver.com" IN  
{
        type master;
        allow-update { none; };
        file "testserver.com.zone";
};
zone "10.168.192.in-addr.arpa." IN
{
        type master;
        file "10.168.192.zone";
        allow-update { none; };
};
options {
directory "/var/named";
};
Create /var/named/testserver.com.zone file
[root]> cat /var/named/testserver.com.zone

$TTL 86400
@                       IN     SOA    localhost     root.localhost (
                                                        42      ; serial
                                                        3H      ; refresh
                                                        15M     ; retry
                                                        1W      ; expiry
                                                        86400 ) ; minimum
                        IN     NS   localhost
localhost		IN      A   127.0.0.1
rac01			IN      A   192.168.10.101
rac02			IN      A   192.168.10.102
rac01-priv		IN      A   192.168.20.101
rac02-priv		IN      A   192.168.20.102
rac01-vip		IN      A   192.168.10.111
rac02-vip		IN      A   192.168.10.112
rac-scan		IN      A   192.168.10.121
rac-scan		IN      A   192.168.10.122
rac-scan		IN      A   192.168.10.123
Create /var/named/10.168.192.zone file
[root]> cat /var/named/10.168.192.zone

$TTL 1H
@                       IN     SOA    dns.testserver.com     root.dns.testserver.com. (
                                                        2
                                                        3H
                                                        1H
                                                        1W
                                                        1H )
10.168.192.in-addr.arpa.       IN      NS      dns.testserver.com.

101     IN  PTR  rac01.testserver.com.
102     IN  PTR  rac02.testserver.com.
111     IN  PTR  rac01-vip.testserver.com.
112     IN  PTR  rac02-vip.testserver.com.
121     IN  PTR  rac-scan.testserver.com.
122     IN  PTR  rac-scan.testserver.com.
123     IN  PTR  rac-scan.testserver.com.
Edit /etc/resolv.conf file
[root]> cat /etc/resolv.conf

search testserver.com
nameserver 192.168.10.100
Edit network manager file to prevent it overwrite the resolv.conf file.
[root]> service NetworkManager stop
[root]> chkconfig NetworkManager off
Copy zone files
[root]> cp /etc/named.conf /var/named/chroot/etc/
[root]> cp /var/named/testserver.com.zone /var/named/chroot/var/named
[root]> cp /var/named/10.168.192.zone /var/named/chroot/var/named
Fix the ownership
[root]> chown named:named /var/named/chroot/etc -R
[root]> chown named:named /var/named/chroot/var/named -R
Start the named service and enable it to start on startup
[root]> service named start
[root]> chkconfig named on


RAC NODE 1 OS INSTALLATION
Now that our DNS server is ready, we can start the installation of the OS for the main server 1st node. After we install the OS and configure everything for the RAC on this server, we will clone it and create the second node out of this installation. So, let's start the installation:





























You should choose the appropriate keyboard for your system. In my case it is Turkish... Yours might be different.


















The following packages should be installed:
Base System > Base
Base System > Compatibility libraries
Base System > Hardware monitoring utilities
Base System > Large Systems Performance
Base System > Network file system client
Base System > Performance Tools
Base System > Perl Support
Servers > Server Platform
Servers > System administration tools
Desktops > Desktop
Desktops > Desktop Platform
Desktops > Fonts
Desktops > General Purpose Desktop
Desktops > Graphical Administration Tools
Desktops > Input Methods
Desktops > X Window System
Applications > Internet Browser
Development > Additional Development
Development > Development Tools



















Install Guest Additions for a smooth transition of the mouse pointer between the host and the guest OSs. Besides, the resolution of the guest system adjusts itself when you resize the guest OS screen if the Guest Additions are installed:








RAC NODE 1 NETWORK CONFIGURATION

Shutdown the server


Change the current Interface on the Virtual Machine for Public Connection




Create a new Adapter on the Virtual Machine for Private Connection


Configure the Network Properties of the Virtual Server RAC01


PUBLIC NETWORK: Adapter 1


PRIVATE NETWORK: Adapter 2


Configure the Network Properties on the Linux Server

After starting the server, configure the following network properties on the server:

Hostname
Edit the /etc/sysconfig/network file and set the hostname to rac01.testserver.com. (This should already be set)
[root@rac01]> cat /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=rac01.testserver.com
NOZEROCONF=yes

Hosts File
Edit /etc/hosts file. The scan ip entries are there just for info. They are commented.
[root@rac01 Desktop]> cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

##########################################################
##################  RAC Configuration  ###################
##########################################################

#Public
192.168.10.101    rac01.testserver.com       rac01
192.168.10.102    rac02.testserver.com       rac02

#Private
192.168.20.101    rac01-priv.testserver.com  rac01-vip
192.168.20.102    rac02-priv.testserver.com  rac02-vip

#Virtual
192.168.10.111    rac01-vip.testserver.com   rac01-vip
192.168.10.112    rac02-vip.testserver.com   rac02-vip

#Scan
#192.168.10.121   rac-scan.testserver.com    rac-scan
#192.168.10.122   rac-scan.testserver.com    rac-scan
#192.168.10.123   rac-scan.testserver.com    rac-scan

DNS File
Edit /etc/resolv.conf to point to the DNS server for name resolution of scan IPs.
[root@rac01]> cat /etc/resolv.conf 

nameserver 192.168.10.100 
search testserver.com
Edit network manager file to prevent it overwrite the resolv.conf file
[root@rac01]> service NetworkManager stop
[root@rac01]> chkconfig NetworkManager off

ETH0
Edit the configuration script for the relevant interface under the /etc/sysconfig/network-scripts/ path. If named directly after the interface eth0 then this would be called ifcfg-eth0, however other names are possible (including ifcfg-Auto_eth0 if using the Gnome Network Manager). If no configuration script exists then you will need to create one. I prefer configuring using the GUI at this point:



If you want to configure manually, you need to edit the relevant eth configuration file:

Edit /etc/sysconfig/network-scripts/ifcfg-Auto_eth0
[root@rac01 network-scripts]> cat /etc/sysconfig/network-scripts/ifcfg-Auto_eth0
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.10.101
PREFIX=24
GATEWAY=192.168.10.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="Auto eth0"
ONBOOT=yes
HWADDR=08:00:27:52:AC:B2
ETH1
Again, while configuring the eth1, either use the GUI or edit the config file as below:

Edit /etc/sysconfig/network-scripts/ifcfg-Auto_eth1
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.20.101
PREFIX=24
GATEWAY=192.168.20.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="Auto eth1"
ONBOOT=yes
HWADDR=08:00:27:22:44:84

HWADDR address in the above configuration is the MAC address of the virtual adapter:


Restart the network
[root@rac01]> service network restart
Check to see whether the IPs are correct. Even reboot the server and see whether the IP configuration is consistent.
root@rac01 network-scripts]> ifconfig

eth0      Link encap:Ethernet  HWaddr 08:00:27:52:AC:B2
          inet addr:192.168.10.101  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe52:acb2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:346 errors:0 dropped:0 overruns:0 frame:0
          TX packets:380 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:29276 (28.5 KiB)  TX bytes:56245 (54.9 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:22:44:84
          inet addr:192.168.20.101  Bcast:192.168.20.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe22:4484/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1002 (1002.0 b)  TX bytes:938 (938.0 b)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:64 errors:0 dropped:0 overruns:0 frame:0
          TX packets:64 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5068 (4.9 KiB)  TX bytes:5068 (4.9 KiB)


RAC NODE 1 OS CONFIGURATION
Install RPMs
Let’s create our own Yum repository and install the RPMs using that. Yum can connect to internet repository but the servers are not always connected to the internet so I find this method more convenient.

Mount the Linux iso file as a CD/DVD


You should see the image file as below:
[root@rac01 media]> pwd
/media
[root@rac01 media]> ll
total 6
drwxr-xr-x. 12 root root 6144 Nov 26  2013 OL6.5 x86_64 Disc 1 20131125
Create a link for the Yum repository
[root@rac01]> mkdir -p /linuxdvd
[root@rac01]> ln -s /media/OL6.5\ x86_64\ Disc\ 1\ 20131125/Server /linuxdvd
Edit the yum repository file
[root@rac01 linuxdvd]> cat /etc/yum.repos.d/dvd.repo

[oel6.5]
name=Linux 6.5 DVD
baseurl=file:///linuxdvd/Server
gpgcheck=0
enabled=1
Check the repository
[root@rac01 yum.repos.d]> yum repolist

Loaded plugins: refresh-packagekit, security
repo id   repo name                 status
oel6.5    Linux 6.5 DVD             3,669
repolist: 3,669
Install the RPMs manually using the Yum and the local repository
yum install binutils -y
yum install compat-libcap1 -y
yum install compat-libstdc++-33 -y
yum install compat-libstdc++-33.i686 -y
yum install glibc -y
yum install glibc.i686 -y
yum install glibc-devel -y
yum install glibc-devel.i686 -y
yum install ksh -y
yum install libaio -y
yum install libaio.i686 -y
yum install libaio-devel -y
yum install libaio-devel.i686 -y
yum install libX11 -y
yum install libX11.i686 -y
yum install libXau -y
yum install libXau.i686 -y
yum install libXi -y
yum install libXi.i686 -y
yum install libXtst -y
yum install libXtst.i686 -y
yum install libgcc -y
yum install libgcc.i686 -y
yum install libstdc++ -y
yum install libstdc++.i686 -y
yum install libstdc++-devel -y
yum install libstdc++-devel.i686 -y
yum install libxcb -y
yum install libxcb.i686 -y
yum install make -y
yum install nfs-utils -y
yum install net-tools -y
yum install smartmontools -y
yum install sysstat -y
yum install unixODBC -y
yum install unixODBC-devel -y
yum install e2fsprogs -y
yum install e2fsprogs-libs -y
yum install libs -y
yum install libxcb.i686 -y
yum install libxcb -y
yum install gcc -y
yum install gcc-c++ -y
yum install libXext -y
yum install libXext.i686 -y
yum install zlib-devel -y
yum install zlib-devel.i686 -y

Set Kernel Parameters

Add or amend the following lines to the "/etc/sysctl.conf" file. [ROOT]
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
kernel.panic_on_oops = 1
Note: Issue the command "/sbin/sysctl -p" after changing kernel values.


Define Security Limits

Edit /etc/security/limits.conf for oracle and grid users. [ROOT]
oracle          hard    memlock 50000000 
oracle          soft    memlock 50000000  
oracle          soft    core    unlimited 
oracle          hard    core    unlimited 
oracle          soft    nproc   2047 
oracle          hard    nproc   16384 
oracle          soft    nofile  4096 
oracle          hard    nofile  65536 
oracle          soft    stack   10240 
grid            hard    memlock 50000000 
grid            soft    memlock 50000000 
grid            soft    core    unlimited 
grid            hard    core    unlimited 
grid            soft    nproc   2047 
grid            hard    nproc   16384 
grid            soft    nofile  4096 
grid            hard    nofile  65536 
grid            soft    stack   10240
Add the following line to /etc/pam.d/login [ROOT]
session    required     pam_limits.so


Create Users & Groups

USER PRIMARY GROUP SECONDARY GROUPS
grid oinstall asmadmin,asmdba,dba
oracle oinstall dba,asmdba
Create groups and users [ROOT]
[root@rac01]> groupadd oinstall 
[root@rac01]> groupadd asmadmin 
[root@rac01]> groupadd asmdba 
[root@rac01]> groupadd dba 
 
[root@rac01]> useradd -g oinstall -G asmadmin,asmdba,dba grid 
[root@rac01]> useradd -g oinstall -G dba,asmdba oracle
Note: Give passwords to the users after creation.

Create Directories
Create the directories [ROOT]
[root@rac01]> mkdir -p /u01/app/grid 
[root@rac01]> mkdir -p /u01/app/12.1.0.2/grid 
[root@rac01]> mkdir -p /u02/app/oracle 
[root@rac01]> mkdir -p /u02/app/oracle/product/12.1.0.2/db_home 
              
[root@rac01]> chown -R grid:oinstall /u01 
[root@rac01]> chown -R oracle:oinstall /u02 
              
[root@rac01]> chmod -R 775 /u01 
[root@rac01]> chmod -R 775 /u02

Create the Profile Files

ORACLE
Edit the /home/oracle/.bash_profile file and append the following lines (with Oracle user)
# Oracle Settings 

TMP=/tmp;export TMP 
TMPDIR=$TMP;export TMPDIR 

ORACLE_HOSTNAME=rac01.testserver.com;export ORACLE_HOSTNAME 
ORACLE_UNQNAME=RAC;export ORACLE_UNQNAME 
ORACLE_BASE=/u02/app/oracle;export ORACLE_BASE 
ORACLE_HOME=/u02/app/oracle/product/12.1.0.2/db_home;export ORACLE_HOME 
ORACLE_SID=RAC1;export ORACLE_SID 
ORACLE_TERM=xterm;export ORACLE_TERM 
BASE_PATH=/usr/sbin:$PATH;export BASE_PATH 
PATH=$ORACLE_HOME/bin:$BASE_PATH;export PATH 
 
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH 
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib;export CLASSPATH

GRID
Edit the /home/oracle/.bash_profile file and append the following lines (with Grid user)
# Grid Settings 

TMP=/tmp;export TMP 
TMPDIR=$TMP;export TMPDIR 
 
ORACLE_HOSTNAME=rac01.testserver.com;export ORACLE_HOSTNAME 
ORACLE_UNQNAME=RAC;export ORACLE_UNQNAME 
ORACLE_BASE=/u01/app/grid;export ORACLE_BASE 
ORACLE_HOME=/u01/app/12.1.0.2/grid;export ORACLE_HOME 
ORACLE_SID=+ASM1;export ORACLE_SID 
ORACLE_TERM=xterm;export ORACLE_TERM 
BASE_PATH=/usr/sbin:$PATH;export BASE_PATH 
PATH=$ORACLE_HOME/bin:$BASE_PATH;export PATH 

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH 
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib;export CLASSPATH

Deconfigure NTP

[root@rac01]> service ntpd stop 
[root@rac01]> chkconfig ntpd off 
[root@rac01]> mv /etc/ntp.conf /etc/ntp.conf.orig

Setup Files

Create setup directory and copy setup files
[root@rac01 ~]> mkdir -p /setup
Copy the setup files using via an FTP tool


Edit the mod of the files and unzip the grid installaiton zips with the grid user:
[root@rac01 setup]> chmod -R 777 /setup
[root@rac01 setup]> chown grid:oinstall linuxamd64_12102_grid*
[root@rac01 setup]> chown oracle:oinstall linuxamd64_12102_database*

[root@rac01 setup]> su - grid
[grid@rac01 ~]$ cd /setup

[grid@rac01 setup]$ unzip linuxamd64_12102_grid_1of2.zip
[grid@rac01 setup]$ unzip linuxamd64_12102_grid_2of2.zip

[grid@rac01 setup]$ rm -rf linuxamd64_12102_grid*

Install the Cluster Verification RPMs

Go to the path /setup/grid/rpm and install the single rpm package residing in this path. [ROOT]
[root@rac01 ~]> cd /setup/grid/rpm/
[root@rac01 rpm]> rpm -Uvh cvuqdisk-1.0.9-1.rpm


RAC NODE 2 CREATION WITH CLONNING

Clone the Virtual Machine

Clone the Virtual Machine RAC01 to create RAC02 as follows:


Reinitializing is important here since both machines will work on the smae network, we need to differentiate them by creating a new MAC address.




Fix the Network
Since our original server RAC01 has been cloned, all of the configurations were also cloned (except the MAC address -> we chose to reinitialize.) and this new server RAC02 is the exact copy of the original server. Therefore, we need to overwrite those configurations:

Change the hostname as seen below -> edit /etc/sysconfig/network [ROOT]
[root@rac01 Desktop]> cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=rac02.testserver.com
NOZEROCONF=yes

Note down the new MAC addresses of the Node2 for adapter1 and adapter2




Edit the persistent rule file /etc/udev/rules.d/70-persistent-net.rules
[root@rac02 Desktop]> cat /etc/udev/rules.d/70-persistent-net.rules

# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:95:87:b0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:3e:de:a8", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
Edit the /etc/sysconfig/network-scripts/ifcfg-Auto_eth0 file
[root@rac02 network-scripts]> cat /etc/sysconfig/network-scripts/ifcfg-Auto_eth0

TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.10.102
PREFIX=24
GATEWAY=192.168.10.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="Auto eth0"
UUID=f683918d-fcd5-444e-83cf-91152682bb28
ONBOOT=yes
HWADDR=08:00:27:95:87:B0
LAST_CONNECT=1545093190
Edit the /etc/sysconfig/network-scripts/ifcfg-Auto_eth1 file
[root@rac02 network-scripts]> cat /etc/sysconfig/network-scripts/ifcfg-Auto_eth1

TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.20.102
PREFIX=24
GATEWAY=192.168.20.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="Auto eth1"
UUID=71607063-cfdd-4973-a46b-3d6ccb89a08e
ONBOOT=yes
HWADDR=08:00:27:3e:de:a8
LAST_CONNECT=1545076856
Reboot the server and check the Ethernet
[root@rac02 ~]> ifconfig

eth0      Link encap:Ethernet  HWaddr 08:00:27:95:87:B0
          inet addr:192.168.10.102  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe95:87b0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:98 errors:0 dropped:0 overruns:0 frame:0
          TX packets:80 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:10207 (9.9 KiB)  TX bytes:10910 (10.6 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:3E:DE:A8
          inet addr:192.168.20.102  Bcast:192.168.20.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe3e:dea8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:818 (818.0 b)

Correct the Bash Profile Files

GRID
[grid@rac02 ~]$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# Grid Settings

TMP=/tmp;export TMP
TMPDIR=$TMP;export TMPDIR

ORACLE_HOSTNAME=rac02.testserver.com;export ORACLE_HOSTNAME
ORACLE_UNQNAME=RAC;export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/grid;export ORACLE_BASE
ORACLE_HOME=/u01/app/12.1.0.2/grid;export ORACLE_HOME
ORACLE_SID=+ASM2;export ORACLE_SID
ORACLE_TERM=xterm;export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH;export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH;export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib;export CLASSPATH
ORACLE
[oracle@rac02 ~]$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# Oracle Settings

TMP=/tmp;export TMP
TMPDIR=$TMP;export TMPDIR

ORACLE_HOSTNAME=rac02.testserver.com;export ORACLE_HOSTNAME
ORACLE_UNQNAME=RAC;export ORACLE_UNQNAME
ORACLE_BASE=/u02/app/oracle;export ORACLE_BASE
ORACLE_HOME=/u02/app/oracle/product/12.1.0.2/db_home;export ORACLE_HOME
ORACLE_SID=RAC2;export ORACLE_SID
ORACLE_TERM=xterm;export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH;export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH;export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib;export CLASSPATH


CREATING THE SHARED DISKS
The table below shows the minumum disk size requirements for the OCR and VOTING disks. In this tutorial we are going to install 2 nodes and use external redundancy. External redundancy means, for the data redundancy we are going to rely on some other external mechanism which is usually the RAID. Not only for test or demo installations (like this tutorial) but also for production environments, external redundancy is often preferred. This is a very detailed subject including modern storage arrays, sequential SAS vs SSD devices, whether to multiplex some files or not,storage pools etc... which is obviously beyond the scope of this tutorial.

Long story short, we are going to use external redundancy for a 2 node RAC. Therfore, 6GB seems to be sufficient, but just to be on the safe side, I am going to allocate 8GB for only the OCR and VOTING purpose.



Shutdown both nodes and perform the following disk operations on one of the nodes on virtual server gui.

On the RAC01 Settings:










Create the following disks as we did above:
DISK GROUP FILE NAME CAPACITY
+OCRVOTING disk_ocrvoting 8GB
+DATA disk_data_01 10GB
+RECO disk_reco_01 10GB
After all disks are created, we need to make them sharable and then attach to both servers:










Finally, the disks on RAC02 also should look like:


Start either of the machines and log in as root.

Partition the following disks sdb, sdc and sdd but DO NOT format them.
[root@rac01 ~]> ls -la /dev/sd*

brw-rw----. 1 root disk 8,  0 Dec 26 12:42 /dev/sda
brw-rw----. 1 root disk 8,  1 Dec 26 12:42 /dev/sda1
brw-rw----. 1 root disk 8,  2 Dec 26 12:42 /dev/sda2
brw-rw----. 1 root disk 8, 16 Dec 26 12:42 /dev/sdb
brw-rw----. 1 root disk 8, 32 Dec 26 12:42 /dev/sdc
brw-rw----. 1 root disk 8, 48 Dec 26 12:42 /dev/sdd

[root@rac01 ~]> fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xed6f38e7.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1044, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1044, default 1044):
Using default value 1044

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
The sequence of answers is "n", "p", "1", "Return", "Return" and "w".

The disks should look like this
[root@rac01 ~]> ls -la /dev/sd*

brw-rw----. 1 root disk 8,  0 Dec 26 12:42 /dev/sda
brw-rw----. 1 root disk 8,  1 Dec 26 12:42 /dev/sda1
brw-rw----. 1 root disk 8,  2 Dec 26 12:42 /dev/sda2
brw-rw----. 1 root disk 8, 16 Dec 26 13:03 /dev/sdb
brw-rw----. 1 root disk 8, 17 Dec 26 13:03 /dev/sdb1
brw-rw----. 1 root disk 8, 32 Dec 26 13:08 /dev/sdc
brw-rw----. 1 root disk 8, 33 Dec 26 13:08 /dev/sdc1
brw-rw----. 1 root disk 8, 48 Dec 26 13:08 /dev/sdd
brw-rw----. 1 root disk 8, 49 Dec 26 13:08 /dev/sdd1

UDEV Configuration

ON NODE 1

Edit the file /etc/scsi_id.config and add this line:
options=-g
Get the scsi ids of the disks
[root@rac01 ~]> /sbin/scsi_id -g -u -d /dev/sdb
1ATA_VBOX_HARDDISK_VB71725e82-baa54082

[root@rac01 ~]> /sbin/scsi_id -g -u -d /dev/sdc
1ATA_VBOX_HARDDISK_VB105ac7b7-71b8ed97

[root@rac01 ~]> /sbin/scsi_id -g -u -d /dev/sdd
1ATA_VBOX_HARDDISK_VB0805b275-9275035b
Using these values, edit the "/etc/udev/rules.d/99-oracle-asmdevices.rules" file
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB71725e82-baa54082", NAME="asmdisk_ocr",  OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB105ac7b7-71b8ed97", NAME="asmdisk_data", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB0805b275-9275035b", NAME="asmdisk_reco", OWNER="grid", GROUP="oinstall", MODE="0660"
Test -> it should not say something like: unable to open device
[root@rac01 ~]> /sbin/udevadm test /block/sdb/sdb1
[root@rac01 ~]> /sbin/udevadm test /block/sdc/sdc1
[root@rac01 ~]> /sbin/udevadm test /block/sdd/sdd1
Reload the UDEV rules and start UDEV
[root@rac01 ~]> /sbin/udevadm control --reload-rules
[root@rac01 ~]> /sbin/start_udev
The disks should now be visible and have the correct ownership using the following command.
[root@rac01 ~]> ls -al /dev/asm*

brw-rw----. 1 grid oinstall 8, 33 Dec 26 13:48 /dev/asmdisk_data
brw-rw----. 1 grid oinstall 8, 17 Dec 26 13:48 /dev/asmdisk_ocr
brw-rw----. 1 grid oinstall 8, 49 Dec 26 13:48 /dev/asmdisk_reco

ON NODE 2

Start the Node2 and do the following tasks on Node2

Edit the file /etc/scsi_id.config and add this line:
options=-g
Edit the "/etc/udev/rules.d/99-oracle-asmdevices.rules" file
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB71725e82-baa54082", NAME="asmdisk_ocr",  OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB105ac7b7-71b8ed97", NAME="asmdisk_data", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB0805b275-9275035b", NAME="asmdisk_reco", OWNER="grid", GROUP="oinstall", MODE="0660"
Test -> it should not say something like: unable to open device
[root@rac02 ~]> /sbin/udevadm test /block/sdb/sdb1
[root@rac02 ~]> /sbin/udevadm test /block/sdc/sdc1
[root@rac02 ~]> /sbin/udevadm test /block/sdd/sdd1
Reload the UDEV rules and start UDEV
[root@rac01 ~]> /sbin/udevadm control --reload-rules
[root@rac02 ~]> /sbin/start_udev


INSTALL THE GRID SOFTWARE
Provide SSH connectivity [ROOT]
Go to the grid installation files path grid/sshsetup and run the ssh setup script (On RAC01)
[root@rac01]> cd /setup/grid/sshsetup 

[root@rac01 sshsetup]> ./sshUserSetup.sh -user oracle -hosts "rac01 rac02" -advanced -exverify -confirm 
[root@rac01 sshsetup]> ./sshUserSetup.sh -user grid -hosts "rac01 rac02" -advanced -exverify -confirm

Verify the cluster structure before the installation [GRID]
[root@rac01]> su - grid
[root@rac01]> cd /setup/grid 
[root@rac01]> ./runcluvfy.sh stage -pre crsinst  -n rac01,rac02  -verbose > result.txt
If you get the following error:
ERROR:

PRVE-0426 : The size of in-memory file system mounted as /dev/shm is "1002" megabytes which is less than the required size of "2048" megabytes on node ""
PRVE-0426 : The size of in-memory file system mounted as /dev/shm is "1002" megabytes which is less than the required size of "2048" megabytes on node ""
Do the following on both nodes:
[root@rac01]> umount /dev/shm
[root@rac01]> fuser -km /dev/shm
[root@rac01]> umount /dev/shm
Edit the /etc/fstab file and modify the line as below by appending ,size=3G
tmpfs		/dev/shm	tmpfs	defaults,size=3G	0 0
then mount again
[root@rac01]> mount /dev/shm
Other thn that, examine the output file "result.txt", at the end of the file we should see the phrase:
Pre-check for cluster services setup was successful.
[root@rac01]> su - grid 
[root@rac01]> cd /setup/grid/ 
[root@rac01]> ./runInstaller



















































INSTALL THE DATABASE SOFTWARE
Create ASM disks for +DATA and +RECO
[grid@rac01 ~]$ asmca





Create the +RECO the same way we did above for the +DATA.
After the creation of the ASM disks is successful, we can exit the asmca...


Now, we can unzip and start the installation of the database software:
[oracle@rac01 ~]$ cd /setup

[oracle@rac01 setup]$ unzip linuxamd64_12102_database_1of2.zip
[oracle@rac01 setup]$ unzip linuxamd64_12102_database_2of2.zip
[oracle@rac01 setup]$ cd database
[oracle@rac01 database]$ ./runInstaller



















Run the root scripts first on node 1 then on node 2 when prompted:






CREATE A DATABASE

[oracle@rac01 database]$ dbca































Later on, we can use the disk group +RECO when we configure the archivelog.