Movimentando Voting Disk e OCR - RAC 10g

Olá jovens.

Hoje vou passar uma dica simples, mas bem útil para movimentar os voting disks e ocr's do seu RAC. 

Sem muita enrolação hoje, vamos ao que interessa.

Primeira parte - Preparação dos novos discos montados via OCFS

1.) Login as root
su -
  
2) Caso necessário, fazer o FDISK nos discos apresentados
ls -lrht /dev/mapper/VV_ORACLE_CRS*

fdisk /dev/mapper/VV_ORACLE_CRS0
n,p,1,enter,enter,w

Se necessário, no nó 2:
partprobe /dev/mapper/VV_ORACLE_CRS0

[+ASM1.zema-ora10g-n1 ~]$ ls -lrht /dev/mapper/VV_ORACLE_CRS?*
brw-rw---- 1 root disk 253, 24 May  8 18:02 /dev/mapper/VV_ORACLE_CRS0
brw-rw---- 1 root disk 253, 25 May  8 18:03 /dev/mapper/VV_ORACLE_CRS1
brw-rw---- 1 root disk 253, 26 May  8 18:03 /dev/mapper/VV_ORACLE_CRS2
brw-rw---- 1 root disk 253, 27 May  8 18:03 /dev/mapper/VV_ORACLE_CRS3
brw-rw---- 1 root disk 253, 28 May  8 18:03 /dev/mapper/VV_ORACLE_CRS4
brw-rw---- 1 root disk 253, 38 May  8 18:04 /dev/mapper/VV_ORACLE_CRS0p1
brw-rw---- 1 root disk 253, 39 May  8 18:04 /dev/mapper/VV_ORACLE_CRS1p1
brw-rw---- 1 root disk 253, 40 May  8 18:04 /dev/mapper/VV_ORACLE_CRS2p1
brw-rw---- 1 root disk 253, 41 May  8 18:04 /dev/mapper/VV_ORACLE_CRS3p1
brw-rw---- 1 root disk 253, 42 May  8 18:04 /dev/mapper/VV_ORACLE_CRS4p1

3) Verificar no /etc/rc.local se estes discos já fazem parte das regras de direitos (nos dois nós)

4) Criar os novos diretórios onde serão montados os discos (nos dois nós)
mkdir /ocfs2_ocr11 /ocfs2_ocr22 /ocfs2_vdsk11 /ocfs2_vdsk22 /ocfs2_vdsk33
  
chown oracle:oinstall /ocfs2_ocr11 /ocfs2_ocr22 /ocfs2_vdsk11 /ocfs2_vdsk22 /ocfs2_vdsk33
  
5) Realizar o mkfs.ocfs2 para formatar os novos discos e montar
mkfs.ocfs2 -b 4k -C 32K -N 4 -L /ocfs2_ocr11  /dev/mapper/VV_ORACLE_CRS0p1
mkfs.ocfs2 -b 4k -C 32K -N 4 -L /ocfs2_ocr22  /dev/mapper/VV_ORACLE_CRS1p1
mkfs.ocfs2 -b 4k -C 32K -N 4 -L /ocfs2_vdsk11 /dev/mapper/VV_ORACLE_CRS2p1
mkfs.ocfs2 -b 4k -C 32K -N 4 -L /ocfs2_vdsk22 /dev/mapper/VV_ORACLE_CRS3p1
mkfs.ocfs2 -b 4k -C 32K -N 4 -L /ocfs2_vdsk33 /dev/mapper/VV_ORACLE_CRS4p1
  
mount -t ocfs2 -o datavolume,nointr /dev/mapper/VV_ORACLE_CRS0p1 /ocfs2_ocr11
mount -t ocfs2 -o datavolume,nointr /dev/mapper/VV_ORACLE_CRS1p1 /ocfs2_ocr22
mount -t ocfs2 -o datavolume,nointr /dev/mapper/VV_ORACLE_CRS2p1 /ocfs2_vdsk11
mount -t ocfs2 -o datavolume,nointr /dev/mapper/VV_ORACLE_CRS3p1 /ocfs2_vdsk22
mount -t ocfs2 -o datavolume,nointr /dev/mapper/VV_ORACLE_CRS4p1 /ocfs2_vdsk33
  
6) Acrescentar as montagens no /etc/fstab (nos dois nós)
/dev/mapper/VV_ORACLE_CRS0p1 /ocfs2_ocr11       ocfs2  datavolume,nointr 0 0
/dev/mapper/VV_ORACLE_CRS1p1 /ocfs2_ocr22       ocfs2  datavolume,nointr 0 0
/dev/mapper/VV_ORACLE_CRS2p1 /ocfs2_vdsk11      ocfs2  datavolume,nointr 0 0
/dev/mapper/VV_ORACLE_CRS3p1 /ocfs2_vdsk22      ocfs2  datavolume,nointr 0 0
/dev/mapper/VV_ORACLE_CRS4p1 /ocfs2_vdsk33      ocfs2  datavolume,nointr 0 0

7) Pronto, é só verificar os discos.
df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VV_ORACLE_CRS0p1 1020M   86M  934M   9% /ocfs2_ocr11
/dev/mapper/VV_ORACLE_CRS1p1 1020M   86M  934M   9% /ocfs2_ocr22
/dev/mapper/VV_ORACLE_CRS2p1 1020M   86M  934M   9% /ocfs2_vdsk11
/dev/mapper/VV_ORACLE_CRS3p1 1020M   86M  934M   9% /ocfs2_vdsk22
/dev/mapper/VV_ORACLE_CRS4p1 1020M   86M  934M   9% /ocfs2_vdsk33


Segunda parte - Movimentação

1) Login as root
su -

2) Verificações
  
cd $GRID_HOME/bin            (/oracle/app/oracle/product/10.2.0/crs/bin/)
-- OCR's
./ocrcheck

Status of Oracle Cluster Registry is as follows :
     Version                  :          2
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       9504
     Available space (kbytes) :     252616
     ID                       : 1593614163
     Device/File Name         : /ocfs2_ocr1/ocr01.ocr
                                Device/File integrity check succeeded
     Device/File Name         : /ocfs2_ocr2/ocr02.ocr
                                Device/File integrity check succeeded

     Cluster registry integrity check succeeded


-- VD
./crsctl query css votedisk
    0.     0    /ocfs2_vdsk1/vdisk01.crs
    1.     0    /ocfs2_vdsk2/vdisk02.crs
    2.     0    /ocfs2_vdsk3/vdisk03.crs

-- Backup's
./ocrconfig -showbackup

israc01  2014/05/16 20:19:08 /oracle/app/oracle/oracle/product/10.2.0/crs/cdata/crs
israc01  2014/05/16 16:19:08 /oracle/app/oracle/oracle/product/10.2.0/crs/cdata/crs
israc01  2014/05/16 12:19:08 /oracle/app/oracle/oracle/product/10.2.0/crs/cdata/crs
israc01  2014/05/15 00:19:07 /oracle/app/oracle/oracle/product/10.2.0/crs/cdata/crs
israc01  2014/05/16 00:19:08 /oracle/app/oracle/oracle/product/10.2.0/crs/cdata/crs

  
-- Executa o backup manual
./ocrconfig -export /oracle/app/oracle/product/10.2.0/crs/cdata/crs/ocr_20140518_1300 -s online

dd if=/ocfs2_vdsk1/vdisk01.ocr of=/oracle/app/oracle/product/10.2.0/crs/cdata/crs/vdisk01.crs.bak bs=4k
2500+0 records in
2500+0 records out
10240000 bytes (10 MB) copied, 0.079659 seconds, 129 MB/s

dd if=/ocfs2_vdsk2/vdisk02.ocr of=/oracle/app/oracle/product/10.2.0/crs/cdata/crs/vdisk02.crs.bak bs=4k
2500+0 records in
2500+0 records out
10240000 bytes (10 MB) copied, 0.088987 seconds, 115 MB/s

dd if=/ocfs2_vdsk3/vdisk03.ocr of=/oracle/app/oracle/product/10.2.0/crs/cdata/crs/vdisk03.crs.bak bs=4k
2500+0 records in
2500+0 records out
10240000 bytes (10 MB) copied, 0.103655 seconds, 98.8 MB/s

[root@zema-ora10g-n1 bin]# ls -lrht /oracle/app/oracle/product/10.2.0/crs/cdata/crs
total 96M
-rw-r--r-- 1 root root 9.5M May 11 06:44 week.ocr
-rw-r--r-- 1 root root 9.5M May 17 06:44 day.ocr
-rw-r--r-- 1 root root 9.5M May 18 02:44 backup02.ocr
-rw-r--r-- 1 root root 9.5M May 18 06:44 week_.ocr
-rw-r--r-- 1 root root 9.5M May 18 06:44 day_.ocr
-rw-r--r-- 1 root root 9.5M May 18 06:44 backup01.ocr
-rw-r--r-- 1 root root 9.5M May 18 10:44 backup00.ocr
-rw-r--r-- 1 root root 223K May 18 13:04 ocr_20140518_1300
-rw-r--r-- 1 root root 9.8M May 18 13:06 vdisk01.crs.bak
-rw-r--r-- 1 root root 9.8M May 18 13:06 vdisk02.crs.bak
-rw-r--r-- 1 root root 9.8M May 18 13:06 vdisk03.crs.bak

[prdmv1.israc01 ~]$ ~/crstat.sh
HA Resource                                   Target     State
-----------                                   ------     -----
ora.zema-ora10g-n1.ASM1.asm                   ONLINE     ONLINE on zema-ora10g-n1
ora.zema-ora10g-n1.LISTENER_ZEMA-ORA10G-N1.lsnr ONLINE     ONLINE on zema-ora10g-n1
ora.zema-ora10g-n1.gsd                        ONLINE     ONLINE on zema-ora10g-n1
ora.zema-ora10g-n1.ons                        ONLINE     ONLINE on zema-ora10g-n1
ora.zema-ora10g-n1.vip                        ONLINE     ONLINE on zema-ora10g-n1
ora.zema-ora10g-n2.ASM2.asm                   ONLINE     ONLINE on zema-ora10g-n2
ora.zema-ora10g-n2.LISTENER_ZEMA-ORA10G-N2.lsnr ONLINE     ONLINE on zema-ora10g-n2
ora.zema-ora10g-n2.gsd                        ONLINE     ONLINE on zema-ora10g-n2
ora.zema-ora10g-n2.ons                        ONLINE     ONLINE on zema-ora10g-n2
ora.zema-ora10g-n2.vip                        ONLINE     ONLINE on zema-ora10g-n2
ora.bdprd.bdprd1.inst                         ONLINE     ONLINE on zema-ora10g-n1
ora.bdprd.bdprd2.inst                         ONLINE     ONLINE on zema-ora10g-n2
ora.bdprd.db                                  ONLINE     ONLINE on zema-ora10g-n1
ora.bdsml.bdsml1.inst                         ONLINE     ONLINE on zema-ora10g-n1
ora.bdsml.bdsml2.inst                         ONLINE     ONLINE on zema-ora10g-n2
ora.bdsml.db                                  ONLINE     ONLINE on zema-ora10g-n1
ora.bdtrn.bdtrn1.inst                         ONLINE     ONLINE on zema-ora10g-n1
ora.bdtrn.bdtrn2.inst                         ONLINE     ONLINE on zema-ora10g-n2
ora.bdtrn.db                                  ONLINE     ONLINE on zema-ora10g-n2
  
Importante: Verificar se não existe SPFILE ou qualquer outro arquivo nas áreas que não serão mais utilizadas. Se houver, alterar.
   
3) Parar todos os serviços do RAC (nos dois nós)
  
-- Nó 2
/etc/init.d/init.crs stop
    Shutting down Oracle Cluster Ready Services (CRS):
    May 18 13:09:03.653 | INF | daemon shutting down
    Stopping resources. This could take several minutes.
    Successfully stopped CRS resources.
    Stopping CSSD.
    Shutting down CSS daemon.
    Shutdown request successfully issued.
    Shutdown has begun. The daemons should exit soon.

-- Nó 1
/etc/init.d/init.crs stop
    Shutting down Oracle Cluster Ready Services (CRS):
    May 18 13:10:04.123 | INF | daemon shutting down
    Stopping resources. This could take several minutes.
    Successfully stopped CRS resources.
    Stopping CSSD.
    Shutting down CSS daemon.
    Shutdown request successfully issued.
    Shutdown has begun. The daemons should exit soon.

--Verificação
./crsctl check crs
    Failure 1 contacting CSS daemon
    Cannot communicate with CRS
    Cannot communicate with EVM

4) Cria os novos VD's para as novas localizações
  
./crsctl add css votedisk /ocfs2_vdsk11/vdisk011.crs -force
    Now formatting voting disk: /ocfs2_vdsk11/vdisk011.crs
    successful addition of votedisk /ocfs2_vdsk11/vdisk011.crs .

./crsctl add css votedisk /ocfs2_vdsk22/vdisk022.crs -force
    Now formatting voting disk: /ocfs2_vdsk22/vdisk022.crs
    successful addition of votedisk /ocfs2_vdsk22/vdisk022.crs .

./crsctl add css votedisk /ocfs2_vdsk33/vdisk033.crs -force
    Now formatting voting disk: /ocfs2_vdsk33/vdisk033.crs
    successful addition of votedisk /ocfs2_vdsk33/vdisk033.crs .
  
Importante: Não esquecer de verificar as permissões dos arquivos criados:
    chown oracle:oinstall /ocfs2_vdsk11/vdisk011.crs
    chown oracle:oinstall /ocfs2_vdsk22/vdisk022.crs
    chown oracle:oinstall /ocfs2_vdsk33/vdisk033.crs
  
  
ls -lrht /ocfs2_vdsk11
total 9.8M
drwxr-xr-x 2 root   root     3.9K May 17 20:47 lost+found
-rw-r--r-- 1 oracle oinstall 9.8M May 18 13:29 vdisk011.crs

ls -lrht /ocfs2_vdsk22
total 9.8M
drwxr-xr-x 2 root   root     3.9K May 17 20:47 lost+found
-rw-r--r-- 1 oracle oinstall 9.8M May 18 13:29 vdisk022.crs

ls -lrht /ocfs2_vdsk33
total 9.8M
drwxr-xr-x 2 root   root     3.9K May 17 20:47 lost+found
-rw-r--r-- 1 oracle oinstall 9.8M May 18 13:31 vdisk033.crs
  
5)     Verifica se a alteração foi realizada
./crsctl query css votedisk

0.     0    /ocfs2_vdsk1/vdisk01.crs
1.     0    /ocfs2_vdsk2/vdisk02.crs
2.     0    /ocfs2_vdsk3/vdisk03.crs
3.     0    /ocfs2_vdsk11/vdisk011.crs            -- Novo
4.     0    /ocfs2_vdsk22/vdisk022.crs            -- Novo
5.     0    /ocfs2_vdsk33/vdisk033.crs            -- Novo

6) Exclui os VD's antigos e verifica novamente

./crsctl delete css votedisk /ocfs2_vdsk1/vdisk01.ocr -force
    successful deletion of votedisk /ocfs2_vdsk1/vdisk01.ocr.

./crsctl delete css votedisk /ocfs2_vdsk2/vdisk02.ocr -force
    successful deletion of votedisk /ocfs2_vdsk2/vdisk02.ocr.

./crsctl delete css votedisk /ocfs2_vdsk3/vdisk03.ocr -force
    successful deletion of votedisk /ocfs3_vdsk1/vdisk03.ocr.

      
./crsctl query css votedisk
    0.     0    /ocfs2_vdsk11/vdisk011.crs
    1.     0    /ocfs2_vdsk22/vdisk022.crs
    2.     0    /ocfs2_vdsk33/vdisk033.crs

7) Agora vamos movimentar os OCR's. Para isso, iniciar o CRS nos dois nós
./crsctl start crs
  
-- Checar se subiu
./crsctl check crs
    CSS appears healthy
    CRS appears healthy
    EVM appears healthy

8) Criar os novos OCR's:

touch /ocfs2_ocr11/ocrs011.crs
touch /ocfs2_ocr22/ocrs022.crs
  
Importante: Não esquecer de deixar os direitos iguais aos antigos

./ocrconfig -replace ocr /ocfs2_ocr11/ocrs011.crs
./ocrconfig -replace ocrmirror /ocfs2_ocr22/ocrs022.crs

-- Verificar a alteração de local
cat /etc/oracle/ocr.loc

9) Verificações
./ocrcheck
  
Status of Oracle Cluster Registry is as follows :
     Version                  :          2
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       3840
     Available space (kbytes) :     258280
     ID                       :  665915271
     Device/File Name         : /ocfs2_ocr11/ocrs011.crs
                                Device/File integrity check succeeded
     Device/File Name         : /ocfs2_ocr22/ocrs022.crs
                                Device/File integrity check succeeded

     Cluster registry integrity check succeeded

10) Verificar todos os serviços, fstab e fazer um restart no CRS apenas para conferência (nos dois nós)
./crsctl stop crs
./crsctl start crs
    
-- Fontes
OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) (Doc ID 428681.1)
Failed to Start CRS stack After Adding a New Voting Disk [ID 460874.1]

É filho, se chegou até aqui e não me xingou é porquê deu tudo certo né?

Mario
Abraço