Linux配置hugepage

对于内存较大的Oracle数据库服务器，配置HugePage是很有必要的，下面是配置Huge的步骤：

我的环境：

OS：CentOS release 6.4 (Final)

Oracle：11.2.0.4 RAC
1.         设置memlock

编辑/etc/security/limits.conf

增加：

* Soft memlock 稍小于RAM值

* hard memlock 稍小于RAM值

Memlock值大于sga没关系的，所以我们可以设置这个值在我们想要的SGA size和物理内存size之间，这个值得单位是kb。

[root@node1 ~]# grep -i memtot /proc/meminfo

MemTotal:       132250576 kB

我们物理内存大小126G，因此我可以设置memlock为100G。

在2个节点设置：

[root@node1 ~]# tail -2 /etc/security/limits.conf

* soft memlock 104857600

* hard memlock 104857600

设置完毕后，以Oracle用户登录验证：

node2-> ulimit -l

104857600

2.         禁用AMM特性：

禁用AMM特性需要将memory_max_target, memory_max_target2个参数重置，而不仅仅是设置为0，最好的办法是创建一个pfile,在pfile中将这2个参数删除，再根据这个pfile创建spfile.

主要内存参数设置为：

node1->

node1-> grep target /tmp/init.ora

*.pga_aggregate_target=1572864000

*.sga_target=64424509440

SQL> create spfile from pfile=‘/tmp/init.ora‘;

File created.

SQL> startup force

ORA-27102: out of memory

Linux-x86_64 Error: 28: No space left on device

Additional information: -134217728

Additional information: 1

SQL>

出现这个报错是因为shmall设置的太小。

Shmall表示共享内存的总大小，单位为pagesize的大小。

根据我的配置，最大能支持8G的SGA：

node1-> getconf PAGE_SIZE

4096

node1->  cat /proc/sys/kernel/shmall

2097152

修改shmall并重新启动数据库，修改的方法很简单，编辑/etc/sysctl.conf：

kernel.shmall= 16252928

即可。

修改完毕kernel.shmall，此时可以重启数据库。

SQL> startup nomount pfile=‘/tmp/init.ora‘;            

ORACLE instance started.

Total System Global Area 6.4137E+10 bytes

Fixed Size                  2269072 bytes

Variable Size            6979321968 bytes

Database Buffers         5.7043E+10 bytes

Redo Buffers              112762880 bytes

SQL>

SQL>

DB启动后重新创建spfile：

SQL>  create spfile=‘+OCR/sszgdb/spfilesszgdb.ora‘ from pfile=‘/tmp/init.ora‘;

File created.

SQL>

3.         根据Oracle的脚本计算hugepage大小（计算时要确保Oracle已经启动），也可自己计算：

Press Enter to proceed...

Recommended setting: vm.nr_hugepages = 30739

node1->

4.         根据脚本的计算值在2个节点编辑/etc/sysctl.conf

加入vm.nr_hugepages = 30739

[root@node2 ~]# grep -i huge /proc/meminfo

AnonHugePages:    602112 kB

HugePages_Total:   30739

HugePages_Free:    30739

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB停止数据库实例

5.         重启服务器，一般重启服务器之后DB也自动启动了。服务器和DB重启后可以通过以下命令验证大页的使用情况：

grep -i Hugepages /proc/meminfo

此时查看Oracle的alert日志，也可以发现：

Starting ORACLE instance (normal)

************************ Large Pages Information *******************

Per process system memlock (soft) limit = UNLIMITED

Total Shared Global Region in Large Pages = 58 GB (96%)

Large Pages used by this instance: 29761 (58 GB)

Large Pages unused system wide = 24 (48 MB)

Large Pages configured system wide = 30739 (60 GB)

Large Page size = 2048 KB

RECOMMENDATION:

  Total System Global Area size is 60 GB. For optimal performance,

  prior to the next instance restart:

  1. Increase the number of unused large pages by

 at least 936 (page size 2048 KB, total size 1872 MB) system wide to

可以发现sga并没有全部放在大页里，这样可能对性能带来负面影响，Oracle建议我们再加入936个大页，可见Oracle的大页脚本计算并不准确，我们需要增加大页数量，并设置参数Use_large_pages=only，该参数确保Oracle的共享内存全部使用大页。

再重启DB，此时alert日志：

************************ Large Pages Information *******************

Parameter use_large_pages = ONLY

Per process system memlock (soft) limit = 100 GB

Total Shared Global Region in Large Pages = 60 GB (100%)

Large Pages used by this instance: 30721 (60 GB)

Large Pages unused system wide = 0 (0 KB)

Large Pages configured system wide = 31675 (62 GB)

可以发现sga全部使用了大页了。

另需确保环境变量DISABLE_HUGETLBFS unset，该变量会禁用Oracle使用Hugepage

附：计算HugePage大小的脚本：

#!/bin/bash

#

# hugepages_settings.sh

#

# Linux bash script to compute values for the

# recommended HugePages/HugeTLB configuration

#

# Note: This script does calculation for all shared memory

# segments available when the script is run, no matter it

# is an Oracle RDBMS shared memory segment or not.

#

# This script is provided by Doc ID 401749.1 from My Oracle Support

# http://support.oracle.com

# Welcome text

echo "

This script is provided by Doc ID 401749.1 from My Oracle Support

(http://support.oracle.com) where it is intended to compute values for

the recommended HugePages/HugeTLB configuration for the current shared

memory segments. Before proceeding with the execution please note following:

 * For ASM instance, it needs to configure ASMM instead of AMM.

 * The ‘pga_aggregate_target‘ is outside the SGA and

   you should accommodate this while calculating SGA size.

 * In case you changes the DB SGA size,

   as the new SGA will not fit in the previous HugePages configuration,

   it had better disable the whole HugePages,

   start the DB with new SGA size and run the script again.

And make sure that:

 * Oracle Database instance(s) are up and running

 * Oracle Database 11g Automatic Memory Management (AMM) is not setup

   (See Doc ID 749851.1)

 * The shared memory segments can be listed by command:

     # ipcs -m

Press Enter to proceed..."

read

# Check for the kernel version

KERN=`uname -r | awk -F. ‘{ printf("%d.%d\n",$1,$2); }‘`

# Find out the HugePage size

HPG_SZ=`grep Hugepagesize /proc/meminfo | awk ‘{print $2}‘`

if [ -z "$HPG_SZ" ];then

    echo "The hugepages may not be supported in the system where the script is being executed."

    exit 1

fi

# Initialize the counter

NUM_PG=0

# Cumulative number of pages required to handle the running shared memory segments

for SEG_BYTES in `ipcs -m | cut -c44-300 | awk ‘{print $1}‘ | grep "[0-9][0-9]*"`

do

    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`

    if [ $MIN_PG -gt 0 ]; then

        NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`

    fi

done

RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`

# An SGA less than 100MB does not make sense

# Bail out if that is the case

if [ $RES_BYTES -lt 100000000 ]; then

    echo "***********"

    echo "** ERROR **"

    echo "***********"

    echo "Sorry! There are not enough total of shared memory segments allocated for

HugePages configuration. HugePages can only be used for shared memory segments

that you can list by command:

    # ipcs -m

of a size that can match an Oracle Database SGA. Please make sure that:

 * Oracle Database instance is up and running

 * Oracle Database 11g Automatic Memory Management (AMM) is not configured"

    exit 1

fi

# Finish with results

case $KERN in

    ‘2.2‘) echo "Kernel version $KERN is not supported. Exiting." ;;

    ‘2.4‘) HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;

           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;

    ‘2.6‘) echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;

    ‘3.8‘) echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;

esac

# End