Install Sun Cluster 3.2 Update 1 between two guest LDOMS

With the latest updates, there are many new things to play with.

The material I've been using is:

- Solaris 10 U5 (5/08).
- Sun Cluster 3.2 U1.
- Logical Domains Manager 1.0.2
- 2 x T2000

I've setup on each T2000:
- 1 Control Domain.
- 1 Guest Domain.

So in this case, the cluster is between two guest domains of different physical systems.

I've used a Quorum Server installed on a small Ultra 5 as the "quorum device".

There's no shared storage so I've setup an Scalable Apache with an external NFS.

Remember that we need two virtual switches inside the guest domains to create the internal network of the Cluster.

Here we go!:


FIRST T2000

root@t2000:~> ldm ls
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv SP 4 4G 0.9% 20m
delibes active -n--- 5000 16 2G 0.0% 7m


root@t2000:~> ldm ls -l delibes
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
delibes active -n--- 5000 16 2G 0.1% 8m

SOFTSTATE
Solaris running

VCPU
VID PID UTIL STRAND
0 4 0.6% 100%
1 5 0.0% 100%
2 6 0.0% 100%
3 7 0.0% 100%
4 8 0.0% 100%
5 9 0.0% 100%
6 10 0.0% 100%
7 11 0.0% 100%
8 12 0.1% 100%
9 13 0.0% 100%
10 14 0.0% 100%
11 15 0.0% 100%
12 16 0.1% 100%
13 17 0.0% 100%
14 18 0.0% 100%
15 19 0.0% 100%

MAU
CPUSET
(4, 5, 6, 7)

MEMORY
RA PA SIZE
0x8000000 0x108000000 2G

VARIABLES
auto-boot?=false
boot-device=/virtual-devices@100/channel-devices@200/disk@0:a disk net
keyboard-layout=US-English

NETWORK
NAME SERVICE DEVICE MAC
vnet0 primary-vsw0@primary network@0 00:14:4f:f8:7c:4b
vnet1 primary-vsw1@primary network@1 00:14:4f:f8:fb:f4
vnet2 primary-vsw2@primary network@2 00:14:4f:fb:89:9e

Three interfaces (it should be four):
- vnet0: public interface
- vnet1 and vnet2: cluster private network.

DISK
NAME VOLUME TOUT DEVICE SERVER
vdisk0 vol0@primary-vds0 disk@0 primary
globaldev globaldev@primary-vds0 disk@1 primary

Both disks are virtualized files. I used globaldev "disk" as the mountpoint for /globaldevices

VCONS
NAME SERVICE PORT
curso primary-vcc0@primary 5000



SECOND T2000:

root@t2000-2:~> ldm ls
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv SP 4 4G 0.6% 21m
melena active -n--- 5000 28 3G 0.0% 5m


root@t2000-2:~> ldm ls -l cardiologa
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
cardiologa active -n--- 5000 28 3G 0.0% 6m

SOFTSTATE
Solaris running

VCPU
VID PID UTIL STRAND
0 4 3.7% 100%
1 5 0.0% 100%
2 6 0.0% 100%
3 7 0.0% 100%
4 8 0.0% 100%
5 9 0.0% 100%
6 10 0.0% 100%
7 11 0.0% 100%
8 12 1.7% 100%
9 13 0.0% 100%
10 14 0.0% 100%
11 15 0.0% 100%
12 16 2.7% 100%
13 17 0.0% 100%
14 18 0.0% 100%
15 19 0.0% 100%
16 20 0.0% 100%
17 21 0.0% 100%
18 22 0.0% 100%
19 23 0.0% 100%
20 24 0.0% 100%
21 25 0.0% 100%
22 26 0.0% 100%
23 27 0.0% 100%
24 28 0.0% 100%
25 29 0.0% 100%
26 30 0.0% 100%
27 31 0.0% 100%

MEMORY
RA PA SIZE
0x8000000 0x108000000 3G

VARIABLES
auto-boot?=false
keyboard-layout=Spanish
NETWORK

NAME SERVICE DEVICE MAC
vnet0 primary-vsw0@primary network@0 00:14:4f:fa:28:88
vnet1 primary-vsw1@primary network@1 00:14:4f:f9:39:76
vnet2 primary-vsw2@primary network@2 00:14:4f:fa:8a:a8

DISK
NAME VOLUME TOUT DEVICE SERVER
vdisk0 vol0@primary-vds0 disk@0 primary
globaldev globaldev@primary-vds0 disk@3 primary

VCONS
NAME SERVICE PORT
cardiologa primary-vcc0@primary 5000


The first guest domain:

root@delibes:/> cat /etc/hosts
#
# Internet host table
#::1 localhost

127.0.0.1 localhost
10.164.17.25 delibes loghost
10.164.17.35 cardiologa


root@cardiologa:/> cat /etc/hosts
#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
10.164.17.35 cardiologa loghost
10.164.17.25 delibes


root@delibes:/> dladm show-link
vnet0 type: non-vlan mtu: 1500 device: vnet0
vnet1 type: non-vlan mtu: 1500 device: vnet1
vnet2 type: non-vlan mtu: 1500 device: vnet2


root@cardiologa:/> dladm show-link
vnet0 type: non-vlan mtu: 1500 device: vnet0
vnet1 type: non-vlan mtu: 1500 device: vnet1
vnet2 type: non-vlan mtu: 1500 device: vnet2

root@delibes:/> scinstall


*** Main Menu ***

Please select from one of the following (*) options:

* 1) Create a new cluster or add a cluster node
2) Configure a cluster to be JumpStarted from this install server
3) Manage a dual-partition upgrade
4) Upgrade this cluster node
5) Print release information for this cluster node

* ?) Help with menu options
* q) Quit

Option: 1

*** New Cluster and Cluster Node Menu ***

Please select from any one of the following options:

1) Create a new cluster
2) Create just the first node of a new cluster on this machine
3) Add this machine as a node in an existing cluster

?) Help with menu options
q) Return to the Main Menu

Option: 1

*** Create a New Cluster ***

This option creates and configures a new cluster.

You must use the Java Enterprise System (JES) installer to install the
Sun Cluster framework software on each machine in the new cluster
before you select this option.

If the "remote configuration" option is unselected from the JES
installer when you install the Sun Cluster framework on any of the new
nodes, then you must configure either the remote shell (see rsh(1)) or
the secure shell (see ssh(1)) before you select this option. If rsh or
ssh is used, you must enable root access to all of the new member
nodes from this node.

Press Control-d at any time to return to the Main Menu.


Do you want to continue (yes/no) [yes]? yes

>>> Typical or Custom Mode <<<
This tool supports two modes of operation, Typical mode and Custom. For most clusters, you can use Typical mode. However, you might need to select the Custom mode option if not all of the Typical defaults can be applied to your cluster. For more information about the differences between Typical and Custom modes, select the Help option from the menu.
Please select from one of the following options:
1) Typical
2) Custom
?) Help
q) Return to the Main Menu
Option [1]: 1

>>> Cluster Name <<<
Each cluster has a name assigned to it. The name can be made up of any characters other than whitespace. Each cluster name should be unique within the namespace of your enterprise.

What is the name of the cluster you want to establish []? ldomcluster

>>> Cluster Nodes <<<
This Sun Cluster release supports a total of up to 16 nodes. Please list the names of the other nodes planned for the initial cluster configuration. List one node name per line.
When finished, type Control-D:

Node name: delibes
Node name: cardiologa
Node name (Control-D to finish): ^D

This is the complete list of nodes: delibes cardiologa
Is it correct (yes/no) [yes]? yes
Attempting to contact "cardiologa" ... done Searching for a remote configuration method ... done
The Sun Cluster framework is able to complete the configuration process without
remote shell access. Press Enter to continue:

>>> Cluster Transport Adapters and Cables <<<
You must identify the cluster transport adapters which attach this node to
the private cluster interconnect.
For node "delibes",
What is the name of the first cluster transport adapter (help) []? vnet1
Will this be a dedicated cluster transport adapter (yes/no) [yes]? yes
Searching for any unexpected network traffic on "vnet1" ... done
Verification completed. No traffic was detected over a 10 second sample period.

For node "delibes",
What is the name of the second cluster transport adapter (help) []? vnet2
Searching for any unexpected network traffic on "vnet2" ... done
Verification completed. No traffic was detected over a 10 second sample period.

>>> Quorum Configuration <<<
Every two-node cluster requires at least one quorum device. By default, scinstall will select and configure a shared SCSI quorum disk device for you. This screen allows you to disable the automatic selection and configuration of a quorum device. The only time that you must disable this feature is when ANY of the shared storage in your cluster is not qualified for use as a Sun Cluster quorum device. If your storage was purchased with your cluster, it is qualified. Otherwise, check with your storage vendor to determine whether your storage device is supported as Sun Cluster quorum device. If you disable automatic quorum device selection now, or if you intend to use a quorum device that is not a shared SCSI disk, you must instead use clsetup(1M) to manually configure quorum once both nodes have joined the cluster for the first time.

Do you want to disable automatic quorum device selection (yes/no) [no]? yes


I did not set automatic quorum device selection, as we'll use a quorum server as the "quorum device".


Is it okay to create the new cluster (yes/no) [yes]? yes

During the cluster creation process, sccheck is run on each of the new cluster nodes. If sccheck detects problems, you can either interrupt the process or check the log files after the cluster has been established.
Interrupt cluster creation for sccheck errors (yes/no) [no]? no

Cluster Creation Log file - /var/cluster/logs/install/scinstall.log.921
Testing for "/globaldevices" on "delibes" ... done
Testing for "/globaldevices" on "cardiologa" ... done
Starting discovery of the cluster transport configuration.

The following connections were discovered:
delibes:vnet1 switch1 cardiologa:vnet1 delibes:vnet2 switch2 cardiologa:vnet2
Completed discovery of the cluster transport configuration.

Started sccheck on "delibes".
Started sccheck on "cardiologa".
sccheck failed for "delibes".
sccheck failed for "cardiologa".
The sccheck command failed on both of the nodes.
Refer to the log file for details. The name of the log file is /var/cluster/logs/install/scinstall.log.921.
Configuring "cardiologa" ... done
Rebooting "cardiologa" ...


Now everything is setup but we still need to set the quorum server. After this I used the clsetup command to remove "install flag" mode to complete the installation.

root@delibes:/> clq add -t quorum_server -p qshost=node12 -p port=9000 node12

root@delibes:/> clq status

=== Cluster Quorum ===

--- Quorum Votes Summary ---

Needed Present Possible
------ ------- --------
2 3 3


--- Quorum Votes by Node ---

Node Name Present Possible Status
--------- ------- -------- ------
cardiologa 1 1 Online
delibes 1 1 Online


--- Quorum Votes by Device ---

Device Name Present Possible Status
----------- ------- -------- ------
node12 1 1 Online


root@delibes:/> scstat
------------------------------------------------------------------

-- Cluster Nodes --

Node name Status
--------- ------
Cluster node: cardiologa Online
Cluster node: delibes Online

------------------------------------------------------------------

-- Cluster Transport Paths --

Endpoint Endpoint Status
-------- -------- ------
Transport path: cardiologa:vnet2 delibes:vnet2 Path online
Transport path: cardiologa:vnet1 delibes:vnet1 Path online

------------------------------------------------------------------

-- Quorum Summary --

Quorum votes possible: 3
Quorum votes needed: 2
Quorum votes present: 3


-- Quorum Votes by Node --

Node Name Present Possible Status
--------- ------- -------- ------
Node votes: cardiologa 1 1 Online
Node votes: delibes 1 1 Online


-- Quorum Votes by Device --

Device Name Present Possible Status
----------- ------- -------- ------
Device votes: node12 1 1 Online

------------------------------------------------------------------

-- Device Group Servers --

Device Group Primary Secondary
------------ ------- ---------


-- Device Group Status --

Device Group Status
------------ ------


-- Multi-owner Device Groups --

Device Group Online Status
------------ -------------

------------------------------------------------------------------
------------------------------------------------------------------

-- IPMP Groups --

Node Name Group Status Adapter Status
--------- ----- ------ ------- ------
IPMP Group: cardiologa sc_ipmp0 Online vnet0 Online

IPMP Group: delibes sc_ipmp0 Online vnet0 Online

------------------------------------------------------------------


As a test, I've also set up an Apache service (a scalable one) using an external NFS server as the shared storage.


root@delibes:/> clrg status

=== Cluster Resource Groups ===

Group Name Node Name Suspended Status
---------- --------- --------- ------
sa-rg delibes No Online
cardiologa No Offline

web-rg delibes No Online
cardiologa No Online


Just to check that the network is properly configured:

root@delibes:/> ifconfig -a
lo0: flags=20010008c9 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
vnet0: flags=9000843 mtu 1500 index 2
inet 10.164.17.25 netmask ffffff00 broadcast 10.164.17.255
groupname sc_ipmp0
ether 0:14:4f:f8:7c:4b
vnet0:1: flags=1040843 mtu 1500 index 2
inet 10.164.17.24 netmask ffffff00 broadcast 10.164.17.255
vnet1: flags=1008843 mtu 1500 index 4
inet 172.16.0.130 netmask ffffff80 broadcast 172.16.0.255
ether 0:14:4f:f8:fb:f4
vnet2: flags=1008843 mtu 1500 index 3
inet 172.16.1.2 netmask ffffff80 broadcast 172.16.1.127
ether 0:14:4f:fb:89:9e
clprivnet0: flags=1009843
mtu 1500 index 5
inet 172.16.4.2 netmask fffffe00 broadcast 172.16.5.255
ether 0:0:0:0:0:2





Everything seems to work properly, but there are some obscure messages when switching the network from one node to the other:

May 9 15:31:58 cardiologa arp: [ID 994748 kern.notice] ar_query_xmit: Could not find the ace

BTW, these new updates offer a lot of flexibility and security, and the chance to recover cluster environments very fast.