Matching Cisco and System p

IBM System p, AIX & Linux Aktuell 2008

Matching Network Parameters between
IBM System p and CISCO Networks

Alexander Paul
Netmakers IT-Consulting
paulalex@de.ibm.com

© 2008 Alexander Paul

Matching Network Parameters between IBM System p and CISCO Networks

Networking on IBM System p

POWER4/AIX5.2 POWER6/AIX6
Network Enhancements Advanced Virtual Networking
• Link Aggregation (EtherChannel, IEEE 802.3ad) • Integrated Virtual Ethernet (IVE)
Enhancements • Advanced VLAN Features
• Virtual LAN (VLAN) & VLAN trunking (dot1q) (GARP, VLAN Filtering)
• IPv6 • Performance Optimization
• Multipath Routing
• Path MTU discovery (PMTU)
• Quality of Service (QoS)
• Virtual IP Addresses (VIPA)

POWER5/AIX5.3
Virtual Networking
• Virtual Ethernet
• Shared Ethernet Adapter (SEA)
• VLAN Trunking (dot1q)
• 10 Gigabit Ethernet & InfiniBand
• High Availability for Virtual Networking

2 Alexander Paul IBM System p, AIX & Linux Aktuell 2008 © 2007 IBM Corporation


There are many interesting connection points …
In this session,
we focus on two topics:

– Link Aggregation (LA):
Discover how LA works by looking on perfomance behavior and
the confguration on both sites (AIX and Cisco).

– Multiple VLANs:
Connection on Cisco catalyst switches to provide a full VLAN
connectivity to the Virtual I/O server.



Link Aggregation

ent0
eth0
ent1 ent3
ent2



What exactly is Link Aggregation (LA)?

LA is a logical grouping of physical ports.

There are different namings for the same things on
AIX and Cisco:

– EtherChannel (AIX/Cisco)
Means a cisco proprietary protocol, named PAgP.
EtherChannel is a trademark of Cisco.
– Aggregate Port (AIX/Cisco)
– Port Channel (Cisco)
– Channel Group (Cisco)



What is Link Aggregation used for?

Eliminates SPOF on physical adapter connections
– Very efficient with less failover latency
– Several tuning points to fit the LA to your network behavior
– Hot adapter add, remove or replace

May improve overall throughput
– More about this in only a few slides…



Link Aggregation protocols

Cisco EtherChannel (PAgP)
– Cisco proprietary
– Uses a logical PortChannel on Cisco site that can bundle up to
8 physical ports.
– Supported by AIX and VIOS
– Supports load balancing mechanisms

IEEE 802.3ad (LACP)
– Supported by Cisco and other switch vendors
– Supported by AIX, VIOS and Linux bonding
– Supports load balancing mechanisms



How does a LA protocol works?

When you configure one end of a link aggregation in either
PAgP or LACP mode, it negotiates with the other end to
determine which ports would become active.

Incompatible ports are automatically suspended.

Protocol negotiation

ent0
Switch ent1 ent3 AIX
ent2



What to do before configuring Link Aggregation?
On AIX site:

Configure speed and duplex settings for each adapter ports.
Know the workload from AIX to switch.

IBM System p

Cisco Catalyst

On Switch site:

Configure speed and duplex settings for each switchport.
A appropriate description on switchports to be aggregate may be helpful in
case of trouble.
Know the workload leaving the switch through the portchannel.

IBM System p

Cisco Catalyst



Configuring Link Aggregation (IEEE 802.3ad)

aix61server2 1. Add a new Link Aggregation:
aix61server2:/#smitty addethch1

LA
ent3
2. Select the adapters for the active channel:
> ent0 active
Physical Adapters ent0 ent1 ent2
> ent1 active
ent2 backup

3. Configure LA-mode, backup and load balacing:

Add An EtherChannel / Link Aggregation

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

[Entry Fields]
EtherChannel / Link Aggregation Adapters ent0,ent1 +
Enable Alternate Address no
Alternate Address []
Enable Gigabit Ethernet Jumbo Frames no
Mode 8023ad
Hash Mode default
Backup Adapter ent2
Automatically Recover to Main Channel yes
Perform Lossless Failover After Ping Failure yes
Internet Address to Ping [9.155.62.1]
Number of Retries [3]
Retry Timeout (sec) [1]



4. Configure IP-Address, DNS and routing:
aix61server2
aix61server2:/#smitty tcpip
│ en0 Standard Ethernet Network Interface
LA │ en2 Standard Ethernet Network Interface
ent3
│ et0 IEEE 802.3 Ethernet Network Interface
Physical Adapters ent0 ent1 ent2 │ et3 IEEE 802.3 Ethernet Network Interface
* HOSTNAME [aix61server2]
* Internet ADDRESS (dotted decimal) [9.155.66.95]
Network MASK (dotted decimal) [255.255.240.0]
* Network INTERFACE en3
NAMESERVER
Internet ADDRESS (dotted decimal) [9.155.62.5]
DOMAIN Name [ibm.com]
Default Gateway
Address (dotted decimal or symbolic name) [9.155.62.1]

5. Show LA-Attributes:
aix61server2:/#lsattr -El ent3
adapter_names ent0,ent1 EtherChannel Adapters True
alt_addr 0x000000000000 Alternate EtherChannel Address True
auto_recovery yes Enable automatic recovery after failover True
backup_adapter ent2 Adapter used when whole channel fails True
hash_mode default Determines how outgoing adapter is chosen True
mode 8023ad EtherChannel mode of operation True
netaddr 9.155.62.1 Address to ping True
noloss_failover yes Enable lossless failover after ping failure True
num_retries 3 Times to retry ping before failing True
retry_time 1 Wait time (in seconds) between pings True
use_alt_addr no Enable Alternate EtherChannel Address True
use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames True



A. Setup the PortChannel No. 1:
Cat_3560_1 FastEthernet 0/9
Cat_3560_1(config)#interface range fastEthernet 0/9 - 10
ent0
ent1 AIX Cat_3560_1(config-if-range)#channel-group 1 mode ?
ent2 active Enable LACP unconditionally
FastEthernet 0/10 auto Enable PAgP only if a PAgP device is detected
desirable Enable PAgP unconditionally
on Enable Etherchannel only
passive Enable LACP only if a LACP device is detected

Cat_3560_1(config-if-range)#channel-group 1 mode passive
Creating a port-channel interface Port-channel 1



B. Show the result
FastEthernet 0/9
Cat_3560_1#show etherchannel summary
ent0 Flags: D - down P - in port-channel
ent1 AIX I - stand-alone s - suspended
ent2 H - Hot-standby (LACP only)
FastEthernet 0/10 R - Layer3 S - Layer2
u - unsuitable for bundling
U - in use f - failed to allocate aggregator
d - default port

Number of channel-groups in use: 1
Number of aggregators: 1

Group Port-channel Protocol Ports
------+-------------+-----------+-------------------------
1 Po1(SU) LACP Fa0/9(Pd) Fa0/10(P)

PortChannel is UP All Switchports are
part of the channel
IEEE 802.3ad



LACP negotiation
A. Debug the creation process:

ent0
Cat_3560_1#debug etherchannel event
ent1 AIX
ent2
%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/9, changed state to down

FEC: lacp_fec_dontbundle_internal: Fa0/10
FEC: lacp_fec_dontbundle_internal: Fa0/9

%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/10, changed state to up

FEC: lacp_fec_unbundle_internal: Fa0/10
FEC: lacp_switch_add_port_to_associated_list: Fa0/10 added to list for Po1
FEC: lacp_fec_unbundle_internal: Fa0/9

FEC: lacp_switch_get_first_agg_id_from_admin_oper_keys: found aggregator Po1 for admin_key
[0x1] and oper_key [0x1]

FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Fa0/10 associated to Po1
FEC: lacp_switch_add_port_to_associated_list: Fa0/9 added to list for Po1


FEC: lacp_switch_add_port_to_agport_internal: Fa0/9 added to aggregator Po1 list
FEC: lacp_switch_add_port_to_agport_internal: Fa0/10 added to aggregator Po1 list


%LINK-3-UPDOWN: Interface Port-channel1, changed state to up



Test AIX Link Aggregation throughput
192.168.1.161 GigabitEthernet 0/1 FastEthernet 0/9 192.168.1.160

ent0
Linux eth0
ent1 ent3 AIX
ent2

Server FastEthernet 0/10
Client
application application
Traffic direction

------------------------------------------------------------
Client connecting to 192.168.1.161, TCP port 5001
TCP window size: 256 KByte (default)
------------------------------------------------------------
One TCP session: [ 3] local 192.168.1.160 port 32838 connected with 192.168.1.161 port 5001
[ 3] 0.0-10.0 sec 112 MBytes 94.2 Mbits/sec

------------------------------------------------------------
------------------------------------------------------------
[ 4] local 192.168.1.160 port 32840 connected with 192.168.1.161 port 5001
Two TCP sessions: [ 3] local 192.168.1.160 port 32839 connected with 192.168.1.161 port 5001
[ 3] 0.0-10.0 sec 56.5 MBytes 45.2 Mbits/sec
[SUM] 0.0-10.0 sec 112 MBytes 93.9 Mbits/sec



Load Balacing for AIX link aggregation

Hash_Mode: Default uses a hash of the last byte of the IP address
to select the output adapter (or MAC address for non-IP frames)

Other possible load balancing methods are:
– Per Layer 4 source port (src_port)
– Per Layer 4 destination sort (dst_port)
– Per Layer 4 source and destination port (src_dst_port)

Can only be changed by removing the LA.



Change load balancing hash mode

ent0
Linux eth0
ent1 ent3 AIX
ent2

aix61server2:/#ifconfig en3 detach
aix61server2:/#rmdev –dl ent3
aix61server2:/#smitty etherchannel

[…]

aix61server2:/#lsattr -El ent3
adapter_names ent0,ent1 EtherChannel Adapters True
alt_addr 0x000000000000 Alternate EtherChannel Address True
auto_recovery yes Enable automatic recovery after failover True
backup_adapter ent2 Adapter used when whole channel fails True
hash_mode src_dst_port Determines how outgoing adapter is chosen True
mode 8023ad EtherChannel mode of operation True
netaddr 192.168.1.111 Address to ping True
noloss_failover yes Enable lossless failover after ping failure True
num_retries 3 Times to retry ping before failing True
retry_time 1 Wait time (in seconds) between pings True
use_alt_addr no Enable Alternate EtherChannel Address True
use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames True



Test AIX Link Aggregation throughput

ent0
Linux eth0
ent1 ent3 AIX
ent2

Server FastEthernet 0/10
Client
Traffic direction

------------------------------------------------------------
------------------------------------------------------------
Two TCP sessions: [ 4] local 192.168.1.160 port 32857 connected with 192.168.1.161 port 5001



Test switch Link Aggregation throughput

ent0
Linux eth0
ent1 ent3 AIX
ent2

Client FastEthernet 0/10
Server
Traffic direction

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 16.0 KByte (default)
One session: ------------------------------------------------------------

Two sessions: [ 4] 0.0-10.0 sec 56.1 MBytes 47.0 Mbits/sec



Test Link Aggregation (IEEE 802.3ad) on Cisco
192.168.1.161
GigabitEthernet FastEthernet 0/9
192.168.1.160
Linux eth0 0/1
ent0
ent1 ent3 AIX
ent2

Linux eth0 GigabitEthernet FastEthernet 0/10
0/2 Server
application
192.168.1.180 Traffic direction

aix61server2:/#iperf -s



Load Balacing for Cisco port channels

Cisco switches uses load balancing by source mac-address.
Cat_3560_2#show etherchannel load-balance
EtherChannel Load-Balancing Operational State (src-mac):
Non-IP: Source MAC address
IPv4: Source MAC address
IPv6: Source IP address

Availability of other load balancing methods depends on switch model:
Cat_3560_2(config)#port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr

Load balancing can be changed dynamically.



Troubleshooting Link Aggregation on AIX

aix61server2:/#entstat -d ent3

[…]
Number of adapters: 2
Operating mode: Standard mode (IEEE 802.3ad)
IEEE 802.3ad Link Aggregation Statistics:
Aggregation status: Aggregated
Received LACPDUs: 18 This indicates a successful LA negotiation.
Transmitted LACPDUs: 18
Received marker PDUs: 0
Transmitted marker PDUs: 0
Received marker response PDUs: 0
Transmitted marker response PDUs: 0
Received unknown PDUs: 0
Received illegal PDUs: 0
Hash mode: Destination IP address
[…]



Troubleshooting on Cisco switches

show etherchannel summary
– Lists the status of a port channel and the associated switchports
– Lists the configured protocol or mode for the port channel.

show etherchannel detail
– Lists detailed information about the switchport behavior.

debug etherchannel event
– Gives live report to the console regarding important etherchannel events.
– Good for tracing a problem during the configuration process.



Link Aggregation and POWER6 IVE

Client LPAR 1 Client LPAR 2 Client LPAR 3

(LA) (LA)

LOG LOG LOG LOG LOG
PORT PORT PORT PORT PORT

HEA Switch HEA Switch HEA Switch HEA Switch
4 Port IVE
Adapter



Virtualization in a switched environment

Cisco Catalyst 6500 Virtual Switching System (VSS) 1440

– Cisco Catalyst 6500 VSS 1440 enables multiple Catalyst 6500 switches
to be pooled into one virtual switch, simplifying network management,
increasing availability, and optimizing switch resources.

– Cisco Catalyst 6500 VSS enables interchassis stateful failover
capabilities while Smart Call Home service reduces resolution time for
Catalyst 6500 issues. In-service software upgrades enhance availability
on both the Catalyst 6500 and Catalyst 4500 platforms.



Multiple VLAN support



Cisco Switch configuration for VIO environment
VIOS 1 Client 1 Client 2

en2 en1
(if) (if)

ent2 en0 en1 ent1 en0
(SEA) (if) (if) VLAN (if)

ent0 ent1 ent0 ent1 ent0
(Phy) (Vir) (Vir) (Vir) (Vir)

VID PVID PVID PVID PVID VID
Hypervisor

2 1 1 2 1 2

This Link must be configured to forward dot1q-tagged
frames.

Cisco Catalyst


Show trunk configuration on switch interfaces
VIOS 1
Switch#show interface Gi0/1 switchport
en2
(if)
[...]
Name: Gi0/1
ent2 Switchport: Enabled
(SEA) Administrative Mode: dynamic auto
Operational Mode: down
Administrative Trunking Encapsulation: negotiate
ent0 ent1 Operational Trunking Encapsulation: native
(Phy) (Vir) Negotiation of Trunking: Off
PVID 1 Access Mode VLAN: 1 (default)
PHYP Trunking Native Mode VLAN: 1 (default)
VID 2
VLAN 1 Administrative Native VLAN tagging: enabled
Voice VLAN: none
Tag VLAN 2 Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Only untagged frames are Administrative private-vlan trunk encapsulation: dot1q
VLAN 1 forwarded to the VIO server and Administrative private-vlan trunk normal VLANs: none
accepted by the switch. Administrative private-vlan trunk private VLANs: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Tag VLAN 2 Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL

VLAN 1



Default switchport configuration

By default Switchport role depends on configuration on the other site.
Possible Roles are:
– Trunk
– Access

Administrative Trunking Encapsulation
– Dot1q is standard on newer switch models and IOS Versions
– Dynamic if ISL is an option

Access Mode VLAN
– By default a switchport is configured as an access port
– VLAN membership: VLAN 1



Default switchport configuration

A default switchport configuration…

– accepts only untagged frames.
– does only support multiple VLANs if neighbor device negotiates a VLAN
trunk.
– will forward all untagged frames from the VIO Server to VLAN 1 on the
switch, regardless of their original VLAN memberships.
– is vulnerable by negotiation messages



Configure multiple VLAN support on Cisco IOS
VIOS 1
en2 switch(config)#interface gi0/1
(if)
switch(config-if)#switchport mode trunk
ent2
(SEA) 5w2d: %LINEPROTO-5-UPDOWN: Line protocol on Interface
GigabitEthernet0/1, changed state to down
ent0 ent1
(Phy) (Vir)
5w2d: %LINEPROTO-5-UPDOWN: Line protocol on Interface
PVID 1
PHYP GigabitEthernet0/1, changed state to up
VID 2
VLAN 1

Tag VLAN 2

VLAN 1
Now the switch forwards untagged frames to the VIO server
and accept them from VIO server as well.
Tag VLAN 2



Configure multiple VLAN support on Cisco IOS
VIOS 1
switch#show interface gi0/1 switchport
en2
Name: Gi0/1
(if)
Switchport: Enabled
Administrative Mode: trunk
ent2 Operational Mode: trunk
(SEA) Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: dot1q
Negotiation of Trunking: On
ent0 ent1
Access Mode VLAN: 1 (default)
(Phy) (Vir)
Trunking Native Mode VLAN: 1 (default)
Voice VLAN: none
Gigabit Ethernet 0/1 Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk private VLANs: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL
Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Appliance trust: none



Define allowed VLANs on a trunk
VIOS 1
en2
(if)

ent2 SEA bridges all tagged frames
(SEA)

ent0 ent1
(Phy) (Vir) VLAN border

PVID 1
PHYP
VLAN 1
VID 2 By default the switch will forward all tagged frames
Tag VLAN 2
to the VIO server
VLAN 1
The Hypervisor switch drops the VLANs which are
Tag VLAN 2 not defined as VID.
Tag VLAN 5 The SEA is still bridging all frames
Tag VLAN 100 Consumes CPU time in VIO server
Tag VLAN 99 May be a security issue



Define allowed VLANs on a trunk
VIOS 1
en2
(if)
switch(config-if)# switchport trunk allowed vlan 1,2
ent2
(SEA)
Switch will only forward frames for defined
ent0 ent1
(Phy) (Vir) VLANs
VLAN 1 PVID 1
PHYP
VID 2
SEA must only bridge frames which are
Tag VLAN 2
configured on the Hypervisor switch

Reduced load on VIO server
VLAN 1
Filtering is done in the physical network layer
Tag VLAN 2

VLAN border

Tag VLAN 5

Tag VLAN 100
Tag VLAN 99


Matching Cisco and System p

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Matching Cisco and System p (20)

More from Andrey Klyachkin (9)

Recently uploaded (20)

Matching Cisco and System p