Class 2: [Day-0] CE - Infrastructure Provisioning ================================================= Objective ~~~~~~~~~~ **Onbord and deploy CE on-prem / data center** .. NOTE:: Onboarding steps/process of CE in UDF(KVM-based) similar to onboarding on VMWare ESXi hypervisor. Step 1: Create Site token ~~~~~~~~~~~~~~~~~~~~~~~~~ You can always click F5 logo on top right to go to main screen. .. image:: ./_static/class2-0.png Input your own token name. For example, if you been allocated with CE name of "ce-01", your token name will be "ce-01-token" .. image:: ./_static/class2-1.png Token generated and will be use for subsequence step .. image:: ./_static/class2-2.png Step 2: Enroll CE Node ~~~~~~~~~~~~~~~~~~~~~~ **2.1 Start CE node enrollment** From UDF main page, RDP to Windows Jumphost Username and password will be provided or obtains from UDF Details tab. .. image:: ./_static/class2-27.png .. image:: ./_static/class2-3.png Perform subsequent task from Windows Jumphost SSH to CE node with PuTTY and select "master-0". Below are info of CE node +------------+---------------+ | master-0 | 10.1.1.4 | +------------+---------------+ | master-1 | 10.1.1.5 | +------------+---------------+ | master-2 | 10.1.1.6 | +------------+---------------+ .. image:: ./_static/class2-4.png Login with the following default credential +----------------+---------------+ | **Username** | admin | +----------------+---------------+ | **Password** | Volterra123 | +----------------+---------------+ You are required to change admin password on first time login. .. image:: ./_static/class2-5.png Input the following details +----------------------+--------------------------------------------------------------------------------------+ | | Value | +======================+======================================================================================+ | Token | Token value generated from previous steps | +----------------------+--------------------------------------------------------------------------------------+ | Site Name | Your CE site name (e.g. ce-01, ce-02, ce03, etc) | +----------------------+--------------------------------------------------------------------------------------+ | Hostname | Hostname for the node. Use "master-0, master-1 or master-2, worker-0, etc" | +----------------------+--------------------------------------------------------------------------------------+ | Latitude | Optional latitude. Determine registration to RE | +----------------------+--------------------------------------------------------------------------------------+ | Longtitude | Optional longtitude. Determine registration to RE | +----------------------+--------------------------------------------------------------------------------------+ | Default Fleet name | Optional. Leave it blank | +----------------------+--------------------------------------------------------------------------------------+ | Certified Hardware | kvm-regular-nic-voltmesh | +----------------------+--------------------------------------------------------------------------------------+ | Primary NIC | eth0 | +----------------------+--------------------------------------------------------------------------------------+ Enter to confirm configuration. Example .. image:: ./_static/class2-6.png **2.2 Approve Registration** Upon successful registration of CE onto F5XC Console, CE node will appear in F5XC Console .. image:: ./_static/class2-7.png UI shown CE node in "Pending Registrations". Administrator approval to enroll is required. .. IMPORTANT:: **DO NOT approve registration if you are doing a multi-node CE cluster**. For multi-node cluster, you will need to wait until all 3 node being enrolled. If you are doing a **single node cluster, you CAN approved here now**. .. image:: ./_static/class2-8.png .. NOTE:: Depend on class instruction. For CE cluster with HA setup, proceed with Cluster Setup steps. Else, skips to "Approve CE node registration" **2.3 Cluster Setup** .. NOTE:: This steps only require if you setup a CE Cluster with HA - 3 CE nodes cluster CE node cluster runs with minimum 3 nodes. **Enroll master-1** SSH via PuTTY to master-1 Repeat same CE enrolment process. You only required to change hostname value. Other value remain the same. .. image:: ./_static/class2-9.png **Enroll master-2** SSH via PuTTY to master-2 Repeat same CE enrolment process. You only required to change hostname value. Other value remain the same. .. image:: ./_static/class2-10.png **2.4 Approve CE node registration** From F5XC Console, approve pending registrations .. NOTE:: Image shown below with CE Cluster with HA. You may only see CE Cluster with single node (e.g. master-0 only). .. image:: ./_static/class2-11.png For Single node CE, Cluster Size will be "1" and for Multi-Node CE (CE Cluster with HA), Cluster size will be "3". .. image:: ./_static/class2-12.png With CE cluster with HA, all 3 nodes have to be approved and present before CE cluster provisioning started. .. image:: ./_static/class2-13.png .. image:: ./_static/class2-14.png **2.5 CE Cluster onboarded and healthy** CE in "Provisioning" State .. image:: ./_static/class2-15.png CE in "Healthy" State .. image:: ./_static/class2-16.png Step 3: Explore CE Status ~~~~~~~~~~~~~~~~~~~~~~~~~ Dashboard - CE-01 .. image:: ./_static/class2-17.png Nodes - CE-01 .. image:: ./_static/class2-18.png Site Status - CE-01 .. image:: ./_static/class2-19.png Node: master-0 .. image:: ./_static/class2-20.png Node: master-1 .. image:: ./_static/class2-21.png Node: master-2 .. image:: ./_static/class2-22.png PoP(RE) Connectivity - CE-01 .. image:: ./_static/class2-23.png Step 4: Update and Upgrade Node ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ CE software are built on demand. Hence, it always uses the current version. OS is depends on the original iso or ova file. Click upgrade to upgrade CE OS. .. image:: ./_static/class2-24.png CE schedule to be upgraded. In a mult-node CE, F5XC intent-based orchstration will upgrade one CE node at a time. Health of a CE node will be validated (Ready) before second node will be upgrade. That will ensure minimum downtime during the OS upgrade. .. image:: ./_static/class2-25.png CE node(s) successfully upgraded and healthy. .. image:: ./_static/class2-26.png Step 5: Setup Cluster VIP ~~~~~~~~~~~~~~~~~~~~~~~~~ .. image:: ./_static/class2-28.png Specify a cluseter IP (VIP). This is in additional to the CE node IP. .. image:: ./_static/class2-29.png .. NOTE:: Cluster VIP is not pingable/alive until a HTTP/TCP LB created to advertise that cluster VIP. Step 6: Create Fleet ~~~~~~~~~~~~~~~~~~~~ **4.1 Create interface** .. NOTE:: For the purpose of this lab, we will experience using 3 different methods to create Interface object. You can create all three interfaces using any of the following method. 1. Click-Ops - Clicking from F5XC Console UI (for master-0) 2. Clone from existing object (for master-1) 3. Copy and Paste from a JSON config (for master-2) .. NOTE:: Replace any reference on ce-01 to the name of your own ce name (e.g. ce-0X) master-0-eth1 Interface (Click from UI) .. image:: ./_static/class2-30.png Specify Interface metadata .. image:: ./_static/class2-31.png Specify Ethernet Interface information .. image:: ./_static/class2-32.png Ensure select "Site Local Network Inside" .. image:: ./_static/class2-33.png Save and Exit to confirm the configuration .. image:: ./_static/class2-34.png master-1-eth1 (Clone from UI) .. image:: ./_static/class2-35.png When you "Clone Object", configuration (except Name) will be pre-populated. Ensure appropriate name. .. image:: ./_static/class2-36.png Update to respective node hostname .. image:: ./_static/class2-37.png Save and Exit. Interface object will be created .. image:: ./_static/class2-38.png master-2-eth1 (Copy and paste) JSON configuration for master-2-eth1 been prepared. Add Network interface. .. image:: ./_static/class2-39.png Clear/delete content in JSON and copy and paste the following prepared JSON configuration onto the text field. .. image:: ./_static/class2-40.png ce-01-master-2-eth1 :: { "metadata": { "name": "ce-01-master-2-eth1", "namespace": "system", "labels": {}, "annotations": {}, "disable": false }, "spec": { "type": "NETWORK_INTERFACE_ETHERNET", "mtu": 0, "dhcp_address": "NETWORK_INTERFACE_DHCP_DISABLE", "static_addresses": [ { "prefix": "10.1.10.12", "plen": 24 } ], "default_gateway": { "default_gateway_mode": "NETWORK_INTERFACE_GATEWAY_DISABLE" }, "DNS_server": { "dns_mode": "NETWORK_INTERFACE_DNS_DISABLE", "dns_server": [] }, "DHCP_server": "NETWORK_INTERFACE_DHCP_SERVER_DISABLE", "vlan_tagging": "NETWORK_INTERFACE_VLAN_TAGGING_DISABLE", "device_name": "master-2/eth1", "vlan_tag": 0, "priority": 0, "interface_ip_map": {}, "is_primary": false, "monitor_disabled": {}, "ethernet_interface": { "device": "eth1", "node": "master-2", "untagged": {}, "static_ip": { "node_static_ip": { "ip_address": "10.1.10.12/24" } }, "no_ipv6_address": {}, "site_local_inside_network": {}, "mtu": 0, "priority": 0, "not_primary": {}, "monitor_disabled": {} } } } Examle pasted content .. image:: ./_static/class2-41.png Interface created .. image:: ./_static/class2-42.png **4.2 Create Fleet** Fleet is used to configure infrastructure components (like nodes) in one or CE sites homogeneously. Fleet configuration includes the following information - Software image release to be deployed on the Fleet - Virtual networks - List of interface and devices to be configured on every node - Connections between the virtual networks - Security policies applied in the Site For this lab, create fleet and assinged those created interface to fleet. .. image:: ./_static/class2-43.png Provide Fleet name and fleet label .. image:: ./_static/class2-44.png Add all three or one (single node) to the fleet. .. image:: ./_static/class2-45.png Fleet created .. image:: ./_static/class2-46.png **4.3 Attach fleet to CE** Attach fleet to CE by adding fleet label to the CE site .. image:: ./_static/class2-47.png Add label and select ves.io/fleet as the key .. image:: ./_static/class2-48.png Label added. Save and Exit. .. image:: ./_static/class2-49.png **4.4 Validate fleet working** master-0 Ensure correct eth1 IP shown on master-0 .. image:: ./_static/class2-50.png master-1 Ensure correct eth1 IP shown on master-1 .. image:: ./_static/class2-51.png master-2 Ensure correct eth1 IP shown on master-2 .. image:: ./_static/class2-52.png Advance BGP Setup (Optional) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default, CE uses Virtual Router Redundancy Protocol (VRRP) to provides High availability for services advertised on CE. F5XC CE support BGP peering with neighbor router to spray traffic to CE via ECMP (Equal Cost Multi Path) Protocol. Create BGP Peering ~~~~~~~~~~~~~~~~~~ Create eth0 interface for respective CE nodes. This eth0 will be use for BGP peering with external router. For simplicity, Copy and paste the following to create network interface. ce-01-master-0-eth0 :: { "metadata": { "name": "ce-01-master-0-eth0", "namespace": "system", "labels": {}, "annotations": {}, "disable": false }, "spec": { "type": "NETWORK_INTERFACE_ETHERNET", "mtu": 0, "dhcp_address": "NETWORK_INTERFACE_DHCP_DISABLE", "static_addresses": [ { "prefix": "10.1.1.4", "plen": 24 } ], "default_gateway": { "default_gateway_mode": "NETWORK_INTERFACE_GATEWAY_USE_CONFIGURED", "default_gateway_address": { "addr": "10.1.1.1" } }, "DNS_server": { "dns_mode": "NETWORK_INTERFACE_DNS_USE_CONFIGURED", "dns_server": [ { "addr": "10.1.1.1" } ] }, "DHCP_server": "NETWORK_INTERFACE_DHCP_SERVER_DISABLE", "vlan_tagging": "NETWORK_INTERFACE_VLAN_TAGGING_DISABLE", "device_name": "master-0/eth0", "vlan_tag": 0, "priority": 0, "interface_ip_map": {}, "is_primary": false, "monitor_disabled": {}, "ethernet_interface": { "device": "eth0", "node": "master-0", "untagged": {}, "static_ip": { "node_static_ip": { "ip_address": "10.1.1.4/24", "default_gw": "10.1.1.1", "dns_server": "10.1.1.1" } }, "no_ipv6_address": {}, "site_local_network": {}, "mtu": 0, "priority": 0, "not_primary": {}, "monitor_disabled": {} } } } ce-01-master-1-eth0 :: { "metadata": { "name": "ce-01-master-1-eth0", "namespace": "system", "labels": {}, "annotations": {}, "disable": false }, "spec": { "type": "NETWORK_INTERFACE_ETHERNET", "mtu": 0, "dhcp_address": "NETWORK_INTERFACE_DHCP_DISABLE", "static_addresses": [ { "prefix": "10.1.1.5", "plen": 24 } ], "default_gateway": { "default_gateway_mode": "NETWORK_INTERFACE_GATEWAY_USE_CONFIGURED", "default_gateway_address": { "addr": "10.1.1.1" } }, "DNS_server": { "dns_mode": "NETWORK_INTERFACE_DNS_USE_CONFIGURED", "dns_server": [ { "addr": "10.1.1.1" } ] }, "DHCP_server": "NETWORK_INTERFACE_DHCP_SERVER_DISABLE", "vlan_tagging": "NETWORK_INTERFACE_VLAN_TAGGING_DISABLE", "device_name": "master-1/eth0", "vlan_tag": 0, "priority": 0, "interface_ip_map": {}, "is_primary": false, "monitor_disabled": {}, "ethernet_interface": { "device": "eth0", "node": "master-1", "untagged": {}, "static_ip": { "node_static_ip": { "ip_address": "10.1.1.5/24", "default_gw": "10.1.1.1", "dns_server": "10.1.1.1" } }, "no_ipv6_address": {}, "site_local_network": {}, "mtu": 0, "priority": 0, "not_primary": {}, "monitor_disabled": {} } } } ce-01-master-2-eth0 :: { "metadata": { "name": "ce-01-master-2-eth0", "namespace": "system", "labels": {}, "annotations": {}, "disable": false }, "spec": { "type": "NETWORK_INTERFACE_ETHERNET", "mtu": 0, "dhcp_address": "NETWORK_INTERFACE_DHCP_DISABLE", "static_addresses": [ { "prefix": "10.1.1.6", "plen": 24 } ], "default_gateway": { "default_gateway_mode": "NETWORK_INTERFACE_GATEWAY_USE_CONFIGURED", "default_gateway_address": { "addr": "10.1.1.1" } }, "DNS_server": { "dns_mode": "NETWORK_INTERFACE_DNS_USE_CONFIGURED", "dns_server": [ { "addr": "10.1.1.1" } ] }, "DHCP_server": "NETWORK_INTERFACE_DHCP_SERVER_DISABLE", "vlan_tagging": "NETWORK_INTERFACE_VLAN_TAGGING_DISABLE", "device_name": "master-2/eth0", "vlan_tag": 0, "priority": 0, "interface_ip_map": {}, "is_primary": false, "monitor_disabled": {}, "ethernet_interface": { "device": "eth0", "node": "master-2", "untagged": {}, "static_ip": { "node_static_ip": { "ip_address": "10.1.1.6/24", "default_gw": "10.1.1.1", "dns_server": "10.1.1.1" } }, "no_ipv6_address": {}, "site_local_network": {}, "mtu": 0, "priority": 0, "not_primary": {}, "monitor_disabled": {} } } } All interfaces created for eth0 (SLO) .. image:: ./_static/class2-53.png Update existing fleet to add eth0 for all nodes .. image:: ./_static/class2-54.png Create F5XC BGP Settings ~~~~~~~~~~~~~~~~~~~~~~~~ .. image:: ./_static/class2-55.png Instead of manually creating, you can have the options to copy and paste config below. **Please ensure you change the BGP name to reflect your CE** ce-01-bgp-to-ext-frr :: { "metadata": { "name": "ce-01-bgp-to-ext-frr", "namespace": "system", "labels": {}, "annotations": {}, "disable": false }, "spec": { "where": { "site": { "ref": [ { "kind": "site", "namespace": "system", "name": "ce-01" } ], "network_type": "VIRTUAL_NETWORK_SITE_LOCAL", "disable_internet_vip": {} } }, "bgp_parameters": { "asn": 64512, "local_address": {}, "bgp_router_id_type": "BGP_ROUTER_ID_FROM_INTERFACE" }, "peers": [ { "metadata": { "name": "ce-01-master-0-to-ext-frr-peer", "disable": false }, "external": { "asn": 64512, "address": "10.1.1.9", "port": 179, "interface": { "namespace": "system", "name": "ce-01-master-0-eth0", "kind": "network_interface" } }, "passive_mode_disabled": {}, "target_service": "frr" }, { "metadata": { "name": "ce-01-master-1-to-ext-frr-peer", "disable": false }, "external": { "asn": 64512, "address": "10.1.1.9", "port": 179, "interface": { "namespace": "system", "name": "ce-01-master-1-eth0", "kind": "network_interface" } }, "passive_mode_disabled": {}, "target_service": "frr" }, { "metadata": { "name": "ce-01-master-2-to-ext-frr-peer", "disable": false }, "external": { "asn": 64512, "address": "10.1.1.9", "port": 179, "interface": { "namespace": "system", "name": "ce-01-master-2-eth0", "kind": "network_interface" } }, "passive_mode_disabled": {}, "target_service": "frr" } ] } } Login to ext_router ~~~~~~~~~~~~~~~~~~~ Configure external router to do iBGP with CE nodes. ext-router :: ubuntu@ext-router:~$ vtysh % Can't open configuration file /etc/frr/vtysh.conf due to 'Permission denied'. Hello, this is FRRouting (version 7.2.1). Copyright 1996-2005 Kunihiro Ishiguro, et al. ext-router# Execute the following command :: ext-router# configure terminal ext-router(config)# router bgp 64512 ext-router(config-router)# neighbor 10.1.1.4 remote-as 64512 ext-router(config-router)# neighbor 10.1.1.5 remote-as 64512 ext-router(config-router)# neighbor 10.1.1.6 remote-as 64512 ext-router(config-router)# end ext-router# wr Note: this version of vtysh never writes vtysh.conf Building Configuration... Integrated configuration saved to /etc/frr/frr.conf [OK] Show running configuration :: ext-router# show running-config Building configuration... Current configuration: ! frr version 7.2.1 frr defaults traditional hostname ext-router log syslog informational no ipv6 forwarding service integrated-vtysh-config ! router bgp 64512 neighbor 10.1.1.4 remote-as 64512 neighbor 10.1.1.5 remote-as 64512 neighbor 10.1.1.6 remote-as 64512 ! line vty ! end show ip bgp summary :: ext-router# show ip bgp summary IPv4 Unicast Summary: BGP router identifier 10.1.20.21, local AS number 64512 vrf-id 0 BGP table version 1 RIB entries 1, using 184 bytes of memory Peers 3, using 61 KiB of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.1.1.4 4 64512 15 14 0 0 0 00:00:59 1 10.1.1.5 4 64512 15 14 0 0 0 00:00:59 1 10.1.1.6 4 64512 15 14 0 0 0 00:00:57 1 Total number of neighbors 3 show ip route :: ext-router# show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued route, r - rejected route K>* 0.0.0.0/0 [0/100] via 10.1.1.1, ens5, src 10.1.1.9, 00:12:21 C>* 10.1.1.0/24 is directly connected, ens5, 00:12:21 K>* 10.1.1.1/32 [0/100] is directly connected, ens5, 00:12:21 B>* 10.1.1.100/32 [200/255] via 10.1.1.4, ens5, 00:01:25 * via 10.1.1.5, ens5, 00:01:25 * via 10.1.1.6, ens5, 00:01:25 C>* 10.1.20.0/24 is directly connected, ens6, 07:38:43 .. NOTE:: BGP route will only shown when a HTTP/TCP LB configured to advertise custom. As shown above, 10.1.1.100 is the advertised VIP. When traffic hit the external router, external router will send to CE node respectively. - master-0 [10.1.1.4] - master-1 [10.1.1.5] - master-2 [10.1.1.6] Validate Console BGP Configuation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. image:: ./_static/class2-61.png .. image:: ./_static/class2-62.png .. image:: ./_static/class2-63.png .. toctree:: :maxdepth: 3 :glob: