Note on Policy Components in Cisco SD-WAN

There are 2 types of policies in Cisco SD-WAN.

1. Centralized

2. Localized

Centralized Policies are applied at the WAN side (Northbound) and Localized Policies are applied at the LAN side (Southbound) of the WAN Edges.

Centralized Polices

These policies are pushed to vSmart from vManage and vSmart will push them to WAN edges.

In order to configure these type of policies, the vSmart should be configured via the vManage templates. Which means if it is configured only via the CLI while it was onboarded, you will have to reconfigure it using templates.

You will not see these policies in the running config of WAN edges but there are commands to view the pushed policies.

There are 2 types of Centralized Polices too.

1. Topology Policies (Control Policies)

2. Traffic Polices (Data Policies)

Topology Policies 

These are there to control the OMP protocol which is basically the routing updates which effects the routing tables. So if you want to do route filtering, change next hop of routes etc. you have to use Topology Policies under Centralized Policies.

Traffic Policies 

These are the policies which effects the data plane only. 

This can be used to alter data plane based on L3/L4 characteristics like source IP / destination IP type of things without effecting to Control Plane. Traffic Data Policies are used to configure local internet breakouts (Direct Internet Access / DIA).

When this is configured based on applications, it is called AAR (Application Aware Routing). In AAR; the WAN edge will look for the application and will override the routing table for that specific traffic according to the Policy configured. Note that it will not change the routing table but will change the way data is forwarded. It can be configured for SLAs with link quality etc. as well.

Configuring Centralized Policies?

You can configure them via Cisco vManage > Configuration > Polices > Centralized Policies

You can configure many Centralized Policies in vManage and There can be only one active Centralized Policy at a time. When you deploy one policy, other policies are deactivated..

There are 4 steps to configure a Centralized Policy.

1. Create Groups of Interest (Objects needed to configure a policy)

2. Configure Topology and VPN Membership (Topology Policy / Control Policy component)

3. Configure Traffic Rules ( Traffic Policy / Data Policy component)

4. Apply Policies to Sites and VPNs

You can bypass some steps if there is none to configure under that step. As an example, an AAR may not need to configure a Topology Policy hence the step 2 will be ignored.

Examples:-

Configuring AAR (Application Aware Routing) Policy in Cisco SD-WAN

Manipulating Traffic Flow using TLOCs in Cisco SD-WAN (Topology Policies)

Configuring Route Filtering using Topology / Control Policies in Cisco SD-WAN

Creating a Controlled / Flexible Hub & Spoke Topology using TLOCs

Configuring Direct Internet Access / Local Breakout in Cisco SD-WAN Edges

Localized Policies

These policies will be also configured in vManage but directly send to the WAN edges and they can be found in the running config in vEdges / sdwan running config in cEdges.

Used to configure QoS in local WAN edge, Configure ACLs or Route Policies for a local WAN edge etc.

Configuring Localized Policies?

There are 5 steps to configure a localized policy.

1. Create Groups of Interest (Objects needed to configure a policy)

2. Configure Forwarding Classes/QoS

3. Configure Access Control Lists

4. Configure Route Policy

5. Policy Overview

Examples:-

Configuring QoS in Cisco SD-WAN Edges by Localized Policies

======================================================================

About Groups of Interests / Lists

The components which are needed while configuring a policy are called Groups of Interests / Lists.

They can be configured before configuring policy as well as on the go.

Just by looking at the above lists you can get an idea of the things which can be configured under each policy type.

Figuring out the Exit Interface of an OMP Route in cEdges by Routing Tables

Let's start from looking into the configured VRF numbers through CLI by command show vrf

(click on the images to view in full size)

As we can see the; VRF 1 is configured in this router. Other 2 VRFs are default configured VRFs. So this VRF 1 is the only Service VPN we can see here which the OMP routes are learned.

Let's check the routing table of VPN 1 by following command,

show ip route vrf 1

In OMP (Overlay Management Protocol), there is this concept called TLOC (Transport Locator) to identify the hop and transport which the traffic should be forwarded to.

TLOC contains of 3 components;

1. TLOC IP (System IP of the Hop)

2. Color (the transport eg:- biz-internet, MPLS, etc)

3. Encapsulation Method

Let's analyze a specific route, 192.168.11.0/24, the next-hop is 10.1.1.111 which is actually the TLOC IP (System IP of the WAN edge router) which is not need to be routed from other WAN edges, just like a OSPF router id. So let's find the public IPs of the TLOC transports by the following command..

show sdwan omp tlocs | b 10.1.1.111

So to reach the TLOC IP of 10.1.1.111, there are 2 public IPs 172.16.1.1 and 10.10.11.1

Now let's check how those IPs can be reached through the default VRF which is actually the underlay VPN (VPN 0) of SD-WAN domain.

show ip route

So as you can see from the above output, the traffic will be load balanced between GE 1 and GE 2 interfaces.  

All above was done to resolve next hop IP and exit interface by examining routing tables. Following command will give you the same information at once and hope now you know how it is derived and that is nothing but the forwarding table.

show sdwan ip fib

Red box shows the next hop addresses of the route in the Service VPN (Overlay VPN).

To find the physical exit interface; 

show ip cef

Note that since AAR (App Aware Routing) in SD-WAN is defining the traffic forwarding in case it is configured. You can simulate to visualize the actual traffic flow in vManage GUI interface.

Figuring out the Exit Interface of an OMP Route in vEdges by Routing Tables

Let's start from show ip routes command in CLI.

(Click on the image to see in actual size)

You can see there are 2 VPNs (0 and 1) and several routes learned via static, connected, OSPF and OMP. All other routes except OMP are straight forward just like in normal Cisco IOS; either there is a next hop or else directly connected. 

In OMP (Overlay Management Protocol), there is this concept called TLOC (Transport Locator) to identify the hop and transport which the traffic should be forwarded to.

TLOC contains of 3 components;

1. TLOC IP (System IP of the Hop)

2. Color (the transport eg:- biz-internet, MPLS, etc)
3. Encapsulation Method

OMP routes are learned via a Service VPN only which in this case the VPN 1. Let's analyze a specific route, 192.168.22.0/24

So there are 2 TLOCs which the traffic should be forwarded to, which has the same TLOC IP but different transports. Which means this subnet is learned from these 2 transports from same WAN edge.

Now let's see how to reach this TLOC IP. 

The thing is that the TLOC IP is not need to be routed from other WAN edges, it's the System IP of the WAN edge just like a OSPF router id. So let's find the public IPs of the TLOC transports by the following command..

show omp tlocs | b 10.1.1.112

Now it's the time to look for the next hops in VPN 0 routes for the above 2 public IPs.

So the traffic is load balanced through the 2 interfaces.

All the above was done to resolve next hop IP and exit interface by examining routing tables. Following command will give you the same information at once and hope now you know how it is derived and that is nothing but the forwarding table.

show ip fib

Red box shows the next hop addresses of the route in the Service VPN (Overlay VPN) and Purple lines show the real next hop addresses in VPN 0 (underlay VPN) if any and the exit physical interfaces.

Note that since AAR (App Aware Routing) in SD-WAN is defining the traffic forwarding in case it is configured. You can simulate to visualize the actual traffic flow in vManage GUI interface.

If AAR is not configured, it will show the same result as we found from above CLI analysis.

Go to vManage > Monitor > Devices > Select the Device

Troubleshooting (left side)  > Simulate Flows under Traffic and enter the values.

Initializing cEdges in EVE-NG

This post is about making cEdges working inside EVE-NG simulation environment which does not focus on basic concepts.

Following is the network diagram. Please note that vManage, vBond and vSmart and vEdges are already initialized here. Also the vEdges and cEdges are added to vManage via serialFile.viptela file. You can find those in previous posts.

cEdge image I am using in this lab is CSR1000v IOS-XE version 16.12.03.

After bootup, you can console it and it will ask to change the password where the default username and passwords are both "admin".

Configuring cEdges is slightly different from configuring vEdges but the functionality is basically the same. According to the lab diagram, following are the configuration needed for vEdge1. It is just like the Cisco CLI.

!

config-transaction

!

hostname cEdge1

!

system

 system-ip 10.1.1.112

 site-id 2

 vbond 10.10.10.20

 organization-name TEST-ORG1

 exit

 clock timezone GST 5 30

!

int GigabitEthernet1

 no shut

 ip address 10.10.22.1 255.255.255.252

 exit

ip route 0.0.0.0 0.0.0.0 10.10.22.2

ip route 0.0.0.0 0.0.0.0 172.16.1.254

!

interface Tunnel1

 no shut

 ip unnumbered GigabitEthernet1

 tunnel source GigabitEthernet1

 tunnel mode sdwan

 exit

!

sdwan

 interface GigabitEthernet1

  tunnel-interface

   encapsulation ipsec

   allow-service all

   allow-service sshd

  commit

!

I have added only the cEdge1 config here; For cEdge2, System IP, Site ID, IP addresses will be changed.

Key Things to Note:-

system-ip is just an ID, which does not need to be routed. It is there to identify the device and it's just a number like OSPF router-id.
organization-name is very important as all the controllers, edges and the controller profile in smart account also need to match..
you can allow all services or just limit to sshd, https etc only.

You will not see "interface Tunnel1" command in context sensitive help but it will work after you enter the command. This tunnel number directly mapping the interface number at the backend. Ex:- Tunnel1 is mapping the GigabitEthernet1.

**You can see the VPN 0 is not defined in configuration, as there is no such concept in cEdges. It has VRF concept instead. When the interfaces are not defined with a VRF, they belong to the default which acts as the transport VPN.

As you can see, there are 2 default routes because each transport must have reachability to vBond and vSmart in order for OMP (Overlay Management Protocol) to work properly on both transports. More abut this and a workaround to overcome this will be discussed in a later post.

Now let's add the cEdge to vManage.

To do that we need to have the Root Cert installed in to the cEdge. We will do it by making CA Server a TFTP server and copy the CA Root Cert from CA Server to cEdge1 via TFTP.

!

copy tftp://10.10.10.40/CARoot.cer flash:CARoot.cer

!

request platform software sdwan root-cert-chain install bootflash:CARoot.cer

!

request platform software sdwan vedge_cloud activate chassis-number <> token <>

!

The chassis number and token can be found in Configuration > Devices > WAN Edge List on vManage.

If everything worked well, we can see the State as "certificate installed" in WAN Edge List on vManage.

Initializing vEdges in EVE-NG

This post is about making vEdges working inside EVE-NG simulation environment which does not focus on basic concepts.

Following is the network diagram. Please note that vManage, vBond and vSmart are already initialized here. Also the vEdges and cEdges are added to vManage via serialFile.viptela file. You can find those in previous posts.

vEdge version used in this lab is 20.7.1

After bootup, you can console it and it will ask to change the password where the default username and passwords are both "admin".

According to the lab diagram, following are the configuration needed for vEdge1. It is just like the Cisco CLI.

!

config t

!

system

 host-name vEdge1

 system-ip 10.1.1.111

 organization-name TEST-ORG1

 site-id 1

 vbond 10.10.10.20

 clock timezone Asia/Colombo

!

vpn 0

 ip route 0.0.0.0/0 10.10.11.2

 ip route 0.0.0.0/0 172.16.1.254

 interface ge0/0

  ip address 10.10.11.1/30

  no shut

  tunnel-interface

   encapsulation ipsec

   allow-service all

   allow-service sshd

   exit

 exit

! 

commit

!

Key Things to Note:-

system-ip is just an ID, which does not need to be routed. It is there to identify the device and it's just a number like OSPF router-id.
organization-name is very important as all the controllers, edges and the controller profile in smart account also need to match..
ge0/0 is the default interface configured for VPN0 (the underlay VPN)

You can allow all services or just limit to sshd, https etc only.

As you can see, there are 2 default routes because each transport must have reachability to vBond and vSmart in order for OMP (Overlay Management Protocol) to work properly on both transports. More abut this and a workaround to overcome this will be discussed in a later post.

Now let's add the vEdges to vManage

To do that we need to have the Root Cert installed in to the vEdge. We will do it by making CA Server a WinSCP server and upload the CA Root from CA Server to vEdge1 via SFTP (port 22).

Default folder in vEdge is /home/admin

Now enter the command "request root-cert-chain install /home/admin/CARoot.cer" in vEdge CLI

Then the activation command "request vedge-cloud activate chassis-number <####> token <####>"

The chassis number and token can be found in Configuration > Devices > WAN Edge List on vManage.

Now what happens at the backend is that the vEdge will communicate the vBond to get it onboarded. You can see the State as "certificate installed" in  WAN Edge List on vManage if everything went well.

Certificate Installed means that the vManage is creating and installing the ID Cert of vEdge into vEdge after onboarding automatically.

Note that this is possible because "WAN Edge Cloud Certificate Authorization" is set to "Automated" in Administration > Settings. If it is set to "Manual" you have to copy the CSR and sign it and install like we did to onboard vBond & vSmart.

Additionally you can test by CLI command "show control connections"

Adding vEdges / cEdges List to vManage in EVE-NG

This post is about making vEdges and cEdges working inside EVE-NG simulation environment which does not focus on basic concepts.

You need to have a Cisco Smart Account with you for this. 

Go to Cisco Software Central > Network Plug and Play Connect

 Select Controller Profiles

  + Add Profile

    Controller type -> VBOND

    Fill the reqired details like Profile Name, Organization Name, Primary Controller IP address which        is the VBOND IP

(click on the image to view in full size)

Then need to add the devices going again to Cisco Software Central > Plug and Play Connect

 Select Devices > +Add Software Devices

In Identify Device you should provide the Base PID (VEDGE-CLOUD-DNA for vEdges and CSR1KV for my cEdge), Quantity and the Controller Profile which you created for VBOND..

Now if you go to Cisco Software Central > Plug and Play Connect > Devices, you can see the devices are provisioned.

From now onwards you can do 2 things to onboard the provisioned devices to vManage.

1. Let vManage sync with your Cisco Smart Account through internet

2. Download serialFile.viptela from Smart Account portal and upload it to vManage manually

I am using the second option.

Go to Cisco Software Central > Plug and Play Connect

 Select Controller Profiles

 Download the "Provisioning File"

Now go to vManage > Configuration > Upload WAN Edge List and upload the downloaded file.

You can see the list of Devices are pushed to the vManage, vBond and vSmart.

You can check this from vBond CLI also by following command.

There is another command which shows same kind of output in vManage and vSmart which is "show control valid-vedges"

Initializing vSmart in EVE-NG

Please note that his post is just about deploying the Cisco SD-WAN components in EVE-NG hence concepts are not discussed.

Following is the lab used in this post.

vSmart version used in this lab is 20.7.1

After bootup, you can console it and it will ask to change the password where the default username and passwords are both "admin".

According to the lab diagram, following are the configuration needed. It is just like the Cisco CLI

!

config t
!
system
 host-name vSmart
 system-ip 10.1.1.103
 organization-name TEST-ORG1
 site-id 100
 vbond 10.10.10.20
 clock timezone Asia/Colombo
!
vpn 0

 ip route 0.0.0.0/0 10.10.10.1
 interface eth0
  ip address 10.10.10.30/24
  no shut
  tunnel-interface
  allow-service all

commit
!

Key Things to Note:-

system-ip is just an ID, which does not need to be routed. It is there to identify the device and it's just a number like OSPF router-id.
organization-name is very important as all the controllers, edges and the controller profile in smart account also need to match.
site-id should also be same in all controllers in order to sync/work.
eth0 is the default interface configured for VPN0 (the underlay VPN), you can allow all services or just limit to sshd, https etc only.
10.10.10.1 which is the default route is inside the Site-1 Core Switch as a VLAN interface.

Now let's add the vSmart to vManage

Go to Configuration > Devices

  Select "Controllers"

    Add Controller

      Select vSmart and give the VPN0 interrface IP address  as Management IP and username and password

      tick the Generate CSR and hit Add""

Now Go to Configuration > Certificates

 Select controllers

 Click on the 3 dots at the right side of the controller and select Generate CSR

 Download the CSR and send it to CA Server to get it signed..

After you have it signed, you can Go to Configuration > Certificates again and hit "Install Certificate" 

and paste the certificate content as text to install the certificate.

After few seconds, the status will turn to green with a Success..

Additionally you can test by CLI command "show control connections"

Initializing vBond in EVE-NG

Please note that his post is just about deploying the Cisco SD-WAN components in EVE-NG hence concepts are not discussed.
Following is the lab used in this post.

vBond in EVE-NG is the same image as vEdge and the version used in this lab is 20.7.1

After bootup, you can console it and it will ask to change the password where the default username and passwords are both "admin".

According to the lab diagram, following are the configuration needed. 

It is just like the Cisco CLI

!

config t

!

system

 host-name vBond

 system-ip 10.1.1.102

 organization-name TEST-ORG1

 site-id 100

 vbond 10.10.10.20 local

 clock timezone Asia/Colombo

!

vpn 0

 ip route 0.0.0.0/0 10.10.10.1

 interface ge0/0

  ip address 10.10.10.20/24

  no shut

  tunnel-interface

   encapsulation ipsec

   allow-service all

commit

!

Key Things to Note:-

system-ip is just an ID, which does not need to be routed. It is there to identify the device and it's just a number like OSPF router-id.

organization-name is very important as all the controllers, edges and the controller profile in smart account also need to match.

ge0/0 is the default interface configured for VPN0 (the underlay VPN), you can allow all services or just limit to sshd, https etc only.

site-id should also be same in all controllers in order to sync/work.

"local" keyword specifies that this is the vBond itself.

Now let's add the vBond to vManage

Go to Configuration > Devices

  Select "Controllers"

    Add Controller

      Select vBond and give the VPN0 interrface IP address  as Management IP and username and password

      tick the Generate CSR and hit Add""

Now Go to Configuration > Certificates

 Select controllers

 Click on the 3 dots at the right side of the controller and select Generate CSR

 Download the CSR and send it to CA Server to get it signed..

After you have it signed, you can Go to Configuration > Certificates again and hit "Install Certificate" 

and paste the certificate content as text to install the certificate.

After few seconds, the status will turn to green with a Success..

Additionally you can test by CLI command "show control connections"

Initializing vManage in EVE-NG

Please note that his post is just about deploying the Cisco SD-WAN components in EVE-NG hence concepts are not discussed.

Following is the lab used in this post.

As you can see, there are 3 WAN sites. All the devices will be configured in upcoming posts.

vManage version used in this lab is 20.7.1

After bootup, you can console it and it will ask to change the password where the default username and passwords are both "admin". Then it will ask to format the disk.

It will take some time to perform the format operation..
According to the lab diagram, following are the configuration needed.
It is just like the Cisco CLI

!

config t

!

system

 host-name vManage

 system-ip 10.1.1.101

 organization-name TEST-ORG1

 site-id 100

 vbond 10.10.10.20

 clock timezone Asia/Colombo

!

vpn 0

 ip route 0.0.0.0/0 10.10.10.1

 interface eth0

  ip address 10.10.10.10/24

  no shut

  tunnel-interface

   allow-service all

   commit

!

Key Things to Note:-

system-ip is just an ID, which does not need to be routed. It is there to identify the device and it's just a number like OSPF router-id.

organization-name is very important as all the controllers, edges and the controller profile in smart account also need to match.

Eth0 is the default interface configured for VPN0 (the underlay VPN), you can allow all services or just limit to sshd, https etc only.

After login in to the GUI, two of the things which were configured from the CLI needs to be configured again.

1. Organization name

2. vBond IP

To do that, Go to Administration > Settings and configure the 1st two options.

Now what is remaining is installing the certificates.

Each SD-WAN device must have 2 certificates.

1. Root Cert

2. ID Cert

ID Cert is the certificate which belongs to the device itself.

Root Cert is the CA Cert which is used to generate the CSR of the ID cert to get signed by the CA.

First the CA Root cert must be downloaded to the vManage.

Download the CA Cert as Base 64

Go to Administration > Settings 

  Edit Controller Certificate Authorization

   Change Certificate Sigining by to Enterprise Root Certificate

   Paste the Root CA as text

   Tick "Set CSR Properties"

    Enter the correct Domain Name, Organization, City, State, Email & Country Code

    Hit "Import & Save"

 

Then the CSR should be generated and send to CA to Sign and it becomes the ID Cert of vManage.

Now Go to Configuration > Certificates

 Select controllers

 Click on the 3 dots at the right side of the controller and select Generate CSR

 Download the CSR and send it to CA Server to get it signed..

After you have it signed, you can Go to Configuration > Certificates again and hit "Install Certificate" 

and paste the certificate content as text to install the certificate.

After few seconds, the status will turn to green with a Success..

Scalability Flexibility and Modularity in Network Design

Scalability

Ability to perform changes without changing the entire architecture / design.

There are 2 scalability approaches for the network designs.

1. Scale Up
2. Scale Out

Scale Up

Increase the existing system resources without adding a new system.

Scale Out

Having one physical router which can add multiple line cards later on is an example for Scale Up type of scalable solution while having 2 physical routers is considered as the Scale Out solution.

image ref: medium.com

Flexibility

Ability to adapt to business changes.

Modularity

Ability to divide by functions or policy boundaries.
There are 3 modularity approaches which provides flexility to a design.

1. Choosing the physical topology

Some topologies such as hierarchical or leaf and spine are easier to work with adding modules than fully meshed etc.

2. Splitting Functions or Geographies

Separating campus, branches, data center, internet edge etc or security policy boundaries make the design easier to upgrade, manage etc.

3. Break into smaller pieces

Creating smaller fault domains so that a failure on a part of the network doesn’t impact whole system. Not extending the spanning tree domain between data centers is an example.

Modular design allows different modules to be managed by different teams. Network Team, Firewall team, Data center team etc in Enterprise Networks or Core network and Access network in Service Provider networks are examples.

Also modular designs can reduce the configuration overhead, template based configuration in SD WAN is an example.

Packet Delivery Parameters and Reliability in Network Design

There are 3 packet delivery parameters.

1. Delay / Latency

Time which a legitimate packet takes to travel from source to destination. 

2. Jitter

Consistency of delay / latency

3. Packet Loss / Drop Ratio

Fraction of packets sent by the source but not received by the destination.

General accepted best practices for the Delay, Jitter and Packet Loss ratio has been defined per type of applications. For example, one way Delay for VoIP (mouth to ear delay) should be less than 150ms, Jitter should be less than 30ms and PLR should be below 1%.

Reliability

Delivering the legitimate packets from source to destination within a reasonable delay / latency which is defined based on the application type and architecture.

Reliability is often mentioned in choosing links. As an example, if you have to utilize a mix of reliable and unreliable links, best practice is to carry VoIP traffic over the reliable links which don’t have packet loss and latency, and use the cheaper unreliable links such as internet to transport packet loss tolerant application traffic.

But reliability should be considered for everything in the path, including links, devices such as switches, routers, firewalls, application delivery controllers, servers, storage systems etc. Even the hardware onboards should be reliable too. That’s why vendors bother with ASICs, Quantum Flow processors etc in their marketing propagandas.