Configuring QoS in Cisco SD-WAN Edges by Localized Policies

There are 8 Queues in Cisco SD-WAN edges defined from 0 - 7.

Queue 0 is the LLQ (Low Latency Queue) which the network control traffic like OMP, BFD uses.

All other traffic is going through the Queue 2 by default and with Localized Policies you can configure the traffic you want to go through any Queue.

Let's take an example using following topology.

For this example I am taking the vEdge1 in Site-1. cEdges are also same in configuration as this is done through vManage.

Send VoIP traffic through LLQ and send Web Traffic as 2nd higher priority and give least priority / best effort to the rest of the traffic. Bandwidth reservation should be 10% to VoIP traffic , 20% to Web Traffic and remaining to the rest of the traffic. Note that the Internet Link is 80 MBps. Do this QoS on Internet Link assuming MPLS link is not congesting.

Since QoS is configured under Localized Policies, following are the steps to create it.

1. Create Groups of Interests

2. Configure Forwarding Classes/QoS

3. Configure Access Control Lists

4. Configure Route Policy

5. Policy Overview

So we can skip step 4 as we don't need to configure route policies to achieve this.

Configuration > Policies > Localized Policies

Add Policy

1. Create Groups of Interests

Configuration > Policies > Custom Options > Localized Policy > Lists > Class-Maps

2. Configure Forwarding Classes/QoS

(click on the image to view in full screen)

3. Configure Access Control Lists

Remember to set Default Action to Accept Enabled to rest of the traffic.

5. Policy Overview

Cloud QoS Service Side needs to be ticked because we are applying it in the Service / LAN side.

If this policy required Deep Packet Inspection to work means if the policy is configured for Applications in Layer 7, Application check box need to be checked.

Applying to Devices

Unlike Centralized Policies, Localized Polices are applied via the templates as they are going to the running configuration of the WAN edges.

First let's send the QoS MAP, Policy Map and ACLs to the devices via Device Templates.

It is under the Additional Templates section in the Device Template.

Then have to apply the ACL in ingress interface and the Policy Map in egress interfaces by Interface Templates.

Note that this should be done after the QoS MAP, Policy Map and ACLs are pushed because it will not work when the Interface configuration tries to refer the ACL name etc if it is not done upfront.

Let's apply the QoS Map to G0/0 interface 1st and this is under the ACL/QOS section of the Interface Template.

Next the ACL to G0/2 interface.

Set Redundancy Through Another WAN Edge using TLOC Extensions

This method becomes handy when there are 2 physical WAN Edges at a branch, each with a separate transport like in the following design. I am only going to explain the logic behind this and how it works, not going to make this a configuration guide.

In this example, cEdge2 is taken in to consideration. As you can see, cEdge2 has only one transport which is a MPLS connection. On the other hand, cEdge1 has another transport to Internet (Biz-Internet). Both has internal connection to Site-2 Core Switch but Gi3 interface of cEdge2 is configured with OSPF only to learn 192.168.22.0/24. So cEdge2 will only advertise 192.168.22.0/24 subnet to OMP via its transport MPLS.

Requirement is like the following..

Make a redundant path for 192.168.22.0/24 subnet through cEdge1 router which means utilize the Internet Link of cEdge1 to give redundancy to the MPLS traffic going through cEdge2.. In other words, 192.168.22.0/24 should be able to access from vEdge1 even after MPLS of cEdge2 is down even though it seems the only path.. 

This can be achieved by configuring a TLOC Extension interface on Gi2 interface of cEdge1.

To do that, following things are required to be done.

1. Gi1 of cEdge2 should be configured as a Tunnel Interface of Biz-Internet color.

2. A default route is needed in cEdge2 to reach controllers and biz-internet cloud; next-hop should be the Gi2 of cEdge1.

3. Gi2 of cEdge1 should be configured as a TLOC Extension Interface, a Non Tunnel Interface.

4. There must be a route from Internet to reach 10.12.12.0/30 subnet, the next-hop should be Gi1 interface of cEdge1 which is for the reverse traffic.

5. There should be NAT enabled on Gi1 interface of cEdge1.

To make the Gi2 Interface of cEdge1, navigate to Advanced in Interface Template of G2.

TLOC Extension field should be configured as "GigabitEthernet1" which specifies the interface it will forward the traffic to; from the TLOC Extension Interface.

NAT Configuration will be like the following in Interface Template of Gi1 of cEdge1.

Routes will be like the following from vEdge1;

Biz-Internet TLOC from 10.1.1.113 will be like the following in show omp tlocs output.

Let's check what will happen to the OMP routes when the MPLS link of cEdge2 is down..

It turned the transport in TLOC to biz-internet.

Notes:-

If the TLOC Extension's transport was not Internet, (a private transport); you may not need to configure NAT but will need to advertise the subnet of interlink to service provider etc though BGP or something for the reverse traffic.

If the inter-link is routed through L3 network, GRE tunnel should be configured, but only cEdges will support, vEdges need direct or L2 connectivity.

BFD session for biz-internet will be down while MPLS link is up, it will come up as soon as MPLS link is down.

Before:

After:

Configuring Direct Internet Access / Local Breakout in Cisco SD-WAN Edges

This is an example of configuring branch offices with internet access from the same WAN edge interfaces which are used also for building tunnels in internet transport.

It is just NAT combined with a Traffic Data Policy under Centralized Policies.

Following is the requirement.

All Site LAN traffic from WAN sites should be routed to internet through Gi1 interface

1st let's configure NAT on Gi1 interfaces via corresponding interface templates.

Then the Policy.

Let's create the Group of Interest required, Data Prefix..

It should be a Traffic Data Policy and It's under the Traffic Rules.

Sequence Type should be Custom.

There must be 2 Sequence Rules to be configured.

1st one should Match the Data Prefix created earlier "LAN" as the source and destination and it should be Accepted means do nothing, normal processing for WAN routing..

2nd one should Match the same Data Prefix and the Set Action should be NAT VPN.

**Also don't forget to select the Default Action under the Sequence to Accept and Enabled otherwise other traffic will be dropped.

Policy Application also should be in Traffic Data, Sites are "ALL-SITES" in my example, VPN should be "DATA". Those values are previously configured by me as Groups of Interests.

Then Save and Activate..

Verification:-

Getting OMP Routes of a TLOC which Cannot Reach Controllers from its Transport

There can be a transport of a WAN edge which will not connect connect to controllers. May be the MPLS type of private transports will not have routing to controllers in the public cloud. In such cases, vSmart will not learn that TLOC and will not be used in OMP routes. This is the default behavior and in most cases you may have seen the designs with a separate router or switch providing reachability for that specific TLOC to controllers.

If you noticed in the following diagram also, I have added a link between MPLS cloud to Internet to counter this behavior.

You can see the mentioned link in red and I am going to remove it and make it work like it is not necessary. 

Before doing this, let's examine the routing tables.

(click on the images to view in full size)

Ok, routes are learned via both transports, removing the link and clearing control connections..

As a result, MPLS routes are gone, now the workaround..

There is a setting called max-control-connection under VPN interface. Default value is 2 and it means 2 controllers can connect to a TLOC. But if we change this value to "0", it will not expect to connect to any controller over that interface and will advertise the TLOC for MPLS to vSmart from other control connections. In the above case, it's Biz-Internet..

Since these WAN egdes are managed by vManage, we need to change the configuration from editing the templates. It is in the Interface Feature template which relates to the MPLS transport interface. 

After configuring this on all WAN edges;

It has worked and learned all MPLS routes.

Note that if it is configured on only one router, that router will send its MPLS TLOCs to vSmart and vSmart will send them to other routers but these other routers will make it an invalid route as their MPLS transport is not included in OMP routing. So in order for this to work, it should have done from other routers as well so that Data Plane tunnels will be established.

Only the Control Plane tunnels to controllers will be established from Biz-Internet transport as a single point of failure now anyway.

Creating a Controlled / Flexible Hub & Spoke Topology using TLOCs

I guess you have seen the following option which allows you to easily create a Hub & Spoke topology under Centralized Policies.

Here you just want to name the Hub site and what should be the spokes and then deploy which will make a Hub & Spoke WAN topology easily.

But there is a draw back for this as if you want to change some traffic as you want, like for a specific hub to talk to a another hub.

So creating a Hub & Spoke type WAN network using TLOCs and Custom Topology Policy makes it handy for real world.

It is just nothing but changing the TLOC of all the routes learned via spokes to a specific TLOC which the Hub should be.

For the following design, let's assume the cEdge1 needs to configure as the hub.

Site List specifying spoke sites and a TLOC List which is the Group of Interest should be configured like the following..

Use the Custom Control (Route & TLOC) as the Topology Type in Control Policy.

 

(click on the image to view in full size)

Match condition should be the created Site List "SPOKES" and the Set condition should set the Action to TLOC and the TLOC List should be "SITE-2-TLOCs" which were created before.

**Also don't forget to select the Default Action under the Sequence to Accept and Enabled otherwise other traffic will be dropped.

Now Apply the Policy with the Outbound Site List as "SPOKES" again.

Now Activate the new policy.

Spokes:-

Hub :-

Configuring Route Filtering using Topology / Control Policies in Cisco SD-WAN

I am goin to create a Topology Policy separately to achieve a route filtering requirement.

This post is also another example for Topology Policies just like the one I did for Traffic Manipulation using TLOCs. Following is the topology for the lab.

Following is the requirement of this lab.

Filter all 10.x.x.x routes from OMP routes. Site LAN switches should not know the inter-link subnets.

First let's see a routing table of a one site.

Yes it seems there are unwanted inter-link routes of 10.xxx which are learned via OSPF from vEdges which originally came from OMP.

Now navigate to Policy Configuration page to create a new policy.

Click on Add Policy.

For this one I am creating new Groups of Interests which are Site List which contains all the site IDs from 1 to 3 and a Prefix List. This can be done directly Custom Options button on the right side of the Policy Configuration Page and selecting Lists under Centralized Policy. 

Add Topology and select Custom Control (Route & TLOC).

Hit + Sequence Type and select Route.

Hit + Sequence Rule

Match condition should be the created Site List "ALL-SITES" and another Match statement should be there under the same sequence for the Prefix List "InterLinks"

The Set condition should set the Action to Reject and Enabled.

**Also don't forget to select the Default Action under the Sequence to Accept and Enabled otherwise other traffic will be dropped.

Now Hit Next, No need to create Traffic rules as this is only a Topology Policy.

Now go to the Policy Application, there you will do the following configuration under CONTROL2 which is the Topology Policy name added to this Policy.

Outbound Site List should be selected only and it should be "ALL-SITES"

Now all done, Save the Policy and Activate it and let's check the routing tables.

Now will it be able to ping Site-2 LAN??

Yes it will but you have to source the VLAN 1.

Important Note:-

I created this as a separate new Centralized Policy because I could not attach it to the previous one. The reason is that while applying this Topology Policy, it conflicts with the previous Topology Policy because of the Sites of Application. You cannot apply 2 Topologies in one Centralized Policy if the Applied Sites overlap. My Previous Policy's applied Site was Site-3 which is included in my new All-Sites Site List.

Manipulating Traffic Flow using TLOCs in Cisco SD-WAN (Topology Policies)

Topology Policies (Control Policies) are one of the 2 main Centralized Policy types which modify the OMP updates to steer the traffic.

The orange lines in the following diagram shows the pathways to reach the 192.168.22.0/24 network from cEdge2 as both the Internet link and MPLS links are load balancing by default.

We can verify this from cEdge2 router by following command output.

(click on the image to view in full size)

Let's see how to cater the following requirement.

Traffic going to 192.168.22.0/24 (SITE2-TOKYO) from cEdge2 (SITE3-SYDNEY) must use MPLS as primary link and Internet Link as the secondary link.

Since I have already created a Centralized Policy for AAR which does not have a Topology (Control) component, I am going configure it under the same policy as there can be only one active Centralized Policy at a time.

Following link shows the AAR example.

https://www.roshanzlab.net/2025/01/configuring-aar-application-aware.html

Still need to create Groups of Interests.

Click on the Custom Options button on the right side of the Policy Configuration Page.

Click on the Lists and the select TLOC to create TLOC Lists.

Create the 2 Sites also.

Now it's time to edit the Centralized Policy.

Go to the Topology and hit Add Topology and select Custom Control (Route & TLOC).

Hit + Sequence Type and select Route.

Hit + Sequence Rule

Match condition should be the created Site List "SITE2-TOKYO" and the Set condition should be the created TLOC List "SITE2-TLOCs"

**Also don't forget to select the Default Action under the Sequence to Accept and Enabled otherwise other traffic will be dropped.

(click on the images to view in full size)

Hit Save Control Policy.

Now go to the Policy Application.

Outbound Site List should be selected only and it should be "SITE3-SYDNEY"

Now all done, Save the Policy and Activate it.

Now let's inspect the routing table of cEdge2 (Site3)

Ok, as we can see the Internet route for 192.168.22.0/24 route is gone. Let's check what will happen after I Pause the MPLS link of cEdge1.. (Pausing can be done in PNET Lab which I am doing this lab on, hopefully EVE-NG also can. It is achieved through right clicking the connection and selecting Suspend)

As you can see, it works.

Note that cEdge2 actually knows both the routes but it only put the highest proffered route in forwarding tables.

You can check this by show sdwan omp routes command.

Big Question :-

How to determine the Match condition Site and Outbound Site in policy?

The green lines indicate how the OMP updates regarding this requirement flows. Because the Centralized Policy is applied to the vSmart and it is doing the route manipulation; site selection in policies should be done with vSmart's point of view. It receives the routes from cEdge1 as "In" and advertises to cEdge2 as "Out". 

Note that since this is a Topology / Control Policy this can only be seen at vSmart CLI by the following command.

show running-config policy