Types & Flaws of OSPF Authentication (to Stop Route Injection & DOS)

OSPF authentication has 3 types.
Type 0 - Null (default)
Type 1 - Simple Password
Type 2 - Cryptic (MD5/SHA)

Let's see how it can be cracked and how it can be  implemented correctly. Authentication is the only protocol level security we can have to secure our OSPF network. Actually in OSPFv2, authentication can only prevent a rouge router from negotiating LSAs to stop route injection & DOS attacks. But still a man in the middle will be able to see the payload of the LSAs which he can compromise the network. Only in OSPFv3, an IPSec tunnel will prevent it happening.

I'm doing it in a simple lab of 2 routers in  area 0, and the wire is sniffed. Let's analyze the packets to see what's really happening in the network layer.
(did't upload initial config files because this is a very simple lab with few basic commands)

Let the 2 routers form adjacency.
Type 0 or Null type is always on by default.
Without doing any authentication configuration, let's capture a Hello packet and analyze them.

You can see the the authentication type is Null (0) and authentication data none.

It means you only need a router configured to match the Hello packet attributes (which u can see) and connect a router to the network to start injecting false routes & carry out a DOS attack.

Let's see about Type 2 now. It can be implemented by issuing the commands,
R1(config-if)#ip ospf authentication
in the interface level or
R1(config-router)#area 0 authentication command in the router config mode, anyhow always the interface level command will over run the router config mode command & the authentication keys and authentication data can only be configured in the interface level.

So if you see something like below in a packet capture,

This is Type 2. Even though it is type 2 you can see authentication data is empty.

It means you only need a router configured to authentication type 2 & match the Hello packet attributes (which u can see) and connect a router to the network to start injecting false routes & carry out a DOS attack.

Now let's see if authentication data is not empty (what if there is a password configured to authentication)
It can be only done in the interface level using the following command after setting the authentication type to 2 like above.
R1(config-if)#ip ospf authentication-key cisco468

The packet will be like this.

Now you can see there is a password and you can clearly see it.

It means you only need a router configured to authentication type 2 & match the password & Hello packet attributes (which u can see) and connect a router to the network to start injecting false routes & carry out a DOS attack.

So basically all above are useless against a hacker. So the correct method will be using the Type 3 with a hashing algorithm.

It can be implemented by issuing the commands,
R1(config-if)#ip ospf authentication message-digest 
in the interface level or
R1(config-router)#area 0 authentication message-digest command.

And the password can be given in interface level by issuing,
R1(config-if)#ip ospf message-digest key 1 md5 cisco321

Capture will be like this

Now the password will not be stated in the packet. Only the hash will be there. So even though the packet is captured it will be hard to crack it. If your platform supports SHA algorithm it is better to use than MD5 as MD5 is again vulnerable to rainbow tables.

Even though the hacker can see the payload of the packet which he can use to build the link state graph to identify the network, he will not be able to do any harm to the network.

So this is the only secure method for doing this. Anything other than this including encrypting the password to level 7 will not defend against a hacker. It can easily be cracked too.