NETWORK SECURITY TOPICS

Click on each topic to open the associated page. 

General Network Security topics 

Paloalto Firewalls 

Read More

TROUBLESHOOTING PROCESS OF ROUTING SWITCHING

OSPF issues

A) When examining a router's configuration, check the following:

1) All interfaces have the correct addresses and masks?

 2) The configure network area statements have the correct inverse masks to match the correct interfaces?

3) The configure network area statements put all interfaces into the correct areas?

B) When examining neighborship, consider the following questions:

1) Are Hellos being sent from both neighbors?

2) Are the timers (Dead and Hello) set the same between neighbors?

3) Are the interfaces configured on the same subnet?

4) Are the neighboring interfaces of the same network type (Broadcast, Point to point,Point to multi-point)

 5) Is a router attempting to form an adjacency with a neighbor's secondary address?

 6) If authentication is being used, is the authentication type the same between neighbors? Are the passwords same?

7) Are any access lists blocking OSPF?

8) If the adjacency is across a virtual link, is the link configured within a stub area?

9) Is unique Router Id configured in the your As?

10) Is MTU size same on the interface?

C) If a neighbor or adjacency is seeing unstable, you can monitor adjacencies using the command "debug ip ospf adj"

D) The state changes of a neighbor can be monitored by adding the command "log-adjacency-changes [detail]" under a router's OSPF configuration.

E) OSPF Errors, Warnings, and Log Messages

Receiving "ospf unknown protocol" error message

Receiving "Mismatch Authentication type"

Receiving "%ospf-4-NONEIGHBOR" error message

Receiving "ospf-5-ADJCHG” error message

Receiving "OSPF: Hello from x.x.x.x with mismatched NSSA option bit"

F) If you suspect that a link-state database is corrupted or that two databases are not synchronized,

you can use the "show ip ospf database database-summary" command to observe the number of LSAs in each router's database.

Tasks for Troubleshooting General OSPF Issues –

1. Verify Port Status – Verify admin state & line state. Line state down can be due to – remote end shut, card error, duplex, speed etc. To test –

2. Verify OSPF Interfaces -

Run the show ospf interface command to verify that the OSPF interfaces are up.  Use the detail keyword to display detailed information about the interface.

3. Check OSPF Adjacency –

◾Down—No information has been received from the neighbor.

◾Attempt—Attempt is valid for neighbors on NBMA network; the neighbor is being contacted, but no information has been received.

◾Init—A Hello packet has been received from the neighbor, but the router is not listed in that Hello packet.

◾2-Way—The 2-way state indicates that the router has identified its own router ID in the Neighbor field of the neighbor’s Hello packet. Receiving a database descriptor packet from a neighbor in the init state also causes a transition to the 2-way state. Receiving a hello with an OSPF neighbor in 2-way state is not a concern in broadcast multiaccess networks at the DR.. Other routers in these networks cannot progress beyond the 2-way state; in these networks, they achieve Full state only with Designated Router (DR) and Backup Designated Router (BDR). The adjacency must pass through 2-way state before proceeding to a Full state. Failure to pass through the 2-way state indicates there might be an error in the database exchange.

◾ExStart—The routers and their Designated Router (DR) and Backup Designated Router (BDR) establish a master-slave relationship and choose the initial sequence number for adjacency formation.

◾Exchange—OSPF routers exchange database descriptor (DD) packets. After the packet header, DD packets contain Link-state advertisement (LSA) headers only. They provide a summary of the sender's database contents to the receiving router.

◾Loading—Routers send link-state request packets. During the database exchange, if a router receives a more recent or missing link-state advertisement (LSA), it requests that LSA by sending a link-state request packet.

◾Full—indicates that routers are fully adjacent with each other. All the router and network LSAs are exchanged, and the router databases are fully synchronized. Full is the normal state for an OSPF router. A router is stuck in another state indicates problems in forming adjacencies but not necessarily for broadcast and NBMA networks.

4. Verify OSPF Neigbhorship –

Run the show ospf neighbor command to verify that you have established adjacency with your neighbors. The configuration on both endpoints (the interface type; for example, point-to-multipoint or P2P) must match; otherwise, you cannot form an adjacency. This is a common issue.

Values other than two-way and full can indicate the following problems:

. Physical issues—If the MTU is too large, the routers attached to the link cannot negotiate the same MTU on the link, and an adjacency cannot form. For example, when the OSPF neighbor state is stuck in an init state, the router can receive packets from the neighbor but not transmit packets to the neighbor, resulting in a one-way link.

. Configuration Issues—If routers attached to the link cannot negotiate the link Area ID, the link Hello protocol timers, the link authentication type, or keys, adjacencies cannot form.

When examining adjacencies, use the following OSPF adjacency checklist to isolate the fault. For more information about how to troubleshoot OSPF adjacencies.

1	Are Hello packets being sent from both neighbors?
2	Are the dead timers set the same between neighbors?
3	Are the interfaces configured on the same subnet (that is, do the address or mask pairs belong to the same subnet)?
4	If authentication is being used, is the authentication type the same between neighbors? Is authentication enabled on all routers within the area? Are the passwords and the keys (in the case of MD5) the same?
5	If the adjacency is across a virtual link, is the link configured within a stub area?
6	Do neighbor MTUs match?
7	Are router IDs unique within the entire internetwork? If not, no neighborship will form.
8	Did you verify that the interface is not defined as passive in OSPF? If it is, no adjacency is formed.
9	Is the network type the same for neighboring interfaces?

5. Display Detailed OSPF Neighbor Information –

Run the show ospf neighbor detail command to display detailed OSPF neighbor information

6. Verify LSAs in the OSPF Database –

Because OSPF is a link-state protocol, the link-state database should be the same for any router in the same area. Each router has a link-state database with information about each router in the network and uses this information to build a network topology and calculate the best routes. All routers within the same area must have the same database content to help them identify the best routes. The LSA advertisement changes frequently, indicated by the fact that the sequence number is significantly higher than that of other LSAs.

You can determine which parts of the network are changing the most by checking the LSA sequence numbers and the age of the LSA in the LS Age field. If the same LSA remains in the database, the LSA age increases. If LSAs are updated frequenly, the age remains low. The LSAge does not need to be the same in both router databases, but the other identifying elements of the LSA header should be the same.

The most common reasons for OSPF to not share the database information about a specific link include:

◾The OSPF neighbor is not advertising routes.

◾The OSPF neighbor (ABR) is not advertising the summary route.

◾The OSPF neighbor is not advertising external routes.

◾The OSPF neighbor is not advertising the default route.

To determine whether routers have a synchronized OSPF database, run the show ospf database command and compare the results of the shared areas:

1.Verify that the summaries of their LSA checksum fields are equal.

2.Determine that the two routers have the same number of LSAs (in the same area) in their link state databases.

If you have routing problems, make sure your interfaces are correctly configured.

When checking an area-wide issue, consider the following issues:

◾Is the ABR correctly configured?

◾Are all the routers configured for the same area type?

◾When summarization is enabled, is it correctly configured?

Two common problems are related to summarization in OSPF:

◦A router is not summarizing interarea routes.

◦A router is not summarizing external routes.

If you suspect that the link-state databases are corrupt or that the databases are not synchronized:

1.Run the show ospf database database-summary command to verify the number of LSAs in each router database. For a given area, the number of each LSA type should be the same in all routers.

2.Run the show ospf database command and verify that each LSA checksum is the same in a given area in every router database.

OSPF sends LSAs to all routers within the area. The LSA contains information about attached interfaces, link metrics and other variables.

In the following example, router rock1200 and two are synchronized (all routers contain the same database) in area 2.0.0.0. On context rock1200, the LSAs highlighted correctly match the LSAs in context two (router two).

To determine if you have the latest version of the LSAs, check the sequence number associated with the LSA in the show ospf database output. On each node, verify that each node OSPF database has synchronized this information to ensure that each router has the same view of your network.

7. Check OSPF Routes –

8. View a Specific OSPF Route Entry with show ip ospf route 11.11.11.0

9. show ip route all

10. Check OSPF Statistics –

Use the show ospf statistics & show ospf statistics interface

11. Examine the OSPF SPF Log –

Run the show ospf spf log command to display the SPF calculation timing calculation log. Verify that SPF ran and investigate network instabilities. The SPF log includes a description of what triggered SPF recalculations—for example, when a new LSA arrives. This command shows which LSAs change most frequently and what triggers the SPF calculations.

Note:  If the SPF log is not running as expected, check the following SPF scheduling configuration parameters: ◾fast-convergence & ◾spf-timers

12. Check OSPF Logs –

Use the show log | grep ospf command to filter the log for entries related to OSPF.

Note:  You must configure the log-neighbor-up-down command to log an informational message when a neighbor transitions to or from full adjacency state.

13. Debug ip ospf –

#	Error Message
1	"OSPF-1 NBR 1.2.3.4: MTU 567 less than our mtu (1500)" Recommended Action: 1. Check MTU configuration by running the show ip interface command on the interfaces that have the mismatched MTUs. 2. Change the router MTU to match the neighbor MTU.
2	"OSPF-1: Duplicate DD packet from slave NBR 1.2.3.4 flags: M,M,I options: 2 seq: 3450293" Recommended Action: The slave in a DD exchange has received a packet with a sequence earlier than the one it already has, which means there is a duplicate. Verify that the values are what you expect for the following options. The values need to match the interfaces between two communicating routers. The initialize(I), more (M) and master(MS) bits—Options field, and DD sequence number contained in the last Database. Description packet received from the neighbor. Used to determine whether the next Database Description packet received from the neighbor is a duplicate. ExternalRoutingCapability—Determines if this is a stub area. E-bit —Describes the way AS-external-LSAs are flooded. MC-bit —Describes whether IP multicast datagrams are forwarded. N/P-bit —This bit describes the handling of Type-7 LSAs. EA-bit —Describes the router's willingness to receive and forward. External-Attributes-LSAs —Describes the router's handling of demand circuits.
3	"OSPF-1: Duplicate DD packet from master NBR 1.2.3.4 flags: M,M,I options: 2 seq: 345234098" Recommended Action: The master in a DD exchange has received a packet with a sequence earlier than the one it already has, which means there is a duplicate. Verify that the values are what you expect for the following options. The values need to match the interfaces between two communicating routers. The initialize(I), more (M) and master(MS) bits—Options field, and DD sequence number contained in the last Database. Description packet received from the neighbor. Used to determine whether the next Database Description packet received from the neighbor is a duplicate. ExternalRoutingCapability—Determines if this is a stub area. E-bit —Describes the way AS-external-LSAs are flooded. MC-bit —Describes whether IP multicast datagrams are forwarded. N/P-bit —This bit describes the handling of Type-7 LSAs. EA-bit —Describes the router's willingness to receive and forward. External-Attributes-LSAs —Describes the router's handling of demand circuits.
4	"OSPF-1 NBR 1.2.3.4: invalid LS recv Type 5 (AS-Ext) for this area/interface" Recommended Action: Make sure the area is not a stub or an NSSA area. Type 5 LSAs are not allowed in these areas.
5	"OSPF-1 LSU AS-Ext 9.10.11.12 [5.6.7.8] from 1.2.3.4: invalid type" Recommended Action: Make sure the area is not a stub or an NSSA area. Type 5 LSAs are not allowed in these areas.
6	"OSPF-1: No current keys in key-chain somePolicyName interface 11.0.0.1 (someIntfName)" Recommended Action: Check the authentication configuration.
7	Dec 13 09:37:11: %OSPF-7-PCK_ERRORS: Recv invalid Hello packet 13.1.1.1->224.0.0.5 area 0.0.0.0: No interface Recommended Action: Verify that the interface for which you are receiving the Hello packet has OSPF enabled.
8	Nov 22 23:14:55.632: [0001]: %OSPF-7-PCK_ERRORS: OSPF-1: Recv int-to-UELKSRPP01 invalid Database Description packet 10.0.9.14->224.0.0.5 area 0.0.0.0: BFD DOWN Nov 22 23:14:11.331: [0001]: %OSPF-7-PCK_ERRORS: OSPF-1: Recv int-to-UELKSRPP01 invalid Hello packet 10.0.9.14->224.0.0.5 area 0.0.0.0: BFD DOWN Recommended Action: Check if BFD keepalives are being exchanged on OSPF neighbor interfaces. If not flap the ports or check if the BFD configuration is complete.

Stuck in a state -

INIT state –

The Init state specifies that the router has received a Hello packet from its neighbor, but the receiving router ID is not included in the Hello packet. When a router receives a Hello packet from a neighbor, it must list the sender's router ID in its Hello packet as an acknowledgment that it received a valid Hello packet.

When the neighbor is stuck in the Init state, a local router likely is not listed in a neighbor's Hello packets because the neighbor has not received Hello packets from the local router. Run the ping and traceroute commands to verify that the links between routers are operational. If a ping between routers is not successful, the link is not functioning properly and you need to troubleshoot it. Consider the following when troubleshooting the Init state:

• If any access lists are defined on the neighbor interface, the destination IP of 224.0.0.5 must be permitted in the access list. OSPF Hello packets have a destination address of 224.0.0.5 (the all ospf routers multicast address).
  
  Recommended Action: To verify the access list, run the show access-group command.
• There may be a Layer 2 or configuration problem preventing multicast packets from reaching the neighboring router.
  
  Recommended Action: Type the ping command to the multicast address to see if responses are received from the neighboring routers if a ping to 224.0.0.5 from the rock1200 context on the rock1200 router does not return a response.
• Authentication is not enabled on both sides. The router on which authentication is not enabled still processes Hello packets from the neighbor and sees the neighbor in the init state.
  
  Recommended Action: To correct this problem, enable authentication on both sides. Make sure they are using the same authentication type.
• Hellos are getting lost on one side at Layer 2.
• For virtual links, authentication is enabled on only one side.
  Recommended Action: Enable authentication both sides.

2-Way State -

• The 2-way state indicates that the router has seen its own router ID in the Neighbor field of the neighbor's Hello packet. Receiving a Database Description (DD) packet from a neighbor in the init state will also a cause a transition to 2-way state. The OSPF neighbor 2-way state is not a cause for concern.
• This state indicates that bidirectional communication has been established between two routers—each router has seen the other's Hello packet, and the router receiving the Hello packet sees its own router ID in the received Hello packet's neighbor field. At this stage, a router can establish adjacency with this neighbor. On broadcast media and nonbroadcast multiaccess networks (NBMA), a router reaches full state only with the designated router (DR) and the backup designated router (BDR); it stays in the 2-way state with all other neighbors. On point-to-point and point-to-multipoint networks, a router reaches full state with all connected routers.
• At the end of this stage, the DR and BDR for broadcast and nonbroadcast multiaccess networks are elected.
• If the router interfaces are not in a waiting state, the router performs DR and BDR election. Once DR and BDR are elected, a router attempts to form a full adjacency with a neighbor if one of the two routers is the DR or BDR. OSPF routers become fully adjacent to routers with which they have successfully completed the database synchronization process. This is how OSPF routers within an area exchange link-state information to populate their databases with the same information. This database synchronization process occurs only if one of the two routers is a DR or BDR in the case of broadcast multiaccess networks. This database synchronization process is only executed between two routers if one of the two routers is the DR or BDR.
• Note:  
• If the OSPF neighbor is stuck in 2-way state, router-priority 0 might be configured on all routers.

Exstart or Exchange State –

A router gets stuck in ExStart or Exchange state more frequently when interoperating with another vendor's router. Once the DR and BDR are elected, link-state information exchange can start between the routers and their DR and BDR. In the ExStart state, the routers and their DR and BDR establish a master-slave relationship and choose the initial sequence number to establish an adjacency. The router with the higher router ID becomes the master and starts the exchange, and is the only router that can increment the sequence number.

Note:  Make sure that the router ID is unique within the entire internetwork; otherwise, OSPF neigborship will "not" form.

In the Exchange state, OSPF routers exchange DD packets. DD contain link-state advertisement (LSA) headers only and describe the contents of the entire link-state database. Each DD packet has a sequence number that can be incremented only by a master that is explicitly acknowledged by the slave. Routers also send link-state request packets and link-state update packets (which contain the entire LSA) in this state. The contents of the DD packets received are compared to the information contained in the routers link-state database to check if new or more current link-state information is available with the neighbor.

OSPF neighbors in Exstart or exchange state are trying to exchange DD packets. The adjacency should continue past this state. If it does not, there is a problem with the DD exchange, such as a maximum transmission unit (MTU) mismatch or receipt of an unexpected DD sequence number.

Note:  Make sure that the router ID is unique within the entire internetwork; otherwise, no neighborship will form.

An OSPF neighbor can be stuck in ExStart or Exchange state for the following reasons:

§  Duplicate router IDs on neighbors. This problem usually manifests before database exchange.

§  Unable to ping without using specific maximum transmission unit (MTU) size.

§  Broken unicast connectivity because of the following:

o    Wrong VC or DLCI mapping in the Frame Relay or ATM switch

o    An access list blocking the unicast

o    Incorrect NAT translation of the unicast packet

§  Mismatched interface MTU

The problem occurs when the MTU settings for neighboring router interfaces do not match. If the router with the higher MTU sends a packet larger than the MTU set on the neighboring router, the neighboring router ignores the packet. When this occurs, the output of the show ospf neighbor detail command displays output similar to what is shown below.

Loading State –

In the loading state, routers send link-state request packets. During the database exchange, if a router receives an outdated or missing LSA, it requests that LSA by sending a link-state request packet. Neighbors that do not transition beyond this state are most likely exchanging corrupted LSAs. The most common causes of this problem include:  

Mismatched MTU

Recommended Action:
1. Run the show ip interface command on the interfaces that have the mismatched MTUs.
2. Change either router MTU to match the neighbor MTU.

Corrupted link-state request packet

Recommended Action: Run the clear ospf neighbor all command to force adjacency reestablishment

show ospf neighbor Output Is Empty –

When the show ospf neighbor detail command displays no output, run the show ip interface and show ip interface detail commands to verify the state of the interfaces.

#	Possible Reason
1	OSPF is not enabled on the interface. Layer 1 or 2 is down.
2	The interface is defined as passive under OSPF.
3	A subnet number or mask has been mismatched over a broadcast links.
4	The Hello or dead interval has been mismatched.
5	The authentication type (plain text versus MD5) has been mismatched.
6	An area ID has been mismatched.
7	Stub, transit, or NSSA area options have been mismatched.
8	An OSPF adjacency exists over an asynchronous interface.
9	No network type or neighbor is defined over NBMA (Frame Relay).

Table 9 lists the common causes for this problem:

#	Possible Reason
1	OSPF is not enabled on the interface. Layer 1 or 2 is down.
2	The interface is defined as passive under OSPF.
3	A subnet number or mask has been mismatched over a broadcast links.
4	The Hello or dead interval has been mismatched.
5	The authentication type (plain text versus MD5) has been mismatched.
6	An area ID has been mismatched.
7	Stub, transit, or NSSA area options have been mismatched.
8	An OSPF adjacency exists over an asynchronous interface.
9	No network type or neighbor is defined over NBMA (Frame Relay).

 ref: -- http://rbdoc.ufanet.ru/en_lzn7830011_1_r4a/13_15451CRA1191170_1Uen.A.html#CHAPTER2.5

Read More