Getting the following error on your Juniper EX/MX/QFX virtual chassis?
fpc1 CMLC: Going disconnected; Routing engine chassis socket closed abruptly
This message is informational and does not necessary indicate any serious issue, for example in a graceful Routing Engine switchover (GRES) context / if you flipped the routing-engine (RE) on purpose. However if this is message is printed repeateadly, without any virtual chassis (VC) topology change or manual intervention, this may indicate a more serious issue that worth investigating.
The most common issue is cabling between your VC members, and most likely to happen if the members are distant from each others connected with SFP+/QSFP+ and fiber rather than the short length DAC cables (it does not mean that no issue can happen with DAC cables, it is just that the elements and distance in the chain of even is reduced).
Here are the common symptom/possibilities on the vc-ports :
- Defective SFP+/QSFP+ optic (dying optic, losing power transmission capability)
- Fiber length too tight for the optic capability (check laser transmit/receive power)
- Damaged fiber/connector, bad fusion point, dirty connector/optic (investigate with light testers, loopback, OTDR, clean the tips…)
- Port flapping (can be caused by all the above)
- If you are using DAC cables and observe CRCs, just swap it out with another one and monitor if there is any change in the situation.
OBSERVATION : When such issue occur and there is a lot of flaps/errors between VC ports members, you may observe a higher load on the CPU/RE than usual and some functionalities such as ICMP drop, SNMP polling issues. If your device is usually very busy and near capacity, more serious service impacting issues may occur on layer 3 services such as BGP and OSPF as well (especially on EX series devices).
In most cases, you will observe Cyclic Redundancy Check (CRC) errors on the vc-port(s). You should check it out using the following command :
show virtual-chassis vc-port statistics extensive
And investigate accordingly based on the tips provided above.
Here is an example output below of a VC with this behavior showing CRC errors (yeah… I knew the fiber that was delivered to me by the cabling technician was bad just by looking at it, I was told it has been tested – trust no one!) :
> show virtual-chassis vc-port statistics extensive
fpc0:
--------------------------------------------------------------------------
RX TX
Port: vcp-255/1/2
Total octets: 3380878201 414651629
Total packets: 3271509 3342492
Unicast packets: 3271509 3062662
Broadcast packets: 0 147643
Multicast packets: 0 132187
MAC control frames: 0 0
CRC alignment errors: 12744
Oversize packets: 0
Undersize packets: 67998
Jabber packets: 0
Fragments received: 367
Ifout errors: 0
64 octets frames: 60
65-127 octets frames: 2710825
128-255 octets frames: 273987
256-511 octets frames: 42794
512-1023 octets frames: 196819
1024-1518 octets frames: 12615
Rate packets per second: 295 280
Rate bytes per second: 1663994 25665
Port: vcp-255/1/3
Total octets: 32445910 124169721
Total packets: 62039 712329
Unicast packets: 62039 224385
Broadcast packets: 0 254986
Multicast packets: 0 232958
MAC control frames: 0 0
CRC alignment errors: 0
Oversize packets: 0
Undersize packets: 2
Jabber packets: 0
Fragments received: 0
Ifout errors: 0
64 octets frames: 0
65-127 octets frames: 8377
128-255 octets frames: 33238
256-511 octets frames: 569
512-1023 octets frames: 6567
1024-1518 octets frames: 12649
Rate packets per second: 27 51
Rate bytes per second: 4948 12619
fpc1:
--------------------------------------------------------------------------
RX TX
Port: vcp-255/1/2
Total octets: 416620156 768802603
Total packets: 3367017 3270937
Unicast packets: 3087273 3270937
Broadcast packets: 147597 0
Multicast packets: 132147 0
MAC control frames: 0 0
CRC alignment errors: 0
Oversize packets: 0
Undersize packets: 68060
Jabber packets: 0
Fragments received: 0
Ifout errors: 0
64 octets frames: 35298
65-127 octets frames: 3030401
128-255 octets frames: 189368
256-511 octets frames: 32934
512-1023 octets frames: 29095
1024-1518 octets frames: 16791
Rate packets per second: 273 284
Message from syslogd@SW-HOSTNAME at Aug 31 12:01:57 ...
SW-HOSTNAME fpc1 CMLC: Going disconnected; Routing engine chassis socket closed abruptly
Rate bytes per second: 26796 40621
Port: vcp-255/1/3
Total octets: 124189980 32448435
Total packets: 712472 62042
Unicast packets: 224444 62042
Broadcast packets: 255026 0
Multicast packets: 233002 0
MAC control frames: 0 0
CRC alignment errors: 0
Oversize packets: 0
Undersize packets: 3
Jabber packets: 0
Fragments received: 0
Ifout errors: 0
64 octets frames: 123265
65-127 octets frames: 360255
128-255 octets frames: 180304
256-511 octets frames: 3119
512-1023 octets frames: 26097
1024-1518 octets frames: 17052
Rate packets per second: 23 1
Rate bytes per second: 4284 660
{master:0}