OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#14762 — vac1
Incident Report for Network & Infrastructure
Resolved
The VAC1 not working properly. Arbor VAC1
seems to have a problem. We just cut it.
The anti-DDoS is made on VAC2 and VAC3.

Update(s):

Date: 2015-09-21 14:46:53 UTC
Everything is in order. We are getting VAC1 on.

Date: 2015-09-21 14:45:38 UTC
This does not look good at all. we started to manage
a hardware failure with arbor.

We have performed different manipulations, we removed cards one by one and it's still not working.
Every time we made the reload of the chassis.
We noticed that during the reload 10G ports did not go DOWN, so we think the software was not properly reloading the chassis.
We torn very violament the chassis power cables (3 cables!!) and plugged them over carefully.
It's better. The chassis has restarted all the cards. We are monitoring.

Date: 2015-09-21 14:39:51 UTC
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:8' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:19' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-0:4' failed on [Errno 111] Connection refused host:apm-0-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:15' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:17' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:19' failed on [Errno 111] Connection refused host:apm-0-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:20' failed on [Errno 111] Connection refused host:apm-0-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:6' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-0:20' failed on [Errno 111] Connection refused host:apm-0-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-1:0' failed on [Errno 111] Connection refused host:apm-1-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-1:3' failed on [Errno 111] Connection refused host:apm-1-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:18' failed on [Errno 111] Connection refused host:apm-0-1

Date: 2015-09-21 14:39:38 UTC
admin@vac1-10-tms:/# services tms show
Peakflow TMS state: stopped
admin@vac1-10-tms:/# services tms start
Starting Peakflow TMS services..done.
admin@vac1-10-tms:/#

Date: 2015-09-21 14:39:26 UTC
Sep 20 18:25:55 (none) python[6468]: [S] #SUBHOSTS-REBOOT found 3 apms
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-0-ipmc
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-1-ipmc
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-2-ipmc

Date: 2015-09-21 14:39:14 UTC
We will reboot.

admin@vac1-10-tms:/# services tms stop
Stopping Peakflow TMS services....................................................done.
admin@vac1-10-tms:/# re
now Reload without confirmation
Reload with confirmation
admin@vac1-10-tms:/# re now
094: Rebooting the system..
Broadcast message from root (pts/8) (Sun Sep 20 18:17:35 2015):

The system is going down for reboot NOW!
Connection to vac1-10-tms closed by remote host.
Connection to vac1-10-tms closed.

Date: 2015-09-21 14:38:33 UTC
Sep 20 09:40:01 apm-2-1 apm-1 pktengine[23602]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-1-0 apm-0 pktengine[3009]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-2-0 apm-0 pktengine[23610]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-2-1 apm-1 pktengine[23610]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-1 apm-1 pktengine[23583]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-0 apm-0 pktengine[23608]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-1 apm-1 pktengine[23618]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable

Date: 2015-09-21 14:38:19 UTC
The Cisco router does not see these DOWN. It must
be an internal problem in Arbor.

Date: 2015-09-21 14:37:32 UTC
TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=ACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161654 Alert 2161654 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=ACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=INACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161653 Alert 2161653 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=ACTIVE tmsx1.2=INACTIVE tmsx1.4=ACTIVE tmsx1.6=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161652 Alert 2161652 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=INACTIVE tmsx1.2=INACTIVE tmsx1.4=INACTIVE tmsx1.6=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161616 Alert 2161616 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=INACTIVE tmsx1.3=ACTIVE tmsx1.5=ACTIVE tmsx1.7=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161615 Alert 2161615 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=INACTIVE tmsx1.3=ACTIVE tmsx1.5=ACTIVE tmsx1.7=ACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161614 Alert 2161614 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=INACTIVE tmsx1.2=ACTIVE tmsx1.4=INACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161613 Alert 2161613 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=ACTIVE tmsx1.2=ACTIVE tmsx1.4=ACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161612 Alert 2161612 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Down' (Logical port INACTIVE, members: tmsx1.1=INACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161611 Alert 2161611 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Down' (Logical port INACTIVE, members: tmsx1.0=INACTIVE tmsx1.2=INACTIVE tmsx1.4=INACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
Posted Sep 21, 2015 - 14:37 UTC