Hi All,

Update from one of our providers:

Zayo's DWDM system between Olympia WA and Tacoma WA is experiencing an outage. We are observing two DWDM systems down. We suspect a fiber cut at this time. Zayo's Field Techs are onsite from the first test site. OSP has proactively dispatched crews. We have also dispatched another field tech to a second site to test from, that tech should be on-site momentarily. We should have a concrete diagnosis of the issue in the next 45 minutes.

Ten minutes later:

Good Afternoon OTDR traces from our Olympia site show an issue 17.3KM outside of the POP. We have techs 10 minutes from the far end of the fiber span who will shoot to help splice crews pinpoint damage. We hope to have an update for you soon. Thanks

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith Keith.Artz@SouthSound911.org
Sent: Thursday, March 11, 2021 10:12 AM
To: Adam Wilhoit adam.wilhoit@piercecountywa.gov
Subject: RE: Scheduled Outage

Thanks Adam.

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D71666.78EE2FF0]

From: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Sent: Thursday, March 11, 2021 9:40 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: RE: Scheduled Outage

FYI, looks like a big Zayo outage affecting IGN service

[cid:image002.png@01D71666.78EE2FF0]

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 9:21 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>
Subject: RE: Scheduled Outage

I've also seen our Zayo service that's unrelated go down, think its an issue with them as the IGN connection also uses them.

Reaching out on our end.

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>
Sent: Thursday, March 11, 2021 9:19 AM
To: Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Subject: Scheduled Outage

We are not receiving routes from the state on the INCTPC ring.  Was there a scheduled outage we are unaware of?

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D71666.78EE2FF0]

Hi All,

Received the following update:

  Zayo OSP crew is continuing to troubleshoot to narrow down the location of damage, no ETR at this time.

Please let me know if you have any questions or concerns.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 11:15 AM
To: 'Artz, Keith' Keith.Artz@SouthSound911.org; 'inctpc-list@list.cityoftacoma.org' inctpc-list@list.cityoftacoma.org
Subject: RE: Scheduled Outage

Hi All,

Update from one of our providers:

Zayo's DWDM system between Olympia WA and Tacoma WA is experiencing an outage. We are observing two DWDM systems down. We suspect a fiber cut at this time. Zayo's Field Techs are onsite from the first test site. OSP has proactively dispatched crews. We have also dispatched another field tech to a second site to test from, that tech should be on-site momentarily. We should have a concrete diagnosis of the issue in the next 45 minutes.

Ten minutes later:

Good Afternoon OTDR traces from our Olympia site show an issue 17.3KM outside of the POP. We have techs 10 minutes from the far end of the fiber span who will shoot to help splice crews pinpoint damage. We hope to have an update for you soon. Thanks

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>
Sent: Thursday, March 11, 2021 10:12 AM
To: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Subject: RE: Scheduled Outage

Thanks Adam.

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D71681.9E420CC0]

From: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Sent: Thursday, March 11, 2021 9:40 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: RE: Scheduled Outage

FYI, looks like a big Zayo outage affecting IGN service

[cid:image002.png@01D71681.9E420CC0]

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 9:21 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>
Subject: RE: Scheduled Outage

I've also seen our Zayo service that's unrelated go down, think its an issue with them as the IGN connection also uses them.

Reaching out on our end.

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>
Sent: Thursday, March 11, 2021 9:19 AM
To: Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Subject: Scheduled Outage

We are not receiving routes from the state on the INCTPC ring.  Was there a scheduled outage we are unaware of?

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D71681.9E420CC0]

Hi All,

Latest update:

  OSP Crews have located the damage caused by tree trimming in the area. We are working on a plan of action for repairs at this time. We will provide details as they become available.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit adam.wilhoit@piercecountywa.gov
Sent: Thursday, March 11, 2021 2:20 PM
To: Keith Artz Keith.Artz@SouthSound911.org; 'inctpc-list@list.cityoftacoma.org' inctpc-list@list.cityoftacoma.org
Subject: [inctpc-list] Re: Scheduled Outage

Hi All,

Received the following update:

  Zayo OSP crew is continuing to troubleshoot to narrow down the location of damage, no ETR at this time.

Please let me know if you have any questions or concerns.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 11:15 AM
To: 'Artz, Keith' <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: RE: Scheduled Outage

Hi All,

Update from one of our providers:

Zayo's DWDM system between Olympia WA and Tacoma WA is experiencing an outage. We are observing two DWDM systems down. We suspect a fiber cut at this time. Zayo's Field Techs are onsite from the first test site. OSP has proactively dispatched crews. We have also dispatched another field tech to a second site to test from, that tech should be on-site momentarily. We should have a concrete diagnosis of the issue in the next 45 minutes.

Ten minutes later:

Good Afternoon OTDR traces from our Olympia site show an issue 17.3KM outside of the POP. We have techs 10 minutes from the far end of the fiber span who will shoot to help splice crews pinpoint damage. We hope to have an update for you soon. Thanks

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>
Sent: Thursday, March 11, 2021 10:12 AM
To: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Subject: RE: Scheduled Outage

Thanks Adam.

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D7168F.78C0E1C0]

From: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Sent: Thursday, March 11, 2021 9:40 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: RE: Scheduled Outage

FYI, looks like a big Zayo outage affecting IGN service

[cid:image002.png@01D7168F.78C0E1C0]

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 9:21 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>
Subject: RE: Scheduled Outage

I've also seen our Zayo service that's unrelated go down, think its an issue with them as the IGN connection also uses them.

Reaching out on our end.

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>
Sent: Thursday, March 11, 2021 9:19 AM
To: Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Subject: Scheduled Outage

We are not receiving routes from the state on the INCTPC ring.  Was there a scheduled outage we are unaware of?

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D7168F.78C0E1C0]

Hi All,

Latest update has a 2AM ETR:

We have located the damage on a 96ct fiber on Pacific Ave SE near Seaton CT. The fibers appear to be burned. We are dispatching aerial crews at this time to start prepping for repairs and construction crews should be onsite within 3 hours. The current ETR is 2am. We will continue to update you as the restoration progresses.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 3:59 PM
To: Keith Artz Keith.Artz@SouthSound911.org; 'inctpc-list@list.cityoftacoma.org' inctpc-list@list.cityoftacoma.org
Subject: RE: Scheduled Outage

Hi All,

Latest update:

  OSP Crews have located the damage caused by tree trimming in the area. We are working on a plan of action for repairs at this time. We will provide details as they become available.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Sent: Thursday, March 11, 2021 2:20 PM
To: Keith Artz <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: [inctpc-list] Re: Scheduled Outage

Hi All,

Received the following update:

  Zayo OSP crew is continuing to troubleshoot to narrow down the location of damage, no ETR at this time.

Please let me know if you have any questions or concerns.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 11:15 AM
To: 'Artz, Keith' <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: RE: Scheduled Outage

Hi All,

Update from one of our providers:

Zayo's DWDM system between Olympia WA and Tacoma WA is experiencing an outage. We are observing two DWDM systems down. We suspect a fiber cut at this time. Zayo's Field Techs are onsite from the first test site. OSP has proactively dispatched crews. We have also dispatched another field tech to a second site to test from, that tech should be on-site momentarily. We should have a concrete diagnosis of the issue in the next 45 minutes.

Ten minutes later:

Good Afternoon OTDR traces from our Olympia site show an issue 17.3KM outside of the POP. We have techs 10 minutes from the far end of the fiber span who will shoot to help splice crews pinpoint damage. We hope to have an update for you soon. Thanks

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>
Sent: Thursday, March 11, 2021 10:12 AM
To: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Subject: RE: Scheduled Outage

Thanks Adam.

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D71695.FBA742E0]

From: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Sent: Thursday, March 11, 2021 9:40 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: RE: Scheduled Outage

FYI, looks like a big Zayo outage affecting IGN service

[cid:image002.png@01D71695.FBA742E0]

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 9:21 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>
Subject: RE: Scheduled Outage

I've also seen our Zayo service that's unrelated go down, think its an issue with them as the IGN connection also uses them.

Reaching out on our end.

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>
Sent: Thursday, March 11, 2021 9:19 AM
To: Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Subject: Scheduled Outage

We are not receiving routes from the state on the INCTPC ring.  Was there a scheduled outage we are unaware of?

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D71695.FBA742E0]

Hi All,

Final update:

Final Update: At this time crew is hands off after repairing the 96ct cable that was burned when a tree limb fell across electrical lines and our fiber."

I confirmed we are receiving routes on our INCTPC router so service should be restored for customers.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 4:46 PM
To: Keith Artz Keith.Artz@SouthSound911.org; 'inctpc-list@list.cityoftacoma.org' inctpc-list@list.cityoftacoma.org
Subject: RE: Scheduled Outage

Hi All,

Latest update has a 2AM ETR:

We have located the damage on a 96ct fiber on Pacific Ave SE near Seaton CT. The fibers appear to be burned. We are dispatching aerial crews at this time to start prepping for repairs and construction crews should be onsite within 3 hours. The current ETR is 2am. We will continue to update you as the restoration progresses.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 3:59 PM
To: Keith Artz <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: RE: Scheduled Outage

Hi All,

Latest update:

  OSP Crews have located the damage caused by tree trimming in the area. We are working on a plan of action for repairs at this time. We will provide details as they become available.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Sent: Thursday, March 11, 2021 2:20 PM
To: Keith Artz <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: [inctpc-list] Re: Scheduled Outage

Hi All,

Received the following update:

  Zayo OSP crew is continuing to troubleshoot to narrow down the location of damage, no ETR at this time.

Please let me know if you have any questions or concerns.

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 11:15 AM
To: 'Artz, Keith' <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: RE: Scheduled Outage

Hi All,

Update from one of our providers:

Zayo's DWDM system between Olympia WA and Tacoma WA is experiencing an outage. We are observing two DWDM systems down. We suspect a fiber cut at this time. Zayo's Field Techs are onsite from the first test site. OSP has proactively dispatched crews. We have also dispatched another field tech to a second site to test from, that tech should be on-site momentarily. We should have a concrete diagnosis of the issue in the next 45 minutes.

Ten minutes later:

Good Afternoon OTDR traces from our Olympia site show an issue 17.3KM outside of the POP. We have techs 10 minutes from the far end of the fiber span who will shoot to help splice crews pinpoint damage. We hope to have an update for you soon. Thanks

Kind regards,

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>
Sent: Thursday, March 11, 2021 10:12 AM
To: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Subject: RE: Scheduled Outage

Thanks Adam.

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D716D5.EC6228A0]

From: Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Sent: Thursday, March 11, 2021 9:40 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; 'inctpc-list@list.cityoftacoma.org' <inctpc-list@list.cityoftacoma.orgmailto:inctpc-list@list.cityoftacoma.org>
Subject: RE: Scheduled Outage

FYI, looks like a big Zayo outage affecting IGN service

[cid:image002.png@01D716D5.EC6228A0]

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Adam Wilhoit
Sent: Thursday, March 11, 2021 9:21 AM
To: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>; Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>
Subject: RE: Scheduled Outage

I've also seen our Zayo service that's unrelated go down, think its an issue with them as the IGN connection also uses them.

Reaching out on our end.

Adam Wilhoit
Pierce County IT - Network Team
direct: (253)798-8522

From: Artz, Keith <Keith.Artz@SouthSound911.orgmailto:Keith.Artz@SouthSound911.org>
Sent: Thursday, March 11, 2021 9:19 AM
To: Reggie Osborne <reggie.osborne@piercecountywa.govmailto:reggie.osborne@piercecountywa.gov>; Adam Wilhoit <adam.wilhoit@piercecountywa.govmailto:adam.wilhoit@piercecountywa.gov>
Subject: Scheduled Outage

We are not receiving routes from the state on the INCTPC ring.  Was there a scheduled outage we are unaware of?

Regards,
Keith Artz
Network Engineer
253.287.4862
Keith.Artz@SouthSound911.orgmailto:artzk@lesa.net
[cid:image001.png@01D716D5.EC6228A0]