Posts

RESOLVED: Internet Outage at SELCO Office

UPDATE: And… we’re back up again. Sounds like someone did something far away from Rochester and brought our ISP down. Service has been restored and everything, including Horizon and our phones, should be working normally now. Apologies for the (thankfully brief) down time. 

 

We are suffering an internet outage at the SELCO office. As a result, all services hosted at SELCO are currently down including:

  • Horizon
  • Pharos
  • Our phone system
Services not hosted at SELCO should still be working. 
 
We are working to diagnose and resolve the problem. More information will be sent out as it becomes available. 
 
Thank you for your patience. 

ILS ALERT RESOLVED: Enterprise is Down

UPDATE 3:27, December 23, 2017: SirsiDynix has resolved the issue that caused Enterprise to go down. Apparently, an issue at their data center in Atlanta caused an outage for numerous customers. Enterprise is now functioning normally. Thanks to SirsiDynix for fixing the problem but, more importantly, thank YOU for your patience today. 

The Enterprise public access catalog is down. We have contacted SirsiDynix emergency support and they are investigating; it looks like a number of their customers are suffering the same issue. We will post further information as the situation develops. 

Thank you for your patience as we work through this issue. 

TECH ALERT: SELCO Office Internet Down – RESOLVED

UPDATE, 12-07-2017, 9:15 am – It appears that our internet access is back up but we have not received confirmation from our Internet Service Provider (ISP). After balancing the risk of sending you a premature “all-clear” with the impact of being down, we have decided to send this tentative message.

All services appear to be functioning normally and you may resume normal operations. Please be aware that, absent a confirmation message from our ISP, we cannot guarantee that things will stay up. But it’s looking pretty good. 
 
If you have been using PC Reliance for offline circulation, please contact the Help Desk to arrange for your files to be processed. Likewise, please contact the Help Desk with any questions or further issues. 
 
Thank you for your patience this morning. 

 

Internet service to the SELCO office is down. We have been in touch with TDS who report that they are unable to contact parts of their Wide Area Network (WAN); this would seem to indicate that the issue is on their end and is not limited to SELCO. Nevertheless, a technician from TDS to en route to our office. 

Despite the information we have from TDS, we have checked our networking equipment and are confident that nothing is wrong there. Fortunately, phone service to the building has not been affected. Since we use Google Apps for Education, our email is also unaffected. 

Horizon, being hosted at SELCO, is unavailable to Online Libraries. Enterprise is up and searchable; real-time status of items, requests, and My Account all require access to Horizon will not work. 

OverDrive and MackinVIA require access to Horizon for patron authentication and thus are not available at the moment. We are contacting both to have authentication temporarily disabled which will bring those services back as soon as possible. 

Without access to the main server at SELCO, Pharos will run in Standalone Mode IF the library has it enabled. 

Since phones and email are working, please contact us with any questions or issues stemming from this outage. If your library wishes to use PC Reliance for offline circulation, please contact the Help Desk to let us know. 

We’ll keep you all posted. Thank you for your patience as we deal with this situation. 

Extensive MediaCom Outage – UPDATE

UPDATE 07-12-2017, 2:00 PM – MediaCom has updated their information and now estimates they will be back up at approximately 6:30 PM.  Please keep in mind this is an estimate and conditions may change without SELCO’s knowledge. We will continue to monitor the situation.

MediaCom has confirmed that they are experiencing an internet outage affecting their customers in Minnesota, western Wisconsin, and northern Iowa. The outage is intermittent; sites are going up and down in an unpredictable manner. MediaCom says they working on the problem but have given no indication on when things will be back to normal. Libraries in the SELCO region using MediaCom for internet are:

  • Chatfield Public Library
  • Caledonia Public Library
  • Cannon Falls Public Library
  • Hokah Public Library
  • Houston Public Library
  • Lanesboro Public Library
  • Preston Public Library
  • Rushford Public Library
  • Spring Valley Public Library

We are monitoring the situation but some of the intervals between being up and down are too short for our monitoring systems to track. We will continue to watch the situation as closely as we can and will post an update when we learn more.

Firewall Testing Resumes on June 19

As discussed in this post on June 02, we are preparing to install a new firewall. This is a complex project that requires a great deal of testing before we launch. Said testing will resume this Monday, June 19, and continue each weekday until further notice. As before, testing will be carried out each morning from 5:00 AM to 7:00 AM. During this time, SELCO technology services may experience brief interruptions. For further information, please see the original June 02 post.

Thank you for your patience as we strive to provide the most robust and stable library technology services possible. Please contact the Help Desk with any questions on the new firewall or our testing process.

Firewall Testing Starts on June 06 With Minor Service Interruptions

Executive Summary: Between 5:00 AM and 7:00 AM from Tuesday, June 06, through Thursday, June 08, SELCO will conduct testing of our new firewall in preparation for putting it into production by July 01. During these periods, technology services may be unavailable for brief periods. We do not expect all services to be down at the same time. It is impossible to predict when or if a given service will be unavailable during the testing period.

More Details: It had a good run but the time has come to retire the firewall in the SELCO office. As you probably know already, a firewall is a device that screens communications between a local network and the internet.* The firewall that protects the SELCO office, its PCs, and servers (including Horizon) is several years old now and, rather than risk it failing unexpectedly some day, we are putting it out to pasture. Which means we need to install a new firewall.

This is not a simple task. Firewalls do their magic through a complex database of rules governing what kinds of internet traffic is allowed in/out and what gets blocked. Misconfiguring a rule can result in serious problems; traffic that you want (like Horizon) might get blocked while stuff you don’t want gets in. In theory, you can copy the rule set from your old firewall to the new one but it’s never that simple. Variations in the way different firewalls operate always require tweaking and testing.

We ordered our new firewall last month and have been configuring it with all the rules it needs to manage our internet traffic. It’s time to start testing live. From Tuesday, June 06, through Thursday, June 08, we will conduct testing between 5:00 AM and 7:00 AM. During this two-hour window, we will:

  • disconnect the old firewall
  • connect the new firewall
  • test our technology services, connections, and operations
  • resolve issues that emerge and test again

We’ll keep this up until we are confident that everything works with the new firewall just as it did with the old. The license on the old firewall expires on July 01, at which point it will stop functioning.  Our goal is to have the new firewall in place no later than June 23.

During the 5:00 AM to 7:00 AM window on Tuesday, June 06, through Thursday, June 08, there will be (hopefully) brief outages of various services. We do not expect all services to be down at the same time and it’s impossible to predict what might go down at any given moment. By conducting our testing early in the morning after schools have largely let out, we hope to avoid excessive inconvenience to libraries and library patrons. Depending on how things go, we may have to add additional testing days; in that event, we will post an update to this news item.

Thank you for your patience as we strive to provide the most robust and stable library technology services possible. Please contact the Help Desk with any questions on the new firewall or our testing process.

*The term is also used for software that protects a PC but that’s a topic for another time.

Network Maintenance on the Evening of Saturday, February 04, 2017

SELCO will be performing maintenance on our internal network on Saturday, February 04, starting at 6:00 pm.  While this work is underway, most services will be unaffected but there are a few exceptions:
  • Requests, placed in both Enterprise and the Horizon staff client, will be unavailable
  • Item statuses in Enterprise will be unavailable
  • My Account in Enterprise will be unavailable
We expect the maintenance to be completed in a few hours barring anything unforeseen.
 
Thank you for your patience as we work to provide the most robust library technology services possible.

Phone Outage – UPDATED

UPDATE, 05/09/2016, 1:03 PM

Just a quick update.  TDS, our phone and data provider (not to be confused with Traverse des Sioux, the public library system to the west of SELCO), has confirmed that a fiber cut has brought down numerous phone lines throughout the area.  There is currently no estimate on repair time.

Please continue to use email, including helpdesk@selco.info, to contact us as much as possible.  507.421.9711 is still available to reach us by phone if absolutely necessary.  We will keep you posted on the situation.
 
Thank you for your patience.

 

We have just found out that the SELCO phone system is non-operational. We are working to resolve the situation but do not have a timeline as yet. This is affecting both the main line and the Help Desk line so neither number will work.

Our email system is still working as normal, including the Help Desk email so we ask that you communicate electronically for the time being. If there is an absolute emergency which requires a phone call, use the Help Desk cell phone at 507.421.9711. Be aware that it is an old phone with no real features so it can only handle one call at a time so please use it only as a last resort.

We will keep you updated as the situation evolves.