Redundant Internet Connection

It has been a long time since I’ve added a post to the ADM Technology blog. It’s not that there has been nothing for me to write about, but rather that it has been busy behind the scenes and there has not been time to write about what has been going on. Much of what goes on in the background in a technology department tends to be low visibility but high impact. This post is about something which should (hopefully!) be almost completely invisible but also provide significant benefit to the staff and students of the ADM district.

With the increasing use of classroom technology the ability to connect to the internet has become an important part of the school experience. ADM has a reasonably fast connection, at 160Mbps. This is more than five times faster than the reported average for Iowa of ~30Mbps. Bigger is better here, and an increase in that number always correlates to a faster connection.

What isn’t always considered when discussing an internet connection is that it must be available when we need it. The fastest internet connection in the world is of little benefit if something goes wrong with it. With many classroom devices and learning software applications offering extensive online functionality the loss of the ability to go online can adversely affect the classroom experience, even if it’s just for a short while. For this reason ADM has implemented a secondary internet connection. Most of the time this does next to nothing, but it provides an important safety net should something happen to our main internet link. It’s like a spare set of house keys. The best scenario is that you don’t need to use them, but should you ever find them necessary you’re very glad you’ve got access to them.

The good news is that our primary internet connection is very reliable. That doesn’t mean it’s perfect though. Late last year we suffered a loss of internet access as a consequence of an attempt to knock out the internet connection of another school district in Central Iowa. In circumstances such as that, or others (eg, if physical cables between our building and the rest of the world got damaged) where we have no direct ability to resolve the problem, then this secondary link comes to the rescue. I touched on the possibility of damaged cables being a potential risk to our internet connection. Because of this possibility our secondary connection comes into the district at our admin building on N. 11th Street (our primary link goes to the middle school/high school campus on Nile Kinnick Drive) so that a hypothetical careless backhoe operator could only break one of the two links.

Our secondary link is very slow in comparison to our primary connection. It is 20Mbps, which is only 1/8 of the speed that our main link offers. It’s not intended to be a full replacement, just a means of ensuring a good level of continuity while the main connection isn’t working properly.

On a technical level the work is handled by our internet facing firewall. This device protects our devices from the nasty things that lurk on the internet. It is set up so that it monitors our primary internet link and if it can’t communicate with the outside world for more than a couple of seconds then it moves to the secondary link. The primary link is monitored and once communication is restored then it is used again. The secondary link is also monitored to make sure we don’t just try to use another broken connection. On our firewall this is known as Policy Based Forwarding, but it goes by other names.

pan-os-683

(Source: https://www.paloaltonetworks.com/documentation/61/pan-os/pan-os/policy/use-case-pbf-for-outbound-access-with-dual-isps)

The changeover process is almost entirely transparent to the end user. What they might notice is a short period of time where it seems like a website isn’t working. This is for approximately two seconds. It seems like nothing, until you actually experience it. On a fixed link data connection, websites will usually respond within 1/500 of a second and the delay beyond that time is surprisingly noticeable. Anyone who has used a satellite internet connection will understand this, but for people who’ve only experienced fast internet you won’t know until it happens. Most people will notice nothing at all. The switch back once things are working again is entirely invisible to the end user.

One of our design considerations when implementing this is that sometimes an internet connection can be working just well enough for you to tell that it’s not working. A situation like this presents a problem as automatic monitoring tends to rely on tests which might not be able to detect that sort of scenario, and thus we could still have no usable internet. For this reason we implemented the ability to manually break our primary internet connection. This can be accomplished in one of two ways.

  1. Technology department staff can remotely do this by making a change to one of network switches. For the geek minded, we use HP switches and the command to do this is: int x disable (where x is the port number our connection runs through.) The corresponding command to fix it is: int x enable
  2. A responsible staff member can disconnect a distinctively coloured, and clearly labelled, cable.

Both of these require manual interaction. The firewall cannot determine whether the primary connection is working again with the connection being physically disabled. It’s a nuisance but these options offer us an extra level of protection which is the goal of this functionality.

While there are absolutely no downsides to having this enabled (when things are fine, they’re as fine as they ever were) there are trade offs when this functionality is active. First, and most obviously, is that things will be slower. During the periods of time that the primary connection is out of service people from the outside world cannot access our public facing websites. Lastly, our public wifi network loses its internet access. All these were design considerations, but the cost to benefit ratio for changing these wasn’t sufficient to justify the expense of making it work. The expected need for the secondary internet link is very low, so for the short durations of it being in use those trade offs were determined to be appropriate. All those down sides vanish the moment the main internet link is working again.

At the end of the day this is something that the vast majority of users won’t even notice. But that is entirely the point. One of the goals of the ADM Technology Department is to continually improve on what we do, and this is a very good example of that methodology at work.

 

Advertisements
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s