Like many organizations, Northwest Multiple Listing Service (NWMLS), a Seattle-based real-estate listing service, relies heavily on the Internet as a critical part of its business success. In turn, more than 14,000 real estate brokers and agents throughout 14 counties in Washington depend on NWMLS for property information to buy, list, sell and analyze their real-estate transactions.
NWMLS also leverages the public network in many ways, including hosting an Internet site with company information and providing Internet access to its workforce for business communication. The company's private extranet application, called Locator, provides a window into the centralized listing database that connects the real-estate industry, providing shared access to all area listings, market analysis, tax information and zoning on a broad range of properties.
The company's reliance on a single Internet service provider (ISP), however, proved a serious flaw in its network operations. When an ISP failure brought down access to its entire site, NWMLS realized that establishing multiple links and ISP providers was critical to ensuring the continued success of its business.
"We had taken care to build resiliency in our network, systems and applications, but that was entirely superseded by the fact that we only had one ISP, one lifeline to our customers," says Brendan Johnson, IT manager at NWMLS.
Losing this access not only negated the firm's critical services, it also stopped other types of important communications, such as FTP traffic, e-mail and employee access to other business partners and data on the Web. While many ISPs offer service-level agreements (SLA), they often fall short of protecting businesses from the financial and reputation damages caused by outages. The fundamental issue for NWMLS, however, became one of control.
"From that point on, we resolved not to rely on a single ISP," Johnson states. "Fundamentally, we wanted to diversify our risk and control our own destiny. When you have a single line or provider, all of their problems essentially become your problems."
THE MULTIHOMING OPTION
To address the problem, NWMLS decided to "multihome" its network. Multihoming means that a site has more than one WAN link or access gateway to the Internet. This establishes another path for traffic to flow in and out of the data center should one link fail. At the time, NWMLS had a single line with one ISP; the company added two additional T-1 lines from two additional ISPs.
According to Johnson, "The real trick to multihoming is defining how your traffic will flow now that you have more than one path. The question becomes not only `How do I steer users down the right link?' but also `How can I determine what the best path is?'"
At the time, the only solution was to use BGP, or border gateway protocol. BGP was created within the Internet Engineering Task Force (IETF) and first implemented by service providers in the early 1990s as a scalable, standardized scheme for moving packets between routers. While BGP serves as a core technology for routing IP packets through the Internet, it was not well suited for this particular task.
"BGP was costly, complex to troubleshoot and created significant traffic distribution problems for us," Johnson says. "It's definitely not turnkey. Sometimes, you have to buy new routers to handle the load and then there are the hidden costs of actually getting it set up."
For many organizations, this often means contracting outside experts or bringing on new staff with extensive skill sets in BGP. Northwest Multiple Listing Service was no exception.
"The skill set for being able to set up and maintain a BGP configuration is high," Johnson explains. "We had to bring additional books, training and resources in-house." He estimates his company spent nearly $3,000 on outside resources to get the BGP configuration up and running, not to mention the time needed for NWMLS employees to get up to speed once it was in place.
PERFORMANCE, COMPLEXITY ISSUES
Then there were the changes that needed to be done to the network. "While we set this (BGP) up ourselves, we had to make significant changes to our network topology and then coordinate with our various ISPs," Johnson says. "In the end, we invested a lot of time and resources to set up the solution, and it introduced latency and performance issues for NWMLS routers because of the increased routing complexity.
Troubleshooting problems associated with BGP also became a difficult chore. "BGP was especially difficult when a provider would have problems," adds Johnson. "We'd be forced to wait while the ISP's tried to diagnose a routing problem and determine whose fault it was.
"There were also instances where some routes were not working well and the only resolution was to shut down our routers and reboot. This would take a long time and force us to bring lines down, which limited our bandwidth and congested other links. Let's just say I had the phone number of my ISP contacts programmed on speed dial for these and other such occasions."
In addition, NWMLS experienced broader fundamental problems trying to control traffic via BGP routing. "We have a specific brokerage house that represents 40% of our traffic for our Locator real-estate application," Johnson says. "Because BGP uses broad groupings to make decisions and then stores the IP routing info in the router caches, all the users from that office ended up getting stuck down one of our pipes. Certain links would be overburdened, while others remained underutilized. Bottom line: the biggest problem was that with BGP we couldn't control our traffic the way we wanted."
In order to build a long-term multihoming solution and to also address its traffic-management issues, NWMLS chose the BIG-IP Link Controller from F5 Networks. The controller is an integrated load-balancing product that replaces legacy routing protocols with faster, more flexible intelligent switching technology.
EASY CONFIGURATION CHANGE
Raymond Williams, network engineer for NWMLS, oversaw the deployment of the controller, which was designed to manage both incoming and outgoing site traffic, and to direct users over the best link.
"We deployed the link load balancer between our firewalls and routers, where it measures the health and capacity of each connection of the real-estate clients accessing our site," Williams says. "Bandwidth, performance and health of each ISP are automatically measured, and then each user is directed down the best pathway.
"We had to make a few network changes," says Williams of the initial setup. "We have a firewall configured on the front end as our defense mechanism. The controller actually sits in front of the firewall, and so, essentially, it took the public domain space out one more level. We had to configure a new network and the external part of the firewall. It was a very small network configuration change and it was easy to do."
For NWLMS, the link load balancer provided several immediate benefits. Its bandwidth control solved NWMLS' BGP and bandwidth scalability problems. With high availability, it ensures that end-users always stay connected, with immediate ISP failover. It improved performance by directing users across the fastest link to avoid congested bandwidth, and it reduced management and support by providing a simplified multihoming architecture.
Since the product helped reduce router load while extending existing network capacity, cost savings were another noticeable benefit. That is, it not only ensured that customers were connected, but it allowed NWMLS to increase user performance and ensure that its corporate bandwidth investments were maximized to their fullest extent.
"When we used BGP, we received quite a few calls a day from realtors complaining about slowness or trouble connecting," according to Johnson. "Since we deployed the Link Controller, call volume about performance of our Internet bandwidth dropped significantly. The product solved our bandwidth-management problems."
To date, NWMLS has passed nearly three terabytes of traffic through the controller, while consistently routing customers and clients around failed or congested ISP links.
"Our customers no longer have to pay the price if one of our providers has problems or is getting flooded," explains Johnson. "There's no more down time or waiting while third-party ISPs try to find the issue. There's no more latency in failing over or waiting for our routers' tables to update through the Internet. It's a whole new level of high availability for Internet connectivity--failover is instantaneous."
NETWORK CAPACITY INCREASED
