Some years ago our law firm migrated away from every old-fashioned landline telephone provider that we used to use, and we moved everything about our telephone service to a Canadian VOIP telephone company called VOIP.MS. I have blogged frequently about my satisfaction generally with the use of VOIP rather than older ways of getting telephone service, and I have blogged frequently about my satisfaction in particular with this company as a provider of such services.
Which then leads to a sense of wonder and frustration to see that somebody has chosen to bring a denial-of-service attack against the VOIP.MS company, and has asked that a bitcoin ransom be paid for the DDOS attack to cease. (See Ars Technica article.) This has led to various disruptions in service for many of the 80,000 or so customers of VOIP.MS, some of which (as for my firm) have been intermittent and some of which (for some customers) have been pretty much continuous.
What can a customer do about this, if anything?
A first thing I suppose is to review briefly the mix of services that one might use to construct a law firm telephone system. This includes:
- DID service
- PBX service
- physical telephones
- outbound calling service
The world of VOIP is sophisticated enough that the customer who wants to pick and choose can use a first company for one of these things, a second company for another of these things, a third company, and a fourth company, and so on.
DID service means “what happens when somebody dials a phone number”. When somebody dials, say, the main telephone number of our law firm, that is DID service. The call goes to whatever company we pay to handle that phone number. The call then passes through a VOIP connect to a PBX and from there to an automated attendant or to voice mail or maybe to a physical phone, depending on how we have configured things.
In our case, we own our own physical telephones, and we program them to connect (via what are called “SIP trunks”) to servers provided by the VOIP.MS company. Some firms choose to bundle a rental of physical phones with other things like the DID service and the outbound calling service, or choose to pay a local service provider for a turnkey solution that includes “all of the above”.
Outbound calling service is what it sounds like. This covers local calls, long-distance calls within one’s own country, and calls to other countries. We choose to get this service from VOIP.MS.
VOIP.MS provides a cloud-based PBX free of charge to its customers, which I think is pretty neat. I have blogged about this.
What exactly is the DDOS attacker doing to try to bring VOIP.MS to its knees? As best I can tell, the attacker is doing two things.
A first thing is that the attacker is trying to make it nearly impossible for anybody to do DNS lookups for anything in the “voip.ms” domain name space. So for example suppose one of our office phones has been configured for many years now that it should “register” its SIP trunk to “denver2.voip.ms”. This means that the phone carries out a registration process once every ten minutes or so. It starts by doing a DNS lookup to find out the IP address for “denver2.voip.ms”. The answer is (usually) 64.27.52.226. The phone then initiates a SIP connection to that IP address. The SIP connection is then used for receiving incoming phone calls and for placing outgoing phone calls.
The VOIP.MS people made a decision, years ago, in their system design to make the TTL (time to live) for DNS lookups quite short — just one hour. The idea was that if one of their servers were to fail, they could change the A record to point to some other server and within an hour, everybody’s phones would have eventually re-registered to the alternative server. They could then do what was needed to repair the broken server and the disruption for customers would not be too terribly lengthy. I think that was quite smart.
Except of course what if a bad person decides to attack the DNS? Then all of a sudden, customer telephones are not able to get the answer to “what is the IP address for denver2.voip.ms”. And customer telephones are unable to place or receive calls.
A similar problem arises if a customer needs to make some change or reconfigure something. The usual way to do this is to log in at the web site www.voip.ms where there is a customer portal. The customer portal is the place for configuring the cloud-based PBX, purchasing new DIDs, and topping up one’s payment account. But if the DNS is disabled, then the customer’s web browser is unable to learn “what is the IP address for www.voip.ms?” and the customer is not able to get anything done.
I have to imagine the attacker is also trying to flood the www.voip.ms web site with TCP SYN packets or some other well-known line of attack that makes it impossible for legitimate users to log in at the web site.
If you click around in Google News you can learn that this is not the first time that some attacker has attacked some VOIP service provider and asked for a ransom. Other VOIP providers in other countries have been attacked in recent months.
So what can or should a customer do if the customer is affected by an attack like this?
One possibility is to reconfigure your phones (or ATAs, or whatever SIP clients you are using) so that they register their SIP trunks using IP addresses instead of domain names. So for example if you usually use “denver2.voip.ms” to identify your SIP server, you would use 64.27.52.226. The advantage to this approach is that you are no longer harmed by the DNS attack. Your phone knows the IP address without having to pose the question to a DNS system that might not be able to answer the question. Of course this approach has the terrible drawback that you will miss out on the automatic redirect to an alternative server that would normally save your service if the usual “denver2” machine were to crash and if the smart people in the VOIP.MS network operations center were to make a plan of redirecting your traffic to a different physical machine.
One colleague of mine wonders if what he should be doing is methodically switching his telephone activity to some other VOIP service provider. He wonders if some other VOIP service provider somehow has a network that is more resistant to DDOS attacks? He wonders if some other VOIP service provider is so small that the attackers will not feel it is worth attacking, and so he should pick that company. He wonders if some other VOIP service provider is so big (he wonders if there are providers who are much bigger than VOIP.MS) that the attackers will not succeed in attacking the much bigger company because they are so much bigger, and so he should pick that company. He wonders if some other VOIP service provider is somehow run by people who are much smarter and somehow could react faster or better to such DDOS attacks?
I do not, of course, know the answers to such questions. But I have lots of guesses. I will share the guesses.
A first guess is that probably every VOIP provider who provides unbundled DID service and unbundled outbound calling service and unbundled PBX service is at risk to pretty much the same kinds of attacks. If you pick a provider who controls everything about your service (they own the phone on your desk, you don’t get to configure anything yourself) then probably they make it so that everything uses hard-coded IP addresses and there is much less vulnerability to DDOS attacks on DNS services.
From this, my guess continues with a guess that if you were to try to migrate your services to any other company that provides more or less the same kinds of services that VOIP.MS provides, you will be migrating to a company that will be largely at risk to the same general kinds of attacks. The new company that you are considering switching to may have had just such a DDOS attack two months ago and you did not know it. Or they may be the company that is next in line for a DDOS attack two months from now and you just don’t happen to know it.
A further guess is that “security through obscurity” is probably not the way to go. By this I mean that picking a new VOIP service provider specifically because they are small and don’t have many customers, in the hopes that the ransom attackers will choose not to attack the company, might not be a good strategy. For one thing, if the attacker does finally get around to attacking the small fry, they probably would have even fewer resources to fight back or react. For another, a small-fry company might have other drawbacks just because they are small. Oh and there are quite a few companies that are small fry companies that are actually basically just resellers from bigger companies but they conceal it really well, and you only figure it out much later if ever. So they would get brought to their knees as soon as some attacker decides to attack the bigger upstream provider that the little company is reselling from.
I am pretty sure there are a few North American VOIP providers that offer the same types of unbundled services that VOIP.MS offers that are bigger than VOIP.MS, but I think there are not many of them. I know of one or two that I suspect are bigger. I have tried to use a couple of them and I have found them much harder to use from a technical point of view. Meanwhile, VOIP.MS is quite big and, I think, is run by very smart people. So I have no particular reason to think that migrating one’s business to a different service provider is going to somehow lead to a situation where smarter people are providing your service.
Our phones are running on the denver2.voip.ms server, and it seems to be running okay just now. But for example the chicago7.voip.ms and chicago3.voip.ms servers seem to be unresponsive. I am not sure why.
For now I am going to just sort of sit tight and not try changing VOIP telephone companies. I am hoping and trusting that the folks at VOIP.MS will figure out ways to deal with the attacks. Maybe moving their DNS hosting to different servers that somehow are more robust, maybe making the TTLs longer for a while. The have already migrated their www.voip.ms portal over to Cloudflare and that seems to be working better.
I welcome thoughts from people. Please post comments below.
Instead of hard-coding the IP address of the VOIP vendor, why not set that address in a local resolver that is consulted as the last in the chain of resolvers only if prior attempts fail. Then, when there is no DDOS attack going on, you get the benefit of any reassignment of host IPs the vendor may use to fix other network problems. I guess that’s only practical on your firms LAN or any VLAN that might be established for home workers accessing your network via VPN.
One of the challenges with the approach that you describe is how to pick a TTL for the A record in your proposed “last resort” resolver. The normal rule is that any local resolver under one’s own control is supposed to merely replicate the DNS record from the authoritative source in all particulars (including replicating the TTL from the authoritative source). But if you do that, you have not made the DDOS problem go away because the expiration of the TTL still means the phone or ATA is stymied and cannot figure out the IP address to reach the SIP server.
So if on the other hand you decide you are going to take matters into your own hands and hard-code the TTL in your “last resort” resolver to be a long-lasting TTL, then I submit that this is not so very different from simply hard-coding the IP address itself into the phone or the ATA.
I guess a chief reason to consider using such a “last resort” resolver with a hard-coded long TTL is that it might simultaneously restore multiple client devices to service. Then when the DDOS attack has eventually ceased or been somehow successfully overcome, you would not need to go back and restore the FQDN into multiple phones or ATAs but would instead only need to restore one resolver to a normal TTL setting.
1. Yes, the reason I propose this is that it eliminates the need to reconfigure devices with an IP address (either with a physical visit or by updating a configuration) and then to reconfigure the devices when the problem has been resolved. If you only have one device, then reconfiguation is no big deal. If you have 20 devices including some offsite that either require a human to make a change or require remote network access, which might not be continuously available, then managing that could be a headache.
2. It makes sense to mimic the zone’s A record TTL in a FIRST-resort resolver, such as a DNS cache, because it honors the operator’s tradeoff between the Zone’s DNS server load and the freshness of the result. (Assuming such a tradeoff was consciously made.) OTOH, for a last-resort resolver in a private network, I couldn’t care less about honoring that tradeoff. I would pick a relatively short, but reasonable TTL. If the zone’s DNS server is able to furnish records when next requested, it will deliver the “real” TTL, and the requesting device on your network will honor it, as will any DNS cache on your network. If the zone’s DNS server is not able to furnish records, it’s probably because the server is down or your request never reaches it, and your device gets an address from your last-resort resolver. Either way, your short last-resort TTL will cause no more load on the zone’s own DNS server than if there were no last result server, though maybe a couple of your requests will add a miniscule additional quantum of congestion to a busy router or link.
As far as I understand the situation you have described is a DDoS on DNS services, not an attack on VoIP services. And the knowledge how to protect DNS Services from DDoS exists, e.g., https://www.a10networks.com/blog/how-to-defend-dns-services-from-all-types-of-ddos-attacks/.
The point is the VoIP providers (at all levels of this infrastructure) should be aware how DNS is important to their activity and use DNS service providers with a good line of defense established or pay some more attention to protecting their own DNS servers.
BTW: We are on VoIP right now, and we see benefits, but the migration process is painful.
Thank you very much for posting. Yes I think part of what you are getting at is that (if the accounts in the media are correct) the attacker is not literally attacking, say, the SIP protocol or the RTSP protocol, but is merely attacking the ordinary mainstream protocols like DNS and HTTP that might just as well have been used in an attack on a non-VOIP target. I think that is a very good point. It also reveals, I think, a limited level of sophistication on the part of the attacker, who has not (apparently) gone to the trouble to try to find specific ways to attack particular protocols relating to the particular line of business of the target, but is instead merely repeating use of tools that the attacker (probably) used in the past against other targets in other lines of business.
An alternative explanation (speculation): In a war of attrition, it benefits the attacker to spread the attack over time. Maybe the attackers have several tools in their toolbox, including attacks specific to VOIP services, and plan to deploy each one serially, using it until defenses to that tool become effective, and then move on to the next. Commenters on an Ars Technica article who claimed to be customers said they intend to stick with the vendor provided the vendor does not pay the ransom, but there’s probably a limit to customer tolerance.
Mitigation is the word I guess (not being a lawyer).
I had my neighbors switch from a local Telco to Voip.Ms. We did not transfer their inbound number away, instead we purchased another number from Voip.Ms and used that number to handle the calls. Thus, when the DDOS attack started we fist just switched servers that our phones went to. Then we switched cities.
When we saw the totality of the issue, we un-forewarded our main number and had calls come in as they did before Voip.Ms
Since the issue continued a bit longer than hoped we then purchased a telephone number from a different provider and now are forwarding our main number to that new number and carrier. Since we use ObiHai/Polycom phones, we have that alternate number appear on a different button on the phones.
What this now gives us is a separate route for our calls from our Telco to the phones. To switch providers to the backup all it takes is one call transfer, and we can change that transfer when Voip.Ms comes back up.
So we now have 3 options, Telco calls, provider 1 or provider 2. If our Telco allowed some type of ring group, we could forward to BOTH Voip providers at the same time.