Internet connection

Phone service connection

SIP connection settings

Internet account settings for AVM FRITZ!Box

Telephone line settings for AVM FRITZ!Box

Telephone number settings for AVM FRITZ!Box

• Error-prone signup process
• Supplied modem/router is heavily compromised
• Phone service is tied to compromised modem/router
• No IPv6 support
• Support staff very inconsistent
• Good support staff hobbled by policy

My Internode connection had become very slow and unstable in hot, dry weather. Strangely it was fine in the rain, and even during flooding. It almost seemed like something needed to be damp to maintain an electrical connection. There’s no way to actually get these kinds of issues resolved, as the ISP and last mile provider will blame each other and the in-building wiring, and charge extortionate rates for technicians to be called out without actually solving the issues. The only other option I have for last mile is TPG. I’d been switching to Telstra LTE on bad days, and to be fair it’s actually not too bad at the moment. It seems to be pretty fast and stable, but I imagine that will get worse as more people start to use the network. But using LTE comes with a number of imitations, and it’s supposed to be my backup, not my day-to-day Internet connection.

Sadly, it seems that Internode may be going downhill since being acquired by TPG. After iiNet acquired Internode, they were allowed to operate independently for the most part. The call centre in Adelaide remained open, Internode continued to offer the same kinds of perks as before, including Usenet servers, Steam content mirrors, native IPv6 connectivity, and more. However, TPG has consolidated iiNet and Internode support and seems to be phasing out Internode perks. They’ve even started selling TPG nbn HFC (DOCSIS cable) under the Internode brand name, providing the same IPv4-only connection and obfuscated SIP phone service.

With the consolidation in the Australian ISP sector, there’s a big reduction in competition. TPG has merged with or acquired Soul, AAPT, Chariot, iiNet, Internode, TransACT, WestNet, PIPE, Westnet, and more. There doesn’t seem to be a good alternative at the moment. There may be an opportunity for an upstart ISP that understands what made “premium” ISPs like Internode successful in the first place.

Sign-up process

I initially tried signing up for the service through the web site, converting an existing dial-up account I’ve had for over a decade. At the end of the process, it gave me a red error message telling me there was a problem and to call customer service. Despite this, it still charged me the setup fee, and not the correct setup fee for the options I’d chosen. Also, there’s no option to choose the delivery address for the supplied modem/router through the web interface: it will always be sent to the billing address, not the service address. This means you need to get it from the billing address to the service address if they aren’t the same.

It took multiple calls to customer service over several weeks to get the incorrect setup fee refunded and get back to where I started again. The telephone support staff seem to vary substantially. Many of them don’t seem to be interested in actually getting issues resolved, and just want to read from a script. I also had support staff promise to call back, and then never do so.

After this, I tried my luck signing up over the phone. The saleswoman insisted that I needed to create a new account, and couldn’t convert my existing dial-up account over. She assured me that my existing TPG e-mail address could be transferred to the new account without any period where mail would be lost. It’s possible to specify a delivery address for the modem/router when signing up over the phone. However, after completing the sign-up process, I was transferred to support who informed me that there was no need to sign up for a new account at all, and it seems to be impossible to transfer the existing e-mail account to the new account without a multi-day period where e-mails will be lost. The call was recorded, so it’s on record that the saleswoman promised me something that they can’t deliver. This issue still hasn’t been resolved.

Supplied modem/router

TPG supplied a Huawei HG659 modem/router. This device is rather lacking in functionality. It lacks DECT base station functionality, it can’t function as a SIP gateway for multiple IP phones, it doesn’t support incoming VPN connections, and numerous other useful features are absent. On top of this, TPG supplies the device with crippled firmware. The predefined “admin” user account is limited to changing basic settings, and it’s not possible to create an account with full access. It’s possible to access some hidden settings (including authentication, encapsulation and VLAN settings) with a JavaScript debugger attack, but trying to access other settings this way drops you back to the login page. It’s completely impossible to access bandwidth settings and telephony settings, or to back up/restore settings.

The modem/router is pre-configured and has TR-069 permanently enabled on VLAN 6. This allows TPG to push configuration or firmware updates to the device at any time. This is a huge problem for stability and security. There’s no way to control if/when updates may be pushed, allowing your connections to be interrupted at any time. A poorly considered or malicious update could cause denial of service, DNS hijacking, communication interception, or a host of other issues. Flaws in TR-069 are actively exploited by the Mirai botnet as well as other malware.

TPG’s justification for this is that it makes it easy to TPG to fix configuration problems, and they make vague claims about doing it for “security” reasons. It’s true that it makes support simpler if the ISP can push out default configuration. However it comes with a massive security risk. They should acknowledge the security risks involved, and give the customer the ability to choose between convenience and security. The real motivation seems to be an effort to hide the SIP settings to prevent customers from using other SIP clients or IP phones. I really don’t understand TPG’s obsession with preventing the customer from using a SIP client of their choice.

It’s possible to put the modem/router into firmware recovery mode by holding the reset button (with a straightened paperclip) for twenty seconds, and then to load a different firmware image. However, Huawei doesn’t seem to distribute a standard firmware image, so you’d need to use a firmware image from another ISP, with its own customisations and potential security issues. If you don’t enable TR-069 after loading a different firmware image, you won’t be able to obtain the SIP settings, so the phone service still won’t be usable. However, if you do enable TR-069, TPG will just push out their firmware image along with the configuration, and you’ll be back where you started.

In summary, it’s impossible to get the modem/router into a clean state where you can fully control it and still use TPG’s phone service. The modem/router supplied by TPG must be treated as a hostile device on your network. As the customer, you can’t prevent malicious configuration or firmware updates being applied, and you can’t verify or change the device’s configuration.

Phone service inflexibility

TPG’s SIP phone service for FTTB customers is limited and inflexible. Unlike other SIP phone services, it’s only accessible from TPG’s network. The server uses the DNS name uni-v1.tpg.com.au which resolves to three private IPv4 addresses – 172.26.0.17, 172.26.0.1, and 172.26.0.65 – accessible via VLAN 6. TPG requires use of the narrowband 8 kbps G.729 voice codec, which provides poor call quality. TPG also actively works to prevent customers from using their own IP phones.

TPG refuses to supply customers with SIP connection details, only pushing them out via TR069. The SIP username and password are different from the username and password used to access e-mail and other TPG services. It seems somewhat strange and pointless to require authentication at all, since the SIP server is only accessible on a TPG connection via a specific VLAN. It would be trivial to identify the customer by the origin of the connection. It seems to be nothing more than a way to force the customer to use the compromised modem/router supplied by TPG. (TPG actually does provide SIP settings for some services on this page. The aphone1 to aphone6 servers resolve to public IP addresses, but they are only accessible from TPG connections. However, there’s nothing to indicate which customers can use these settings – it’s definitely not applicable to FTTB services.)

It was previously possible to use a JavaScript debugger attack on the supplied Huawei modem/router to back up settings, and extract the SIP settings, including the password, from the resulting file. However, new firmware made that impossible. It would be possible to buy a VDSL DSLAM, emulate the SIP server, and steal the credentials that way, but this is prohibitively expensive. It may be possible to connect to VLAN 6 with a different modem/router, use software to emulate the TR-069 client, and obtain the VoIP settings that way. It may also be possible to open the supplied modem router, solder in a serial or JTAG header, and dump the Flash filesystem. Desoldering the Flash chips and dumping the data directly is another option. All of these options are a lot of work just to be able to use a service that you pay for, without having to allow a compromised device on your network.

There’s no way to unbundle the phone service from the Internet service. So if you decide that the risk of using a compromised modem/router is too high and the workarounds are too impractical, you’re still forced to pay for a phone service you can’t use.

All this effort to prevent customers from using SIP clients other than the supplied modem/router seems rather strange. There doesn’t seem to be a technical reason for it, as the service seems to use standard protocols, and customers who’ve managed to extract the details from their modem/router haven’t had issues using other SIP clients. The lack of any plausible explanation almost seems like TPG wants to have devices they control on customers’ networks for some malicious purpose.

The decision to require G.729 seems odd as well. With ever-increasing line speeds, a 32 kbps codec like G.726 shouldn’t be a problem. In particular, G.726 would allow lossless forwarding to cordless DECT handsets. Only allowing access from TPG’s network is also artificially limiting. Packets are cheap to forward – there’s no real reason not to allow access from other networks. It can still be limited to one or two concurrent calls and/or concurrent registrations. Call quality will suffer if there’s unpredictable latency or packet loss in the path, but the customer can deal with that.

NodePhone SIP service, ironically owned by TPG, can be used from anywhere on the Internet. I’ve successfully used it from as far away as Hong Kong and Shanghai with good results. Right now I’m using a NodePhone service over my TPG FTTB connection as it’s a better option than using a compromised modem/router.

Lack of password verification

TPG requires your VDSL modem to be configured to use PAP authentication. However, the password is not verified. They assume that by being physically patched to the DSLAM port, you are authorised to use the service. This isn’t a safe assumption. In most apartment buildings, tradesmen and/or residents can easily access the main distribution frame (MDF) and change the patches. For services with the DSLAM located in a roadside cabinet or telephone exchange, there are further points along the path where a technician could unintentionally or maliciously patch the DSLAM port assigned to you to another line.

This appears to be to make support simpler. If the password is not verified, a dummy password can be used in settings pre-configured or pushed out to the customer’s modem/router via TR-069, and support staff can walk a customer through the process of setting up a modem/router without either of them having to know the password. However, it’s another security hole, and given the metadata retention laws and the aggressive behaviour of copyright holders, it’s unwise to make it in any way simpler for someone to impersonate the customer.

Lax e-mail security

TPG’s mail servers support explicit and opportunistic SSL/TLS encryption. However, as of the time of writing, TPG’s relevant support pages don’t make any mention of enabling encryption, and the step-by-step guides for Apple Mail and Android phones show settings that will result in usernames, passwords, and mail contents being transmitted in plain text.

This shows blatant disregard for customers’ security. A customer following TPG’s instructions for configuring Apple Mail or an Android phone will expose their account name and password to anyone with the ability to sniff packets between them and TPG’s mail servers. On a public WiFi network, this includes anyone in the vicinity who can use packet capture software.

No IPv6 support

TPG does not officially support IPv6 and has no timeline for IPv6 rollout. There are rumours that they’re testing IPv6 with selected customers, but there’s no way to voluntarily join the test group. IPv6 is not a new technology. RFC 2460 was published in late 1998, almost twenty years ago. Microsoft began requiring applications to work in a pure IPv6 environment (no IPv4) for logo certification beginning with Windows Vista in 2006, over ten years ago. All major operating systems and most network equipment provides IPv6 support.

TPG is really behind here. Internode (now owned by TPG) has provided dual stack IPv4/IPv6 since 2008 (ten years ago), assigning a static /56 subnet and a dynamic /64 subnet to each connection. Even Telstra, not known for being on the cutting edge, has rolled out IPv6 for NBN and ADSL customers. With iiNet, you at least have the option of using a 6rd service to provide IPv6 connectivity, although it suffers from some limitations compared to a true dual stack deployment.

Phone support

The quality of service provided by the phone support staff varies enormously. You often need to work your way through multiple people before you reach someone who seems to actually care or be interested in helping. Even then, the staff are hobbled by processes and policies, and may not be able to really do much. I’ve experienced this multiple times with the support and engineering teams. One time, the guy said something to the effect of, “Well, I understand what you’re saying, but I don’t set the policy. The call’s recorded, I’ll mark this as a complaint, hopefully someone in Sydney will actually hear it.”

There are definitely some people at TPG who seem to want to do the right thing by the customers. Ace and Joy from support, in the unlikely event that you’re reading this, I’d like you to know I think you’re great. You’ve both got back to me when you said you would, tried to understand the issues I raised, and tried to get things resolved as well as you can. It’s not your fault TPG’s policies are hostile to the customer, or some of the other people on the support team don’t seem to care.

Closing thoughts

I’ve had TPG Internet accounts for over twenty years now. Back in the dial-up days, TPG was the ISP to beat. They provided national service at competitive rates, and it just worked with no fuss. Now everything’s a nightmare. It seems TPG wants to sell to people who just use their Internet connection for Facebook and YouTube. There’s definitely a market for that, but the trouble is they’ve absorbed the ISPs who catered for people who wanted a little more, and soon there may not be any other options left. It’s sad to see the Australian ISP landscape go this way.

I needed a brand-new service, as there was no existing POTS or DSL line – only Telstra cable and CATV. There was a lengthy waiting period, and after the installation date, I called an electrician to wire up a socket. It turned out the MDF hadn’t been tagged. After much arguing, Internode sent someone out to tag it properly. However, I had to call out (and pay) an electrician to jumper it. So if your ISP tells you your MDF or boundary point is tagged, don’t believe them – check for yourself before you call out an electrician.

At this point, I had a socket connected to the correct cable and pair, but still no DSL. Internode insisted that I find an analog telephone to listen to the line. I want naked DSL – why should I need an analog telephone? Anyway, I discovered that I had a POTS service of some kind, and even found out what its number was, and told Internode. They informed me that they needed a technician to come and “perform tests”. It took another week for the guy to come out, and he didn’t arrive on time. He just confirmed what I’d told them: my socket was connected to the correct cable and pair, but had POTS service. Apparently they don’t believe their customers.

After this, it took another day for the exchange to be patched correctly. I now have a working Internet connection, but my high-speed ADSL2+ here is barely faster than my plain ADSL1 in Melbourne, and I now have to fight for a refund for the period when I was being billed for a service that didn’t work. If you’re thinking of getting naked ADSL, save yourself the trouble and get something where a single vendor is responsible for the whole solution. Cable Internet or ADSL with a Telstra DSLAM would be a whole lot less trouble.

Now with WordPress 2.3, all that’s in the past. WordPress now seems to be able to do the right thing with text encodings. It’s just too bad the upgrade script can’t clean up the rot left from previous versions. After running the upgrade script, I found that every piece of Japanese text, every typographical quote, every accented character, in fact everything outside 7-bit ASCII, was horribly mangled. Now I had a number of options for going forward:

• Go back to a previous version of WordPress
• Leave it and hope no-one minds.
• Delete every post that got mangled.
• Manually fix every affected post.
• Come up with a l33t way to solve it without manual effort.

Now the first option would have been easy. I had a complete backup (like you should, too), and I could have rolled it back in a matter of minutes. But I like to be on the curve, and I like to have all the newest features, even if I never use them. Also, having the latest security updates is nice.

The second option wouldn’t fly, because even if the readers wouldn’t mind, I’d mind. The third option would probably mean deleting every post, since I’m in the habit of using typographical quotes, non-breaking spaces and dashes (as opposed to hyphens). Deleting all my posts after an upgrade would defeat the purpose of keeping a blog. The fourth option would be excessively time-consuming, and I’d have to play fill the blanks, which may not even be possible if important things were mangled.

So the only way to fix it would be to call on my inner geek. I had a quick look at the database contents in phpMyAdmin, and had a look at the database code in WordPress. I noted that the new tables created by WordPress 2.3 had the collation utf8_general_ci while the upgraded tables had the collation latin1_swedish_ci, and WordPress was asking MySQL to communicate in UTF-8. Armed with this, I downloaded a UTF-8 SQL dump of the database.

The rest of it was actually reasonably simple: I opened the SQL dump in TextWrangler, which correctly identified it as being UTF-8 with no “byte order mark”. I then found all references to the latin1 character set and replaced them with utf8. That would fix the issue with the upgraded tables having the wrong collation, but not the corrupted data.

So here’s the trick: you need to convert the UTF-8 representation of what was in in the database back to its old binary representation, and then interpret that as UTF-8. I tried to save the file as ISO Latin 1 (ISO 8859-1), but TextWrangler complained about unmappable characters. It turns out that MySQL’s latin1 is actually Windows Latin 1 (code page 1252). So I saved the file in this encoding, and then told TextWrangler to reinterpret it as UTF-8. It all went smoothly, and I had my data back! I could play the SQL dump back on the server, and everything is as it should be.

So what are the morals of the story?

• Always keep backups – particularly when you plan to do something drastic like an upgrade. Even though I didn’t actually need the backup this time, it was comforting to know it was there.
• Don’t trust upgrade/migration scripts – always check the result to ensure it’s actually what you want.
• Store data in appropriate formats – hacks will always come back and bite you. I shouldn’t have been using WordPress when I knew it was doing the wrong thing with my data.
• When you’re writing a migration script, try to ensure that it actually works! Then you don’t risk infuriating and/or losing your users.

Take its text rendering, for example. Since the primary purpose of a web browser is to get text on the screen, you’d think they’d have that right. But no, apparently version 2.0 is still too early to expect decent text rendering. Compare these two snaps:

Firefox	Safari

How has Firefox managed to screw up the fixed-pitch text so badly? It’s just plain illegible! I have absolutely know idea, but however they really should have fixed this kind of thing before version 1.0 – not left it in at 2.0. How about italic text. Maybe they could get that right:

Firefox	Safari

Once again, Safari has rendered it beautifully, but Firefox looks like it’s using a synthetic oblique style, the way System 6 used to when you didn’t have an italic version of the font available – it’s most noticeable in the capital S. And then when you select the text, some of the last italic letter gets cut off. Come on, this is pretty basic stuff, guys!

On the topic of selection, Firefox won’t use Mac conventions in handling double-click and drag. It’s supposed to select whole words, but Firefox selects just one whole word. The same goes for triple-click and drag for lines. (Yes, I know Safari’s selection behaviour isn’t quite standard, and there’s no excuse for that, either.) And speaking of things being non-standard, why can’t Firefox use standard OS widgets? For example, the drop-down menus from items on the bookmark bar don’t respond to clicks in the same way as regular menus (items with submenus, in particular). The controls on forms just look like horrible Windows wannabes:

Firefox	Safari

Safari is more usable, too. In Firefox’s bookmarks window, why can’t I drag object from the tree view in the left-hand pane? Why can’t I rename or edit a link in-place, rather than clicking the Properties or Rename button? Speaking of which, why are there two buttons, when both of them bring up exactly the same sheet, with exactly the same text field selected? And on the topic of bookmarks, when I try to drag the URL to the bookmarks bar, Firefox puts a tooltip in the way to thwart my efforts:

When you have lots of tabs on the same site, Safari cuts off any common prefix it can find in the titles, so you have more chance of knowing which is which:

Firefox
Safari

And on top of this, there are the small things. Like ripping a YouTube video, for example – in Safari, you can easily find it in the activity window, and hold option and double-click. Firefox won’t use the system-wide spelling dictionaries. Firefox takes longer to launch.

So if I hate it, why do I use it? First of all, on Linux and Windows there’s no meaningful competition. On the Mac, Safari has three flaws that are too bad to live with: it can’t deal with table cells spanning multiple rows in XHTML (although it can in regular HTML – this mystifies me), it doesn’t deal with character entities correctly in XHTML and it gives you the spinning pinwheel of death far too often.

While I’m on the topic, I’ll put in a plug for my hosting provider Selpaw Services in Perth. Their support is always speedy and first-rate. Thanks for everything, Luke.

Suppose I’m a bot herder (I hope it’s obvious that this is purely hypothetical). I unleash malware that takes over computers and “calls home,” allowing me to send spam for my paying clients. Since the number of people who read spam is very low, and the number of people who buy products advertised in spam is even lower, I need to send huge volumes of spam to make my services worthwhile for my clients. And sending lots of spam requires lots of compromised computers.

As software vendors patch vulnerabilities in their software, I have to find new vulnerabilities and write new malware to exploit them. This requires considerable effort on my part, and takes away from time I could spend doing things I enjoy. Also, as more users become more security-conscious, there are less machines left open to attack.

Suppose for a moment Windows, Linux and Mac OS X are all equally exploitable, and writing a piece of malware for each takes the same amount of time. What am I going to do? Am I going to write three sets of malware to attack the three platforms, or will I pick one to concentrate on?

Of course, the answer depends on market share. The more even the market share, the more likely I would be to write malware for multiple platforms. Also, it’s worth thinking about where the machines are primarily used.

Linux is used primarily in server and professional environments. Machines that are critical for business operations run by tech-savy operators means the machines are more likely to be secured properly and suspicious software will be removed promptly. So scratch Linux. I want to target home users with DSL or cable internet.

So I’m left with a choice of targeting Windows or OS X. What do I do? I look at market share. I know these figures are probably wrong, but suppose OS X runs on 5% of my target machines and Windows runs on 90%. What am I going to target?

The answer should be obvious. I’ll target Windows. I could target OS X as well, but then I’d be spending twice as much time writing malware for less than 6% more compromised machines to send spam from. It just doesn’t make business sense.