Warning upfront – This post is a bit of a rant and ends with hope for new beginnings. But I just had to get this off of my chest.
Yesterday at approximately 4:21pm my AT&T Internet service went out. I have lived my life in technology and a substantial part of it in Technology Operations, so I get it. Stuff happens. A few hours later I got a text stating that AT&T was aware of the outage in my area and were projecting the outage to last about 8 hours. Okay. Happy that I got a text on status (I had not remembered signing up for something like that – but still applaud the outreach). Interestingly though, that was a weirdly specific notification and at least to me led me to believe that the issue was something physical, in the field. Again – While minor-ly inconvenienced (not for the first time) I have learned to just roll with the punches or else I would likely go insane on such things. As the ‘fix’ time was after my normal bed time I did not give it much thought.
I got the first notification was restored around 6:05am this morning. As those of you who know me well would know (especially those that call me the Dark Lord of Automation – you know who you are) you might imagine that I have monitoring and observability services built around the availability of my home network. Because of course I do. What followed was a period of instability until it finally settled down and the Internet was restored. Well beyond that 8 hour notice of course. And while I was thrilled to get the initial status about the outage – no other statuses on progress were ever sent. To be fair I was likely sleeping through most of the event, but it was an exceptionally long outage.
One might tempted to say, well this is a one time deal – get over it. Except it is not, has not been, nor likely will continue to be a transient thing. The Internet service to our neighborhood is not good bordering on terrible. It is the primary reason I got Star Link (see previous post) which I have just received and will be installing soon. But first a little history.
Resistance is futile
We moved into our neighborhood a few years ago. Its a lovely community of about 80 homes filled with mostly great people. Its fair to say that the general make up the neighborhood are folks who are relatively well-off, established professionals in their field, and probably higher up on the socio-economic food chain. Maybe not 1%’ers but definitely top 10%’ers. The homes are well kept and as far as the surrounding county goes is likely the highest property tax basis in the area. Despite this idyllic setting the one service that has been significantly lacking since the establishment of the sub-division has been its Internet service. Despite having incredibly high speed residential fiber in the street that runs adjacent to it, the service that was ultimately delivered was sub-par by even technology from 2000 standards. Worse, the farther away from that main street you got – the more the signal degraded – so bandwidth got worse the farther you went into the neighborhood. By the time you got to my house, you were lucky to get about 8mbps. Thats like one television watching Netflix and one kid playing a video game online these days.
I get it. This is first world problems. But this post is more about why big telecom is going to die and why whether or not you like Elon Musk or Star Link, or 5G or whatever – they are going to eat the lunch of these ancient non-responsive dinosaurs.
About two years ago, the neighborhood had about 200 pairs of AT&T dark fiber installed for business-grade service. There are no businesses in the neighborhood, but there is a tower for basic 911 and 5G services there. 200 pairs. There are only 80 homes. Plenty of room for whatever they needed to do and still accommodate a massive upgrade in service (from 15 – 20 year old copper lines) for everyone in the neighborhood. But alas, that fiber was from AT&T Business, not AT&T residential and apparently never the twain shall meet across their own internal siloes and boundaries.
The neighborhood first approached AT&T with pleas to upgrade the service in the neighborhood. That was declined outright. Per some folks familiar with those talks, they were told it was not financially viable to upgrade for just 80 homes. So the neighbors came back and a large majority committed to multi-year contracts with outs if they put it better service. No deal.
After the business fiber was installed another attempt was made. Knowing that most of such costs are always in the pull, trenching and related groundwork this seemed like a good solution. Most of the cost burden had already been taken on (but by another branch of AT&T). Yet again the answer was no. Apologies but that was business fiber not residential fiber. It is fair to say that they will never be able to recoup that cost as their will never be enough businesses to consume anywhere near that amount of fiber in the ground). So yet again the neighborhood came back and promised long term commitments and contracts to help incentivize AT&T to do the right thing. Please let us give you more money! Nope.
Attempts were made to try and do something similar with competitive services like Comcast – who also declined stating that while they could ride the AT&T fiber, 80 homes was just not enough. What if we all committed to pay in multi-year contracts? Nope. A line would not be crossed. We attempted at looking at micro-wave options which also did not work out. No matter what, the entire neighborhood was stuck. AT&T had it within their power to fix, to make an entire neighborhood of people happy and truly grateful. A neighborhood likely willing to pay a premium for such a thing. But they didnt.
Enter the Young Turks
Most people have suffered with sub-par service for years. But now, there are new players in town. It started with a few neighbors opting for 5G routers for their homes. Services like T-Mobile and others are targeting these use cases pretty aggressively. At about the same time Elon Musk and Starlink had announced service for which many people signed up. In the intervening time Verizon and others have also announced emergent solutions to take on this market with their own 5G services (although not yet available in our area). Its important to note that AT&T has a similar service, but I would be doubtful to think this may be an option given the multiple years of poor treatment and deaf ears. The young turks of technology have arrived and I for one celebrate their arrival. They are changing the game, getting creative with service offerings, being flexible and incredibly customer focused. They are hungry and not complacent. They are not in the middle of their own way in driving success.
The bottom line is that the outage last night was nothing more than the final nail in the coffin. At least for me and from what I am hearing many of my neighbors. AT&T had it within its power to be the hero, get paid more, and longer but in the end could not get out of its own way. If it were some regulatory issue, they could have sold some of the fiber to its residential business. Instead the giant dinosaur could not evolve past its own front feet. Its tiny little T-Rex arms could not collect the service opportunities right in front of them. I wonder how many other similar stories like there are out there. It truly is unfortunate.
As for myself, I received my StarLink kit earlier this week. I am waiting for the opportunity to put it all together and plan on documenting the process for those interested here. I have some work travel and things to do so it may take me a few weeks to get things up and running and posted. But stay tuned. Rant Over. Please proceed to your regularly scheduled blog postings.