Is the internet broken beyond repair?

Thirty years ago Oasis topped the charts, my waist was still slim and the world wide web was new and exciting. It was a simpler time and the web was a much smaller and much less commercial environment. Internet shopping was in its infancy with Amazon having only just been formed and still predominantly selling books rather than everything under the sun.

The internet (or more accurately, the world wide web) was at a tipping point; about to move from a nerdy special interest into a widely used everyday phenomenon. Up to this point, you could still get away with navigating the web by starting at a directory of web sites such as Yahoo or DMOZ and finding a site that seemed to cover the topic you were interested in. But as the number of websites sky-rocketed, so it became impossible for a human-maintained yellow pages of the web to keep up-to-date. This was the birth of the search engine age.

Early attempts at search were of mixed capabilities. Webcrawler, AltaVista and Lycos could usually be persuaded to give you the answers you sought but often you would have to try one and then another until you hit upon the best one for that query. Enter stage left: Google. It's hard to believe it now but Google was revolutionary. It had one search box and nine times out of ten it would lead you directly to the web page you were looking for. It was so accurate it even had an "I feel lucky" button that would skip the search results page and just take you to the first link.

PageRank was Google's secret sauce. Larry Page had realised that there was wisdom in crowds. A web page that was good would end up being linked from more web sites than a poor one. By taking the link texts and weighting them by popularity, Google could find the most relevant page for any query. It democratised knowledge as evidenced by Google's mission statement : "to organise all the world's information".

It's all been downhill from there.

The money men realised the potential of the web. Huge sums of money were thrown at any college kid who could throw a website together and make it sound like an exciting new idea. "X but on the web" was the mantra as "Uber for X" would be twenty years later.

And where there is money to be made, there is advertising space to be sold. As Google increasingly became the front door of the internet, getting on the first page of results became the make or break difference for ecommerce businesses. You could try search engine optimisation but that was time-consuming and unreliable. Much easier was to throw money at the problem: pay Google to put your ads front and centre.

The new millennium dawned, the dotcom boom went bust and slowly from the ashes came Web 2.0 and social media. By now, the ad-driven web was well established. Entire business plans were predicated on the fact that companies would pay to have their ads in front of millions of internet users. In Field of Dreams style, if you built it (a popular website), they (the advertisers) would come.

The real value proposition of internet advertising was that you could capture people at the moment that they were searching for a product like yours. Audiences could be segmented like never before. Big companies like Google, Facebook and DoubleClick were stretching their tentacles around the web. They would allow smaller sites to show their ads in return for a cut of the profits. But what they really wanted was to track and profile the users of those sites. The same code that displayed an ad was also dropping a tracking cookie in the browser so that the same user could be identified as they went from one website to another. Click by click web users were creating a profile of themselves and their interests for the ad brokers. And the ad brokers could then market those profiles back to their advertisers. The process is summed up in the expression "if you're not paying for the product, you are the product".

The awareness of tracking increased, as did the annoying nature of the ads, leading to the rise of the ad blocker. An arms race ensued as new ways of tracking users were invented and new ways of blocking them followed suit.

Well-intentioned attempts at reining in the ad trackers were introduced in heavy-handed fashions : cookie disclaimers on every single website, GDPR privacy options that are deliberately designed to make "accept" an easier option to click than "decline". The web became a maze of annoyances at every turn.

And then along came AI. Google, the company that opened up the web, has announced that it will now offer an AI summary of search results rather than linking to the results themselves. Where users were once sent to the page that had the best-regarded information, they will now get their information in a mangled, mixed up and often misleading machine-made summary.

Cory Doctorow coined the term "enshittification" to describe this process by which quality is pushed out in favour of pernicious technology that grinds the user up for sale to advertisers. AI is the ultimate enshittification, it takes human-created works and churns them out as AI slop to be consumed again as mere compost in which to grow advertising profits. Why would anyone create content when Google will just regurgitate it as its own work and not even send any web traffic back to the original source?

And so the web has been shittified. In the space of twenty years we've gone from all the world's information being available at your fingertips to a sludge of possibly accurate, possibly hallucinated slop.

PageRank, the algorithm which once brought order to the chaotic web, has long since been retired. Marketers learnt how to fudge it. Search engine optimisers realised that they could build more incoming links to their pages and mislead Google into increasing the importance of their pages. As Goodhart's Law puts it "When a measure becomes a target, it ceases to be a good measure".

So, is the web irredeemably broken? As with everything else, money talks. While Google keeps getting users and keeps getting advertising money, it will keep doing what it's doing. While Meta can monetise your social connections, it will keep doing what it's doing. For the web to change, web users have to change. Check out alternative search engines; decentralised social media and video sites, open source file sharing, independent media. There is another way, but we have to build it ourselves.