Why can't we find anything around here?
On the decay of search quality, URLs that are too long, and Jerry and David's Guide to the World Wide Web
Anyone fortunate enough to have experienced the early days of the commercial Internet may recall how personal the entire experience could feel. In the brief window of time between the emergence of the Mosaic browser experience and the arrival of Google, the Internet might have been global in its ambitions, but it also took a whole lot of human curation just to function.
■ Yahoo today may be everything from an email service provider to an original news outlet, but for a brief while, it was just Jerry and David's Guide to the World Wide Web: An attempt to categorize the individual pages on the Internet and document their general contents. It was something like a hybrid between a telephone book and the card catalog in a library, attempting to list what addresses the individual user needed to look up in order to find things (a much bigger issue before the dot-com boom, when domain names cost a relative fortune), as well as what one was likely to discover upon arriving there.
■ It looks quaint today, when webpages are dynamically generated, apps serve up much of the Internet's consumer material, and artificial intelligence is being enlisted to flood the zone with a never-ending pipeline of new content. But sincere efforts to survey and map the world of the all-new World Wide Web were useful. And their imprimatur was vital if you, the content creator (though nobody called you that yet), wanted to be found.
■ What made Google such a spectacular success was its ability to supplant the slow process of looking through curated directories by returning the thing the user was most likely to want based upon no more effort than typing out a question. Search engines ate the world, of course. But careful observers are beginning to note that the search-engine structure is showing signs of frailty. NBC News tech reporter Kat Tenbarge laments, "All the search functions around the internet seem to be in a prolonged state of collapse. It feels impossible to find specific things anymore".
■ Content-management tools have a big part to play in making this problem worse. What makes it easy for people to add new content to the Internet also generally makes it easy for them to mess up or destroy what is already there. Even worse, URLs generally are an unholy mess. Nobody builds rules-based file trees anymore, because they're too busy packing URLs full of search-friendly words.
■ And thus there are few sites left that commit to a logical (and self-perpetuating) file structure that fits inside the address bar. Instead, most fresh pages are identified by either too many words (e.g., "https://www.kosu.org/energy-environment/2023-06-26/some-cities-in-central-u-s-are-asking-residents-to-conserve-water") or by none at all (e.g., "https://www.bbc.com/news/av/world-us-canada-66037455").
■ It's not clear whether there is a real solution available that anyone will adopt voluntarily -- but if search engines really do begin to break down in their real user efficacy, then it's possible we might see a revival of sorts in Internet directories. Humans wouldn't have to curate them entirely by hand -- artificial intelligence may well be ideal for much of the task -- but some sort of thoughtful editorial review may be useful. Rarely does anyone see what's coming next with any accuracy, but if search quality really does fall into a broad sort of decline, people aren't simply going to stop looking to the Internet for answers.
