Search Engines: How Do They Do That?

You already know that you can find just about anything on the Internet. Do you need a recipe for chocolate macadamia nut cookies? Does your daughter have a report due on the wildlife native to Australia? Do you want to find the best pizza in town? Chances are, you bring up your favorite search engine on your desktop computer, laptop, smart phone, or tablet, and type a few words into the search bar. Like magic, the search engine gives you a list of relevant links to click on, chock-full of the information that you need, and then some.

How do they do that? You know that it’s all computerized, and not a live person handing you your results, which explains why sometimes you get a suggestion for a really irrelevant web page. Out of the hundreds of millions of websites, how do Google, Yahoo, Bing, and other search engines weed through them all to give you precisely the information you desire? Let’s find out!

Spiders and Other Creepy Crawlies

The World Wide Web is the part of the Internet that holds millions and millions of pages of information. What crawls on actual webs? Right, spiders. Well, spiders also crawl all over the World Wide Web. These spiders are computerized robots that visit nearly all websites. They index all of the information that they encounter, grouping them by the words that are on each page. This process is called Web crawling.

Much as a spiderweb starts off as a small circle, then rapidly expands, web-spiders start off on one web page, then rapidly spiral outward in different directions. For example, let’s start with a website about cooking. On the main page, there may be 100 different links for various recipes. As the spiders crawl through each of those 100 links, they might encounter 20 more on each page, then 50 more on each of those 20, and so on. When you type “chicken marsala recipes” into your search engine of choice, the spiders have already found hundreds or thousands of pages that hold chicken marsala recipes. The most popular ones will be listed at the top of your search page. Within seconds of your search, you can start to take out your ingredients for dinner.

How do the spiders know which results will be most relevant? That is where meta tags come in.

Tag, You’re It!

As a human being, you know the difference between wanting to “book a reservation” for a flight and wanting to “reserve a book” at the library. A computer, however, might not be programmed to know the difference. This is how meta tags can help the spiders to index pages as they crawl all over the World Wide Web.

When a website owner creates a web page, he has the option of using meta tags, which are the keywords and concepts that he wants the spiders to index his page under. If you have a website that allows users to book airplane reservations, you will choose very different keywords than those which a library might use on its site. Even though the spiders will see the words “book,” “reserve,” and “reservation” on both sites, if the keywords are used correctly, users wanting to book a flight to Cancun should not end up on a library’s page explaining how to reserve a book about Mexico.

Now, there’s always the possibility that a website owner might accidentally or purposefully send a spider in the wrong direction with ineffective or misleading meta tags. Placing meta tags about weight loss, for example, on your airline reservation website, could potentially drive would-be dieters to your site. For this reason, the spiders are programmed to match the actual content with the meta tags. If the meta tags and the content don’t go together, then those misleading meta tags are disregarded. It’s not a perfect system, of course, and this is why you occasionally end up on an obscure site which has nothing to do with the search terms you used.

Website owners also have the option of excluding a page from the World Wide Web spiders. This is called robot exclusion protocol, and is helpful in certain circumstances. If you write a book, for example, and want certain pages only accessible to those who have the direct link to the pages listed in the book, you might purposely exclude these pages via your meta tags.

Ranking and Indexing

Now you understand a bit about how the search engines know which sites to bring up when you type your query into the search bar. But, you may be asking yourself, how do they know what results to put at the top of the page, and which are buried on page nine of the list? The search engines use various algorithms to determine page rank. The better a particular page’s rank, the higher up on the list of results it will appear.

What goes into this formula? Many factors, the sum of which comprise what is called search engine optimization, commonly referred to as SEO. These algorithms are not an exact science, and page ranks may fluctuate day by day. Keyword density, content quality, the correct use of grammar and spelling, meta tag usage, and many other factors all play a part. Depending on the search engine you use, your results may vary when searching for a precise term.

Boolean Operators

To maximize your chances of getting exactly the results you want when performing a search, you can use Boolean operators to include or exclude certain pages. This gives you control over what types of pages come up on the first page of your search results. Being more specific is almost always better than being less specific if you know exactly what you are looking for. Of course, typing in more general terms is also useful in some cases, particularly when you know little about the topic you are researching, and are depending on the search engine to tell you what you are looking for!

Here are some common Boolean operators:

*AND: If are looking for beekeepers in Savannah, for example, typing in beekeepers AND Savannah will give you more precise results than simply typing in beekeepers Savannah. With the latter search term, you may come up with results pertaining to beekeepers but not those located in Savannah, as well as results pertaining to Savannah, but having nothing to do with beekeepers.

*OR: Maybe you want to visit a beach in either Tampa or Sarasota. In that case, you could type in beaches AND Tampa OR Sarasota, which will give you the results for both places.

*NOT: If you are looking for a restaurant, but are violently allergic to shrimp, you might want to avoid those that serve seafood. In that case, you may have good luck searching restaurant NOT shrimp, or something similar, to leave out menus that mention the tasty crustacean.

*Quotation Marks: By putting a group of words in quotation marks, the search engine will know to search only for that particular phrase. Searching for “Betty Brown” will give you different results than searching for Betty Brown, without the quotation marks. Try it and see!

Remember that this is not an exact science, and depending on what you are looking for, you might need to play with Boolean operators a bit to get your desired results.

Thinking Like a Person

While search engines are robots, they are programmed by human beings, and they have a type of artificial intelligence about them. In many ways, they do have the capability to “understand” what you are looking for when you type in a search query, even if you don’t use Boolean operators. If you type in a question or a phrase, the search engine can often decipher what type of results you need. For example, typing in only the word fall could bring up results about the season of autumn, a man named Dr. Fall, falling in love, a story about a woman who fell down the stairs, and other potentially random and irrelevant results. If you type in fall in love, however, or slip and fall, then you will not see results about camping in New England in October. This capability is called concept-based searching, and is evolving rapidly.

Looking Into the Future

There is much more to learn about search engine optimization, writing good web content, page ranking, and other items pertaining to Internet search engines. The reality is that the information available on this topic is always changing, and the future of search engine capability is vast and full of possibilities. Keeping up with the changing trends in the way we can interact with the World Wide Web is both important and nearly impossible; with so much more information available each day, today’s new capability may be outdated a year from now. Whether you are browsing the Internet, looking for something specific, or building your own website with the hopes of ending up on the front page of Google’s search results for your keyword, however, the possibilities are nearly limitless. Have fun with technology and spend some time getting to know the ins and outs of Internet search engines while broadening your own knowledge base. Now, like never before, the world is literally at your fingertips.