Search engines use spiders/bots to crawl the internet and find documents and web addresses, then index these pages by adding them to the search engine's database. The indexing process determines whether to include pages in search results based on meta tags like index or no-index. When users search with keywords, the search algorithm searches the indexed database to return relevant results.