Search engines use computer programs called spiders or robots to crawl the web and index pages, while subject directories are organized by humans. When searching, it is important to use techniques like keywords, Boolean operators, phrase searching and field searching to narrow results and find the most relevant information. These techniques include using AND, OR and AND NOT to combine search terms effectively as well as searching specific fields like title, domain, host, URL and links.