Real-time Chat GPT – A look at New Bing

It's no secret Microsoft has bet on #chatGPT and OpenAI heavily. After a long wait, I finally got a chance to try, how well the integration to Bing works in practice.

I started out by enrolling in Microsoft's wait list. I also took some of the steps to expedite my access. Microsoft is taking the opportunity to force feed eager early adopters Bing, their search extension and the #Edge browser, which they say can move you up the wait list.

After a few weeks of waiting, I got an email message to let me know I'm in. Unfortunately the New Bing only works on MS Edge – I tried spoofing my user agent too, but it didn't seem to work. Edge it is.

Now I got to run some tests. I'll focus here on documenting use cases and behaviors which are new, rather than reiterating what we all know advanced language models can do.

New thing #1: Latest information

As I'm sure everyone knows by now, ChatGPT's training data has not been updated since September 2021:

This is not the case with Bing. I asked news about the Russian war of aggression in Ukraine

The resulting summary contains details, which have become available today, like the full scale war's anniversary and the UN resolution. Bing's version of chat GPT seems to automatically combine Bing's search results with the text generation and general information included in the language model.

New thing #2: Page summaries

This doesn't always work as you'd expect though. I asked the New Bing for what it can tell us about our product, which is a whistleblowing channel for small and medium companies.

Bing seems to assume that the site is a general-purpose informational site, and not about a product. A worse issue is, that Bing tells us that the page is maintained by the Finnish Border Guard (whose whistleblowing channel appears on the 1st page of results when you search bing for ilmoittajansuojelu.fi).

Speculation

It would appear that New Bing's chat mode effectively constructs a set of search queries, the results of which are then used as the input for Microsoft's variant of ChatGPT. This means that limitations in Bing's search algorithm and index will be reflected in the results of the chat. I expect these issues to solve themselves, as the search volume in Bing increases, Microsoft gets more data about searches and is able to improve Bing's search algorithm. Some other problems, like interpreting the customs office and our website as having a common owner is likely an issue with the integration between Bing search and ChatGPT and should be easily fixed.

Conclusion

Chat-based search is an improvement over conventional search engines, but I still find myself going back to a regular search engine for many things. In a lot of cases, I'm simply searching for the right website, without the layer of abstraction introduced by an AI agent. In Microsoft's case, part of the reason may be that it only works on the Edge browser, but I don't find myself using the alternatives, like you.com either. I still use the original ChatGPT very much though, somehow the fact that it's not trying to perform a search makes it more useful, not less.