Brand sentiment, not keyword rankings is what matters on AI Search
The SEO industry is heading into uncharted territory with the rise of Google’s AI Overviews and ChatGPT. Historically, SEO strategy and metrics have revolved around keyword rankings to a large degree. This type of measurement is much less relevant when it comes to AI Search responses. What matters is what do these machine learning algorithms think of your brand.
Once these systems have formed a point of view of your brand then that’s likely what will determine which prompts will mention your brand and what is said about your brand. Unlike traditional SERP results on Google, where almost any mention is considered desirable because the results are largely coming from your own site, with AI the responses can just as easily say factually incorrect statements about you or have negative sentiment which can hurt users’ perception of your brand. Monitoring what is being said can help you better understand what you might want to focus on.
The above chart illustrates a good example of the importance of LLM brand monitoring. Just from looking at this Looker Studio dashboard, I can already tell that while ChatGPT mostly gets it right about my brand name, it is also getting somewhat confused for another person named Raymond Wong. He is a famous film producer from Hong Kong. I don’t know him personally and obviously do not want him being mentioned in the context of my brand name though I probably wouldn’t mind being mentioned in the context of his brand name. Ha!
How do I monitor what ChatGPT is saying about my brand?
I’ve tested a few LLM AI Search tools to-date, and the most promising one I’ve come across is Waikay which was created by Dixon Jones. The goal of this tool is to determine what popular LLM’s are saying about your brand in addition to competitor monitoring. The underlying technology is based off of their InLinks platform which I’m a proponent of. If you want a SAAS tool I would recommend checking them out.
As for myself, I am currently doing my own testing and building dashboards to monitor LLM brand sentiment for my other business. This may eventually be a service that I offer to my clients. Here’s the LLM brand monitoring process that I’m using:
- Run a Python script in Google Colab that leverages the OpenAI API and Google Natural Language API to prompt ChatGPI & Gemini about my brand then extract entities from the responses
- Create a Google BigQuery dataset to store prompt responses
- Run another Python script that visualizes the entities with time series charts
- Run another Python script that analyzes the prompt responses with Google NLP’s sentiment analysis and visualize the results with time series charts
- Run another Python script that visualizes entity by salience score. Salience is a metric generated by Google’s NLP API that measures the importance of an entity to the text that it was generated from. This is a good summarization metric to show the most relevant entities.
- Alternative Option: Import the BigQuery data table into Looker Studio to create dashboards. This can be a simper no-code option if you need simpler reporting that can be shared with your clients or executive team.
- The steps above describe a manual process for daily brand monitoring. At some point it could make sense to automate this through Google Cloud Scheduler if I decide I want to do this project long-term.
How many prompts you use to monitor your brand is up to you but the idea is to see if the trends over time align with your brand. For one prompt, I went with asking about ChatGPT to tell me about (brand name). For another prompt, I asked ChatGPT to describe my (website domain) in one paragraph and provide five bullet points. Those responses were uploaded to my BigQuery data table which I’ve stored and can keep the charts up-to-date as time goes by.
Prompt responses can change from session to session, even anonymized sessions so perhaps prompting once is not enough. But LLM responses can also be completely random so I think once per day is probably enough for my own testing. Some tools prompt with much greater variety and velocity but until traffic starts to become significant I’m totally fine with limited prompting for brand monitoring. Other factors to consider are the monetary and environmental cost of running excessive prompts.
I initially started off using Python charts to visualize my LLM brand entities because there is a lot of flexibility to create custom charts. It was too cumbersome to be practical outside of business presentations so I’ve built out a Looker Studio connection to my BigQuery database that I can easily use on a daily basis. There are less visualization options within Looker Studio but it’s more than sufficient for me at this time.
Another area I haven’t explored yet is how to extract external website links from the API response. Doing this at scale would certainly would help to understand where these LLMs are getting their information from. When I was doing some non-logged in prompts I found that some of my brand citations were coming from a press release that I did for another one of my websites. This indicated that ChatGPT smart enough to connect the dots there however about the association between my two websites despite me not explicitly stating that in the press release.
Another feature within Google Cloud’s Natural Language Processing API that can be useful for brand monitoring is sentiment analysis. By analyzing the LLM prompt responses you can get a brand sentiment score. +1 represents a very positive sentiment while -1 represents a very negative sentiment. Tracking LLM brand sentiment over time can be quite useful especially as notoriety increases.
As AI search evolves we’ll be seeing more and more tools come into the marketplace. In my opinion, paying for SAAS tools makes sense for larger brands but for smaller brands and solo businesses it’s certainly possible to build your own proprietary brand monitoring through the use of API’s and databases like I’ve shown in this article.