Detecting vaccine misinformation in a custom Bluesky feed, using LaunchDarkly and OpenAI featured image

Bluesky is a social network that is currently growing at a rate of a million users a day. Pour one out for their SRE team.

One reason for Bluesky’s popularity is how much flexibility and control it gives back to users. For example, moderation tools are highly configurable. Starter packs make it easy to find people who share your interests, be they financial economics, astrophotography, or Flavor Flav’s faves.

You can even customize what kinds of posts you’d like to see. Bluesky provides tools to build custom feeds using the open source ATProtocol. Free yourself from the tyranny of the algorithm by changing ~10 lines of code! 

I’m a bit of a medical nerd myself. I want to keep up to date about public health and vaccinations. Unfortunately, the vaccine disinformation movement is also growing. It’s only a matter of time before anti-vaxxers crash the Bluesky party.

I could use a large language model to try and determine whether a given post contains misinformation or denialism. But do I have to? AI compute costs real dollars. What if I had the flexibility to enable LLM filtering on my feed when anti-vax posts spike, without needing to deploy anything?

LaunchDarkly’s feature flags can help. Decoupling deployment from feature management is especially useful on rapidly scaling platforms where things are changing fast.

In this tutorial, you’ll create a Bluesky custom feed in Python that filters posts based on keywords. You’ll use an OpenAI model to do basic sentiment analysis on the posts, discarding the ones that are likely to be misinformation. Then you’ll wrap the LLM call in a LaunchDarkly feature flag so you can quickly enable it when misinformation surges.

Prerequisites

Building a custom Bluesky feed with Python

Clone this repository, which is a fork of MarshalX’s Flask bluesky-feed-generator. Thanks MarshalX

git clone https://github.com/launchdarkly-labs/bluesky-custom-feed-python

Set up and activate your virtual environment using these commands:

cd bluesky-custom-feed-python
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Rename your .env.example file to .env. Save the file.

Start the server. Make a note of this command, we’ll use it throughout the tutorial:

flask run

You should see a “firehose” of post output in your terminal.

INFO:server.logger:NEW POST [CREATED_AT=2024-11-19T03:11:47.328Z][AUTHOR=did:plc:r5skxbp6uq27pa4ooxxzztel][WITH_IMAGE=False]: ひるおん
INFO:server.logger:NEW POST [CREATED_AT=2024-11-19T03:11:47.817Z][AUTHOR=did:plc:nejecdxp6bkzicn5dckezodc][WITH_IMAGE=False]: Esse sorriso 🤏🥹
INFO:server.logger:NEW POST [CREATED_AT=2024-11-19T03:11:46.992Z][AUTHOR=did:plc:53tl6yttplruxe2layewdm3k][WITH_IMAGE=False]: Alright, since some of you are quick on the draw, I’ll get started. Please don’t think less of me for any of these.   *clears throat* in no particular order…  1. George Michael. He was my favorite since I was little. His voice was like butter. And this video was a brilliant response to the tabloids.

If you get a SSL certificate error, you may need to install an additional package on your local machine. See this Stack Overflow post for details.

pip install --upgrade certifi

Stop the app. Let’s disable logging every single post. It was fun for a hot second to make sure everything is working, but we don’t want to drown in input forever. 


Open server/data_filter.py. Comment out the following lines:

       # print all texts just as demo that data stream works
       # post_with_images = isinstance(record.embed, models.AppBskyEmbedImages.Main)
       # inlined_text = record.text.replace('\n', ' ')
       # logger.info(
       #     f'NEW POST '
       #     f'[CREATED_AT={record.created_at}]'
       #     f'[AUTHOR={author}]'
       #     f'[WITH_IMAGE={post_with_images}]'
       #     f': {inlined_text}'
       # )

Before the operations_callback function definition, create a set of keywords:

KEYWORDS = {
   "publichealth",
   "vaccine",
   "vaccines",
   "vaccination",
   "mrna",
   "booster"
}

This is an incredibly naive way of creating a feed but it works.


For reasons I can’t get into here, Bluesky’s code examples all create Alf-themed feeds. Let’s tweak the operations_callback function to use our keyword set instead.

Delete these two lines:

       # only alf-related posts
       if 'alf' in record.text.lower():

Replace them with:

    if any(keyword in record.text.lower() for keyword in KEYWORDS):

Restart your server from the command line with flask run.

Now you should see a log of posts that match these keywords. Sick! (Pun intended.)

INFO:server.logger:NEW POST [CREATED_AT=2024-11-19T03:11:56.177Z][AUTHOR=did:plc:g3tob4ohsflqrputc7wr3fke][WITH_IMAGE=False]: I got five vaccines and one blood draw in one sitting today.  I really feel like there should have been more fanfare about that milestone.  Like a personal best certificate or something.
INFO:server.logger:Added to feed: 1

Sentiment analysis is your friend: filtering your Bluesky feed with a LLM

This bit requires an OpenAI API key. Go to your OpenAI dashboard and create a new key named “Bluesky custom feed”. Copy the key. Paste it into your .env file. Save the file. 


Create a new file, server/openai_client.py. Paste the following code into it:

import os
import openai
from server.logger import logger

from dotenv import load_dotenv
load_dotenv()
client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def detect_vaccine_denialism(text: str) -> bool:
   """
   Uses OpenAI's API to determine if a post contains vaccine denialism.
   Returns True if vaccine denialism is detected, False otherwise.
   """
   try:
       response = client.chat.completions.create(
           model="gpt-3.5-turbo",
           messages=[
               {"role": "system", "content": "You are an expert at detecting vaccine misinformation and denialism. Respond with only 'true' if the text contains vaccine denialism or 'false' if it does not."},
               {"role": "user", "content": text}
           ],
           temperature=0,
           max_tokens=10
       )
      
       result = response.choices[0].message.content.strip().lower()
       print("detect_vaccine_denialism result: ", result)
       return result == "true"
      
   except Exception as e:
       logger.error(f"Error detecting vaccine denialism: {e}")
       return False

For optional fun, you can run some sample inputs through this function. In your terminal, open an interactive Python session. I have truncated some output for brevity.

from server.openai_client import detect_vaccine_denialism
detect_vaccine_denialism(“Vaccines cause autism.”)
…
detect_vaccine_denialism result:  true
detect_vaccine_denialism(“mRNA vaccines for AIDS are in large scale trials.”)
…
Detect_vaccine_denialism result: false

Edit operations_callback (beginning on line 43) so that it calls our new function by replacing the following lines of code:

    if any(keyword in record.text.lower() for keyword in KEYWORDS):
       contains_denialism = detect_vaccine_denialism(record.text)
       if not contains_denialism:
           post_with_images = isinstance(record.embed, models.AppBskyEmbedImages.Main)
           inlined_text = record.text.replace('\n', ' ')
           logger.info(
               f'NEW POST '
               f'[CREATED_AT={record.created_at}]'
           f'[AUTHOR={author}]'
               f'[WITH_IMAGE={post_with_images}]'
           f': {inlined_text}'
           )
           reply_root = reply_parent = None
           if record.reply:
               reply_root = record.reply.root.uri
               reply_parent = record.reply.parent.uri

           post_dict = {
               'uri': created_post['uri'],
               'cid': created_post['cid'],
               'reply_parent': reply_parent,
               'reply_root': reply_root,
           }
           posts_to_create.append(post_dict)

The remaining lines of the function should stay as-is.

Now when we run our server we should see the following:

INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
detect_vaccine_denialism result:  false
INFO:server.logger:NEW POST [CREATED_AT=2024-11-20T00:52:38.939Z][AUTHOR=did:plc:j3a762gtes2jdm4f2kd27gfr][WITH_IMAGE=False]: I also wanna get my MMR titers checked, make sure I don't need a booster.
INFO:server.logger:Added to feed: 1

As soon as you have verified the output, kill the server lest you run up your OpenAI bill unnecessarily. 💸

Adding a LaunchDarkly feature flag to a Bluesky feed

Head over to the LaunchDarkly app to create a new feature flag.

Use the following configuration:

  • Name: vaccine_disinformation_filter
  • Description: Flag that enables LLM filtering of Bluesky posts that might contain vaccine disinformation.
  • Configuration: custom
  • Flag type: Boolean

Screenshot showing flag configuration for a custom Bluesky feed misinformation filter.

Variations:

  • True: name, true. Value, true.
  • False: name, false. Value, false.
  • Serve when targeting is on: true. Serve when targeting is off: false.

Screenshot showing part 2 of Bluesky custom feed flag configuration.

Click “Create flag.” On the next screen, click the … menu. Go down to “SDK key” to copy your SDK key. Paste it into your .env file. Save the file.

Screenshot demonstrating how to copy the SDK key after creating a flag.

In openai_client.py, add these import statements to the top of the file:

import ldclient
from ldclient import Context
from ldclient.config import Config

Update the detect_vaccine_denialism function to check the value of the flag:

def detect_vaccine_denialism(text: str) -> bool:
   """
   Uses OpenAI's API to determine if a post contains vaccine denialism.
   Returns True if vaccine denialism is detected, False otherwise.
   """

   ldclient.set_config(Config(os.getenv("LAUNCHDARKLY_SDK_KEY")))
   ld_client = ldclient.get()

   context = Context.builder("vaccine-filter-user").build()
   if not ld_client.variation("vaccine_disinformation_filter", context, False):
       print("LaunchDarkly flag is not enabled")
       return False
   try:
       print("flag is on!!!!")
       response = client.chat.completions.create(
           model="gpt-3.5-turbo",
           messages=[
               {"role": "system", "content": "You are an expert at detecting vaccine misinformation and denialism. Respond with only 'true' if the text contains vaccine denialism or 'false' if it does not."},
               {"role": "user", "content": text}
           ],
           temperature=0,
           max_tokens=10
       )
      
       result = response.choices[0].message.content.strip().lower()
       print("detect_vaccine_denialism result: ", result)
       return result == "true"
      
   except Exception as e:
       logger.error(f"Error detecting vaccine denialism: {e}")
       return False

Start the server. Log output should reflect that the flag isn’t enabled yet.

INFO:ldclient.util:Started LaunchDarkly Client: OK
LaunchDarkly flag is not enabled
INFO:server.logger:NEW POST [CREATED_AT=2024-11-19T02:51:47.141Z][AUTHOR=did:plc:sj2wpqwrsq4uob5hq5vfb4u7][WITH_IMAGE=False]: So what’s your explanation for the anti-vaccine rhetoric?!?  www.washingtonpost.com/world/2024/1...

In the LaunchDarkly app, turn your flag on.

Screenshot showing how to turn on a flag to enable LLM filtering in a custom Bluesky feed.

Output should now look like this example:

LaunchDarkly flag is on!!!!
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
detect_vaccine_denialism result:  false
INFO:server.logger:NEW POST [CREATED_AT=2024-11-19T02:50:35.766Z][AUTHOR=did:plc:takc745rvwdh3b7v3tke2fs2][WITH_IMAGE=False]: www.cell.com/heliyon/full...  The Impact of Vaccination Status on Post-acute Sequelae in Hospitalized COVID-19 Survivors using a multi-disciplinary approach: an observational single center study.

Score one for science! 🔬💉🧬Excellent work.

What’s next for Bluesky, custom feeds, Python, feature flags, LLMs, and the world at large?

In this post, you’ve learned how to:

  • Create a custom Bluesky feed in Python that performs keyword-based filtering
  • Use an LLM to do rough sentiment analysis on Bluesky posts
  • Conditionally enable LLM functionality with a LaunchDarkly feature flag

This demo is very basic. Some upgrades to consider:

  • Using ML classification to better detect vaccine-related content
  • Using LaunchDarkly’s new AI configs: experiment with different models and prompts to improve sentiment analysis
  • Deploying your feed to production so other users can benefit from it


Thanks so much for reading. Hit me up on Bluesky if you found this tutorial useful. You can also reach me via email (tthurium@launchdarkly.com) or LinkedIn.

Like what you read?
Get a demo
Related Content
November 27, 2024