Blog

A comprehensive guide on getting the most out of your Chinese topic models, from preprocessing to interpretation.

With our recent paper on discourse dynamics in European Chinese diaspora media, our team has tapped into an almost unanimous frustration with the quality of topic modelling approaches when applied to Chinese data. In this article, I will introduce you to our novel topic modelling method, KeyNMF, and how to apply it most effectively to Chinese textual data.

Topic Modelling with Matrix Factorization

Before diving into practicalities, I would like to give you a brief introduction to topic modelling theory, and motivate the advancements introduced in our paper.

Topic modelling is a discipline of Natural Language Processing for uncovering latent topical information in textual corpora in an unsupervised manner, that is then presented to the user in a human-interpretable way (usually 10 keywords for each topic).

There are many ways to formalize this task in mathematical terms, but one rather popular conceptualization of topic discovery is matrix factorization. This is a rather natural and intuitive way to tackle the problem, and in a minute, you will see why. The primary insight behind topic modelling as matrix factorization is the following: Words that frequently occur together, are likely to belong to the same latent structure. In other words: Terms, the occurrence of which are highly correlated, are part of the same topic.

You can discover topics in a corpus, by first constructing a bag-of-words matrix of documents. A bag-of-words matrix represents documents in the following way: Each row corresponds to a document, while each column to a unique word from the model’s vocabulary. The values in the matrix are then the number of times a word occurs in a given document.

This matrix can be decomposed into the linear combination of a topic-term matrix, which indicates how important a word is for a given topic, and a document-topic matrix, which indicates how important a given topic is for a given document. A method for this decomposition is Non-negative Matrix Factorization, where we decompose a non-negative matrix to two other strictly non-negative matrices, instead of allowing arbitrary signed values.

NMF is not the only method one can use for decomposing the bag-of-words matrix. A method of high historical significance, Latent Semantic Analysis, utilizes Truncated Singular-Value Decomposition for this purpose. NMF, however, is generally a better choice, as:

1. The discovered latent factors are of different quality from other decomposition methods. NMF typically discovers localized patterns or parts in the data, which are easier to interpret.
2. Non-negative topic-term and document-topic relations are easier to interpret than signed ones.

Using NMF with just BoW matrices, however attractive and simple it may be, does come with its setbacks:

1. NMF typically minimizes the Frobenius norm of the error matrix. This entails an assumption of Gaussianity of the outcome variable, which is obviously false, as we are modelling word counts.
2. BoW representations are just word counts. This means that words won’t be interpreted in context, and syntactical information will be ignored.

KeyNMF

To account for these limitations, and with the help of new transformer-based language representations, we can significantly improve NMF for our purposes.

The key intuition behind KeyNMF is that most words in a document are semantically insignificant, and we can get an overview of topical information in the document by highlighting the top N most relevant terms. We will select these terms by using contextual embeddings from sentence-transformer models.

The KeyNMF algorithm consists of the following steps:

1. Embed each document using a sentence-transformer, along with all words in the document.
2. Calculate cosine similarities of word embeddings to document embeddings.
3. For each document, keep the highest N words with positive cosine similarities to the document.
4. Arrange cosine similarities into a keyword-matrix, where each row is a document, each column is a keyword, and values are cosine similarities of the word to the document.
5. Decompose the keyword matrix with NMF.

This formulation helps us in multiple ways. a) We substantially reduce the model’s vocabulary, thereby having less parameters, resulting in faster and better model fit b) We get continuous distribution, which is a better fit for NMF’s assumptions and c) We incorporate contextual information into our topic model.

Chinese Topic Modelling with KeyNMF

Now that you understand how KeyNMF works, let’s get our hands dirty and apply the model in a practical context.

Preparation and Data

First, let’s install the packages we are going to use in this demonstration:

pip install turftopic[jieba] datasets sentence_transformers topicwizard

Then let’s get some openly available data. I chose to go with the SIB200 corpus, as it is freely available under the CC-BY-SA 4.0 open license. This piece of code will fetch us the corpus.

from datasets import load_dataset

# Loads the dataset
ds = load_dataset("Davlan/sib200", "zho_Hans", split="all")
corpus = ds["text"]

Building a Chinese Topic Model

There are a number of tricky aspects to applying language models to Chinese, since most of these systems are developed and tested on English data. When it comes to KeyNMF, there are two aspects that need to be taken into account.

Firstly, we will need to figure out how to tokenize texts in Chinese. Luckily, the Turftopic library, which contains our implementation of KeyNMF (among other things), comes prepackaged with tokenization utilities for Chinese. Normally, you would use a CountVectorizer object from sklearn to extract words from text. We added a ChineseCountVectorizer object that uses the Jieba tokenizer in the background, and has an optionally usable Chinese stop word list.

from turftopic.vectorizers.chinese import ChineseCountVectorizer

vectorizer = ChineseCountVectorizer(stop_words="chinese")

Then we will need a Chinese embedding model for producing document and word representations. We will use the paraphrase-multilingual-MiniLM-L12-v2 model for this, as it is quite compact and fast, and was specifically trained to be used in multilingual retrieval contexts.

from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")

We can then build a fully Chinese KeyNMF model! I will initialize a model with 20 topics and N=25 (a maximum of 15 keywords will be extracted for each document)

from turftopic import KeyNMF

model = KeyNMF(
    n_components=20,
    top_n=25,
    vectorizer=vectorizer,
    encoder=encoder,
    random_state=42, # Setting seed so that our results are reproducible
)

We can then fit the model to the corpus and see what results we get!

document_topic_matrix = model.fit_transform(corpus)
model.print_topics()

┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Topic ID ┃ Highest Ranking                                                                              ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        0 │ 旅行, 非洲, 徒步旅行, 漫步, 活动, 通常, 发展中国家, 进行, 远足, 徒步                         │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│        1 │ 滑雪, 活动, 滑雪板, 滑雪运动, 雪板, 白雪, 地形, 高山, 旅游, 滑雪者                           │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│        2 │ 会, 可能, 他们, 地球, 影响, 北加州, 并, 它们, 到达, 船                                       │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│        3 │ 比赛, 选手, 锦标赛, 大回转, 超级, 男子, 成绩, 获胜, 阿根廷, 获得                             │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│        4 │ 航空公司, 航班, 旅客, 飞机, 加拿大航空公司, 机场, 达美航空公司, 票价, 德国汉莎航空公司, 行李 │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│        5 │ 原子核, 质子, 能量, 电子, 氢原子, 有点像, 原子弹, 氢离子, 行星, 粒子                         │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│        6 │ 疾病, 传染病, 疫情, 细菌, 研究, 病毒, 病原体, 蚊子, 感染者, 真菌                             │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│        7 │ 细胞, cella, 小房间, cell, 生物体, 显微镜, 单位, 生物, 最小, 科学家                          │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│        8 │ 卫星, 望远镜, 太空, 火箭, 地球, 飞机, 科学家, 卫星电话, 电话, 巨型                           │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│        9 │ 猫科动物, 动物, 猎物, 狮子, 狮群, 啮齿动物, 鸟类, 狼群, 行为, 吃                             │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       10 │ 感染, 禽流感, 医院, 病毒, 鸟类, 土耳其, 病人, h5n1, 家禽, 医护人员                           │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       11 │ 抗议, 酒店, 白厅, 抗议者, 人群, 警察, 保守党, 广场, 委员会, 政府                             │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       12 │ 旅行者, 文化, 耐心, 国家, 目的地, 适应, 人们, 水, 旅行社, 国外                               │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       13 │ 速度, 英里, 半英里, 跑步, 公里, 跑, 耐力, 月球, 变焦镜头, 镜头                               │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       14 │ 原子, 物质, 光子, 微小, 粒子, 宇宙, 辐射, 组成, 亿, 而光                                     │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       15 │ 游客, 对, 地区, 自然, 地方, 旅游, 时间, 非洲, 开车, 商店                                     │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       16 │ 互联网, 网站, 节目, 大众传播, 电台, 传播, toginetradio, 广播剧, 广播, 内容                   │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       17 │ 运动, 运动员, 美国, 体操, 协会, 支持, 奥委会, 奥运会, 发现, 安全                             │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       18 │ 火车, metroplus, metro, metrorail, 车厢, 开普敦, 通勤, 绕城, 城内, 三等舱                    │
├──────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│       19 │ 投票, 投票箱, 信封, 选民, 投票者, 法国, 候选人, 签名, 透明, 箱内                             │
└──────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘

As you see, we’ve already gained a sensible overview of what there is in our corpus! You can see that the topics are quite distinct, with some of them being concerned with scientific topics, such as astronomy (8), chemistry (5) or animal behaviour (9), while others were oriented at leisure (e.g. 0, 1, 12), or politics (19, 11).

Visualization

To gain further aid in understanding the results, we can use the topicwizard library to visually investigate the topic model’s parameters.

Since topicwizard uses wordclouds, we will need to tell the library that it should be using a font that is compatible with Chinese. I took a font from the ChineseWordCloud repo, that we will download and then pass to topicwizard.

import urllib.request
import topicwizard

urllib.request.urlretrieve(
    "https://github.com/shangjingbo1226/ChineseWordCloud/raw/refs/heads/master/fonts/STFangSong.ttf",
    "./STFangSong.ttf",
)
topicwizard.visualize(
    corpus=corpus, model=model, wordcloud_font_path="./STFangSong.ttf"
)

This will open the topicwizard web app in a notebook or in your browser, with which you can interactively investigate your topic model:

Conclusion

In this article, we’ve looked at what KeyNMF is, how it works, what it’s motivated by and how it can be used to discover high-quality topics in Chinese text, as well as how to visualize and interpret your results. I hope this tutorial will prove useful to those who are looking to explore Chinese textual data.

For further information on the models, and how to improve your results, I encourage you to check out our Documentation. If you should have any questions or encounter issues, feel free to submit an issue on Github, or reach out in the comments :))

All figures presented in the article were produced by the author.

---

Contextual Topic Modelling in Chinese Corpora with KeyNMF was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Use LangGraph, mlx and Florence2 to build an agent that answers complex image questions, with the option to run everything locally.

In this article we’ll use LangGraph in conjunction with several specialized models to build a rudimentary agent that can answer complex questions about an image, including captioning, bounding boxes and OCR. The original idea was to build this using local models only, but after some iteration I decided to add connections to cloud based models too (i.e. GPT4o-mini) for more reliable results. We’ll explore that aspect too, and all the code for this project can be found here.

Over the past year, multimodal large language models — LLMs that take reasoning and generative capabilities beyond text to media such as images, audio and video — have become increasingly powerful, accessible and usable within production ML systems.

Closed source, cloud-based models like GPT-4o, Claude Sonnet and Google Gemini are remarkably capable at reasoning over image inputs and are much cheaper and faster than the multimodal offerings just a few months ago. Meta has joined the party by releasing the weights of multiple competitive multimodal models in its Llama 3.2 series. In addition, cloud computing services like AWS Bedrock and Databricks Mosaic AI are now hosting many of these models, allowing developers to quickly try them out without having to manage hardware requirements and cluster scaling themselves. Finally, there is an interesting trend towards a myriad of smaller, specialized open source models which are available for download from repositories like Hugging Face. A smart agent with access to these models should be able to choose which ones to call in what order to get a good answer, which bypasses the need for a single giant, general model.

One recent example with fascinating image capabilities is Florence2. Released in June 2024, it is somewhat of a foundational model for image-specific tasks such as captioning, object detection, OCR and phrase grounding (identifying objects from provided descriptions). By LLM standards it’s also small — 0.77B parameters for the most capable version — and therefore runnable on a modern laptop. Florence2 like beats massive multimodal LLMs like GPT4o at specialized image tasks, because while these larger models are great at answering general questions in text, they’re not really designed to provide numerical outputs like bounding box coordinates. With the right training data at the instruction fine tuning stage they can certainly improve — GPT4o can be fine tuned to become good at object detection, for example — but many teams don’t have the resources to do this. Intriguingly, Gemini is in fact advertised as being capable of object detection out of the box, but Florence2 is still more versatile in terms of the range of image tasks it can accomplish.

Reading about Florence2 spawned the idea for this project. If we can connect it to a text-only LLM that’s good at reasoning (Llama 3.2 3B for example) and a multimodal LLM that’s good at answering general questions about images (such as Qwen2-VL) then we could build a system that answers complicated questions over an image. It would do so by first planning which models to call with what inputs, then running those tasks and assembling the results. The agent orchestration library LangGraph, which I explored in a recent project article here, provides a great framework for designing such a system.

Also I recently purchased a new laptop: An M3 Macbook with 24GB of RAM. Such a machine can run the smallest versions of these models with reasonable latency, making it possible for local development of an image agent. This combination of increasingly capable hardware and shrinking models (or smart ways of compressing/quantizing large models) is very impressive! But it has practical challenges: For a start when Florence2-base-ft, Llama-3.2–3B-Instruct-4bit, and Qwen2-VL-2B-Instruct-4bit are all loaded up, I barely have enough RAM for anything else. That’s fine for development, but it would be a big problem for an application that people might actually find useful. Also, as we’ll see Llama-3.2–3B-Instruct-4bit is not great at producing reliable structured outputs, which caused me to switch to GPT4-o-mini for the reasoning step during development of this project.

The image agent

But what exactly is this project? Let’s introduce it with a tour and demo of the system that we’ll build. The StateGraph (take a look at this article for an intro) looks like this, and every query consists of an image and text input.

We proceed through the stages, each of which is associated with a prompt.

• Planning. Here the goal is just to formulate a plan in text about how to best answer the questions with the tools available. The prompt contains a list of the tools and their various modes. A more complex system might use RAG at this stage to gather the list of tools most appropriate for the problem and craft a plan
• Structure the plan. The aim here is to create a list of plan components that the agent can step through. We take the plan text and force the model to generate a list that is consistently formatted according the Pydantic models here. Its useful to keep both the plan text and the structured plan for evaluation purposes.
• Execute the plan. Each element in the structured plan contains a tool name and inputs. We then proceed to call these tools in sequence and gather their results. Our agent has just two available tools: special vision (which calls Florence2) or general vision (which calls Qwen2 or GPT4o) and the routing node is used to keep track of the current plan stage
• Assess the result. Once each step of the plan has been executed, a model gets to see the inputs and outputs and make an assessment of whether or not the the question was answered. If not, we go back to the planning step and try to amend the old plan using these new insights. If yes, we proceed to the end. If the model revises the plan too many times, a timeout is triggered that allows the loop to break.

There are many possible improvements and extensions here, for example in the current implementation the agent just spits out the results of all the prior steps, but an LLM could be called here to formulate them into a nice answer if desired.

As an example just to see it in action, we give it the following image and input text, which is a fairly complex set of questions

query = """
What city is in this photo? Tell me what the tallest building is, then find all the skyscrapers and bridges
"""

After going through all the steps, this is the result we get.

[
{1: 'This photo is of Austin, Texas. The tallest building in the image is likely The Independent, also known as the "Jenga Tower."'},
{2: 
  '{"bboxes": [
    [961.9750366210938, 293.1780090332031, 1253.5550537109375, 783.6420288085938], 
    [77.67500305175781, 41.65800094604492, 307.1150207519531, 401.64599609375], 
    [335.7950134277344, 310.4700012207031, 474.4150085449219, 753.7739868164062], 
    [534.1650390625, 412.6499938964844, 684.7350463867188, 774.2100219726562], 
    [1365.885009765625, 510.114013671875, 1454.3150634765625, 774.2100219726562], 
    [1824.76513671875, 583.9979858398438, 1927.5350341796875, 758.489990234375]
  ], "labels": ["skyscraper", "skyscraper", "skyscraper", "skyscraper", "skyscraper", "skyscraper"]}'},
{3:
  '{"bboxes": [
    [493.5350341796875, 780.4979858398438, 2386.4150390625, 1035.1619873046875]
  ], "labels": ["bridge"]}'}
}
]

And we can plot these results to confirm the bounding boxes.

Impressive! One could argue about whether or not it really found all the skyscrapers here but I feel like such a system has the potential to be quite powerful and useful, especially if we were to add the ability to crop the bounding boxes, zoom in and continue the conversation.

In the following sections, let’s dive into the main steps in a bit more detail. My hope is that some of them might be informative for your projects too.

The agent state, nodes and edges

My previous article contains a more detailed discussion of agents and LangGraph, so here I’ll just touch in the agent state for this project. The AgentState is made accessible to all the nodes in the LangGraph graph, and it’s where the information relevant to a query gets stored.

Each node can be told to write to one of more variables in the state, and by default they get overwritten. This is not the behavior we want for the plan output, which is supposed to be a list of results from each step of the plan. To ensure that this list gets appended as the agent goes about its work we use the add reducer, which you can read more about here.

Each of the nodes in the graph above is a method in the class AgentNodes. They take in state, perform some action (typically calling an LLM) and output their updates to the state. As an example, here’s the node used to structure the plan, copied from the code here.

   def structure_plan_node(self, state: dict) -> dict:

        messages = state["plan"]
        response = self.llm_structure.call(messages)
        final_plan_dict = self.post_process_plan_structure(response)
        final_plan = json.dumps(final_plan_dict)

        return {
            "plan_structure": final_plan,
            "current_step": 0,
            "max_steps": len(final_plan_dict),
        }

The routing node is also important because it’s visited multiple times over the course of plan execution. In the current code it’s very simple, just updating the current step state value so that other nodes know which part of the plan structure list to look at.

   def routing_node(self, state: dict) -> dict:

        plan_stage = state.get("current_step", 0)
        return {"current_step": plan_stage + 1}

An extension here would be to add another LLM call in the routing node to check if the output of the previous step of the plan warrants any modifications to the next steps or early termination of the question has been answered.

Finally we need to add two conditional edges, which use data stored in the AgentStateto determine which node should be run next. For example, the choose_model edge looks at the name of the current step in the plan_structure object carried in AgentState and then uses a simple if stagement to return the name of corresponding node that should be called at that step.

    def choose_model(state: dict) -> str:

        current_plan = json.loads(state.get("plan_structure"))
        current_step = state.get("current_step", 1)
        max_step = state.get("max_steps", 999)

        if current_step > max_step:
            return "finalize"
        else:
            step_to_execute = current_plan[str(current_step)]["tool_name"]
            return step_to_execute

The entire agent structure looks like this.

edges: AgentEdges = AgentEdges()
nodes: AgentNodes = AgentNodes()
agent: StateGraph = StateGraph(AgentState)

## Nodes
agent.add_node("planning", nodes.plan_node)
agent.add_node("structure_plan", nodes.structure_plan_node)
agent.add_node("routing", nodes.routing_node)
agent.add_node("special_vision", nodes.call_special_vision_node)
agent.add_node("general_vision", nodes.call_general_vision_node)
agent.add_node("assessment", nodes.assessment_node)
agent.add_node("response", nodes.dump_result_node)

## Edges
agent.set_entry_point("planning")
agent.add_edge("planning", "structure_plan")
agent.add_edge("structure_plan", "routing")
agent.add_conditional_edges(
            "routing",
            edges.choose_model,
            {
                "special_vision": "special_vision",
                "general_vision": "general_vision",
                "finalize": "assessment",
            },
        )
agent.add_edge("special_vision", "routing")
agent.add_edge("general_vision", "routing")
agent.add_conditional_edges(
            "assessment",
            edges.back_to_plan,
            {
                "good_answer": "response",
                "bad_answer": "planning",
                "timeout": "response",
            },
        )
agent.add_edge("response", END)

And it can be vizualized in a notebook using the turorial here.

The orchestration model

The planning, structure and assessment nodes are ideally suited to a text-based LLM that can reason and produce structured outputs. The most straightforward option here is to go with a large, versatile model like GPT4o-mini, which has the benefit of excellent support for JSON output from a Pydantic schema.

With the help of some LangChain functionality, we can make class to call such a model.

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

class StructuredOpenAICaller:
   def __init__(
       self, api_key, system_prompt, output_model, temperature=0, max_tokens=1000
   ):
       self.temperature = temperature
       self.max_tokens = max_tokens
       self.system_prompt = system_prompt
       self.output_model = output_model
       self.llm = ChatOpenAI(
           model=self.MODEL_NAME,
           api_key=api_key,
           temperature=temperature,
           max_tokens=max_tokens,
       )
       self.chain = self._set_up_chain()


   def _set_up_chain(self):
       prompt = ChatPromptTemplate.from_messages(
           [
               ("system", self.system_prompt.system_template),
               ("human", "{query}"),
           ]
       )
       structured_llm = self.llm.with_structured_output(self.output_model)
       chain = prompt | structured_llm


       return chain
   def call(self, query):
       return self.chain.invoke({"query": query})

To set this up, we supply a system prompt and an output model (see here for some examples of these) and then we can just use the call method with an input string to get a response that conforms to the structure of the output model that we specified. With the code set up like this we’d need to make a new instance of StructuredOpenAICaller with every different system prompt and output model we used in the agent. I personally prefer this to keep track of the different models being used, but as the agent becomes more complex it could be modified with another method to directly update the system prompt and output model in the single instance of the class.

Can we do this with local models too? On Apple Silicon, we can use the MLX library and MLX community on Hugging Face to easily experiment with open source models like Llama3.2. LangChain also has support for MLX integration, so we can follow the structure of the class above to set up a local model.

from langchain_core.output_parsers import PydanticOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_community.llms.mlx_pipeline import MLXPipeline
from langchain_community.chat_models.mlx import ChatMLX

class StructuredLlamaCaller:
   MODEL_PATH = "mlx-community/Llama-3.2-3B-Instruct-4bit"

   def __init__(
       self,
       system_prompt: Any,
       output_model: Any,
       temperature: float = 0,
       max_tokens: int = 1000,
   ) -> None:


       self.system_prompt = system_prompt
       # this is the name of the Pydantic model that defines
       # the structure we want to output
       self.output_model = output_model
       self.loaded_model = MLXPipeline.from_model_id(
           self.MODEL_PATH,
           pipeline_kwargs={"max_tokens": max_tokens, "temp": temperature, "do_sample": False},
       )
       self.llm = ChatMLX(llm=self.loaded_model)
       self.temperature = temperature
       self.max_tokens = max_tokens
       self.chain = self._set_up_chain()

   def _set_up_chain(self) -> Any:
       # Set up a parser
       parser = PydanticOutputParser(pydantic_object=self.output_model)


       # Prompt
       prompt = ChatPromptTemplate.from_messages(
           [
               (
                   "system",
                   self.system_prompt.system_template,
               ),
               ("human", "{query}"),
           ]
       ).partial(format_instructions=parser.get_format_instructions())


       chain = prompt | self.llm | parser
       return chain

def call(self, query: str) -> Any:
   return self.chain.invoke({"query": query})

There are a few interesting points here. For a start, we can just download the weights and config for Llama3.2 as we would any other Hugging Face model, then under the hood they are loaded into MLX using the MLXPipeline tool from LangChain. When the models are first downloaded they are automatically placed in the Hugging Face cache. Sometimes it’s desirable to list the models and their cache locations, for example if you want to copy a model to a new environment. The util scan_cache_dir will help here and can be used to make a useful result with this function.

from huggingface_hub import scan_cache_dir

def fetch_downloaded_model_details():

    hf_cache_info = scan_cache_dir()

    repo_paths = []
    size_on_disk = []
    repo_ids = []
    
    for repo in sorted(
        hf_cache_info.repos, key=lambda repo: repo.repo_path
    ):
        repo_paths.append(str(repo.repo_path))
        size_on_disk.append(repo.size_on_disk)
        repo_ids.append(repo.repo_id)
    repos_df = pd.DataFrame({
        "local_path":repo_paths,
        "size_on_disk":size_on_disk,
        "model_name":repo_ids
    })

    repos_df.set_index("model_name",inplace=True)
    return repos_df.to_dict(orient="index")

Llama3.2 does not have a built-in support for structured output like GPT4o-mini, so we need to use the prompt to force it to generate JSON. LangChain’s PydanticOutputParser can help, although it it also possible to implement your own version of this as shown here.

In my experience, the version of Llama that I’m using here, namely Llama-3.2–3B-Instruct-4bit, is not reliable for structured output beyond the simplest schemas. It’s reasonably good at the “plan generation” stage of our agent given a prompt with a few examples, but even with the help of the instructions provided by PydanticOutputParser, it often fails to turn that plan into JSON. Larger and/or less quantized versions of Llama will likely be better, but they would run into RAM issues if run alongside the other models in our agent. Therefore going forwards in the project, the orchestration mdoel is set to be GPT4o-mini.

A model for general vision: Qwen2-VL

To be able to answer questions like “What’s going on in this image?” or “what city is this?”, we need a multimodal LLM. Arguably Florence2 in image captioning mode might be to give good responses to this type of question, but it’s not really designed for conversational output.

The field of multimodal models small enough to run on a laptop is still in its infancy (a recently compiled list can be found here), but the Qwen2-VL series from Alibaba is a promising development. Furthermore, we can make use of MLX-VLM, an extension of MLX specifically designed for tuning and inference of vision models, to set up one of these models within our agent framework.

from mlx_vlm import load, apply_chat_template, generate

class QwenCaller:
   MODEL_PATH = "mlx-community/Qwen2-VL-2B-Instruct-4bit"

   def __init__(self, max_tokens=1000, temperature=0):
       self.model, self.processor = load(self.MODEL_PATH)
       self.config = self.model.config
       self.max_tokens = max_tokens
       self.temperature = temperature

   def call(self, query, image):
       messages = [
           {
               "role": "system",
               "content": ImageInterpretationPrompt.system_template,
           },
           {"role": "user", "content": query},
       ]
       prompt = apply_chat_template(self.processor, self.config, messages)
       output = generate(
           self.model,
           self.processor,
           image,
           prompt,
           max_tokens=self.max_tokens,
           temperature=self.temperature,
       )
       return output

This class will load the smallest version of Qwen2-VL and then call it with an input image and prompt to get a textual response. For more detail about the functionality of this model and others that could be used in the same way, check out this list of examples on the MLX-VLM github page. Qwen2-VL is also apparently capable of generating bounding boxes and object pointing coordinates, so this capability could also be explored and compared with Florence2.

Of course GPT-4o-mini also has vision capabilities and is likely more reliable than smaller local models. Therefore when building these sorts of applications it’s useful to add the ability to call a cloud based alternative, if anything just as a backup in case one of the local models fails. Note that input images must be converted to base64 before they can be sent to the model, but once that’s done we can also use the LangChain framework as shown below.

import base64
from io import BytesIO
from PIL import Image
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser


def convert_PIL_to_base64(image: Image, format="jpeg"):
   buffer = BytesIO()
   # Save the image to this buffer in the specified format
   image.save(buffer, format=format)
   # Get binary data from the buffer
   image_bytes = buffer.getvalue()
   # Encode binary data to Base64
   base64_encoded = base64.b64encode(image_bytes)
   # Convert Base64 bytes to string (optional)
   return base64_encoded.decode("utf-8")


class OpenAIVisionCaller:
   MODEL_NAME = "gpt-4o-mini"

   def __init__(self, api_key, system_prompt, temperature=0, max_tokens=1000):
       self.temperature = temperature
       self.max_tokens = max_tokens
       self.system_prompt = system_prompt
       self.llm = ChatOpenAI(
           model=self.MODEL_NAME,
           api_key=api_key,
           temperature=temperature,
           max_tokens=max_tokens,
       )
       self.chain = self._set_up_chain()

   def _set_up_chain(self):
       prompt = ChatPromptTemplate.from_messages(
           [
               ("system", self.system_prompt.system_template),
               (
                   "user",
                   [
                       {"type": "text", "text": "{query}"},
                       {
                           "type": "image_url",
                           "image_url": {"url": "data:image/jpeg;base64,{image_data}"},
                       },
                   ],
               ),
           ]
       )

       chain = prompt | self.llm | StrOutputParser()
       return chain


   def call(self, query, image):
       base64image = convert_PIL_to_base64(image)
       return self.chain.invoke({"query": query, "image_data": base64image})

A specialized model for Vision: Florence2

Florence2 is seen as a specialist model in the context of our agent because while it has many capabilities its inputs must be selected from a list of predefined task prompts. Of course the model could be fine tuned to accept new prompts, but for our purposes the version downloaded directly from Hugging Face works well. The beauty of this model is that it uses a single training process and set of weights, but yet achieves high performance in multiple image tasks that previously would have demanded their own models. The key to this success lies in its large and carefully curated training dataset, FLD-5B. To learn more about the dataset, model and training I recommend this excellent article.

In our context, we use the orchestration model to turn the query into a series of Florence task prompts, which we then call in a sequence. The options available to us include captioning, object detection, phrase grounding, OCR and segmentation. For some of these options (i.e. phrase grounding and region to segmentation) an input phrase is needed, so the orchestration model generates that too. In contrast, tasks like captioning need only the image. There are many use cases for Florence2, which are explored in code here. We restrict ourselves to object detection, phrase grounding, captioning and OCR, though it would be straightforward to add more by updating the prompts associated with plan generation and structuring.

Florence2 appears to be supported by the MLX-VLM package, but at the time of writing I couldn’t find any examples of its use and so opted for an approach that uses Hugging Face transformers as shown below.

from transformers import AutoModelForCausalLM, AutoProcessor
import torch
 
def get_device_type():


   if torch.cuda.is_available():
       return "cuda"
   else:
       if torch.backends.mps.is_available() and torch.backends.mps.is_built():
           return "mps"
       else:
           return "cpu"


class FlorenceCaller:

   MODEL_PATH: str = "microsoft/Florence-2-base-ft" 
   # See https://huggingface.co/microsoft/Florence-2-base-ft for other modes 
   # for Florence2 
   TASK_DICT: dict[str, str] = {
       "general object detection": "<OD>",
       "specific object detection": "<CAPTION_TO_PHRASE_GROUNDING>",
       "image captioning": "<MORE_DETAILED_CAPTION>",
       "OCR": "<OCR_WITH_REGION>",
   }


   def __init__(self) -> None:
       self.device: str = (
           get_device_type()
       )  # Function to determine the device type (e.g., 'cpu' or 'cuda').


       with patch("transformers.dynamic_module_utils.get_imports", fixed_get_imports):
           self.model: AutoModelForCausalLM = AutoModelForCausalLM.from_pretrained(
               self.MODEL_PATH, trust_remote_code=True
           )
           self.processor: AutoProcessor = AutoProcessor.from_pretrained(
               self.MODEL_PATH, trust_remote_code=True
           )
           self.model.to(self.device)


   def translate_task(self, task_name: str) -> str:
       return self.TASK_DICT.get(task_name, "<DETAILED_CAPTION>")


   def call(
       self, task_prompt: str, image: Any, text_input: Optional[str] = None
   ) -> Any:


       # Get the corresponding task code for the given prompt
       task_code: str = self.translate_task(task_prompt)

       # Prevent text_input for tasks that do not require it
       if task_code in [
           "<OD>",
           "<MORE_DETAILED_CAPTION>",
           "<OCR_WITH_REGION>",
           "<DETAILED_CAPTION>",
       ]:
           text_input = None

       # Construct the prompt based on whether text_input is provided
       prompt: str = task_code if text_input is None else task_code + text_input

       # Preprocess inputs for the model
       inputs = self.processor(text=prompt, images=image, return_tensors="pt").to(
           self.device
       )

       # Generate predictions using the model
       generated_ids = self.model.generate(
           input_ids=inputs["input_ids"],
           pixel_values=inputs["pixel_values"],
           max_new_tokens=1024,
           early_stopping=False,
           do_sample=False,
           num_beams=3,
       )

       # Decode and process generated output
       generated_text: str = self.processor.batch_decode(
           generated_ids, skip_special_tokens=False
       )[0]

       parsed_answer: dict[str, Any] = self.processor.post_process_generation(
           generated_text, task=task_code, image_size=(image.width, image.height)
       )

       return parsed_answer[task_code]

On Apple Silicon, the device becomes mps and the latency of these model calls is tolerable. This code should also work on GPU and CPU, though this has not been tested.

Another example and some limitations

Let’s run through another example to see the agent outputs from each step. To call the agent on an input query and image we can use the Agent.invoke method, which follows the same process as described in my previous article to add each node output to a list of results in addition to saving outputs in a LangGraph InMemoryStore object.

We’ll be using the following image, which presents an interesting challenge if we ask a tricky question like “Are there trees in this image? If so, find them and describe what they are doing”

from image_agent.agent.Agent import Agent
from image_agent.utils import load_secrets

secrets = load_secrets()

# use GPT4 for general vision mode
full_agent_gpt_vision = Agent(
openai_api_key=secrets["OPENAI_API_KEY"],vision_mode="gpt"
)

# use local model for general vision 
full_agent_qwen_vision = Agent(
openai_api_key=secrets["OPENAI_API_KEY"],vision_mode="local"
)

In an ideal world the answer is straightforward: There are no trees.

However this turns out to be a difficult question for the agent and it’s interesting to compare the responses when it using GPT-4o-mini vs. Qwen2 as the general vision model.

When we call full_agent_qwen_vision with this query, we get a bad result: Both Qwen2 and Florence2 fall for the trick and report that trees are present (interestingly, if we change “trees” to “dogs”, we get the right answer)

Plan: 
Call generalist vision with the question 'Are there trees in this image? If so, what are they doing?'. Then call specialist vision in object specific mode with the phrase 'cat'.

Plan_structure: 
{
  "1": {"tool_name": "general_vision", "tool_mode": "conversation", "tool_input": "Are there trees in this image? If so, what are they doing?"}, 
  "2": {"tool_name": "special_vision", "tool_mode": "specific object detection", "tool_input": "tree"}
}

Plan output:
[
  {1: 'Yes, there are trees in the image. They appear to be part of a tree line against a pink background.'}
[
  {2: '{"bboxes": [[235.77601623535156, 427.864501953125, 321.7920227050781, 617.2275390625]], "labels": ["tree"]}'}
]

Assessment: 
The result adequately answers the user's question by confirming the presence of trees in the image and providing a description of their appearance and context. The output from both the generalist and specialist vision tools is consistent and informative.

Qwen2 seems subject to blindly following the prompts hint that here might be trees present. Florence2 also fails here, reporting a bounding box when it should not

If we call full_agent_gpt_visionwith the same query, GPT4o-mini doesn’t fall for the trick, but the call to Florence2 hasn’t changed so it still fails. We then see the query assessment step in action because the generalist and specialist models have produced conflicting results.

Node : general_vision
Task : plan_output
[
  {1: 'There are no trees in this image. It features a group of dogs sitting in front of a pink wall.'}
]

Node : special_vision
Task : plan_output
[
  {2: '{"bboxes": [[235.77601623535156, 427.864501953125, 321.7920227050781, 617.2275390625]], "labels": ["tree"]}'}
]

Node : assessment
Task : answer_assessment
The result contains conflicting information. 
The first part states that there are no trees in the image, while the second part provides a bounding box and label indicating that a tree is present. 
This inconsistency means the user's question is not adequately answered.

The agent then tries several times to restructure the plan, but Florence2 insists on producing a bounding box for “tree”, which the answer assessment nodes always catches as inconsistent. This is a better result than the Qwen2 agent, but points to a broader issue of false positives with Florence2. This could be addressed by having the routing node evaluate the plan after every step and then only call Florence2 if absolutely necessary.

With the basic building blocks in place, this system is ripe for experimentation, iteration and improvement and I may continue to add to the repo over the coming weeks. For now though, this article is long enough!

Thanks for making it to the end and I hope the project here prompts some inspiration for your own projects! The orchestration of multiple specialist models within agent frameworks is a powerful and increasingly accessible approach to putting LLMs to work on complex tasks. Clearly there is still a lot of room for improvI for one look forward to seeing how ideas in this field develop over the coming year.

---

A Multimodal AI Assistant: Combining Local and Cloud Models was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Understanding the Evolution of ChatGPT: Part 2 — GPT-2 and GPT-3

Scaling from 117M to 175B: Insights into GPT-2 and GPT-3.

This is the second article of our GPT series, where we will dive into the development of GPT-2 and GPT-3, with model size increased from 117M to a staggering 175B.

In case you are interested in the first article which covers GPT-1 as well as the techniques inspired it, check this link: Understanding the Evolution of ChatGPT: Part 1 — An In-Depth Look at GPT-1 and What Inspired It.

We choose to cover GPT-2 and GPT-3 together not just because they share similar architectures, but also they were developed with a common philosophy aimed at bypassing the finetuning stage in order to make LLMs truly intelligent. Moreover, to achieve that goal, they both explored several key technical elements such as task-agnostic learning, scale hypothesis and in-context learning, etc. Together they demonstrated the power of training large models on large datasets, inspired further research into emergent capabilities, established new evaluation protocols, and sparked discussions on enhancing the safety and ethical aspects of LLMs.

Below are the contents we will cover in this article:

• Overview: The paradigm shift towards bypassing finetuning, and the three key elements made this possible: task-agnostic learning, the scaling hypothesis, and in-context learning.
• GPT-2: Model architecture, training data, evaluation results, etc.
• GPT-3: Core concepts and new findings.
• Conclusions.

Overview

The Paradigm Shift Towards Bypassing Finetuning

In our previous article, we revisited the core concepts in GPT-1 as well as what had inspired it. By combining auto-regressive language modeling pre-training with the decoder-only Transformer, GPT-1 had revolutionized the field of NLP and made pre-training plus finetuning a standard paradigm.

But OpenAI didn’t stop there.

Rather, while they tried to understand why language model pre-training of Transformers is effective, they began to notice the zero-shot behaviors of GPT-1, where as pre-training proceeded, the model was able to steadily improve its performance on tasks that it hadn’t been finetuned on, showing that pre-training could indeed improve its zero-shot capability, as shown in the figure below:

This motivated the paradigm shift from “pre-training plus finetuning” to “pre-training only”, or in other words, a task-agnostic pre-trained model that can handle different tasks without finetuning.

Both GPT-2 and GPT-3 are designed following this philosophy.

But why, you might ask, isn’t the pre-training plus finetuning magic working just fine? What are the additional benefits of bypassing the finetuning stage?

Limitations of Finetuning

Finetuning is working fine for some well-defined tasks, but not for all of them, and the problem is that there are numerous tasks in the NLP domain that we have never got a chance to experiment on yet.

For those tasks, the requirement of a finetuning stage means we will need to collect a finetuning dataset of meaningful size for each individual new task, which is clearly not ideal if we want our models to be truly intelligent someday.

Meanwhile, in some works, researchers have observed that there is an increasing risk of exploiting spurious correlations in the finetuning data as the models we are using become larger and larger. This creates a paradox: the model needs to be large enough so that it can absorb as much information as possible during training, but finetuning such a large model on a small, narrowly distributed dataset will make it struggle when generalize to out-of-distribution samples.

Another reason is that, as humans we do not require large supervised datasets to learn most language tasks, and if we want our models to be useful someday, we would like them to have such fluidity and generality as well.

Now perhaps the real question is that, what can we do to achieve that goal and bypass finetuning?

Before diving into the details of GPT-2 and GPT-3, let’s first take a look at the three key elements that have influenced their model design: task-agnostic learning, the scale hypothesis, and in-context learning.

Task-agnostic Learning

Task-agnostic learning, also known as Meta-Learning or Learning to Learn, refers to a new paradigm in machine learning where the model develops a broad set of skills at training time, and then uses these skills at inference time to rapidly adapt to a new task.

For example, in MAML (Model-Agnostic Meta-Learning), the authors showed that the models could adapt to new tasks with very few examples. More specifically, during each inner loop (highlighted in blue), the model firstly samples a task from a bunch of tasks and performs a few gradient descent steps, resulting in an adapted model. This adapted model will be evaluated on the same task in the outer loop (highlighted in orange), and then the loss will be used to update the model parameters.

MAML shows that learning could be more general and more flexible, which aligns with the direction of bypassing finetuning on each individual task. In the follow figure the authors of GPT-3 explained how this idea can be extended into learning language models when combined with in-context learning, with the outer loop iterates through different tasks, while the inner loop is described using in-context learning, which will be explained in more detail in later sections.

The Scale Hypothesis

As perhaps the most influential idea behind the development of GPT-2 and GPT-3, the scale hypothesis refers to the observations that when training with larger data, large models could somehow develop new capabilities automatically without explicit supervision, or in other words, emergent abilities could occur when scaling up, just as what we saw in the zero-shot abilities of the pre-trained GPT-1.

Both GPT-2 and GPT-3 can be considered as experiments to test this hypothesis, with GPT-2 set to test whether a larger model pre-trained on a larger dataset could be directly used to solve down-stream tasks, and GPT-3 set to test whether in-context learning could bring improvements over GPT-2 when further scaled up.

We will discuss more details on how they implemented this idea in later sections.

In-Context Learning

As we show in Figure 3, under the context of language models, in-context learning refers to the inner loop of the meta-learning process, where the model is given a natural language instruction and a few demonstrations of the task at inference time, and is then expected to complete that task by automatically discovering the patterns in the given demonstrations.

Note that in-context learning happens in the testing phase with no gradient updates performed, which is completely different from traditional finetuning and is more similar to how humans perform new tasks.

In case you are not familiar with the terminology, demonstrations usually means exemplary input-output pairs associated with a particular task, as we show in the “examples” part in the figure below:

The idea of in-context learning was explored implicitly in GPT-2 and then more formally in GPT-3, where the authors defined three different settings: zero-shot, one-shot, and few-shot, depending on how many demonstrations are given to the model.

In short, task-agnostic learning highlights the potential of bypassing finetuning, while the scale hypothesis and in-context learning suggest a practical path to achieve that.

In the following sections, we will walk through more details for GPT-2 and GPT-3, respectively.

GPT-2

Model Architecture

The GPT-2 model architecture is largely designed following GPT-1, with a few modifications:

• Moving LayerNorm to the input of each sub-block and adding an additional LayerNorm after the final self-attention block to make the training more stable.
• Scaling the weights of the residual layers by a factor of 1/sqrt(N), where N is the number of residual layers.
• Expanding the vocabulary to 50257, and also using a modified BPE vocabulary.
• Increasing context size from 512 to 1024 tokens and using a larger batch size of 512.

In the GPT-2 paper, the authors trained four models with approximately log-uniformly spaced sizes, with number of parameter ranging from 117M to 1.5B:

Training Data

As we scale up the model we also need to use a larger dataset for training, and that is why in GPT-2 the authors created a new dataset called WebText, which contains about 45M links and is much larger than that used in pre-training GPT-1. They also mentioned lots of techniques to cleanup the data to improve its quality.

Evaluation Results

Overall, GPT-2 achieved good results on many tasks, especially for language modeling related ones. However, for tasks like reading comprehension, translation and QA, it still performed worse than the respective SOTA models, which partly motivates the development of GPT-3.

GPT-3

Model Architecture

GPT-3 adopted a very similar model architecture to that of GPT-2, and the only difference is that GPT-3 used an alternating dense and locally banded sparse attention patterns in Transformer.

GPT-3 trained 8 models with different sizes, with number of parameters ranging from 125M to 175B:

Training Data

GPT-3 model was trained on even larger datasets, as listed in the table below, and again the authors did some cleanup work to improve data quality. Meanwhile, training datasets were not sampled in proportion to their size, but rather according to their quality, with high-quality dataset sampled more frequently during training.

Evaluation Results

By combining larger model with in-context learning, GPT-3 achieved strong performance on many NLP datasets including translation, question-answering, cloze tasks, as well as tasks require on-the-fly reasoning or domain adaptation. The authors presented very detailed evaluation results in the original paper.

A few findings that we want to highlight in this article:

Firstly, during training of GPT-3 they observed a smooth scaling trend of performance with compute, as shown in the figure below, where the validation loss decreases linearly as compute increasing exponentially.

Secondly, when comparing the three in-context learning settings (zero-shot, one-shot and few-shot), they observed that larger models appeared more efficient in all the three settings:

Following that, they plotted the aggregate performance for all the three settings, which further demonstrated that larger models are more effective, and few-shot performance increased more rapidly than the other two settings.

Conclusions

The development of GPT-2 and GPT-3 bridges the gap between the original GPT-1 with more advanced versions like InstructGPT, reflecting the ongoing refinement of OpenAI’s methodology in training useful LLMs.

Their success also paves the way for new research directions in both NLP and the broader ML community, with many subsequent works focusing on understanding emergent capabilities, developing new training paradigms, exploring more effective data cleaning strategies, and proposing effective evaluation protocols for aspects like safety, fairness, and ethical considerations, etc.

In the next article, we will continue our exploration and walk you through the key elements of GPT-3.5 and InstructGPT.

Thanks for reading!

---

Understanding the Evolution of ChatGPT: Part 2 — GPT-2 and GPT-3 was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Education and Training Quality Authority (BQA) plays a critical role in improving the quality of education and training services in the Kingdom Bahrain. BQA reviews the performance of all education and training institutions, including schools, universities, and vocational institutes, thereby promoting the professional advancement of the nation’s human capital. In this post, we explore how BQA used the power of Amazon Bedrock, Amazon SageMaker JumpStart, and other AWS services to streamline the overall reporting workflow.

This post shows how MuleSoft introduced a generative AI-powered assistant using Amazon Q Business to enhance their internal Cloud Central dashboard. This individualized portal shows assets owned, costs and usage, and well-architected recommendations to over 100 engineers.