Tag: AI

Estimate the Unobserved: Moving-Average Model Estimation with Maximum Likelihood in Python

How unobserved covariates’ coefficients can be estimated with MLE

For those experienced with time series data and forecasting, terms like regressions, AR, MA, and ARMA should be familiar. Linear Regression is a straightforward model with a closed-form parametric solution obtained through OLS. AR models can also be estimated using OLS. However, things become more complex with MA models, which form the second component of the more advanced ARMA and ARIMA models.

The plan for this story:

1. Introducing Moving Average models
2. Discussing why there’s no closed solution for MA model
3. Introducing Maximum Likelihood Estimation method
4. MA(1) ML estimation — Theory and Python code

Moving Average Model

MA models can be described by the following formula:

Here, the thetas represent the model parameters, while the epsilons are the error terms, assumed to be mutually independent and normally distributed with constant variance. The intuition behind this formula is that our time series, X, can always be described by the last q shocks that occurred in the series. It is evident from the formula that each shock impacts only the subsequent q values of X, in contrast to AR models where the effect of a shock persists indefinitely, although gradually diminishing over time.

Closed estimation of a simple model — Linear Regression

As a reminder, the general form of a linear regression equation looks like this:

For forecasting tasks, we typically aim to estimate all the model’s parameters (the betas) using a sample set of x’s and y’s. Given a few assumptions about our model, the Gauss–Markov theorem states that the ordinary least squares (OLS) estimators for the betas have the lowest sampling variance among the class of linear unbiased estimators. In simpler terms, OLS provides us with the best possible estimation of the betas.

So, what is OLS? It is a closed-form solution to a loss function minimization problem:

where the loss function, S, is defined as follows –

In this context, y and X are our sample data and are observable vectors of numbers (as in time series). Therefore, it is straightforward to calculate the function S, determine its derivative, and find the beta that solves the minimization problem.

Closed estimation of MA(q)

It is should be clear why applying a method like OLS to estimate MA(q) models is problematic — the dependent variable, the time series values, are described by unobservable variables, the epsilons. This raises the question: how can these models be estimated at all?

Maximum Likelihood (MLE)

Likelihood Function

Statistical distributions typically depend on one or more parameters. For instance, the normal distribution is characterized by its mean and variance, which define its “height” and “center of mass” —

Suppose we have a dataset, X={x_1,…x_n}, comprising samples drawn from an unknown normal distribution, with its parameters unknown. Our objective is to determine the mean and variance values that would characterize the specific normal distribution from which our dataset X is most likely to have been sampled.

MLE provides a framework that precisely tackles this question. It introduces a likelihood function, which is a function that yields another function. This likelihood function takes a vector of parameters, often denoted as theta, and produces a probability density function (PDF) that depends on theta.

The probability density function (PDF) of a distribution is a function that takes a value, x, and returns its probability within the distribution. Therefore, likelihood functions are typically expressed as follows:

The value of this function indicates the likelihood of observing x from the distribution defined by the PDF with theta as its parameters.

The goal

When constructing a forecast model, we have data samples and a parameterized model, and our goal is to estimate the model’s parameters. In our examples, such as Regression and MA models, these parameters are the coefficients in the respective model formulas.

The equivalent in MLE is that we have observations and a PDF for a distribution defined over a set of parameters, theta, which are unknown and not directly observable. Our goal is to estimate theta.

The MLE approach involves finding the set of parameters, theta, that maximizes the likelihood function given the observable data, x.

We assume our samples, x, are drawn from a distribution with a known PDF that depends on a set of parameters, theta. This implies that the likelihood (probability) of observing x under this PDF is essentially 1. Therefore, identifying the theta values that make our likelihood function value close to 1 on our samples, should reveal the true parameter values.

Conditional likelihood

Notice that we haven’t made any assumptions about the distribution (PDF) on which the likelihood function is based. Now, let’s assume our observation X is a vector (x_1, x_2, …, x_n). We’ll consider a probability function that represents the probability of observing x_n conditional on that we have already observed (x_1, x_2, …, x_{n-1}) —

This represents the likelihood of observing just x_n given the previous values (and theta, the set of parameters). Now, we define the conditional likelihood function as follows:

Later, we will see why it is useful to employ the conditional likelihood function rather than the exact likelihood function.

Log-Likelihood

In practice, it is often convenient to use the natural logarithm of the likelihood function, referred to as the log-likelihood function:

This is more convenient because we often work with a likelihood function that is a joint probability function of independent variables, which translates to the product of each variable’s probability. Taking the logarithm converts this product into a sum.

Estimating MA(1) with MLE

For simplicity, I’ll demonstrate how to estimate the most basic moving average model — MA(1):

Here, x_t represents the time-series observations, alpha and beta are the model parameters to be estimated, and the epsilons are random noise drawn from a normal distribution with zero mean and some variance — sigma, which will also be estimated. Therefore, our “theta” is (alpha, beta, sigma), which we aim to estimate.

Let’s define our parameters and generate some synthetic data using Python:

import pandas as pd
import numpy as np

STD = 3.3
MEAN = 0
ALPHA = 18
BETA = 0.7
N = 1000

df = pd.DataFrame({"et": np.random.normal(loc=MEAN, scale=STD, size=N)})
df["et-1"] = df["et"].shift(1, fill_value=0)
df["xt"] = ALPHA + (BETA*df["et-1"]) + df["et"]

Note that we have set the standard deviation of the error distribution to 3.3, with alpha at 18 and beta at 0.7. The data looks like this —

Likelihood function for MA(1)

Our objective is to construct a likelihood function that addresses the question: how likely is it to observe our time series X=(x_1, …, x_n) assuming they are generated by the MA(1) process described earlier?

The challenge in computing this probability lies in the mutual dependence among our samples — as evident from the fact that both x_t and x_{t-1} depend on e_{t-1) — making it non-trivial to determine the joint probability of observing all samples (referred to as the exact likelihood).

Therefore, as discussed previously, instead of computing the exact likelihood, we’ll work with a conditional likelihood. Let’s begin with the likelihood of observing a single sample given all previous samples:

This is much simpler to calculate because —

All that remains is to calculate the conditional likelihood of observing all samples:

applying a natural logarithm gives:

which is the function we should maximize.

Code

We’ll utilize the GenericLikelihoodModel class from statsmodels for our MLE estimation implementation. As outlined in the tutorial on statsmodels’ website, we simply need to subclass this class and include our likelihood function calculation:

from scipy import stats
from statsmodels.base.model import GenericLikelihoodModel
import statsmodels.api as sm

class MovingAverageMLE(GenericLikelihoodModel):
    def initialize(self):
        super().initialize()
        extra_params_names = ['beta', 'std']
        self._set_extra_params_names(extra_params_names)

        self.start_params = np.array([0.1, 0.1, 0.1])

    def calc_conditional_et(self, intercept, beta):
        df = pd.DataFrame({"xt": self.endog})
        ets = [0.0]
        for i in range(1, len(df)):
            ets.append(df.iloc[i]["xt"] - intercept - (beta*ets[i-1]))

        return ets

    def loglike(self, params):
        ets = self.calc_conditional_et(params[0], params[1])
        return stats.norm.logpdf(
            ets,
            scale=params[2],
        ).sum()

The function loglike is essential to implement. Given the iterated parameter values paramsand the dependent variables (in this case, the time series samples), which are stored as class members self.endog, it calculates the conditional log-likelihood value, as we discussed earlier.

Now let’s create the model and fit on our simulated data:

df = sm.add_constant(df) # add intercept for estimation (alpha)
model = MovingAverageMLE(df["xt"], df["const"])
r = model.fit()
r.summary()

and the output is:

And that’s it! As demonstrated, MLE successfully estimated the parameters we selected for simulation.

Wrapping Up

Estimating even a simple MA(1) model with maximum likelihood demonstrates the power of this method, which not only allows us to make efficient use of our data but also provides a solid statistical foundation for understanding and interpreting the dynamics of time series data.

Hope you liked it !

References

[1] Andrew Lesniewski, Time Series Analysis, 2019, Baruch College, New York

[2] Eric Zivot, Estimation of ARMA Models, 2005

Unless otherwise noted, all images are by the author

---

Estimate the unobserved — Moving-Average Model Estimation with Maximum Likelihood in Python was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Math Behind Risk — Part 1

Does the attack really have an advantage in the game of world conquest?

A good friend of mine (Hey, Aron) recently asked me what the probabilities are for the attack and defense in the board game Risk. I know that the conquests of Alexander the Great and Genghis Khan cannot be fully explained by the mechanics of Risk, but it still seemed like an intriguing question, and it provides an excellent illustration of the capacity of probability and data visualization to inform and power intelligent decision making.

For those who don’t know, Risk is a board game of world conquest where the attacker rolls (up to) three dice and the defender rolls (up to) two dice. The player whose highest roll is lower loses a soldier, and a tie goes to the defender. We will refer to this as the first battle. If both players are rolling at least two dice, then the player whose second highest roll is lower will also lose a soldier. Once again, a tie goes to the defender. We will call this the second battle.

Of course, it’s clearly an advantage to throw 3 dice instead of 2, and the favorability of winning in the case of a tie is equally obvious. What’s less obvious is how these advantages stack up against each other.

(Here you can find code in which I confirm the below probabilities.)

Before analyzing the relative probabilities of defense and attack, let’s first analyze each of them in isolation.

The defender’s turn is simpler as he rolls only two dice, so let’s start there. There are 11 permutations yielding a highest roll of 6. This can be calculated by considering all the possibilities: {(1,6), (2,6), (3,6), (4,6), (5,6), (6,1), (6,2), (6,3), (6,4), (6,5), (6,6)}. By similar logic, the number of permutations giving highest rolls of 5, 4, 3, 2 or 1 can be calculated as 9, 7, 5, 3 and 1 respectively. Since there are a total of 6 x 6 = 36 total permutations for 2 dice, we need only divide each permutation count by 36 to obtain probabilities. The below graphics should be helpful. Please note that in all the graphics, red denotes attack and blue denotes defense, in keeping with the color scheme of Risk itself.

The highest roll of the attacker is slightly more complex, as he rolls 3 dice. To calculate how many permutations yield a highest roll of 6, let’s start by fixing the highest roll at 6 and let’s further stipulate that it occurs at die 1. It’s then clear that dice two and three can be anything up to 6, for a total of 6*6 = 36 outcomes. If we next fix the highest outcome of 6 at die 2, we must limit the first die to not being six, since we already considered that possibility. The third die can be anything, for a total of 6×5=30 additional outcomes. Finally, we can fix the highest outcome of 6 at the third die, and we know that we must limit the first two dice to not being six, since those possibilities have already been counted. This yields a further 25 outcomes, for a total of 36+30+25 = 91 of possibilities.

We can generalize this to calculate the number of outcomes yielding a highest roll of x. If the highest roll occurs at die 1, the second and third die can take any outcome up to and including x, for a total of x² outcomes. If, alternatively, the highest roll occurs at die 2, die 1 can take any value up to x-1, (since we have already considered the case where the first die takes a value of x) and the third die can take any value up to x, for a total of x(x-1)=x²-x additional outcomes. Finally, we consider the option of die three taking the value x. Then the only options not yet counted are for the first and second dice to each take a value up to x-1, adding (x-1)*(x-1) = x²–2x+1 outcomes. Adding everything up, we obtain a total of 3x²–3x+1 ways¹ to obtain a highest roll of x.

Note that the total number of permutations is 216, as expected, since 6³ = 216, which gives confidence that these calculations are correct.

Next, let’s directly compare the chances for attack and defense.

We can see that the defense has a higher chance of getting 1, 2, 3 and 4 as a highest roll than attack does, while attack has a higher chance of getting 5 or 6 as the highest roll than his opponent. This is the 3rd die working in the attack’s favor.

Enough stalling, let’s get to the battle.

We must consider the two battles separately. In Part 1, we will analyze the first battle, of who will win the highest roll, and we will leave the analysis of the second battle, for the 2nd highest roll, for Part 2.

To calculate the probability of the attacker winning the highest roll, we will first count the permutations in which the defender achieves a highest roll of x and calculate how many of those permutations would result in an outright win for the defense, a tie, which goes to the defense, or a win for the attack.

For example, since, as calculated above, there is a 9/36 chance of the defense’s highest roll being 5, and there are a total of 6⁵ = 7776 permutations, clearly (9/36) * 7776 = 1944 of those permutations will yield a highest defense roll of 5. To win, the attack then needs to get a highest roll of 6, the probability of which is 91/216, as calculated above, so (91/216) * 1944 = 819 of the 1944 permutations which yielded a highest defense roll of 6 will result in a victory for attack. To achieve a tie, attack must roll a highest roll of 5, the probability of which is 61/216, so (61/216) * 1944 = 549 of those permutations will result in ties, and the remainder (1944–819–549 = 576) will result in outright defense wins.

We can make similar calculations for all possible outcomes for defense. See the below table.

We can then calculate the conditional² probability of a defense victory, of a tie, and of an attack victory, by dividing the number of permutations yielding the selected outcome (e.g. attack wins) by the count of the broader group of outcomes (e.g. defense rolls a highest roll of 5).

We can also visualize the conditional probabilities.

Chart 4 gives the same false impression that most people have initially, namely that attack has a big advantage overall. But this is because it ignores the probabilities of the highest defense rolls themselves. In fact, higher rolls are significantly more likely than lower rolls, as can be seen in Table 3.

For this reason, total probabilities are a more effective measure. It is even simpler to calculate the total probability of each outcome. We simply divide each permutation count by the total number of possible permutations, which is 7776.

We can then sum the total permutations which result in a victory for the attack (3667) and divide by the total possible permutations (7776) to obtain a win probability of 47.15% for the attack.

Below is a chart of total win probabilities by highest defensive roll. We include a tie as part of a defense victory for simplicity.

Finally, we will calculate the joint³ probabilities of each possible highest roll for both defense and attack. Since the highest roll for attack and defense are independent, we can simply multiply the probabilities together to obtain the joint probability. Note that in the below two graphics, red indicate victory for the attack, while blue denotes a defensive victory and pale blue indicates a tie, which goes to the defense.

We can also graph this data visually. Please note the configurations of the axes in the below chart, which have been configured to allow for maximal visibility.

Conclusions

1. The probability of a victory for the attack is 47.15%. This can be seen clearly in Table 5. We go through the formal math in the footnotes⁴.
2. The probability of an outright victory for the defense is 28.07%. This can be seen clearly in Table 5.
3. The probability of a tie, which goes to the defense of course, is 24.77%. This can be seen clearly in Table 5.
4. If the defense’s highest roll is four, he/she has only a 29.63% chance of winning. This can be calculated by summing the defense win probabilities on row 4 of Table 6 and dividing by the row total.
5. In order for the defense to have a greater chance of winning outright than attack, he needs to get a highest roll of 6. This can be seen on rows 5 and 6 of Table 3 and Table 4.
6. The most likely overall outcome, at 12.87% is for both attack and defense to roll highest rolls of 6, giving defense the victory. This can be seen clearly in Table 6.
7. Of the cases in which the attack wins, the most commonly occurring way for this to happen is for the defense to roll a highest roll of 4. This case constitutes 29.02% of attack’s victories. This can be calculated by observing the attack percentage for a defense roll of 4 in Table 5 and dividing by the column total.
8. Only 5.86% of attack’s victories come in cases where the defense’s highest roll is 1. The subtle, almost paradoxical point here is that if defense’s highest roll was 1, you can be almost sure (99.54%) that attack wins, but if attack wins, you can still be confident (94.14%) that defense did not roll a highest roll of 1, although admittedly, not as confident as you were (97.2%) before you found out that attack won. (Probability is awesome.) The first number can be calculated from Table 5. The second number is taken directly from Table 4. The third is the complement of the first, and the fourth can be calculated from Table 1.

So the attack is actually at a disadvantage for the first battle. So how was Genghis Khan able to rout his enemies so effectively in war? Perhaps this points to greater subtleties awaiting us in the matter of the second battle. Or maybe this demonstrates a breakdown in the ability of Risk to explain history’s greatest conquests. Or perhaps, even more tantalizingly, both?

See you in Part 2.

1. For the mathematically inclined, this seems to indicate that the sum of 3x²-3x+1 for all x up to and including k is equal to k³, to satisfy the obvious requirement that summing the permutations yielding a highest value of x, for all x up to k, must equal the total number of permutations available with 3 dice yielding k alternatives. And indeed, a simple proof by induction yields this result. Clearly this holds for the base case of 1, since 3(1)²-3(1)+1 = ¹³. Then we must prove that the sum of 3x²-3x+1 for all x up to and including k is equal to k³, using the assumption that the sum of 3x²-3x+1 for all x up to and including k-1 is equal to (k-1)³. Since (k-1)³ = k³-3k²+3k-1, clearly adding 3k²-3k+1 will equal k³. QED.
2. Conditional probability means the probability given some condition. For example, if the defense rolls a highest roll of 6, then the attack has a conditional probability of victory of 0%, since he cannot win in that case. But that’s only if the defense rolls a highest roll of 6.
3. Joint probability simply means the probability of two events co-occurring.
4. To formalize the math, we need some notation. Let A₁ and D₁ refer to the highest dice rolls of the attack and defense, respectively. Furthermore, let d indicate possible highest rolls of defense, which obviously can take any value between one and six. Then the probability of a win for attack can be calculated by calculating the below summation:

---

The Math Behind Risk — Part 1 was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

NinjaTech AI’s mission is to make everyone more productive by taking care of time-consuming complex tasks with fast and affordable artificial intelligence (AI) agents. We recently launched MyNinja.ai, one of the world’s first multi-agent personal AI assistants, to drive towards our mission. MyNinja.ai is built from the ground up using specialized agents that are capable of completing tasks on your behalf, including scheduling meetings, conducting deep research from the web, generating code, and helping with writing. These agents can break down complicated, multi-step tasks into branched solutions, and are capable of evaluating the generated solutions dynamically while continually learning from past experiences. All of these tasks are accomplished in a fully autonomous and asynchronous manner, freeing you up to continue your day while Ninja works on these tasks in the background, and engaging when your input is required.

A comprehensive tutorial: Using advanced techniques like prompt adaptation and adapters to transform open-source unimodal models into multimodal ones, including all variants of LLaMA-Adapters, LLaVa, MiniGPT-4, and more.

Authors: Elahe Aghapour, Salar Rahili

INTRODUCTION

With recent advancements in large language models (LLMs), AI has become the spotlight of technology. We’re now more eager than ever to reach AGI-level intelligence. Yet, achieving a human-like understanding of our surroundings involves much more than just mastering language and text comprehension. Humans use their five senses to interact with the world and act based on these interactions to achieve goals. This highlights that the next step for us is to develop large models that incorporate multimodal inputs and outputs, bringing us closer to human-like capabilities. However, we face two main obstacles. First, we need a multimodal labeled dataset, which is not as accessible as text data. Second, we are already pushing the limits of compute capacity for training models with textual data. Increasing this capacity to include other modalities, especially high-dimensional ones like images and videos, is incredibly challenging.

These limitations have been a barrier for many AI researchers aiming to create capable multimodal models. So far, only a few well-established companies like Google, Meta, and OpenAI have managed to train such models. However, none of these prominent models are open source, and only a few APIs are available for public use. This has forced researchers, especially in academia, to find ways to build multimodal models without massive compute capabilities, relying instead on open-sourced pre-trained models, which are mostly single modal.

In this blog, we focus on successful, low-effort approaches to creating multi-modal models. Our criteria are centered on projects where the compute costs remain a few thousand dollars, assuming this is within the budget a typical lab can afford.

1- Parameter-Efficient Fine-Tuning (PEFT)

Before we dive into the proposed approaches for integrating and aligning two pre-trained models, we need to discuss the mechanics of fine-tuning a large model with limited compute power. Therefore, we’ll start by exploring Parameter-Efficient Fine-Tuning (PEFT) and then describe how these methods can be further used to align pre-trained models and build open-source multimodal models.

As model sizes continue to grow, the need for efficient fine-tuning methods becomes more critical. Fine-tuning all parameters in a large-scale pre-trained model is often impractical due to the substantial computational resources and time required. Parameter-efficient fine-tuning (PEFT) addresses this challenge by freezing the model’s parameters and only training the injected modules with a small number of parameters. Hence, only one copy of the large Transformer is stored with learned task specific lightweight PEFT modules, yielding a very small overhead for each additional task. This approach not only reduces resource demands but also accelerates the adaptation of models to new tasks, making it a practical and effective strategy in the era of ever-expanding models. PEFT approaches are very commonly used in LLMs and giant vision models and can be mainly divided into three categories as shown in Fig. 1: Among several methods that have been proposed, three have gotten significant attention from the community.

1- adapters: An adapter is essentially a small module, typically consisting of a downsample layer, nonlinearity, and an upsample layer with a skip connection to preserve the original input. This module is inserted into a pretrained model, with only the adapters being trained during fine-tuning.

2- LoRA injects trainable low-rank decomposition matrices into the model to approximate weight updates, significantly reducing the number of trainable parameters for downstream tasks. For a pre-trained weight matrix W of dimensions d×k, LoRA represents its update with a low-rank decomposition: W+ΔW=W+DU
where D​ has dimensions d×r and U has dimensions r×k. These matrices D and U are the tunable parameters. LoRA can be applied to the attention matrices and/or the feedforward module for efficient finetuning.

3- P*-tuning (prefix-tuning, prompt tuning) typically prepend a set of learnable prefix vectors or tokens to the input embedding, and only these so-called “soft prompts” are trained when fine-tuning on downstream tasks. The philosophy behind this approach is to assist the pre-trained models in understanding downstream tasks with the guidance of a sequence of extra “virtual tokens” information. Soft prompts are sequences of vectors that do not correspond to actual tokens in the vocabulary. Instead, they serve as intermediary representations that guide the model’s behavior to accomplish specific tasks, despite having no direct linguistic connection to the task itself.

Evaluating PEFT Techniques: Strengths and Limitations:

Adapters add a small number of parameters (3–4% of the total parameters) which makes them more efficient than full fine-tuning but less than prompt tuning or LoRA. However, they are capable of capturing complex task-specific information effectively due to the additional neural network layers and often achieve high performance on specific tasks by learning detailed task-specific features. On the downside, this approach makes the model deeper, which can complicate the optimization process and lead to longer training times.

LoRa adds only a small fraction of parameters (0.1% to 3%), making it highly efficient and scalable with very large models, making it suitable for adapting state-of-the-art LLMs and VLMs. However, LoRA’s adaptation is constrained to what can be expressed within the low-rank structure. While efficient, LoRA might be less flexible compared to adapters in capturing certain types of task-specific information.

P*- tuning is extremely parameter-efficient (often requiring less than 0.1%), as it only requires learning additional prompt tokens while keeping the original model parameters unchanged, thereby preserving the model’s generalization capabilities. However, it may not be able to capture complex task-specific information as effectively as other methods.

So far, we’ve reviewed new methods to fine-tune a large model with minimal compute power. This capability opens the door for us to combine two large models, each with billions of parameters, and fine-tune only a few million parameters to make them work together properly. This alignment allows one or both models to generate embeddings that are understandable by the other. Next, we’ll discuss three main approaches that demonstrate successful implementations of such a training regime.

2.1 Prompt adaptation:

LLaMA-Adapter presents a lightweight adaptation method to efficiently fine-tune the LLaMA model into an instruction-following model. This is achieved by freezing the pre-trained LLaMA 7B model and introducing a set of learnable adaptation prompts (1.2M parameters) into the topmost transformer layers. To avoid the initial instability and effectiveness issues caused by randomly initialized prompts, the adaptation prompts are zero-initialized. Additionally, a learnable zero-initialized gating factor is introduced to adaptively control the importance of the adaptation prompts.
Furthermore, LLaMA-Adapter extends to multi-modal tasks by integrating visual information using a pre-trained visual encoder such as CLIP. Given an image as visual context, the global visual features are acquired through multi-scale feature aggregation and then projected into the dimension of the LLM’s adaptation prompt via a learnable projection network. The resulting overall image token is repeated K times, and element-wisely added to the K-length adaptation prompts at all L inserted transformer layers. Fine-tuning with LLaMA-Adapter takes less than one hour on 8 A100 GPUs. A similar approach is used in RobustGER, where LLMs are fine-tuned to perform denoising for generative error correction (GER) in automatic speech recognition. This process takes 1.5–4.5 hours of training on a single NVIDIA A40 GPU.

LLaMA-Adapter V2 focuses on instruction-following vision models that can also generalize well on open-ended visual instructions. To achieve this goal, three key improvements are presented over the original LLaMA-Adapter. First, it introduces more learnable parameters (14M) by unfreezing all the normalization layers in LLaMA and adding a learnable bias and scale factor to all linear layers in the transformer, which distributes the instruction-following capability across the entire model. Second, visual tokens are fed into the early layers of the language model, while the adaptation prompts are added to the top layers. This improves the integration of visual knowledge without disrupting the model’s instruction-following abilities (see Fig. 2). Third, a joint training paradigm for both image-text captioning data and language-only instruction data is employed. The visual projection layers are trained for image-text captioning data while the late adaptation prompts and the unfrozen norms are trained from the instruction-following data. Additionally, expert models like captioning and OCR systems are integrated during inference, enhancing image understanding without additional training costs. We weren’t able to find specific details on GPU requirements and the time needed for training. However, based on information from GitHub, it takes approximately 100 hours on a single A100 GPU.

2.2 Intermediate Module Training:

To create a multi-modal model, two or more unimodal foundation models can be connected through a learnable projection module. This module maps features from one modality to another, enabling the integration of different data types. For instance, a vision encoder can be connected to a large language model (LLM) via a projection module. Hence, as illustrated in Fig. 3, the LLM’s input consists of a sequence of projected image features and text. The training process typically involves two stages:

1. Pretraining: The projection module is pretrained on a large dataset of paired examples to achieve cross-modality alignment.
2. Fine-tuning: The projection module (with one or more unimodal models) is fine-tuned for specific downstream tasks, such as instruction-following tasks.

MiniGPT-4, aligns a frozen visual encoder, ViT-G/14, with a frozen LLM, Vicuna, using one projection layer. For visual encoder, the same pretrained visual perception component of BLIP-2 is utilized which consists of a ViT-G/14 and Q-former network. MiniGPT-4 adds a single learnable projection layer where its output is considered as a soft prompt for the LLM in the following format:
“###Human: <Img><ImageFeatureFromProjectionLayer></Img> TextTokens. ###Assistant:”.
Training the projection layer involves two stages. First, pretrain the projection layer on a large dataset of aligned image-text pairs to acquire vision-language knowledge. Then, fine-tune the linear projection layer with a smaller, high-quality dataset. In both stages, all other parameters are frozen. As a result, MiniGPT-4 is capable of producing more natural and reliable language outputs. MiniGPT-4 requires training approximately 10 hours on 4 A100 GPUs.

Tuning the LLMs to follow instructions using machine-generated instruction-following data has been shown to improve zero-shot capabilities on new tasks. To explore this idea in the multimodality field, LLaVA connects LLM Vicuna, with a vision encoder, ViT-L/14, using a single linear layer for vision-language instruction following tasks. In the first stage, the projection layer is trained on a large image-text pairs dataset while the visual encoder and LLM weights are kept frozen. This stage creates a compatible visual tokenizer for the frozen LLM. In the second stage, the pre-trained projection layer and LLM weights are fine-tuned using a high-quality generated dataset of language-image instruction-following data. This stage enhances the model’s ability to follow multimodal instructions and perform specific tasks. LLaVA uses 8× A100s GPUs. The pretraining takes 4 hours, and the fine-tuning takes 4–8 hours depending on the specific task dataset. It showcases commendable proficiency in visual reasoning capabilities, although it falls short on academic benchmarks requiring short-form answers.

To improve the performance of LLaVA, in LLaVa-1.5:

1. A two-layer MLP is added to connect the LLM to the vision encoder.
2. The input image resolution has been scaled up using CLIP-ViT-L-336px, allowing for better detail perception.
3. The Vicuna model has been scaled up to 13B parameters.
4. Academic-task-oriented VQA data has been added, along with specific response formatting prompts to indicate the desired output format. When prompting for short-form answers, the prompt “Answer the question using a single word or phrase.” is appended to the VQA question.

The training finishes in ∼1 day on a single 8-A100 GPU and achieves state-of-the-art results on a wide range of benchmarks.

Video-ChatGPT focuses on creating a video-based conversational agent. Given the limited availability of video-caption pairs and the substantial resources required for training from scratch, it uses the pretrained image-based visual encoder, CLIP ViT-L/14 for video tasks and connect it with pretrained LLM Vicuna through a learnable linear projection model. ViT-L/14 encodes images, so for a given video sample with T frames, it generates T frame-level embeddings with dimensions h*w*D. As illustrated in Fig. 4. The process of obtaining video-level features involves two key steps:

• Spatial Video Features: These are obtained by average-pooling frame-level features across the temporal dimension to achieve an h*w*D dimension.
• Temporal Video Features: These are obtained by average-pooling across the spatial dimension to achieve T*D dimensions.

These temporal and spatial features are concatenated to form video-level features, which are then projected into the textual embedding space by a learnable linear layer. The model is trained on their curated, high-quality dataset of video-text pairs, and the training of the linear projection layer takes around 3 hours on 8 A100 40GB GPUs. This approach allows Video-ChatGPT to generate detailed and coherent conversations about video content.

PandaGPT, while not connecting unimodal models, introduces the first general-purpose model capable of instruction-following by integrating the pretrained LLM Vicuna with the multimodal encoder ImageBind through a linear projection layer (see Fig. 5). The linear projection layer is trained, and Vicuna’s attention modules are fine-tuned using LoRA on 8×A100 40G GPUs for 7 hours, leveraging only image-language (multi-turn conversation) instruction-following data. Despite being trained exclusively on image-text pairs, PandaGPT exhibits emergent, zero-shot, cross-modal capabilities across multiple modalities by leveraging the binding property across six modalities (image/video, text, audio, depth, thermal, and IMU) inherited from the frozen ImageBind encoders. This enables PandaGPT to excel in tasks such as image/video-grounded question answering, image/video-inspired creative writing, visual and auditory reasoning, and more.

2.3 Adapter Mixture:

Cheap&Quick adopts lightweight adapters to integrate large language models (LLMs) and vision models for vision-language tasks. The paper proposes a Mixture-of-Modality Adapters (MMA), designed to facilitate switching between single- and multi-modal instructions without compromising performance. A learnable token t is proposed as the modality selector token. This token indicates the input features’ modality (i.e., unimodal or multimodal input) and informs the router module on how to combine the output of the learned adapters, as illustrated in Fig 6. The adapter is formulated as :
Z′=Z+s⋅router(f(Z),g(Z); t)
where Z is the input features, either unimodal or concatenated multimodal (image-text) features. Modules f and g share a common unimodal adapter architecture. s is a scaling factor, and the router(⋅) function determines the routing path based on the modality token t.
To demonstrate the effectiveness of MMA, the authors connected LLaMA and CLIP-ViT with a single linear layer and inserted MMA into both ViT and LLaMA before the multi-head attention modules. The adapters and projection layer (only 3.8M parameters) were trained with a mixture of text-only and text-image data on 8 A100 GPUs for 1.4 hours. This approach showed a significant reduction in training costs while maintaining high performance on vision-language tasks.

2.4 A Modality as Grounding Without Training
Up to this point, we have discussed papers that connect unimodal models to create a multimodal model. However, advancing toward AGI requires a multimodal model capable of handling data from different modalities for diverse tasks, ranging from calculus to generating images based on descriptions.

Recently, many papers have explored integrating pre-trained multi-modal models via language prompting into a unified model capable of handling various tasks across different modalities without additional training. In this approach, language serves as an intermediary for models to exchange information. Through prompt engineering (e.g., Visual ChatGPT, MM-REACT) or fine-tuning (e.g., Toolformer, GPT4Tools), LLMs can invoke specialized foundation models to handle various modality-specific tasks. While this topic is beyond the scope of our current blog post, you can refer to these papers for more detailed information.

In another similar work, MAGIC proposes a novel, training-free, plug-and-play framework and uses image embedding, through pre-trained CLIP, as the grounding foundation. This framework connects GPT-2 with CLIP to perform image-grounded text generation (e.g., image captioning) in a zero-shot manner. By incorporating the similarity between the image embeddings from CLIP and the top-k generated tokens from a pre-trained LLM at each time step into the decoding inference, the model effectively leverages visual information to guide text generation. Without any additional training, this approach demonstrates the capability to generate visually grounded stories given both an image and a text prompt.

3. High quality Curated data:

We have discussed various methods of aligning different modalities up to this point; however, it is important to remember that having curated, high-quality data in such training regimes is equally crucial. For instance, detailed and accurate instructions and responses significantly enhance the zero-shot performance of large language models on interactive natural language tasks. In the field of interactive vision-language tasks, the availability of high-quality data is often limited, prompting researchers to develop innovative methods for generating such data.

MiniGPT-4 proposes a two-stage method to curate a detailed, instruction-following image description dataset:
1-Data Generation: The pre-trained model from the first stage of training is used to generate detailed descriptions. For a given image, a carefully crafted prompt is used to enable the pre-trained model to produce detailed and informative image descriptions in multiple steps,

2- Post-Processing and Filtering: The generated image descriptions contain noisy or incoherent descriptions. In order to fix these issues, ChatGPT is employed to refine the generated descriptions according to specific post-processing requirements and standards, guided by a designed prompt. The refined dataset is then manually verified to ensure the correctness and quality of the image-text pairs.

LLaVA proposes a method to generate multimodal instruction-following data by querying ChatGPT/GPT-4 based on widely available image-text pair data. They designed a prompt that consists of an image caption, bounding boxes to localize objects in the scene, and a few examples for in-context learning. This method leverages the existing rich dataset and the capabilities of ChatGPT/GPT-4 to produce highly detailed and accurate multimodal data.

Video-ChatGPT utilized two approaches for generating high-quality video instruction data.
1- Human-Assisted Annotation: Expert annotators enrich given video-caption pairs by adding comprehensive details to the captions
2- Semi-Automatic Annotation: This involves a multi-step process leveraging several pretrained models:

1. Pretrained BLIP-2 and GRiT models are used for analyzing key frames in the videos. BLIP-2 generates frame-level captions, while GRiT provides detailed descriptions of scene objects. Additionally, the pretrained Tag2Text model generates tags for each key frame of the video.
2. A specialized filtering mechanism is employed to remove any captions from BLIP-2 or GRiT that do not match the Tag2Text frame-level tags.
3. The GPT-3.5 model is used to merge the filtered captions and generate a singular, coherent video-level caption.

MIMIC-IT generated a dataset of 2.8 million multimodal instruction-response pairs, aimed at enhancing Vision-Language Models (VLMs) in perception, reasoning, and planning. To demonstrate the importance of high-quality data, they fine-tuned OpenFlamingo using the MIMIC-IT dataset on 8 A100 GPUs over 3 epochs in one day. The resulting model outperforms OpenFlamingo, demonstrating superior in-context and zero-shot learning capabilities.

The opinions expressed in this blog post are solely our own and do not reflect those of our employer.

References:

[1] LLaMA-Adapter: Zhang, Renrui, et al. “Llama-adapter: Efficient fine-tuning of language models with zero-init attention.” (2023).
[2] LLaMA-Adapter V2: Gao, Peng, et al. “Llama-adapter v2: Parameter-efficient visual instruction model.” (2023).
[3] MiniGPT-4: Zhu, Deyao, et al. “Minigpt-4: Enhancing vision-language understanding with advanced large language models.” (2023).
[4] LLaVA: Liu, Haotian, et al. “Visual instruction tuning.” (2024).
[5] LLaVa-1.5: Liu, Haotian, et al. “Improved baselines with visual instruction tuning.” (2024).
[6] Video-ChatGPT: Maaz, Muhammad, et al. “Video-chatgpt: Towards detailed video understanding via large vision and language models.” (2023).
[7] PandaGPT: Su, Yixuan, et al. “Pandagpt: One model to instruction-follow them all.” (2023).
[8] Cheap&Quick: Luo, Gen, et al. “Cheap and quick: Efficient vision-language instruction tuning for large language models.” (2024).
[9] RobustGER: Hu, Yuchen, et al. “Large Language Models are Efficient Learners of Noise-Robust Speech Recognition.” (2024).
[10] MAGIC: Su, Yixuan, et al. “Language models can see: Plugging visual controls in text generation.” (2022).
[11] Visual ChatGPT: Wu, Chenfei, et al. “Visual chatgpt: Talking, drawing and editing with visual foundation models.” (2023).
[12] MM-REACT: Yang, Zhengyuan, et al. “Mm-react: Prompting chatgpt for multimodal reasoning and action.” (2023).
[13] Toolformer: Schick, Timo, et al. “Toolformer: Language models can teach themselves to use tools.” (2024).
[14] GPT4Tools: Yang, Rui, et al. “Gpt4tools: Teaching large language model to use tools via self-instruction.” (2024).
[15] MIMIC-IT: Li, Bo, et al. “Mimic-it: Multi-modal in-context instruction tuning.” (2023).
[16] He, Junxian, et al. “Towards a unified view of parameter-efficient transfer learning.” (2021).

---

From Unimodals to Multimodality: DIY Techniques for Building Foundational Models was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.