Category: Artificial Intelligence

Your multi-account strategy is the core of your foundational environment on AWS. Design decisions around your multi-account environment are critical for operating securely at scale. Grouping your workloads strategically into multiple AWS accounts enables you to apply different controls across workloads, track cost and usage, reduce the impact of account limits, and mitigate the complexity […]

Discover how AI tools can transform intricate concepts into clear, practical frameworks and diagrams

Introduction

AI tools like Chat GPT are transforming how we approach complex ideas. One of the things I enjoy doing with Chat GPT is integrating perspectives and ideas from different thinkers, as well as differentiating between them to better understand their nuances. This is easily one of my favorite applications of AI.

Motivation

Napkin AI caught my eye because it generates interesting diagrams automatically from text input, making it highly flexible and easy to use. I’ve been looking for good concept mapping and knowledge mapping software, and this seemed like a reasonable place to start.

Goal

This article is the first in a series where I will explore different methods of examining, integrating, and visualizing complex ideas and perspectives. Although I’ve now moved on to creating my own tool, experimenting with Napkin AI was an important first step. In this series, I will document the journey that will involve exploring some of the tools out there, making new ones easy to use, and sharing my experiences.

The Process

1. Setting up the Prompt

After some basic setup, Napkin AI asked me for a prompt. I started with something I happened to be exploring at the time:

“The most unique insights from Alan Watts”

It generated some basic information, maybe about as powerful as Chat GPT-4. I previewed a few diagrams and then decided to challenge it further:

“The most unique insights from Alan Watts, Robert M. Pirsig, Spinoza, and Marcus Aurelius, emphasizing their overlaps with each other and their detailed underpinnings in very specific ideas from philosophy, science, and psychology, organized into practical frameworks and the logical progression towards such frameworks.”

The results were geared more towards the process of creating a framework, which in a broad sense is great, but I wanted it to go further. It was easy enough to start a new Napkin and change a few words of the prompt to focus more on building what I wanted:

… organized into a single practical step by step iterative framework.

2. Creating and Customizing Visuals

There are a number of templates to choose from, and a few styles. The content of each choice varied slightly, sometimes going into different levels of detail, reflecting the format of the diagram.

I selected the diagram template I felt suited each section best. I tended to prefer a few of the templates that were best suited for this project, but I opted towards a balance between consistency and variety. Here are a few:

3. Publishing the Output

Upon completing each document, I exported them as PDFs. Find them here.

Reflections and Applications

I am definitely glad I went ahead with this tool. What I created is not the type of concept map I was looking for, but it did help me visualize complex topics. It even helped me integrate them into practical frameworks with its Draft with AI option, although you can also start a blank document and provide your own text. If you want to edit the document or add your own images, that’s just as easy.

I imagine this could be used in a wide variety of presentational contexts, including business, education, personal growth, and content creation. It empowers individual creators to turn complex ideas into practical frameworks, supporting innovation and growth. Most importantly, it creates powerful diagrams that can be highly useful for business communications and strategy development.

Next Steps

This was a great beginning, but I have more to share. Aside from using other tools for diagramming, I am developing a custom tool tailored to better integrate and visualize complex ideas, including the detailed relationships between their components. This series will cover those journeys. Stay tuned!

Resources

• Napkin AI
• Concept Maps and Diagrams | GitHub repository

All images made by author in Napkin AI

---

Visualizing and Integrating Complex Ideas with LLMs, Part 1: Napkin AI was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Seeking advice from a panel of specialists

Large Language Models (LLMs) have undoubtedly taken the tech industry by storm. Their meteoric rise was fueled by a large corpora of data from Wikipedia, web pages, books, troves of research papers and, of course, user content from our beloved social media platforms. The data and compute hungry models have been feverishly incorporating multi-modal data from audio and video libraries, and have been using 10s of thousands of Nvidia GPUs for months to train the state-of-the-art (SOTA) models. All this makes us wonder whether this exponential growth can last.

The challenges facing these LLMs are numerous but let’s investigate a few here.

• Cost and Scalability: Larger models can cost tens of millions of dollars to to train and serve, becoming a barrier to adoption by the swath of day-to-day applications. (See Cost of training GPT-4)
• Training Data Saturation: Publicly available datasets will exhaust soon enough and may need to rely on slowly generated user content. Only companies and agencies that have a steady source of new content will be able to generate improvements.
• Hallucinations: Models generating false and unsubstantiated information is going to be a deterrent with users expecting validation from authoritative sources before using for sensitive applications.
• Exploring unknowns: LLMs are now being used for applications beyond their original intent. For example LLMs have shown great ability in game play, scientific discovery and climate modeling. We will need new approaches to solve these complex situations.

Before we start getting too worried about the future, let’s examine how AI researchers are tirelessly working on ways to ensure continued progress. The Mixture-of-Experts (MoE) and Mixture-of-Agents (MoA) innovations show that hope is on the horizon.

First introduced in 2017, Mixture-of-Experts technique showed that multiple experts and a gating network that can pick a sparse set of experts can produce a vastly improved outcome with lower computational costs. The gating decision allows to turn off large pieces of the network enabling conditional computation, and specialization improves performance for language modeling and machine translational tasks.

The figure above shows that a Mixture-of-Experts layer is incorporated in a recurrent neural network. The gating layer activates only two experts for the task and subsequently combines their output.

While this was demonstrated for select benchmarks, conditional computation has opened up an avenue to see continued improvements without resorting to ever growing model size.

Inspired by MOE, Mixture-of-Agents technique leverages multiple LLM to improve the outcome. The problem is routed through multiple LLMs aka agents that enhance the outcome at each stage and the authors have demonstrated that it produces a better outcome with smaller models versus the larger SOTA models.

The figure shows 4 Mixture-of-Agents layers with 3 agents in each layer. Selecting appropriate LLMs for each layer is important to ensure proper collaboration and to produce high quality response. (Source)

MOA relies on the fact that LLMs collaborating together produce better outputs as they can combine responses from other models. The role of LLMs is divided into proposers that generate diverse outputs and aggregators that can combine them to produce high-quality responses. The multi-stage approach will likely increase the Time to First Token (TTFT), so mitigating approaches will need to be developed to make them suitable for broad applications.

MOE and MOA have similar foundational elements but behave differently. MOE works on the concept of picking a set of experts to complete a job where the gating network has the task of picking the right set of experts. MOA works on teams building on the work of the previous teams, and improving the outcome at each stage.

Innovations for MOE and MOA have created a path of innovation where a combination of specialized components or models, collaborating and exchanging information, can continue to provide better outcomes even when linear scaling of model parameters and training datasets is no longer trivial.

While it is only with hindsight we will know whether the LLM innovations can last, I have been following the research in the field for insights. Seeing what is coming out of universities and research institutions, I am extremely bullish on what is next to come. I do feel we are just warming up for the onslaught of new capabilities and applications that will transform our lives. We don’t know what they are but we can be fairly certain that coming days will not fail to surprise us.

“We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.”. -Amara’s Law

References

[1] Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2024). Mixture-of-Agents Enhances Large Language Model Capabilities. arXiv [Preprint]. https://arxiv.org/abs/2406.04692

[2] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.

---

MOE & MOA for Large Language Models was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Key steps to kick off an AI journey in your current job

I have heard many times data scientists frustrated due to the lack of cool projects to work on within their company. Convincing business stakeholders and management to start AI projects can be challenging. While it is not usually the data scientist’s responsibility to think and propose the projects that need to be prioritized, I’ve seen how data scientists together with data managers and product managers can influence the roadmaps and help introduce more innovative and impactful projects.

In this blog post I am going to share some of the steps and strategies that I’ve seen successfully influence the team or company culture towards introducing more innovative ML or AI based projects. Be aware this is not something that happens from one day to another, but a journey in which your knowledge and motivation can help others in your company to think outside of the box and see the potential of ML and AI.

These key steps and strategies for pitching innovation and AI in your company are: raising awareness, inspiring through use cases, finding sponsors & ideas, and prioritization.

1. Raise awareness about AI

The first step is to raise awareness on your organization about what AI can and cannot do. Many people have limited understanding of AI, which can lead both to skepticism and unrealistic expectations.

The end goal at this first step would be to help people around you to gain sensibility about AI. This sensibility can include: what is the difference between ML and AI, what type of problems can I solve with traditional ML (classification, regression, time series…), what new opportunities appear now with GenAI (text generation, image generation, few shot classification…). Some strategies to reach this awareness are:

• Workshops and trainings: these can be organized in-house, or you can also recommend online courses. The second option is usually faster and less expensive; courses like “AI For Everyone” and “Generative AI For Everyone” from deeplearning.ai are always a good start.
• Empower & encourage everyone to use GenAI: this can be done by casually explaining how you leverage GenAI yourself, by sharing images and poems obtained through it, or by challenging why they haven’t used it yet. Try to understand if there are specific concerns that are holding people back (e.g. “I don’t trust it with my own data”), and share tools or techniques that can help mitigate those perceived risks.
• Showcase ML / AI projects: actively participate in your company demos, All Hands, or internal knowledge sharing sessions. You can share ML or AI projects you or your team have already implemented. It is important to ensure the right level of technical details to allow people to follow your presentation, and highlight the project’s potential, impact, and learnings. It can also be interesting to share how these projects differentiate from “traditional software development” or the other type of projects from the company.

2. Inspire through relevant use cases

People around you have already some awareness and sensibility about AI and ML, the types of models that exist, their potential, and how those types of projects work, great! The next step is to start introducing use cases that can inspire projects for your company. These use cases can come from competitors or analogous industries, but also from the general use cases that apply to most companies (user segmentation, client / user churn prediction, selling forecasts…).

Demonstrating how competitors or other companies are leveraging AI can be powerful to illustrate its potential and inspire next steps. When showcasing use cases, you can focus on the problem that use case solves, the tangible benefits it achieved, and make the analogy on how something similar could be applicable in your company. Similarly, for more general use cases, such as user segmentation, it can be interesting to showcase the type of application that could come out of it in your specific company (dynamic pricing, personalization, improved communications…).

If there are already some teams doing competitors analysis (usually User Researchers), make sure they are also taking into account ML / AI features. Help them gain the sensibility on how those solutions might work underneath to enrich further their research and detect AI opportunities for your company.

3. Find your sponsors & use cases

There is now awareness on what AI is and what types of problems and use cases it can help solve in your company. If you’ve done it right, you should have been able to get some people really excited about all this potential!

This excitement can translate into people coming directly to you to share other use cases, ask questions, or even ask wether something is feasible to solve with AI or not. These are your sponsors: champions within the organization who can support and advocate for AI initiatives. Depending on the size of the company and how big the culture change needs to be, this sponsorship might be close enough to influence decision-making at the highest levels. However, getting to inspire business stakeholders can also be good enough, as they can push to solve their own objectives through AI.

You’ve planted the seeds for AI project ideas to come out. You can now start proposing specific AI solutions for specific company problems or objectives. Thanks to all the previous work on awareness, use cases, and sponsors, these proposals should be now much more well received!

What is most interesting from this step though, is waiting for the use cases to also come to you. Your AI sponsors and other people in the company are now able to link problems and objectives to AI solutions. It might surprise you how much use cases can appear from this direction. The awareness you’ve built will naturally lead to more informed and relevant suggestions.

4. Assessing and prioritizing use cases

At this point you might have been able to collect several ideas of initiative and have the buy-in from the management to dedicate some time to work on them. But how do you decide with where to start? It might make sense to start with the initiative with biggest potential, but predicting the ROI of innovation, and particularly of AI projects, can be challenging due to their inherent uncertainty. However, there are some key points to take into account that can help on that end:

• Focus on specific strategic pain points or opportunities within the company.
• Use industry benchmarks to estimate success rates and potential revenues.
• Asses potential benefit, but also feasibility and risks.
• Differentiate between exploratory (high uncertainty, long-term) and exploitative (low uncertainty, short-term) projects.

Try to start with exploitative ideas (quick wins), to prove value faster, gain traction and build trust. Once that is managed, maybe you can start introducing explorative ideas (moonshots) that aim longer term & bigger transformation, but also involve higher risks to fail. Balancing a continuous delivery and improvement with moonshots is key to maintain the trust long term while also exploring real innovation.

In a previous post “Starting ML Product Initiatives on the Right Foot”, I deep-dived into how to successfully start with ML initiatives and manage their inherent uncertainty from the beginning.

Starting ML Product Initiatives on the Right Foot

Wrapping it up

Pitching AI in your company is a long term journey, not something that will happen overnight. From my experience, it is important to start generating awareness and education, showcasing use cases, and aligning with sponsors in the right positions. Only then, proposing use cases will listened; even other people might come to you with relevant ideas! Once some use cases have been gathered and there is some bandwidth and buy-in to prioritize some dedication, it is time to focus on strategic problems, quantify well the opportunity and potential, and balance between quick wins and long term moonshots.

We are in a moment in time where everybody is talking about AI. In particular, companies are trying to think about their (Gen)AI strategies and how this new technology will change the business and ways of working. This plays in your favor: it should be a good moment to start introducing this steps, as people are particularly keen to learn, play, and leverage AI.

---

Pitching (AI) Innovation in Your Company was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

An end-2-end empirical sharing of multi-step quantile forecasting with Tensorflow, NeuralForecast, and Zero-shot LLMs.

Content

1. Short Introduction
2. Data
3. Build a Toy Version of Quantile Recurrent Forecaster
4. Quantile Forecasting with the State-of-Art Models
5. Zero-shot Quantile Forecast with LLMs
6. Conclusion

Short Introduction

Quantile forecasting is a statistical technique used to predict different quantiles (e.g., the median or the 90th percentile) of a response variable’s distribution, providing a more comprehensive view of potential future outcomes. Unlike traditional mean forecasting, which only estimates the average, quantile forecasting allows us to understand the range and likelihood of various possible results.

Quantile forecasting is essential for decision-making in contexts with asymmetric loss functions or varying risk preferences. In supply chain management, for example, predicting the 90th percentile of demand ensures sufficient stock levels to avoid shortages, while predicting the 10th percentile helps minimize overstock and associated costs. This methodology is particularly advantageous in sectors such as finance, meteorology, and energy, where understanding distribution extremes is as critical as the mean.

Both quantile forecasting and conformal prediction address uncertainty, yet their methodologies differ significantly. Quantile forecasting directly models specific quantiles of the response variable, providing detailed insights into its distribution. Conversely, conformal prediction is a model-agnostic technique that constructs prediction intervals around forecasts, guaranteeing that the true value falls within the interval with a specified probability. Quantile forecasting yields precise quantile estimates, whereas conformal prediction offers broader interval assurances.

The implementation of quantile forecasting can markedly enhance decision-making by providing a sophisticated understanding of future uncertainties. This approach allows organizations to tailor strategies to different risk levels, optimize resource allocation, and improve operational efficiency. By capturing a comprehensive range of potential outcomes, quantile forecasting enables organizations to make informed, data-driven decisions, thereby mitigating risks and enhancing overall performance.

Data

To demonstrate the work, I chose to use the data from the M4 competition as an example. The data is under CC0: Public Domain license which can be accessed here. The data can also be loaded through datasetsforecast package:

# Install the package
pip install datasetsforecast
# Load Data
df, *_ = M4.load('./data', group='Weekly')
# Randomly select three items
df = df[df['unique_id'].isin(['W96', 'W100', 'W99'])]
# Define the start date (for example, "1970-01-04")
start_date = pd.to_datetime("1970-01-04")
# Convert 'ds' to actual week dates
df['ds'] = start_date + pd.to_timedelta(df['ds'] - 1, unit='W')
# Display the DataFrame
df.head()

The original data contains over 300 unique time series. To demonstrate, I randomly selected three time series: W96, W99, and W100, as they all have the same history length. The original timestamp is masked as integer numbers (i.e., 1–2296), I manually converted it back to normal date format with the first date to be January 4th, 1970. The following figure is a preview of W99:

Build a Toy Version of Quantile Recurrent Forecaster

First, let’s build a quantile forecaster from scratch to understand how the target data flows through the pipeline and how the forecasts are generated. I picked the idea from the paper A Multi-Horizon Quantile Recurrent Forecaster by Wen et al. The authors proposed a Multi-Horizon Quantile Recurrent Neural Network (MQ-RNN) framework that combines Sequence-to-Sequence Neural Networks, Quantile Regression, and Direct Multi-Horizon Forecasting for accurate and robust multi-step time series forecasting. By leveraging the expressiveness of neural networks, the nonparametric nature of quantile regression, and a novel training scheme called forking-sequences, the model can effectively handle shifting seasonality, known future events, and cold-start situations in large-scale forecasting applications.

We cannot reproduce everything in this short blog, but we can try to replicate part of it using the TensorFlow package as a demo. If you are interested in the implementation of the paper, there is an ongoing project that you can leverage: MQRNN.

Let’s first load the necessary package and define some global parameters. We will use the LSTM model as the core, and we need to do some preprocessing on the data to obtain the rolling windows before fitting. The input_shape is set to (104, 1) meaning we are using two years of data for each training window. In this walkthrough, we will only look into an 80% confidence interval with the median as the point forecast, which means the quantiles = [0.1, 0.5, 0.9]. We will use the last 12 weeks as a test dataset, so the output_steps or horizon is equal to 12 and the cut_off_date will be ‘2013–10–13’.

# Install the package
pip install tensorflow

# Load the package
from sklearn.preprocessing import StandardScaler
from datetime import datetime
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, LSTM, Dense, concatenate, Layer

# Define Global Parameters
input_shape = (104, 1)
quantiles = [0.1, 0.9]
output_steps = 12
cut_off_date = '2013-10-13'
tf.random.set_seed(20240710)

Next, let’s convert the data to rolling windows which is the desired input shape for RNN-based models:

# Preprocess The Data
def preprocess_data(df, window_size = 104, forecast_horizon = 12):
    # Ensure the dataframe is sorted by item and date
    
    df = df.sort_values(by=['unique_id', 'ds'])
    # List to hold processed data for each item
    X, y, unique_id, ds = [], [], [], []
    # Normalizer
    scaler = StandardScaler()
    # Iterate through each item
    for key, group in df.groupby('unique_id'):
        demand = group['y'].values.reshape(-1, 1)
        scaled_demand = scaler.fit_transform(demand)
        dates = group['ds'].values
        # Create sequences (sliding window approach)    
        for i in range(len(scaled_demand) - window_size - forecast_horizon + 1):
            X.append(scaled_demand[i:i+window_size])
            y.append(scaled_demand[i+window_size:i+window_size+forecast_horizon].flatten())
            unique_id.append(key)
            ds.append(dates[i+window_size:i+window_size+forecast_horizon])
    X = np.array(X)
    y = np.array(y)
    return X, y, unique_id, ds, scaler

Then we split the data into train, val, and test:

# Split Data
def split_data(X, y, unique_id, ds, cut_off_date):
    cut_off_date = pd.to_datetime(cut_off_date)
    val_start_date = cut_off_date - pd.Timedelta(weeks=12)
    train_idx = [i for i, date in enumerate(ds) if date[0] < val_start_date]
    val_idx = [i for i, date in enumerate(ds) if val_start_date <= date[0] < cut_off_date]
    test_idx = [i for i, date in enumerate(ds) if date[0] >= cut_off_date]

    X_train, y_train = X[train_idx], y[train_idx]
    X_val, y_val = X[val_idx], y[val_idx]
    X_test, y_test = X[test_idx], y[test_idx]

    train_unique_id = [unique_id[i] for i in train_idx]
    train_ds = [ds[i] for i in train_idx]
    val_unique_id = [unique_id[i] for i in val_idx]
    val_ds = [ds[i] for i in val_idx]
    test_unique_id = [unique_id[i] for i in test_idx]
    test_ds = [ds[i] for i in test_idx]
    
    return X_train, y_train, X_val, y_val, X_test, y_test, train_unique_id, train_ds, val_unique_id, val_ds, test_unique_id, test_ds

The authors of the MQRNN utilized both horizon-specific local context, essential for temporal awareness and seasonality mapping, and horizon-agnostic global context to capture non-time-sensitive information, enhancing the stability of learning and the smoothness of generated forecasts. To build a model that sort of reproduces what the MQRNN is doing, we need to write a quantile loss function and add layers that capture local context and global context. I added an attention layer to it to show you how the attention mechanism can be included in such a process:

# Attention Layer
class Attention(Layer):
    def __init__(self, units):
        super(Attention, self).__init__()
        self.W1 = Dense(units)
        self.W2 = Dense(units)
        self.V = Dense(1)
    def call(self, query, values):
        hidden_with_time_axis = tf.expand_dims(query, 1)
        score = self.V(tf.nn.tanh(self.W1(values) + self.W2(hidden_with_time_axis)))
        attention_weights = tf.nn.softmax(score, axis=1)
        context_vector = attention_weights * values
        context_vector = tf.reduce_sum(context_vector, axis=1)
        return context_vector, attention_weights

# Quantile Loss Function
def quantile_loss(q, y_true, y_pred):
    e = y_true - y_pred
    return tf.reduce_mean(tf.maximum(q*e, (q-1)*e))

def combined_quantile_loss(quantiles, y_true, y_pred, output_steps):
    losses = [quantile_loss(q, y_true, y_pred[:, i*output_steps:(i+1)*output_steps]) for i, q in enumerate(quantiles)]
    return tf.reduce_mean(losses)

# Model architecture
def create_model(input_shape, quantiles, output_steps):
    inputs = Input(shape=input_shape)
    lstm1 = LSTM(256, return_sequences=True)(inputs)
    lstm_out, state_h, state_c = LSTM(256, return_sequences=True, return_state=True)(lstm1)
    context_vector, attention_weights = Attention(256)(state_h, lstm_out)
    global_context = Dense(100, activation = 'relu')(context_vector)
    forecasts = []
    for q in quantiles:
        local_context = concatenate([global_context, context_vector])
        forecast = Dense(output_steps, activation = 'linear')(local_context)
        forecasts.append(forecast)
    outputs = concatenate(forecasts, axis=1)
    model = Model(inputs, outputs)
    model.compile(optimizer='adam', loss=lambda y, f: combined_quantile_loss(quantiles, y, f, output_steps))
    return model

Here are the plotted forecasting results:

We also evaluated the SMAPE for each item, as well as the percentage coverage of the interval (how much actual was covered by the interval). The results are as follows:

This toy version can serve as a good baseline to start with quantile forecasting. The distributed training is not configured for this setup nor the model architecture is optimized for large-scale forecasting, thus it might suffer from speed issues. In the next section, we will look into a package that allows you to do quantile forecasts with the most advanced deep-learning models.

Quantile Forecasting with the SOTA Models

The neuralforecast package is an outstanding Python library that allows you to use most of the SOTA deep neural network models for time series forecasting, such as PatchTST, NBEATs, NHITS, TimeMixer, etc. with easy implementation. In this section, I will use PatchTST as an example to show you how to perform quantile forecasting.

First, load the necessary modules and define the parameters for PatchTST. Tuning the model will require some empirical experience and will be project-dependent. If you are interested in getting the potential-optimal parameters for your data, you may look into the auto modules from the neuralforecast. They will allow you to use Ray to perform hyperparameter tuning. And it is quite efficient! The neuralforecast package carries a great set of models that are based on different sampling approaches. The ones with the base_window approach will allow you to use MQLoss or HuberMQLoss, where you can specify the quantile levels you are looking for. In this work, I picked HuberMQLoss as it is more robust to outliers.

# Install the package
pip install neuralforecast

# Load the package
from neuralforecast.core import NeuralForecast
from neuralforecast.models import PatchTST
from neuralforecast.losses.pytorch import HuberMQLoss, MQLoss

# Define Parameters for PatchTST
PARAMS = {'input_size': 104, 
          'h': output_steps, 
          'max_steps': 6000, 
          'encoder_layers': 4,
          'start_padding_enabled': False,
          'learning_rate': 1e-4,
          'patch_len': 52,  # Length of each patch
          'hidden_size': 256,  # Size of the hidden layers
          'n_heads': 4,  # Number of attention heads
          'res_attention': True,
          'dropout': 0.1,  # Dropout rate
          'activation': 'gelu',  # Activation function
          'dropout': 0.1,
          'attn_dropout': 0.1,
          'fc_dropout': 0.1,
          'random_seed': 20240710,
          'loss': HuberMQLoss(quantiles=[0.1, 0.5, 0.9]),
          'scaler_type': 'standard',
          'early_stop_patience_steps': 10}

# Get Training Data
train_df = df[df.ds<cut_off_date] 

# Fit and predict with PatchTST
models = [PatchTST(**PARAMS)]
nf = NeuralForecast(models=models, freq='W') 
nf.fit(df=train_df, val_size=12)
Y_hat_df = nf.predict().reset_index()

Here are plotted forecasts:

Here are the metrics:

Through the demo, you can see how easy to implement the model and how the performance of the model has been lifted. However, if you wonder if there are any easier approaches to do this task, the answer is YES. In the next section, we will look into a T5-based model that allows you to conduct zero-shot quantile forecasting.

Zero-shot Quantile Forecast with LLMs

We have been witnessing a trend where the advancement in NLP will also further push the boundaries for time series forecasting as predicting the next word is a synthetic process for predicting the next period’s value. Given the fast development of large language models (LLMs) for generative tasks, researchers have also started to look into pre-training a large model on millions of time series, allowing users to do zero-shot forecasts.

However, before we draw an equal sign between the LLMs and Zero-shot Time Series tasks, we have to answer one question: what is the difference between training a language model and training a time series model? It would be “tokens from a finite dictionary versus values from an unbounded.” Amazon recently released a project called Chronos which well handled the challenge and made the large time series model happen. As the authors stated: “Chronos tokenizes time series into discrete bins through simple scaling and quantization of real values. In this way, we can train off-the-shelf language models on this ‘language of time series,’ with no changes to the model architecture”. The original paper can be found here.

Currently, Chronos is available in multiple versions. It can be loaded and used through the autogluon API with only a few lines of code.

# Get Training Data and Transform
train_df = df[df.ds<cut_off_date] 
train_df_chronos = TimeSeriesDataFrame(train_df.rename(columns={'ds': 'timestamp', 'unique_id': 'item_id', 'y': 'target'}))

# Zero-shot forecast with Chronos
predictor = TimeSeriesPredictor(prediction_length=output_steps, freq='W', quantile_levels = [0.1, 0.9]).fit(
            train_df_chronos, presets="chronos_base", 
            random_seed = 20240710
        )
Y_hat_df_chronos = predictor.predict(train_df_chronos).reset_index().rename(columns={'mean': 'Chronos', 
                                                                                     '0.1': 'P10', 
                                                                                     '0.9': 'P90',
                                                                                     'timestamp': 'ds',
                                                                                     'item_id': 'unique_id'})

Here are the plotted forecasts:

Here are the metrics:

As you can see, Chronos showed a very decent performance compared to PatchTST. However, it does not mean it has surpassed PatchTST, since it is very likely that Chronos has been trained on M4 data. In their original paper, the authors also evaluated their model on the datasets that the model has not been trained on, and Chronos still yielded very comparable results to the SOTA models.

There are many more large time series models being developed right now. One of them is called TimeGPT which was developed by NIXTLA. The invention of this kind of model not only made the forecasting task easier, more reliable, and consistent, but it is also a good starting point to make reasonable guesses for time series with limited historical data.

Conclusion

From building a toy version of a quantile recurrent forecaster to leveraging state-of-the-art models and zero-shot large language models, this blog has demonstrated the power and versatility of quantile forecasting. By incorporating models like TensorFlow’s LSTM, NeuralForecast’s PatchTST, and Amazon’s Chronos, we can achieve accurate, robust, and computationally efficient multi-step time series forecasts. Quantile forecasting not only enhances decision-making by providing a nuanced understanding of future uncertainties but also allows organizations to optimize strategies and resource allocation. The advancements in neural networks and zero-shot learning models further push the boundaries, making quantile forecasting a pivotal tool in modern data-driven industries.

Note: All the images, numbers and tables are generated by the author. The complete code can be found here: Quantile Forecasting.

---

From Scratch to Deep Quantile Forecasting was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Using a GloVe embedding-based algorithm to achieve 100% accuracy on the popular party game “Codenames”

Introduction

Codenames is a popular party game for 2 teams of 2 players each, where each team consists of a spymaster and an operative. Each team is allocated a number of word cards on a game board. During each turn, the spymaster provides a word clue, and the number of word cards it corresponds to. The operative would then have to guess which words on the game board belong to his/her team. The objective is for the spymaster to provide good clues, such that the operative can use fewer number of turns to guess all the words, before the opponent team. In addition, there will be an “assassin” card, which upon being chosen by the operative, causes the team to lose immediately.

In this project, we are going to use a simple word vector algorithm using pre-trained word vectors in machine learning to maximize our accuracy in solving the game in as few tries as possible.

Below are examples of what the game board looks like:

In the card arrangement for the Spymaster, the color of each card represents Red Team, Blue Team, Neutral (Beige) and Assassin Card (Black).

Automating the spymaster and operative

We will be creating an algorithm that can take on both the roles of spymaster and operative and play the game by itself. In a board of 25 cards, there will be 9 “good” cards and 16 “bad” cards (including 1 “assassin” card).

Representing meaning with GloVe embeddings

In order for the spymaster to give good clues to the operative, our model needs to be able to understand the meaning of each word. One popular way to represent word meaning is via word embeddings. For this task, we will be using pre-trained GloVe embeddings, where each word is represented by a 100-dimensional vector.

We then score the similarity between two words using cosine similarity, which is the dot product of two vectors divided by their magnitudes:

Operative: Decoding Algorithm

During each turn, the operative receives a clue c, and an integer n representing the number of corresponding words to guess. In other words, the operator has to decode a {c, n} pair and choose n words one at a time without replacement, until a wrong word is reached and the turn ends.

Our decoder is a straightforward greedy algorithm: simply sort all the remaining words on the board based on cosine similarity with the clue word c, and pick the top n words based on similarity score.

Spymaster: Encoding Algorithm

On each turn, based on the remaining “good” and “bad” words, the spymaster has to pick n words and decide on a clue c to give to the operative. One assumption we make here is that the spymaster and operative agree on the decoding strategy mentioned above, and hence the operator will pick the optimal {c, n} that will maximize the number of correct words chosen.

At this point, we can make an observation that the clue c is an information bottleneck, because it has to summarize n words into a single word c for the operative to decode. The word vector of the encoded clue lies in the same vector space as each of the original word vectors.

Generating clue candidates

Word embeddings have the property of enabling us to represent composite meanings via the addition and subtraction of different word vectors. Given set of “good” words G and set of “bad” words B, we can use this property to obtain an “average” meaning of a “good” word, with reference to the “bad” words, by computing a normalized mean of the vectors, with “good” word vectors being added, and “bad” word vectors being subtracted. This average vector enables us to generate clue candidates:

glove = api.load("glove-wiki-gigaword-100")

good_words = [g_1, g_2, ..., g_n]
bad_words = [b_1, b_2, ..., b_n]

candidates = glove.most_similar(positive=good_words,negative=bad_words,topn=20)

Negative sampling

As the number of “bad” words usually exceeds the number of “good” words, we perform negative sampling, by randomly sampling an equal number of “bad” words compared to the “good” words in our computation of our average word vector. This also contributes to more randomness in the clues generated, which improves the diversity of clue candidates.

After we find the average word vector, we use the most_similar() function in Gensim to obtain the top closest words from the entire GloVe vocabulary with respect to the average word vector, based on cosine similarity.

Score function

Now, we have a method to generate candidates for clue c, given n words. However, we still have to decide which candidate c to pick, which n words to choose, and how to determine n.

Thereafter, we generate all possible combinations of k, k-1, …, 1 words from the k “good” words remaining on the board, as well as their respective clue word candidates c, working backwards from k. To pick the best {c, n}, we score all the candidates from each possible combination of the remaining “good” words, through the decoding algorithm. Then we obtain the maximum number of words guessed correct given clue c, count(c), since we know what strategy the operative will use.

Results

In each game, 25 words are sampled from a list of 400 common Codenames words. Overall, after 100 experiments, our method chooses the correct word 100% of the time, completing the game within an average of 1.98 turns, or 4.55 guesses per turn (for 9 correct words), taking at most 2 turns.

In other words, this algorithm takes two turns almost every game, except for a few exceptions where it guesses all the words in a single turn.

Let’s view a sample distribution of word embeddings of the clues and guesses we made.

Although the clues generated do provide some level of semantic summary of the words that the operative eventually guessed correctly, these relationships between clues and guesses may not be obvious from a human perspective. One way to make to clues more interpretable is to cap the maximum number of guesses per turn, which generates clues with better semantic approximation of the guesses.

Even so, our algorithm promotes a good clustering outcome for each of the words, to enable our decoder to get more words correct by providing good clues that lie close to the target words.

Conclusion

In conclusion, this greedy GloVe-based algorithm performs well as both the spymaster and operative in the Codenames game, by offering an effective way to encode and decode words via a clue and number.

In our model, the encoder and decoder share a common strategy, which works in similar ways as a shared encryption key. A possible limitation of this is that the encoder and decoder will not work as well separately, as a human player may not be able to interpret the generated clues as effectively.

Understanding the mechanics behind word embeddings and vector manipulation is a great way to get started in Natural Language Processing. It is interesting to see how simple methods can also perform well in such semantic clustering and classification tasks. To enhance the gameplay even further, one can consider using adding elements of reinforcement learning or training an autoencoder to achieve better results.

Github Repository: https://github.com/jzh001/codenames

References

1. Koyyalagunta, D., Sun, A., Draelos, R. L., & Rudin, C. (2021). Playing codenames with language graphs and word embeddings. Journal of Artificial Intelligence Research, 71, 319–346. https://doi.org/10.1613/jair.1.12665
2. Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/10.3115/v1/d14-1162
3. Li, Y., Yan, X., & Shaw, C. (2022). Codenames AI https://xueweiyan.github.io/codenames-ai-website/
4. Friedman, D., & Panigrahi, A. (2021). Algorithms for Codenames https://www.cs.princeton.edu/~smattw/Teaching/521FA21/FinalProjectReports/FriedmanPanigrahi.pdf
5. Jaramillo, C., Charity, M., Canaan , R., & Togelius, J. (2020). Word Autobots: Using Transformers for Word Association in the Game Codenames. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 16(1), 231–237. https://doi.org/10.1609/aiide.v16i1.7435

---

Hacking “Codenames” with GloVe Embeddings was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.