Tag: AI

Host production-ready LLMs endpoints at twice the speed but one fifth the cost.

Disclosure: I am a Data Engineer with Singapore’s Government Technology Agency (GovTech) Data Science and Artificial Intelligence Division (DSAID). As one of the key developers working on Analytics.gov, I work with agencies across the entire public sector to develop Data Science and AI/ML capabilities for public good.

Table of Contents

1. Preamble
2. Why use open-source models?
3. Blockers for Hosting Open-source LLMs
4. What is quantisation and how can it help?
5. How do AWS Sagemaker Endpoints work?
6. Hosting a Quantised Model in AG Sagemaker
7. Benchmarks
8. Conclusion

1. Preamble

If you haven’t read our previous publications, you can peruse them here!

• Accelerating Machine Learning and AI impact with MLOps on Analytics.gov
• Productionising LLMs and ML Models with Analytics.gov: MOM’s Journey into AI Solution Deployment

Analytics.gov (AG), developed by GovTech Singapore’s Data Science and Artificial Intelligence Division (DSAID), is a Central Machine Learning Operations (MLOps) platform that productionises ML and AI use cases for the Whole-of-Government (WOG). Hosted on Government Commercial Cloud (GCC) 2.0, it utilises best-practice network and security configurations to provide a safe and secure environment for all data science and AI needs. Through AG, government officers are able to access compute resources, managed AI services and other utilities directly from their government issued laptops without the need for managing or developing new infrastructure, thereby fast-tracking AI/ML initiatives across the whole of government.

AG provides custom functionalities to create and manage production-ready inference endpoints for quantised models through the capabilities offered by AWS Sagemaker Endpoints. With just a few lines of code, end users can quickly set up their own private inference endpoints for quantised models, reducing what could have taken days or weeks of work into mere minutes. This substantially lowers the barrier of entry for agencies across the whole of government to leverage the power of GenAI with greater efficiency and cost-effectiveness.

In this article, we will explore how AG enables government agencies to run LLMs efficiently and cost-effectively. Our goal is to demystify model quantisation, illustrate how we streamlined the process of hosting quantised open-source LLMs in AWS Sagemaker, and provide benchmarks to gauge the gains in performance and cost-efficiency.

2. Why use open-source models?

For a brilliant read on Open LLMs, please view Sau Sheong’s publication here! (Note: its a medium member-only story)

Programming with AI — Open LLMs

I highly recommend it, as it sheds light on hosting open-source LLMs as APIs, providing a great complement to this article.

Security & Sensitivity

Open-source models can be hosted privately on your own devices or cloud environments, meaning that queries to your model do not get sent to third-party providers. This is particularly crucial with government data, as a large majority of it contains sensitive information.

Controlled Output Generation

Usage of open-sourced models can be controlled on a more granular level. Closed-sourced models have to be interfaced via exposed commercial APIs which abstracts out complexity but reduces the degree of control over the model. Locally hosted open-sourced models allow for full control over the output generation, this is important as many useful libraries such as LMQL and Guidance work better with locally hosted models.

Variety

As of writing, there are over 600k models in HuggingFace, including the models posted by major players such as Meta and Google and individual contributors who publish their own variants. Some variants are fine-tuned for specific purposes/tasks, which can be used out of the box. Users can simply reuse these models instead of fine-tuning their own.

For example, AiSingapore’s SEA-LION model is instruct-tuned for the Southeast Asia (SEA) region languages, where its training dataset consists of diverse languages from Malay to Thai. Utilising this model would save the effort in obtaining large amounts of datasets in different languages and computational cost of fine-tuning.

3. Blockers for Hosting Open-source LLMs

Language Models come in many shapes and sizes, popular models range from TinyLlama (1.1B) to the upcoming Llama-3 400B+. While Small Language Models (SLM) like TinyLlama works well for smaller and more straightforward use cases, complex use cases usually require the “smarter” Large Language Models (LLM). It goes without saying that all GenAI applications would benefit from having better output quality from the larger LLMs, however with extra size also comes with extra tradeoffs.

To maximise the speed of inference, models have to be fully loaded in GPU memory as any movement between disk and GPU memory or CPU and GPU memory would introduce overheads that can substantially slow down inference speeds.

LLMs require massive amounts of memory to host, the bigger the LLM, the more GPU memory is required to host it. Most large models demand multiple GPUs to fully host in memory, making it an extremely resource intensive and expensive task.

Naturally, as the size of the model increases, more computation is required for each inference task. Consequently, the larger the LLMs, the lower the inference speed.

Just how big are these models?

The size of these LLMs can be estimated with the following formula (Note, this is a naive estimation and model sizes are almost always slightly larger.)

Using the formula we can estimate the model size for some popular models:

Note: The formula merely estimates the model size, real world GPU requirements will certainly be much larger and are different depending on other factors. (As you will see in the later section on Benchmarks, the actual GPU requirements completely blows these estimates out of the water). “BF16” stands for the number format brain float 16, while “FP16” stands for floating point 16.

The upcoming Meta’s Llama-3 400B+ will be one of the biggest open-source models available when it is released. We can estimate that this beast would be as big as 800 GB. For context, 800 GB would require at least 10 x A100 80GB GPU cards to host even if we naively assume zero hosting overheads.

Another popular but more reasonably sized model — Llama-3 70B published at bf16 or 16 bits per weight (bpw) precision, would still require 141.2 GB of GPU memory to host for inference.

Why is Large GPU Memory Requirements an issue?

As GPUs are in short supply and high demand currently, it’s not easy to find multiple GPU chips for cheap. Hosting LLMs in their raw and unquantised format can thus be a very expensive business that is only available to the privileged few that can afford it. This can be limiting for projects that require the wisdom of LLMs but is not valuable enough to warrant the use of multiple scarce and expensive GPUs.

Slower inference speeds from larger LLM sizes also results in:

1. Worse user experience due to slow output.
2. Reduced total possible throughput that can be extracted by downstream applications. For applications that are heavy on token usage such as text-summarisation or report generation, the reduced throughput can seriously hurt the viability of the application.

Slow inference and expensive costs are debilitating factors for production-grade use cases, hence each GenAI application will need to make the tradeoff between output quality, inference speed and cost.

4. What is quantisation and how can it help?

What is Quantisation?

For a more rigorous explanation on Quantisation, please refer to to these two fantastic guides: https://www.tensorops.ai/post/what-are-quantized-llms, https://www.semianalysis.com/p/neural-network-quantization-and-number

For simplicity, the following section will only refers to Post-Training Quantisation (PTQ)

In simple terms, in the domain of AI/ML, Quantisation is a technique for reducing the size of a model. Underneath the hood, model weights are stored as numbers. Typically these weights are stored in number formats like floating point 16 (FP16) or brain float 16 (BF16), which as the name suggests, takes 16 bits to store a number.

Quantisation reduces the number of bits required to store each number, this allows the storage size of the model to be reduced as less bits are used to store each model weight.

However, using fewer bits per weight means the precision of the weights is reduced. This is why Quantisation is aptly described by most articles as “reducing the precision of model weights”.

For visual learners here is π represented in different precisions:

You can try for yourself using this floating point calculator.

Note: Modern quantisation methods may use bespoke number formats rather than FP series to quantise models. These can go as low as 1 bit quantisation (Q1).

As seen in the table, the precision of π is reduced as the number of bits decreases. This not only affects the number of decimal places, but also in the approximation of the number itself.

For example, 3.141592502593994 cannot be represented exactly in FP8, so it has to be rounded off to the nearest possible value that FP8 can represent — 3.125, this is also known as Floating Point Error.

How does it help?

As the number of bits per weight decreases, total GPU memory requirement is also reduced. For instance, a FP16 to 8-bit Quantisation (Q8) reduces the amount of bits required to store each number from 16 bits to 8 bits. This reduces the size of the model by 50%.

To put this in an example, an unquantised FP16 Mistral 7B is estimated to be about 14.48 GB in size, while a Q8 Mistral 7B is only 7.24 GB. A Q4 Mistral 7b will only be a mere 3.62 GB, making it possible to load into some mobile devices.

Not only does reduction in memory reduce the minimum computation requirements to host a model, it also improves inference speeds.

What’s the catch?

Of course there is no free lunch in this world! Reduction in precision will impact the output quality of the model. Relating to our previous Table on Representation of π, a π represented in FP16 would probably be accurate enough for passing a math test, but a FP8 π will give you an F.

Luckily most LLMs are not too sensitive to reduction at higher precisions. As a general rule of thumb, 8-bit Quantisation or Q8 models are nearly as good as the raw ones. This is shown in the following benchmarks from “How Good Are Low-bit Quantized LLAMA3 Models? An Empirical Study”.

In short, this means that you can get a 50% reduction in model size for almost free just by quantising model weights to Q8.

For a 75% reduction in model size, i.e Q4, the model is still decent using the smarter quantisation techniques like AWQ, albeit with visible loss in quality.

Anything below Q4 and you may run into severe degradation of model output quality.

Do note that the effects of quantisation on model quality may vary from model to model. The best way to determine the best quantisation level is really based on your own usage and testing.

What Quantisation Framework to choose?

For more rigorous discourse on choosing Quantisation frameworks please see: https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/ , https://www.reddit.com/r/LocalLLaMA/comments/1anb2fz/guide_to_choosing_quants_and_engines/

There are many quantisation frameworks available, some of the more popular ones are GGUF, GPTQ, EXL2 and AWQ. The best quantisation framework for you will depend on your use case. The following are my personal recommendations from what I’ve observed in my usage. What’s best for you will depend on your use case and your mileage may vary.

GGUF

Created by Georgi Gerganov with the goal of enabling LLM inference with minimal setup and state-of-the-art performance on any hardware locally or in the cloud, GGUF has become a mainstay for AI/ML enthusiasts looking to host LLMs due to its ease of use.

If you need to host models on commodity hardware or CPU only systems, then GGUF is the most suitable as it is the only framework that has CPU hosting support. GGUF also allows you to run newer models on older GPUs as well. GGUF is also the most stable framework due to how it packages the model weights as a single file in a unified format. If you need to host a quantised model reliably on any machine i.e. even your laptop, then GGUF is the way to go.

The caveat for GGUF is that it’s older quants (Qx_0) uses more simple methods of quantisation such as round-to-nearest (RTN) quantisation. This may reduce model output quality to some extent, but it’s less affected at higher quantisation levels. Newer quantisation methods in GGUF (Qx_K or IQx_S) are better at preserving model quality at lower quantisation levels.

GPTQ, EXL2 and AWQ

GPTQ, EXL2 and AWQ are specialised for GPU usage, they are all based on the GPTQ format. These frameworks tend to be much faster than GGUF as they are specially optimised for running on GPU. EXL2 allows for mixing quantisation levels within a model. AWQ tends to have the best output quality as it uses even “smarter” quantisation techniques than GPTQ. Both EXL2 and AWQ attempt to reduce degradation at lower quantisation levels. GPTQ tends to be the most supported for downstream inference engines.

In conclusion, choose GGUF for ease of hosting, EXL2 for mixed quantisation levels, AWQ for output quality and GPTQ if your choice of inference engine does not support the rest.

5. How do AWS Sagemaker Endpoints work?

Now that we understand what quantisation is, how do we bring it into our users on AG’s AWS Sagemaker so that they will be able to host their own production-ready models inference endpoints for their use case?

What are Sagemaker Endpoints?

AWS Sagemaker Endpoints are the native tools within AWS Sagemaker to host model inference. Its advantages are:

1. Easy to configure Auto Scaling: It only takes a few lines to add auto scaling to existing endpoints.
2. Zero Downtime Updates: Updates to Sagemaker Endpoints uses BlueGreen Deployment by default.
3. Flexibility & Customisation: Sagemaker Endpoints are able to use customised containers.
4. Access to AWS Services: Sagemaker Endpoints are able to access AWS services like S3 which can allow for more flexibility in adding additional steps to process inference requests.

This helps to save time and expertise for users who just want to deploy a model and not think about the engineering work required to manage it on a production scale, turning what could be days/weeks of work into mere minutes.

How does Sagemaker Endpoints work?

Underneath the hood, Sagemaker Endpoints utilises special inference containers based on the Sagemaker-Inference-Toolkit library for hosting model APIs. These containers provide a quick and easy method of running inference without needing to build your own container images and supports many different frameworks from simple scikit-learn models using their scikit-learn container to even complex LLMs (and also their AWQ/GPTQ quantised variants) using the TensorRT-LLM container.

However GGUF and EXL2 quants will still require heavy customised inference frameworks. Thankfully, Sagemaker provides the flexibility to use custom containers and Sagemaker Endpoints make it very simple to do so. There are only a few details to keep in mind to make this work:

1. Container must listen on port 8080.
2. Container must respond to /ping and /invocations
3. Container will be run with the ‘docker run <image> serve’ command, containers are expected to use ENTRYPOINT instead of CMD
4. Model artifacts are brought into the ‘/opt/ml/model’ direction by specifying the S3 path to a tar.gz containing the model artifacts. This happens right before the runtime of the container.

Customise for an open-source inference engine

The above diagram represents a container pre-packed with Sagemaker-Inference-Toolkit. To use our own serving engine, we can simply replace the pre-packed packages with our own custom packages.

For instance, one of the custom containers we curated enables users to host GGUF models through using Abetlen’s Llama-cpp-python as the inference engine. This library is open-source and under the permissive MIT license.

In our dockerfile, we only needed to write a few lines of code to conform to sagemaker endpoint requirements:

1. Change listening port to 8080
2. Add routes for /ping and /invocations
3. Run on ENTRYPOINT

6. Hosting a Quantised Model in AG Sagemaker

Using the custom containers, hosting a quantised LLM in AG’s Sagemaker environment is reduced down to a few lines of code.

# Code will vary depending on how you have curated your own custom container.

from sagemaker.model import Model

endpoint_name = "<Name of endpoint>"
image_uri = "<ECR Image URI to Llama-cpp-python Image>"
model_artifact_location = "<S3 Path to Model Artifacts>"
model = "<Path to model file>"


# All other ENV variables defined in documentation
model_endpoint = Model(
  image_uri = image_uri,
  model_data = model_artifact_location,
  role = role,
  env = {
    "MODEL": model_file_path_in_container,
    "N_GPU_LAYERS": "999",
    "INVOCATIONS_ROUTE": "/v1/completions"
  }
)

model_endpoint.deploy(
  initial_instance_count=1,
  instance_type="ml.g4dn.xlarge",
  endpoint_name=endpoint_name
)

That’s it, short and simple. With this, our users can focus on developing their LLM use cases without being encumbered by the complexity behind the scenes.

7. Benchmarks

The following are some benchmarks for the average tokens generated per second based on single query inference tested 5 times over 30 prompts i.e. each candidate is based on an average of 150 tests. For all tests, we used the CodeLlama model as it is available in many sizes, namely 7, 13, 34 and 70 billion parameters. We tested both quantised and unquantised models with different inference engines, using Transformers as the baseline as it’s typically the normal way for running unquantised models.

The following are the specifications for the benchmarking:

Note ExllamaV2 refers to the inference engine, while EXL2 is the quantisation format native to the ExllamaV2, in this case, ExllamaV2 also supports inference for GPTQ. ExllamaV2 will only be benchmarked with Q4_0 as some Q8_0 quants are not found on HuggingFace.

Unquantised via Transformers (Baseline)

BF16:

All multiples in the following tests are based on using Transformers as a baseline. For instance, the GPTQ 7b Q4_0 model has a “(3.42x)” multiple in the “Tokens per second” column, this means that GPTQ is 3.42 times as fast as the Transformers baseline for the 7b model.

GGUF via Llama-cpp-python

GGUF can support hosting on older Nvidia T4s from the g4dn instance families, so we added extra tests that optimises for cost using g4dn instance types when possible:

Q4_0

Q8_0

Using newer Nvidia A10g from the g5 instance family:

Q4_0

Q8_0

In every single case, GGUF can run the Models much cheaper or at the same price but significantly faster. For instance, the Q8 13B model is 74% faster than the baseline but at one fifth the cost!

GPTQ — Via ExllamaV2

ExllamaV2 only supports the newer hosting on the newer Nvidia A10g from the g5 instance family and not the g4dn instance family.

Q4_0

GPTQ on ExllamaV2 takes the performance improvements to a whole new level, with more than triple the speeds from the baseline for every model size quantised in Q4_0.

AWS Sagemaker Jumpstart

Natively AWS also provides a service called JumpStart that allows deployment of pretrained models with a few clicks. These AWS Sagemaker containers implement the Sagemaker Inference Toolkit and have various inference engines pre-installed. In this case, it’s using the HuggingFace’s Text Generation Inference (TGI) Framework as the inference engine.

BF16:

Notice how 13B is faster than 7B. This is because the TGI container is able to utilise more GPU memory to increase the speed of inference. On larger parameter sizes like 34B and 70B, using AWS Sagemaker Jumpstart with TGI containers can even outperform GPTQ on ExllamaV2.

8. Conclusion

Quantisation offers substantial benefits for LLMs as it reduces memory requirements for hosting them. The reduction in memory requirements increases inference speeds and reduces costs. Higher bit quantisation can be achieved with almost zero loss in output quality, substantial gains in speed and reduced cost — essentially a Pareto improvement over using unquantised LLMs.

With auxiliary functionalities provided by AG on top of AWS Sagemaker Endpoints, agencies across the entire public sector can easily access capabilities to create and manage production-ready quantised Open LLM APIs. By streamlining the process of deploying quantised large language models, AG significantly lowers the barrier of entry for producing efficient and cost-effective GenAI applications, allowing government agencies to focus on innovating and developing technology for public good.

Dovetailing with this, AG will continue to further its GenAI endeavours by providing access to closed-source models like Azure OpenAI and VertexAI’s Gemini via secured cross-cloud integration, alongside our existing services with AWS Bedrock. Through robust and comprehensive offerings, AG empowers users to rightsize models for their use cases, resulting in better, faster and cheaper GenAI applications in the public sector.

References

[1] Sau Sheong, Programming with AI — Open LLMs (2024), https://sausheong.com/programming-with-ai-open-llms-28091f77a088

[2] S. Stoelinga, Calculating GPU memory for serving LLMs (2023), https://www.substratus.ai/blog/calculating-gpu-memory-for-llm/

[3] M.C. Neves, What are Quantized LLMs? (2023), https://www.tensorops.ai/post/what-are-quantized-llms

[4] D. Patel, Neural Network Quantization & Number Formats From First Principles (2024), https://www.semianalysis.com/p/neural-network-quantization-and-number

[5] W. Huang, How Good Are Low-bit Quantized LLAMA3 Models? An Empirical Study (2024), https://arxiv.org/pdf/2404.14047

[6] Oobabooga, A detailed comparison between GPTQ, AWQ, EXL2, q4_K_M, q4_K_S, and load_in_4bit: perplexity, VRAM, speed, model size, and loading time. (N.A.), https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/

[7] Sgsdxzy, Guide to choosing quants and engines (2024), https://www.reddit.com/r/LocalLLaMA/comments/1anb2fz/guide_to_choosing_quants_and_engines/

[8] Amazon Web Services, Use Your Own Inference Code with Hosting Services (N.A.), https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html

---

Applied LLM Quantisation with AWS Sagemaker | Analytics.gov was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Top 10 engineering lessons every data analyst should know

The data analyst job combines skills from different domains:

• We need to have business understanding and domain knowledge to be able to solve actual business problems and take into account all the details.
• Maths, statistics, and fundamental machine learning skills help us perform rigorous analyses and reach reliable conclusions from data.
• Visualisation skills and storytelling allow us to deliver our message and influence the product.
• Last but not least, computer science and the basics of software engineering are key to our efficiency.

I’ve learned a lot about computer science at university. I’ve tried at least a dozen programming languages (from low-level assembler and CUDA to high-level Java and Scala) and countless tools. My very first job offer was for a backend engineer role. I’ve decided not to pursue this path, but all this knowledge and principles have been beneficial in my analytical career. So, I would like to share the main principles with you in this article.

Code is not for computers. It’s for people

I’ve heard this mantra from software engineers many times. It’s well explained in one of the programming bibles, “Clean Code”.

Indeed, the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code.

In most cases, an engineer prefers more wordy code that is easy to understand to the idiomatic one-liner.

I must confess that I sometimes break this rule and write extra-long pandas one-liners. For example, let’s look at the code below. Do you have any idea what this code is doing?

# ad-hoc only code
df.groupby(['month', 'feature'])[['user_id']].nunique()
  .rename(columns = {'user_id': 'users'})
  .join(df.groupby(['month'])[['user_id']].nunique()
  .rename(columns = {'user_id': 'total_users'})).apply(
    lambda x: 100*x['users']/x['total_users'], axis = 1)
  .reset_index().rename(columns = {0: 'users_share'})
  .pivot(index = 'month', columns = 'feature', values = 'users_share')

Honestly, it’ll probably take me a bit to get up to speed with this code in a month. To make this code more readable, we can split it into steps.

# maintainable code
monthly_features_df = df.groupby(['month', 'feature'])[['user_id']].nunique()
    .rename(columns = {'user_id': 'users'})

monthly_total_df = df.groupby(['month'])[['user_id']].nunique()
    .rename(columns = {'user_id': 'total_users'})

monthly_df = monthly_features_df.join(monthly_total_df).reset_index()
monthly_df['users_share'] = 100*monthly_df.users/monthly_df.total_users

monthly_df.pivot(index = 'month', columns = 'feature', values = 'users_share')

Hopefully, now it’s easier for you to follow the logic and see that this code shows the percentage of customers that use each feature every month. The future me would definitely be way happier to see a code like this and appreciate all the efforts.

Automate repetitive tasks

If you have monotonous tasks that you repeat frequently, I recommend you consider automation. Let me share some examples from my experience that you might find helpful.

The most common way for analysts to automate tasks is to create a dashboard instead of calculating numbers manually every time. Self-serve tools (configurable dashboards where stakeholders can change filters and investigate the data) can save a lot of time and allow us to focus on more sophisticated and impactful research.

If a dashboard is not an option, there are other ways of automation. I was doing weekly reports and sending them to stakeholders via e-mail. After some time, it became a pretty tedious task, and I started to think about automation. At this point, I used the basic tool — cron on a virtual machine. I scheduled a Python script that calculated up-to-date numbers and sent an e-mail.

When you have a script, you just need to add one line to the cron file. For example, the line below will execute analytical_script.py every Monday at 9:10 AM.

10 9 * * 1 python analytical_script.py

Cron is a basic but still sustainable solution. Other tools that can be used to schedule scripts are Airflow, DBT, and Jenkins. You might know Jenkins as a CI/CD (continuous integration & continuous delivery) tool that engineers often use. It might surprise you. It’s customisable enough to execute analytical scripts as well.

If you need even more flexibility, it’s time to think about web applications. In my first team, we didn’t have an A/B test tool, so for a long time, analysts had to analyse each update manually. Finally, we wrote a Flask web application so that engineers could self-serve. Now, there are lightweight solutions for web applications, such as Gradio or Streamlit, that you can learn in a couple of days.

You can find a detailed guide for Gradio in one of my previous articles.

Master your tools

Tools you use every day at work play a significant role in your efficiency and final results. So it’s worth mastering them.

Of course, you can use a default text editor to write code, but most people use IDEs (Integrated Development Environment). You will be spending a lot of your working time on this application, so it’s worth assessing your options.

You can find the most popular IDEs for Python from the JetBrains 2021 survey.

I usually use Python and Jupyter Notebooks for my day-to-day work. In my opinion, the best IDE for such tasks is JupyterLab. However, I’m trying other options right now to be able to use AI assistants. The benefits of auto-completion, which eliminates lots of boilerplate code, are invaluable for me, so I’m ready to take on switching costs. I encourage you to investigate different options and see what suits your work best.

The other helpful hack is shortcuts. You can do your tasks way faster with shortcuts than with a mouse, and it looks cool. I would start with Googling shortcuts for your IDE since you usually use this tool the most. From my practice, the most valuable commands are creating a new cell in a Notebook, running this cell, deleting it, and converting the cell into markdown.

If you have other tools that you use pretty often (such as Google Sheets or Slack), you can also learn commands for them.

The main trick with learning shortcuts is “practice, practice, practice” — you need to repeat it a hundred times to start doing it automatically. There are even plugins that push you to use shortcuts more (for example, this one from JetBrains).

Last but not least is CLI (command-line interface). It might look intimidating in the beginning, but basic knowledge of CLI usually pays off. I use CLI even to work with GitHub since it gives me a clear understanding of what’s going on exactly.

However, there are situations when it’s almost impossible to avoid using CLI, such as when working on a remote server. To interact confidently with a server, you need to learn less than ten commands. This article can help you gain basic knowledge about CLI.

Manage your environment

Continuing the topic of tools, setting up your environment is always a good idea. I have a Python virtual environment for day-to-day work with all the libraries I usually use.

Creating a new virtual environment is as easy as a couple of lines of code in your terminal (an excellent opportunity to start using CLI).

# creating venv
python -m venv routine_venv

# activating venv
source routine_venv/bin/activate

# installing ALL packages you need 
pip install pandas plotly 

# starting Juputer Notebooks
jupyter notebook

You can start your Jupyter from this environment or use it in your IDE.

It’s a good practice to have a separate environment for big projects. I usually do it only if I need an unusual stack (like PyTorch or yet another new LLM framework) or face some issues with library compatibility.

The other way to save your environment is by using Docker Containers. I use it for something more production-like, like web apps running on the server.

Think about program performance

To tell the truth, analysts often don’t need to think much about performance. When I got my first job in data analytics, my lead shared the practical approach to performance optimisations (and I have been using it ever since). When you’re thinking about performance, consider the total time vs efforts. Suppose I have a MapReduce script that runs for 4 hours. Should I optimise it? It depends.

• If I need to run it only once or twice, there’s not much sense in spending 1 hour to optimise this script to calculate numbers in just 1 hour.
• If I plan to run it daily, it’s worth the effort to make it faster and stop wasting computational resources (and money).

Since the majority of my tasks are one-time research, in most cases, I don’t need to optimise my code. However, it’s worth following some basic rules to avoid waiting for hours. Small tricks can lead to tremendous results. Let’s discuss such an example.

Starting from the basics, the cornerstone of performance is big O notation. Simply put, big O notation shows the relation between execution time and the number of elements you work with. So, if my program is O(n), it means that if I increase the amount of data 10 times, execution will be ~10 times longer.

When writing code, it’s worth understanding the complexity of your algorithm and the main data structures. For example, finding out if an element is in a list takes O(n) time, but it only takes O(1) time in a set. Let’s see how it can affect our code.

I have 2 data frames with Q1 and Q2 user transactions, and for each transaction in the Q1 data frame, I would like to understand whether this customer was retained or not. Our data frames are relatively small — around 300-400K rows.

As you can see, performance differs a lot.

• The first approach is the worst one because, on each iteration (for each row in the Q1 dataset), we calculate the list of unique user_ids. Then, we look up the element in the list with O(n) complexity. This operation takes 13 minutes.
• The second approach, when we calculate the list first, is a bit better, but it still takes almost 6 minutes.
• If we pre-calculate a list of user_ids and convert it into the set, we will get the result in a blink of an eye.

As you can see, we can make our code more than 10K times faster with just basic knowledge. It’s a game-changer.

The other general advice is to avoid using plain Python and prefer to use more performant data structures, such as pandas or numpy. These libraries are faster because they use vectorised operations on arrays, which are implemented on C. Usually, numpy would show a bit better performance since pandas is built on top of numpy but has some additional functionality that slows it down a bit.

Don’t forget the DRY principle.

DRY stands for “Don’t Repeat Yourself” and is self-explanatory. This principle praises structured modular code that you can easily reuse.

If you’re copy-pasting a chunk of code for the third time, it’s a sign to think about the code structure and how to encapsulate this logic.

The standard analytical task is data wrangling, and we usually follow the procedural paradigm. So, the most apparent way to structure the code is functions. However, you might follow objective-oriented programming and create classes. In my previous article, I shared an example of the objective-oriented approach to simulations.

The benefits of modular code are better readability, faster development and easier changes. For example, if you want to change your visualisation from a line chart to an area plot, you can do it in one place and re-run your code.

If you have a bunch of functions related to one particular domain, you can create a Python package for it to interact with these functions as with any other Python library. Here’s a detailed guide on how to do it.

Leverage testing

The other topic that is, in my opinion, undervalued in the analytical world is testing. Software engineers often have KPIs on the test coverage, which might also be useful for analysts. However, in many cases, our tests will be related to the data rather than the code itself.

The trick I’ve learned from one of my colleagues is to add tests on the data recency. We have multiple scripts for quarterly and annual reports that we run pretty rarely. So, he added a check to see whether the latest rows in the tables we’re using are after the end of the reporting period (it shows whether the table has been updated). In Python, you can use an assert statement for this.

assert last_record_time >= datetime.date(2023, 5, 31) 

If the condition is fulfilled, then nothing will happen. Otherwise, you will get an AssertionError . It’s a quick and easy check that can help you spot problems early.

The other thing I prefer to validate is sum statistics. For example, if you’re slicing, dicing and transforming your data, it’s worth checking that the overall number of requests and metrics stays the same. Some common mistakes are:

• duplicates that emerged because of joins,
• filtered-out None values when you’re using pandas.groupby function,
• filtered-out dimensions because of inner joins.

Also, I always check data for duplicates. If you expect that each row will represent one user, then the number of rows should be equal to df.user_id.nunique() . If it’s false, something is wrong with your data and needs investigation.

The trickiest and most helpful test is the sense check. Let’s discuss some possible approaches to it.

• First, I would check whether the results make sense overall. For example, if 1-month retention equals 99% or I got 1 billion customers in Europe, there’s likely a bug in the code.
• Secondly, I will look for other data sources or previous research on this topic to validate that my results are feasible.
• If you don’t have other similar research (for example, you’re estimating your potential revenue after launching the product in a new market), I would recommend you compare your numbers to those of other existing segments. For example, if your incremental effect on revenue after launching your product in yet another market equals 5x current income, I would say it’s a bit too optimistic and worth revisiting assumptions.

I hope this mindset will help you achieve more feasible results.

Encourage the team to use Version Control Systems

Engineers use version control systems even for the tiny projects they are working on their own. At the same time, I often see analysts using Google Sheets to store their queries. Since I’m a great proponent and advocate for keeping all the code in the repository, I can’t miss a chance to share my thoughts with you.

Why have I been using a repository for 10+ years of my data career? Here are the main benefits:

• Reproducibility. Quite often, we need to tweak the previous research (for example, add one more dimension or narrow research down to a specific segment) or just repeat the earlier calculations. If you store all the code in a structured way, you can quickly reproduce your prior work. It usually saves a lot of time.
• Transparency. Linking code to the results of your research allows your colleagues to understand the methodology to the tiniest detail, which brings more trust and naturally helps to spot bugs or potential improvements.
• Knowledge sharing. If you have a catalogue that is easy to navigate (or you link your code to Task Trackers), it makes it super-easy for your colleagues to find your code and not start an investigation from scratch.
• Rolling back. Have you ever been in a situation when your code was working yesterday, but then you changed something, and now it’s completely broken? I’ve been there many times before I started committing my code regularly. Version Control systems allow you to see the whole version history and compare the code or rollback to the previous working version.
• Collaboration. If you’re working on the code in collaboration with others, you can leverage version control systems to track and merge the changes.

I hope you can see its potential benefits now. Let me briefly share my usual setup to store code:

• I use git + Github as a version control system, I’m this dinosaur who is still using the command line interface for git (it gives me the soothing feeling of control), but you can use the GitHub app or the functionality of your IDE.
• Most of my work is research (code, numbers, charts, comments, etc.), so I store 95% of my code as Jupyter Notebooks.
• I link my code to the Jira tickets. I usually have a tasks folder in my repository and name subfolders as ticket keys (for example, ANALYTICS-42). Then, I place all the files related to the task in this subfolder. With such an approach, I can find code related to (almost) any task in seconds.

There are a bunch of nuances of working with Jupyter Notebooks in GitHub that are worth noting.

First, think about the output. When committing a Jupyter Notebook to the repository, you save input cells (your code or comments) and output. So, it’s worth being conscious about whether you actually want to share the output. It might contain PII or other sensitive data that I wouldn’t advise committing. Also, the output might be pretty big and non-informative, so it will just clutter your repository. When you’re saving 10+ MB Jupyter Notebook with some random data output, all your colleagues will load this data to their computers with the next git pull command.

Charts in output might be especially problematic. We all like excellent interactive Plotly charts. Unfortunately, they are not rendered on GitHub UI, so your colleagues likely won’t see them. To overcome this obstacle, you might switch the output type for Plotly to PNG or JPEG.

import plotly.io as pio
pio.renderers.default = "jpeg"

You can find more details about Plotly renderers in the documentation.

Last but not least, Jupyter Notebooks diffs are usually tricky. You would often like to understand the difference between 2 versions of the code. However, the default GitHub view won’t give you much helpful info because there is too much clutter due to changes in notebook metadata (like in the example below).

Actually, GitHub has almost solved this issue. A rich diffs functionality in feature preview can make your life way easier — you just need to switch it on in settings.

With this feature, we can easily see that there were just a couple of changes. I’ve changed the default renderer and parameters for retention curves (so a chart has been updated as well).

Ask for a code review

Engineers do peer reviews for (almost) all changes to the code. This process allows one to spot bugs early, stop bad actors or effectively share knowledge in the team.

Of course, it’s not a silver bullet: reviewers can miss bugs, or a bad actor might introduce a breach into the popular open-source project. For example, there was quite a scary story of how a backdoor was planted into a compression tool widely used in popular Linux distributions.

However, there is evidence that code review actually helps. McConnell shares the following stats in his iconic book “Code Complete”.

… software testing alone has limited effectiveness — the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent.

Despite all these benefits, analysts often don’t use code review at all. I can understand why it might be challenging:

• Analytical teams are usually smaller, and spending limited resources on double-checking might not sound reasonable.
• Quite often, analysts work in different domains, and you might end up being the only person who knows this domain well enough to do a code review.

However, I really encourage you to do a code review, at least for critical things to mitigate risks. Here are the cases when I ask colleagues to double-check my code and assumptions:

• When I’m using data in a new domain, it’s always a good idea to ask an expert to review the assumptions used;
• All the tasks related to customer communications or interventions since errors in such data might lead to significant impact (for example, we’ve communicated wrong information to customers or deactivated wrong people);
• High-stakes decisions: if you plan to invest six months of the team’s effort into the project, it’s worth double- and triple-checking;
• When results are unexpected: the first hypothesis to test when I see surprising results is to check for an error in code.

Of course, it’s not an exhaustive list, but I hope you can see my reasoning and use common sense to define when to reach out for code review.

Stay up-to-date

The famous Lewis Caroll quote represents the current state of the tech domain quite well.

… it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that.

Our field is constantly evolving: new papers are published every day, libraries are updated, new tools emerge and so on. It’s the same story for software engineers, data analysts, data scientists, etc.

There are so many sources of information right now that there’s no problem to find it:

• weekly e-mails from Towards Data Science and some other subscriptions,
• following experts on LinkedIn and X (former Twitter),
• subscribing to e-mail updates for the tools and libraries I use,
• attending local meet-ups.

A bit more tricky is to avoid being drowned by all the information. I try to focus on one thing at a time to prevent too much distraction.

Summary

That’s it with the software engineering practices that can be helpful for analysts. Let me quickly recap them all here:

• Code is not for computers. It’s for people.
• Automate repetitive tasks.
• Master your tools.
• Manage your environment.
• Think about program performance.
• Don’t forget the DRY principle.
• Leverage testing.
• Encourage the team to use Version Control Systems.
• Ask for a code review.
• Stay up-to-date.

Data analytics combines skills from different domains, so I believe we can benefit greatly from learning the best practices of software engineers, product managers, designers, etc. By adopting the tried-and-true techniques of our colleagues, we can improve our effectiveness and efficiency. I highly encourage you to explore these adjacent domains as well.

Thank you a lot for reading this article. I hope this article was insightful for you. If you have any follow-up questions or comments, please leave them in the comments section.

Reference

All the images are produced by the author unless otherwise stated.

Acknowledgements

I can’t miss a chance to express my heartfelt thanks to my partner, who has been sharing his engineering wisdom with me for ages and has reviewed all my articles.

---

From Code to Insights: Software Engineering Best Practices for Data Analysts was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, we are excited to announce that the Jina Embeddings v2 model, developed by Jina AI, is available for customers through Amazon SageMaker JumpStart to deploy with one click for running model inference. This state-of-the-art model supports an impressive 8,192-tokens context length. You can deploy this model with SageMaker JumpStart, a machine learning (ML) hub […]