From tutorial to practical examples of Python Singleton design pattern
Originally appeared here:
You May Know This Design Pattern In Python, But When To Use It?
Go Here to Read this Fast! You May Know This Design Pattern In Python, But When To Use It?
Scripting around a pandas DataFrame can turn into an awkward pile of (not-so-)good old spaghetti code. Me and my colleagues use this package a lot and while we try to stick to good programming practices, like splitting code in modules and unit testing, sometimes we still get in the way of one another by producing confusing code.
I have gathered some tips and pitfalls to avoid in order to make pandas code clean and infallible. Hopefully you’ll find them useful too. We’ll get some help from Robert C. Martin’s classic “Clean code” specifically for the context of the pandas package. TL;DR at the end.
Let’s begin by observing some faulty patterns inspired by real life. Later on, we’ll try to rephrase that code in order to favor readability and control.
Pandas DataFrames are value-mutable [2, 3] objects. Whenever you alter a mutable object, it affects the exact same instance that you originally created and its physical location in memory remains unchanged. In contrast, when you modify an immutable object (eg. a string), Python goes to create a whole new object at a new memory location and swaps the reference for the new one.
This is the crucial point: in Python, objects get passed to the function by assignment [4, 5]. See the graph: the value of df has been assigned to variable in_df when it was passed to the function as an argument. Both the original df and the in_df inside the function point to the same memory location (numeric value in parentheses), even if they go by different variable names. During the modification of its attributes, the location of the mutable object remains unchanged. Now all other scopes can see the changes too — they reach to the same memory location.
Actually, since we have modified the original instance, it’s redundant to return the DataFrame and assign it to the variable. This code has the exact same effect:
Heads-up: the function now returns None, so be careful not to overwrite the df with None if you do perform the assignment: df = modify_df(df).
In contrast, if the object is immutable, it will change the memory location throughout the modification just like in the example below. Since the red string cannot be modified (strings are immutable), the green string is created on top of the old one, but as a brand new object, claiming a new location in memory. The returned string is not the same string, whereas the returned DataFrame was the exact same DataFrame.
The point is, mutating DataFrames inside functions has a global effect. If you don’t keep that in mind, you may:
We’ll fix that problem later, but here is another don’t before we pass to do’s
The design from the previous section is actually an anti-pattern called output argument [1 p.45]. Typically, inputs of a function will be used to create an output value. If the sole point of passing an argument to a function is to modify it, so that the input argument changes its state, then it’s challenging our intuitions. Such behavior is called side effect [1 p.44] of a function and those should be well documented and minimized because they force the programmer to remember the things that go in the background, therefore making the script error-prone.
When we read a function, we are used to the idea of information going in to the function through arguments and out through the return value. We don’t usually expect information to be going out through the arguments. [1 p.41]
Things get even worse if the function has a double responsibility: to modify the input and to return an output. Consider this function:
def find_max_name_length(df: pd.DataFrame) -> int:
df["name_len"] = df["name"].str.len() # side effect
return max(df["name_len"])
It does return a value as you would expect, but it also permanently modifies the original DataFrame. The side effect takes you by surprise – nothing in the function signature indicated that our input data was going to be affected. In the next step, we’ll see how to avoid this kind of design.
To eliminate the side effect, in the code below we have created a new temporary variable instead of modifying the original DataFrame. The notation lengths: pd.Series indicates the datatype of the variable.
def find_max_name_length(df: pd.DataFrame) -> int:
lengths: pd.Series = df["name"].str.len()
return max(lengths)
This function design is better in that it encapsulates the intermediate state instead of producing a side effect.
Another heads-up: please be mindful of the differences between deep and shallow copy [6] of elements from the DataFrame. In the example above we have modified each element of the original df[“name”] Series, so the old DataFrame and the new variable have no shared elements. However, if you directly assign one of the original columns to a new variable, the underlying elements still have the same references in memory. See the examples:
df = pd.DataFrame({"name": ["bert", "albert"]})
series = df["name"] # shallow copy
series[0] = "roberta" # <-- this changes the original DataFrame
series = df["name"].copy(deep=True)
series[0] = "roberta" # <-- this does not change the original DataFrame
series = df["name"].str.title() # not a copy whatsoever
series[0] = "roberta" # <-- this does not change the original DataFrame
You can print out the DataFrame after each step to observe the effect. Remember that creating a deep copy will allocate new memory, so it’s good to reflect whether your script needs to be memory-efficient.
Maybe for whatever reason you want to store the result of that length computation. It’s still not a good idea to append it to the DataFrame inside the function because of the side effect breach as well as the accumulation of multiple responsibilities inside a single function.
I like the One Level of Abstraction per Function rule that says:
We need to make sure that the statements within our function are all at the same level of abstraction.
Mixing levels of abstraction within a function is always confusing. Readers may not be able to tell whether a particular expression is an essential concept or a detail. [1 p.36]
Also let’s employ the Single responsibility principle [1 p.138] from OOP, even though we’re not focusing on object-oriented code right now.
Why not prepare your data beforehand? Let’s split data preparation and the actual computation in separate functions.:
def create_name_len_col(series: pd.Series) -> pd.Series:
return series.str.len()
def find_max_element(collection: Collection) -> int:
return max(collection) if len(collection) else 0
df = pd.DataFrame({"name": ["bert", "albert"]})
df["name_len"] = create_name_len_col(df.name)
max_name_len = find_max_element(df.name_len)
The individual task of creating the name_len column has been outsourced to another function. It does not modify the original DataFrame and it performs one task at a time. Later we retrieve the max element by passing the new column to another dedicated function. Notice how the aggregating function is generic for Collections.
Let’s brush the code up with the following steps:
def compute_length(word: str) -> int:
return len(word)
def prepare_data(df: pd.DataFrame) -> pd.DataFrame:
return pd.concat([
df.copy(deep=True), # deep copy
df.name.apply(compute_length).rename("name_len"),
...
], axis=1)
The way we have split the code really makes it easy to go back to the script later, take the entire function and reuse it in another script. We like that!
There is one more thing we can do to increase the level of reusability: pass column names as parameters to functions. The refactoring is going a little bit over the top, but sometimes it pays for the sake of flexibility or reusability.
def create_name_len_col(df: pd.DataFrame, orig_col: str, target_col: str) -> pd.Series:
return df[orig_col].str.len().rename(target_col)
name_label, name_len_label = "name", "name_len"
pd.concat([
df,
create_name_len_col(df, name_label, name_len_label)
], axis=1)
Did you ever figure out that your preprocessing was faulty after weeks of experiments on the preprocessed dataset? No? Lucky you. I actually had to repeat a batch of experiments because of broken annotations, which could have been avoided if I had tested just a couple of basic functions.
Important scripts should be tested [1 p.121, 7]. Even if the script is just a helper, I now try to test at least the crucial, most low-level functions. Let’s revisit the steps that we made from the start:
1. I am not happy to even think of testing this, it’s very redundant and we have paved over the side effect. It also tests a bunch of different features: the computation of name length and the aggregation of result for the max element. Plus it fails, did you see that coming?
def find_max_name_length(df: pd.DataFrame) -> int:
df["name_len"] = df["name"].str.len() # side effect
return max(df["name_len"])
@pytest.mark.parametrize("df, result", [
(pd.DataFrame({"name": []}), 0), # oops, this fails!
(pd.DataFrame({"name": ["bert"]}), 4),
(pd.DataFrame({"name": ["bert", "roberta"]}), 7),
])
def test_find_max_name_length(df: pd.DataFrame, result: int):
assert find_max_name_length(df) == result
2. This is much better — we have focused on one single task, so the test is simpler. We also don’t have to fixate on column names like we did before. However, I think that the format of the data gets in the way of verifying the correctness of the computation.
def create_name_len_col(series: pd.Series) -> pd.Series:
return series.str.len()
@pytest.mark.parametrize("series1, series2", [
(pd.Series([]), pd.Series([])),
(pd.Series(["bert"]), pd.Series([4])),
(pd.Series(["bert", "roberta"]), pd.Series([4, 7]))
])
def test_create_name_len_col(series1: pd.Series, series2: pd.Series):
pd.testing.assert_series_equal(create_name_len_col(series1), series2, check_dtype=False)
3. Here we have cleaned up the desk. We test the computation function inside out, leaving the pandas overlay behind. It’s easier to come up with edge cases when you focus on one thing at a time. I figured out that I’d like to test for None values that may appear in the DataFrame and I eventually had to improve my function for that test to pass. A bug caught!
def compute_length(word: Optional[str]) -> int:
return len(word) if word else 0
@pytest.mark.parametrize("word, length", [
("", 0),
("bert", 4),
(None, 0)
])
def test_compute_length(word: str, length: int):
assert compute_length(word) == length
4. We’re only missing the test for find_max_element:
def find_max_element(collection: Collection) -> int:
return max(collection) if len(collection) else 0
@pytest.mark.parametrize("collection, result", [
([], 0),
([4], 4),
([4, 7], 7),
(pd.Series([4, 7]), 7),
])
def test_find_max_element(collection: Collection, result: int):
assert find_max_element(collection) == result
One additional benefit of unit testing that I never forget to mention is that it is a way of documenting your code, as someone who doesn’t know it (like you from the future) can easily figure out the inputs and expected outputs, including edge cases, just by looking at the tests. Double gain!
These are some tricks I found useful while coding and reviewing other people’s code. I’m far from telling you that one or another way of coding is the only correct one — you take what you want from it, you decide whether you need a quick scratch or a highly polished and tested codebase. I hope this thought piece helps you structure your scripts so that you’re happier with them and more confident about their infallibility.
If you liked this article, I would love to know about it. Happy coding!
TL;DR
There’s no one and only correct way of coding, but here are some inspirations for scripting with pandas:
Dont’s:
– don’t mutate your DataFrame too much inside functions, because you may lose control over what and where gets appended/removed from it,
– don’t write methods that mutate a DataFrame and return nothing because that’s confusing.
Do’s:
– create new objects instead of modifying the source DataFrame and remember to make a deep copy when needed,
– perform only similar-level operations inside a single function,
– design functions for flexibility and reusability,
– test your functions because this helps you design cleaner code, secure against bugs and edge cases and document it for free.
The graphs were created by me using Miro. The cover image was also created by me using the Titanic dataset and GIMP (smudge effect).
Pandas: From Messy To Beautiful was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
Originally appeared here:
Pandas: From Messy To Beautiful
How a tried and tested solution can yield excellent results in tackling a day-to-day ML problem
Originally appeared here:
Unlocking Insights: Random Forests for PCA and Feature Importance
Go Here to Read this Fast! Unlocking Insights: Random Forests for PCA and Feature Importance
For many researchers, outliers are rogue waves that can dramatically alter the course of the analysis or “confound” some expected effects. I prefer to use the term “extreme observations” and leave the term outlier for observations that are not truly part of the population being studied. For example, in my field (brain ischemia research), an outlier is an animal that does not have ischemia (when it should have), while extreme observations are animals with small or large ischemias that are very different from the others.
Traditional (frequentist) statistical models are built on the strong foundation of Gaussian distributions. This has a significant limitation: an inherent assumption that all data points will cluster around a central mean in a predictable pattern (based on the central limit theorem). This may be true in Plato’s world of ideas, but we, scientists in the biomedical field, are aware it’s challenging to rely on this assumption given the limited sampling (number of animals) we have available to make observations.
Gaussian distributions are very sensitive to extreme observations, and their use leads scientists to believe that eliminating extreme observations is the best way to get “clearer” or “cleaner” results (whatever that means). As I once commented in an article as reviewer 2, “The problem is not the extreme observations that may “hide” your effects, but the fact that you are using a statistical model that (I believe) is inappropriate for your purposes”.
It should be noted that no statistical model is the “right” or “appropriate” one, but we can estimate that, given the data, there are certain statistical models that are more likely to generate the observed data (generative models) than others.
Fortunately, nothing forces us to be bound by the assumptions of the Gaussian models, right? We have other options, such as the Student’s t-distribution (1). I see it as a more adaptable vessel to navigate the turbulent seas of real-world biomedical data. The Student’s t-distribution provides a robust alternative to acknowledge that our data may be populated by extreme observations that are normal biological responses that we can expect in any context. There may be patients or animals that don’t respond or overreact to treatment, and it is valuable that our modeling approach recognizes these responses as part of the spectrum. Therefore, this tutorial explores the modeling strategies using Student’s t-distributions through the lens of the brms package for R (2)—a powerful ally for Bayesian modeling
A Student’s t-distribution is nothing more than a Gaussian distribution with heavier tails. In other words, we can say that the Gaussian distribution is a special case of the Student’s t-distribution. The Gaussian distribution is defined by the mean (μ) and the standard deviation (σ). The Student t distribution, on the other hand, adds an additional parameter, the degrees of freedom (df), which controls the “thickness” of the distribution. This parameter assigns greater probability to events further from the mean. This feature is particularly useful for small sample sizes, such as in biomedicine, where the assumption of normality is questionable. Note that as the degrees of freedom increase, the Student t-distribution approaches the Gaussian distribution. We can visualize this using density plots:
# Load necessary libraries
library(ggplot2)
# Set seed for reproducibility
set.seed(123)
# Define the distributions
x <- seq(-4, 4, length.out = 200)
y_gaussian <- dnorm(x)
y_t3 <- dt(x, df = 3)
y_t10 <- dt(x, df = 10)
y_t30 <- dt(x, df = 30)
# Create a data frame for plotting
df <- data.frame(x, y_gaussian, y_t3, y_t10, y_t30)
# Plot the distributions
ggplot(df, aes(x)) +
geom_line(aes(y = y_gaussian, color = "Gaussian")) +
geom_line(aes(y = y_t3, color = "t, df=3")) +
geom_line(aes(y = y_t10, color = "t, df=10")) +
geom_line(aes(y = y_t30, color = "t, df=30")) +
labs(title = "Comparison of Gaussian and Student t-Distributions",
x = "Value",
y = "Density") +
scale_color_manual(values = c("Gaussian" = "blue", "t, df=3" = "red", "t, df=10" = "green", "t, df=30" = "purple")) +
theme_classic()
Note in Figure 1 that the hill around the mean gets smaller as the degrees of freedom decrease as a result of the probability mass going to the tails, which are thicker. This property is what gives the Student’s t-distribution a reduced sensitivity to outliers. For more details on this matter, you can check this blog.
We load the required libraries:
library(ggplot2)
library(brms)
library(ggdist)
library(easystats)
library(dplyr)
library(tibble)
library(ghibli)
So, let’s skip data simulations and get serious. We’ll work with real data I have acquired from mice performing the rotarod test.
First, we load the dataset into our environment and set the corresponding factor levels. The dataset contains IDs for the animals, a groping variable (Genotype), an indicator for two different days on which the test was performed (day), and different trials for the same day. For this article, we model only one of the trials (Trial3). We will save the other trials for a future article on modeling variation.
As the data handling implies, our modeling strategy will be based on Genotype and Day as categorical predictors of the distribution of Trial3.
In biomedical science, categorical predictors, or grouping factors, are more common than continuous predictors. Scientists in this field like to divide their samples into groups or conditions and apply different treatments.
data <- read.csv("Data/Rotarod.csv")
data$Day <- factor(data$Day, levels = c("1", "2"))
data$Genotype <- factor(data$Genotype, levels = c("WT", "KO"))
head(data)
Let’s have an initial view of the data using Raincloud plots as shown by Guilherme A. Franchi, PhD in this great blog post.
edv <- ggplot(data, aes(x = Day, y = Trial3, fill=Genotype)) +
scale_fill_ghibli_d("SpiritedMedium", direction = -1) +
geom_boxplot(width = 0.1,
outlier.color = "red") +
xlab('Day') +
ylab('Time (s)') +
ggtitle("Rorarod performance") +
theme_classic(base_size=18, base_family="serif")+
theme(text = element_text(size=18),
axis.text.x = element_text(angle=0, hjust=.1, vjust = 0.5, color = "black"),
axis.text.y = element_text(color = "black"),
plot.title = element_text(hjust = 0.5),
plot.subtitle = element_text(hjust = 0.5),
legend.position="bottom")+
scale_y_continuous(breaks = seq(0, 100, by=20),
limits=c(0,100)) +
# Line below adds dot plots from {ggdist} package
stat_dots(side = "left",
justification = 1.12,
binwidth = 1.9) +
# Line below adds half-violin from {ggdist} package
stat_halfeye(adjust = .5,
width = .6,
justification = -.2,
.width = 0,
point_colour = NA)
edv
Figure 2 looks different from the original by Guilherme A. Franchi, PhD because we are plotting two factors instead of one. However, the nature of the plot is the same. Pay attention to the red dots, these are the ones that can be considered extreme observations that tilt the measures of central tendency (especially the mean) toward one direction. We also observe that the variances are different, so modeling also sigma can give better estimates. Our task now is to model the output using the brms package.
Here we fit our model with Day and Genotype as interacting categorical predictors for the distribution of Trial 3. Let’s first fit a typical Gaussian model, which is analogous to an ordinary least squares (OLS) model from the frequentist framework, since we are using the default flat brms priors. Priors are beyond the scope of this article, but I promise we’ll cover them in a future blog.
Once we have results from the Gaussian model, we can compare them to the large results from the Student’s t model. We then addsigma to the equation to account for the difference in the variance of the data.
Our Gaussian model is built under the typical (and often incorrect) assumption of homoscedasticity (3). In other words, we assume that all groups have the same (or very similar) variance. I do not recall seeing this as a researcher.
Gaussian_Fit1 <- brm(Trial3 ~ Day * Genotype,
data = data,
family = gaussian(),
# seed for reproducibility purposes
seed = 8807,
control = list(adapt_delta = 0.99),
# this is to save the model in my laptop
file = "Models/20240222_OutliersStudent-t/Gaussian_Fit1.rds",
file_refit = "never")
# Add loo for model comparison
Gaussian_Fit1 <-
add_criterion(Gaussian_Fit1, c("loo", "waic", "bayes_R2"))
Before proceeding, it’s a good idea to do some simple model diagnostics to compare the actual observations with the predictions made by our model. We can do this in several ways, but the most common is to plot full densities. We can achieve this using the pp_check function from brms.
set.seed(8807)
pp_check(Gaussian_Fit1, ndraws = 100) +
labs(title = "Gaussian model") +
theme_classic()
Figure 3 suggests that our observations (dark blue) are not meaningfully different from the model predictions. Below, I leave you with additional code to check other pp_check alternatives with their respective graphs.
set.seed(88071)
pp_check(Gaussian_Fit1, group = "Genotype", type = "dens_overlay_grouped", ndraws = 100) +
labs(title = "Density by Genotype") +
theme_classic()
pp_check(Gaussian_Fit1, type = "stat_grouped", group = "Genotype", stat = "var", binwidth = 3) +
coord_cartesian(xlim = c(0, 300)) +
ggtitle("Grouped variance") +
theme_classic()
pp_check(Gaussian_Fit1, type = "stat", stat = "var", binwidth = 3) +
coord_cartesian(xlim = c(0, 600)) +
ggtitle("How well we captured the variace") +
theme_classic()
pp_check(Gaussian_Fit1, type = "stat", stat = "mean", binwidth = 2) +
coord_cartesian(xlim = c(0, 50)) +
ggtitle("How well we captured the mean") +
theme_classic()
Now, we use the describe_posterior function from the bayestestR package (4) to see the results:
describe_posterior(Gaussian_Fit1,
centrality = "mean",
dispersion = TRUE,
ci_method = "HDI",
test = "rope",
)
Let’s focus here on the ‘intercept’, which is the value for WT at 1 DPI, and ‘GenotypeKO’, the estimated difference for KO animals at the same time point. We see that WT animals spend about 37 seconds in the rotarod, while their KO counterparts spend less than a second (0.54) more. As a researcher in this field, I can say that this difference is meaningless and that genotype has no effect on rotarod performance. Even the effect of day, which is 2.9, seems meaningless to me under this model. We can easily visualize these estimates using the wonderful conditional_effects function from brms.
# We create the graph for convex hull
Gaussian_CondEffects <-
conditional_effects(Gaussian_Fit1)
Gaussian_CondEffects <- plot(Gaussian_CondEffects,
plot = FALSE)[[3]]
Gaussian_CondEffects +
geom_point(data=data, aes(x = Day, y = Trial3, color = Genotype), inherit.aes=FALSE) +
Plot_theme +
theme(legend.position = "bottom", legend.direction = "horizontal")
In Figure 8 we can see the estimates and uncertainty for the interaction terms. I have customized the plot with a number of ggplot elements, which you can check in the original Quarto Notebook. Note the similar uncertainty for both time points, even though the dispersion is larger on day 1 than on day 2. We will address this point in a small snippet at the end of this article.
Now let’s see how much our understanding changes when we model the same data using a student-t distribution.
It’s time to use the student-t distribution in our `brms` model.
Student_Fit <- brm(Trial3 ~ Day * Genotype,
data = data,
family = student,
# seed for reproducibility purposes
seed = 8807,
control = list(adapt_delta = 0.99),
# this is to save the model in my laptop
file = "Models/20240222_OutliersStudent-t/Student_Fit.rds",
file_refit = "never")
# Add loo for model comparison
Student_Fit <-
add_criterion(Student_Fit, c("loo", "waic", "bayes_R2"))
We plot the model diagnostics as done before:
Figure 9 shows that the mean shape and the peak of the observations and the predictions match. It’s important to note that our model seems to predict values below 0. This is an important research issue that we will skip for now. However, it does imply the use of informative priors or distribution families that set a lower bound at 0, such as the log_normal’,hurdle_lognormal’, or `zero_inflated_poisson’, depending on the case. Andrew Heiss (5) offers a great example in this regard.
Let’s take a look at the posterior distribution:
describe_posterior(Student_Fit,
centrality = "mean",
dispersion = TRUE,
ci_method = "HDI",
test = "rope",
)
Under this model, we can see that our estimates have changed moderately, I would say. Our estimate for the intercept (WT at 1 day) is reduced by 7 seconds. And why is that? Because the extreme values we discovered at the beginning have less influence on the measures of central tendency of the data. Thus, this is a more accurate measure of the “typical” WT animal on day 1. We also observe a substantial increase in the effect of day, with almost 10 seconds more than our initial Gaussian estimates. Importantly, the effect of our KO genotype appears to be more notorious, increasing about 10 times from 0.52 in our Gaussian model to 5.5 in our student-t model. From my perspective, given the context of these data, the contrasts between the two models are notorious.
Let’s see it in graphical terms using conditional_effects:
Student_CondEffects <-
conditional_effects(Student_Fit)
Student_CondEffects <- plot(Student_CondEffects,
plot = FALSE)[[3]]
Student_CondEffects +
geom_point(data=data, aes(x = Day, y = Trial3, color = Genotype), inherit.aes=FALSE) +
Plot_theme +
theme(legend.position = "bottom", legend.direction = "horizontal")
Can we get better estimates? For this particular example, I think we can. From the start, it was easy to notice the difference in the variance of the data, especially when we compare the first and second-day visuals. We improved our estimates using the student-t distribution, and we can improve them further by developing a model for heteroscedasticity that predicts sigma (the residual variance).
In this way, the model does not assume that your residual variance is equal across your grouping variables, but it becomes a response that can be modeled by predictors.
This is the little point we left for the end.
We include sigma as a response variable using thebf function from brms. In this case, we are going to model this parameter using the same predictors Day and Genotype.
Student_Mdl2 <- bf (Trial3 ~ Day * Genotype,
sigma ~ Day * Genotype)
Student_Fit2 <- brm(
formula = Student_Mdl2,
data = data,
family = student,
# seed for reproducibility purposes
seed = 8807,
control = list(adapt_delta = 0.99),
# this is to save the model in my laptop
file = "Models/20240222_OutliersStudent-t/Student_Fit2.rds",
file_refit = "never")
# Add loo for model comparison
Student_Fit2 <-
add_criterion(Student_Fit2, c("loo", "waic", "bayes_R2"))
Figure 11 looks good, except for the uncomfortable predictions below 0. For this case, I judge that this does not strongly bias the estimates and their uncertainty. However, this is an aspect I will take into account when doing actual research.
Now, let’s take a look at the posterior distribution.
We see more parameters compared to the other two fitted models because the response for sigma is now included as a main effect in the model. Under this scheme, we see that the intercepts are closer to those of the Gaussian model and the effect of genotype (GenotypeKO) is reduced by half.
There is one aspect to note, however. In our first Student-t model, the uncertainty for the intercept was 24.1–37.4. On the other hand, in the last model, the uncertainty increases to 24.3–46.1. This means that when we consider the different variances, we are less certain of this (and other) parameters. The same is true for day, for example, which changes from 1.2–18.9 to -5.6–18.1. In this case, we are now less certain that the second day is associated with an increase in time spent on the rotarod.
Don’t worry, the purpose of statistical modeling is to provide the best possible quantification of the uncertainty in a measurement, and that’s what we’re doing right now. Of course, our uncertainty increases when we have extreme values that are part of our sample and therefore part of our population.
In this example, we see that accounting for the different variances in our data gives us a very different idea of our results.
Finally, we can see that sigma, plotted on the log scale, varies meaningfully with day and genotype:
Student_CondEffects2 <-
conditional_effects(Student_Fit2)
Student_CondEffects2 <- plot(Student_CondEffects2,
plot = FALSE)[[3]]
Student_CondEffects2 +
geom_point(data=data, aes(x = Day, y = Trial3, color = Genotype), inherit.aes=FALSE) +
Plot_theme +
theme(legend.position = "bottom", legend.direction = "horizontal")
Student_CondEffects3 <-
conditional_effects(Student_Fit2, dpar = "sigma")
Student_CondEffects3 <- plot(Student_CondEffects3,
plot = FALSE)[[3]]
Student_CondEffects3 +
Plot_theme +
theme(legend.position = "bottom", legend.direction = "horizontal")
What we see in the second graph is sigma, which effectively accounts for the variance in this parameter between days and genotypes. We see a much higher uncertainty at day 1, especially for WT mice, while the parameter is analogous at day 2.
We can conclude this article by comparing the three models for out-of-sample predictions.
We perform model comparisons using the WAIC criteria (6)for estimating the out-of-sample prediction error. By considering both the log-likelihood of the observed data and the effective number of parameters, it provides a balance between model fit and complexity. Unlike some other criteria, WAIC inherently accounts for the posterior distribution of the parameters rather than relying on point estimates, making it particularly suited to Bayesian analyses.
Given a data set and a Bayesian model, the WAIC is calculated as:
WAIC=−2×(LLPD−pWAIC)
Where: LLPD is the log pointwise predictive density, calculated as the average log-likelihood for each observed data point across the posterior samples. WAIC is the effective number of parameters, computed as the difference between the average of the log-likelihoods and the log-likelihood of the averages across posterior samples.
We use the compare_performance function from the performance package, part of the easystats environment (4, 7, 8).
Fit_Comp <-
compare_performance(
Gaussian_Fit1,
Student_Fit,
Student_Fit2,
metrics = "all")
Fit_Comp
The output shows that our Student-t model predicting sigma is the least penalized (WAIC = 497) for out-of-sample prediction. Note that there is no estimate for sigma in this model because it was included as a response variable. This table also shows that the student-t model has less residual variance (sigma) than the Gaussian model, which means that the variance is better explained by the predictors. We can visualize the same results as a graph:
Fit_Comp_W <-
loo_compare(
Gaussian_Fit1,
Student_Fit,
Student_Fit2,
criterion = "waic")
# Generate WAIC graph
Fit_Comp_WAIC <-
Fit_Comp_W[, 7:8] %>%
data.frame() %>%
rownames_to_column(var = "model_name") %>%
ggplot(
aes(x = model_name,
y = waic,
ymin = waic - se_waic,
ymax = waic + se_waic)
) +
geom_pointrange(shape = 21) +
scale_x_discrete(
breaks=c("Gaussian_Fit1",
"Student_Fit",
"Student_Fit2"),
labels=c("Gaussian_Fit1",
"Student_Fit",
"Student_Fit2")
) +
coord_flip() +
labs(x = "",
y = "WAIC (score)",
title = "") +
Plot_theme
Fit_Comp_WAIC
Figure 14 shows that our last model is less penalized for out-of-sample prediction.
You can find an updated version of this post on my GitHub site. Let me know if this journey was useful to you, and if you have any constructive comments to add to this exercise.
*Unless otherwise noted, all images are generated by the author using R code.
1.M. Ahsanullah, B. M. G. Kibria, M. Shakil, Normal and student´s t distributions and their applications (Atlantis Press, 2014; http://dx.doi.org/10.2991/978-94-6239-061-4).
2. P.-C. Bürkner, Brms: An r package for bayesian multilevel models using stan. 80 (2017), doi:10.18637/jss.v080.i01.
3. K. Yang, J. Tu, T. Chen, Homoscedasticity: an overlooked critical assumption for linear regression. General Psychiatry. 32, e100148 (2019).
4. D. Makowski, M. S. Ben-Shachar, D. Lüdecke, bayestestR: Describing effects and their uncertainty, existence and significance within the bayesian framework. 4, 1541 (2019).
5. A. Heiss, A guide to modeling proportions with bayesian beta and zero-inflated beta regression models (2021), (available at http://dx.doi.org/10.59350/7p1a4-0tw75).
6. A. Gelman, J. Hwang, A. Vehtari, Understanding predictive information criteria for Bayesian models. Statistics and Computing. 24, 997–1016 (2013).
7. D. Lüdecke, M. S. Ben-Shachar, I. Patil, P. Waggoner, D. Makowski, Performance: An r package for assessment, comparison and testing of statistical models. 6, 3139 (2021).
8. D. Makowski, M. Ben-Shachar, D. Lüdecke, bayestestR: Describing effects and their uncertainty, existence and significance within the bayesian framework. Journal of Open Source Software. 4, 1541 (2019).
Do not over-think about ‘outliers’, use a student-t distribution instead was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
Originally appeared here:
Do not over-think about ‘outliers’, use a student-t distribution instead
Go Here to Read this Fast! Do not over-think about ‘outliers’, use a student-t distribution instead
Exploring and exploiting the seemingly innocent theorem behind Double Machine Learning
Originally appeared here:
Causal Machine Learning: What Can We Accomplish with a Single Theorem?
Go Here to Read this Fast! Causal Machine Learning: What Can We Accomplish with a Single Theorem?
Let me introduce you to Sarah, a talented and passionate data scientist, who just landed her dream job at GreenEnv, a large company that makes eco-friendly cleaning products. GreenEnv has tons of data on customers, products, and other areas of the business. They hired Sarah to unlock the hidden potential within this data, uncovering market trends, competitive advantages, and more.
Her first task: analyze customer demographics and buying habits to create targeted marketing campaigns. Confident in her abilities and excited to apply data science methods, Sarah dived into the customer database. But her initial excitement quickly faded. The data was a mess — inconsistent formatting, misspelled names, and duplicate entries everywhere. Data quality was terrible. There were variations of names like “Jhon Smith” and “Micheal Brown” alongside entries like “Jhonn Smtih” and “Michealw Brown.” Emails had extra spaces and even typos like “gnail.com” instead of “gmail.com.” along with many other inaccuracies. Sarah realized the hard job ahead of her — data cleaning.
Inconsistent formatting, missing values, and duplicates would lead to skewed results, giving an inaccurate picture of GreenEnv’s customer base. Days turned into weeks as Sarah tirelessly cleaned the data, fixing inconsistencies, filling in gaps, and eliminating duplicates. It was a tedious process, but essential to ensure her analysis was built on a solid foundation.
Who cares about data quality?
Every year, poor data quality costs organizations an average of $12.9 million. [1]
Thankfully, after weeks of cleaning and organizing this messy data, Sarah was able to get the job done…or at least for this part..
Her next challenge came when she ventured into product data, aiming to identify top-selling items and recommend future opportunities. However, she encountered a different problem — a complete lack of metadata. Product descriptions were absent, and categories were ambiguous. Basically, there wasn’t enough data to help Sarah to understand the product’s data. Sarah realized the importance of metadata management — structured information about the data itself. Without it, understanding and analyzing the data was almost impossible.
Research Shows Most Data Has Inaccuracies
Research by Experian reveals that businesses believe around 29% of their data is inaccurate in some way. [2]
Frustrated but determined, Sarah reached out to different departments to piece together information about the products. She discovered that each department used its own internal jargon and classification systems. Marketing and sales refer to the same cleaning product with different names.
As Sarah delved deeper, she found that datasets were kept in separate applications by different departments, outdated storage systems struggling to handle the growing volume of data, and Sarah had to wait for a long time for her queries to be executed. Sarah noticed also there are no clear rules on who can access what data and under what terms, without centralized control and proper access controls, the risk of unauthorized access to sensitive information increases, potentially leading to data breaches and compliance violations. The lack of data governance, a set of rules and procedures for managing data, was evident.
Data Breaches Can Be Costly
According to the Ponemon Institute, the average cost of a data breach in 2023 is $4.45 million globally, an all-time high record, with costs varying by industry and location. [3]
Each of the above issues and hurdles in Sarah’s story highlighted the interconnectedness of many pillars — data quality, metadata management, and data governance all played a crucial role in accessing and utilizing valuable insights at GreenEnv.
Sarah’s journey is a common one for data scientists and analysts. Many organizations have massive amounts of data, and everyone knows the saying: “Data is the new electricity.” Every organization wants to make the most of their data, as it’s a very valuable asset. But most people mistakenly (and practically) believe that simply hiring a data analyst or data scientist is enough to unlock this value. There are many pillars to getting the most value from data, and organizations need to account for and pay attention to these. The keyword here is data management.
Did you know..
86% of organizations say they believe investing in data management directly impacts their business growth[4]
Generally speaking, data management is the overall practice of handling the organization’s data. from acquiring and storing the data to processing, securing, and analyzing it. The goal is to ensure the data is accessible, usable, accurate, reliable, and of high quality to achieve the state of data-informed organization and ultimately achieve our organizational objectives.
The Pillars of Data Management
There are several key dimensions that work together to achieve successful data utilization, and while these pillars might be slightly different from one framework to another, the underlying concepts are the same.
In addition to data security and data integration as pillars we also have:
You can notice that these dimensions are interconnected and also closely related. Addressing weaknesses in one area can often impact others. Some frameworks have been developed to address the relationship between the different dimensions of data management and how they interact and affect each other.
Data Management Frameworks: The Aiken’s Pyramid
Many established frameworks like DAMA-DMBOK2, IGF, and EDM offer structured guidance, standardized terminology, and maturity assessments for data management
One conceptual framework worth mentioning here, and the one I like the most, is Aiken’s pyramid of Data Management. It outlines the different stages of data management processes. Developed by Robert Aiken, a data management pioneer, this framework describes the situation in which many organizations find themselves. In trying to leverage the full potential of their data, many organizations go through a similar progression of steps:
The Aiken Pyramid helps organizations understand how data management activities interconnect, how each one builds on the others, and how to prioritize their efforts for effective data utilization.
Reflecting on learning and experience with data management(although I’m not a data management expert XD), I’ve come to appreciate and favor the following points regarding data management and its best practices, especially if we are focusing on data quality.
– For example, a non-technical process that generates bad data might involve database creation by developers who solely focus on the technical aspects. There might be a lack of documentation or column descriptions for instance. A good practice in my opinion would be to engage data analysts and other relevant stakeholders in the design process to ensure adherence to data management best practices. The data management team can decide if we would go forward with a certain application design or not.
– An application’s design can also be a technical process for bad data generation. A well-designed application should enforce data quality proactively during data entry. For instance, instead of a text box for entering gender, a dropdown menu could be used. Another example might be predefining email types, where the user only needs to add their username before automatically receiving “@gmail.com” or another domain extension.
There are many data management certifications out there you might consider. The best choice depends on your specific goals and experience but here are a few ones I came across:
Ultimately the choice of which one is worth the time and effort requires a more in-depth analysis of the credibility, content, and purpose of each certification.
As you would guess, there is an endless number of tools available to address different aspects of data management, and it would be impractical to talk about all these tools, each tool would have pros and cons, and situations where it might be effective, or specific data management dimensions where it would be helpful.
In my next article, I will delve deeper into two open-source tools for data management, DataHub and Great Expectations(GX), and provide a step-by-step guide on how to integrate them to create a robust, cost-effective, scalable, and unified environment for data discovery, metadata management, data quality, data governance, data lineage, and impact analysis.
Data is arguably an organization’s most valuable asset these days. However, many still lack proper data management, limiting their ability to leverage its true potential. Issues like data quality, governance, security, and metadata management are all central pillars to get the most value out of your organization’s data.
Remember Sarah’s story at the beginning? Hiring a data scientist isn’t enough. They often spend a significant amount of time cleaning and organizing messy data before they can even begin analysis. Thankfully, frameworks like the Aiken Pyramid can guide organizations on their data management journey, and help in communicating data management initiatives across the different teams and different stakeholders easily and regardless of their technical level.
Thanks for reading all the way here! I hope you found this informative and enjoyable. Please feel free to point out any mistakes or share your thoughts — your feedback is always appreciated.
The Many Pillars of Getting the Most Value From Your Organization’s Data was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
Originally appeared here:
The Many Pillars of Getting the Most Value From Your Organization’s Data
Go Here to Read this Fast! The Many Pillars of Getting the Most Value From Your Organization’s Data
From Sankey diagrams and spider plots to ridge plots, this article spans various visualizations, each with its unique use case and insight.
Originally appeared here:
Mastering the Versatility and Depth of Python’s Rich Plot Collection(with Code)
This is how to use XGBoost in a forecasting scenario, from theory to practice
Originally appeared here:
Hands On Monotonic Time Series Forecasting with XGBoost, using Python
Go Here to Read this Fast! Hands On Monotonic Time Series Forecasting with XGBoost, using Python
