library(tidyverse)
library(tidytext)
library(skimr)
library(stopwords)
library(lubridate)
library(corrr)
library(forcats)
library(here)

df <- read_csv(here("data", "clean.csv"))

The fields we will be interested in:

• created_at
• language
• size
• license
• stars
• open_issues_count
• has_

The description will have its own analysis.

All analyses should be faceted by language.

Univariates

Let’s first look at repo stars.

df %>%
  ggplot(aes(stars)) +
  geom_density() +
  scale_x_log10() +
  facet_wrap(vars(language), scales = "free")

The stars distributions are very skewed, even after a log transform. This also makes sense as the majority of projects on Github are not well known. Javascript is weird as it has a less defined peak.

df %>%
  mutate(language = fct_reorder(language, stars)) %>%
  ggplot(aes(log(stars), language)) +
  geom_boxplot()

The distribution of stars among the languages is about what you would expect for the Github crowd. Super general Python and web-friendly Javascript on top, highly specialized R on bottom.

List of the most popular projects for each language (under 100000 stars):

df %>%
  group_by(language) %>%
  slice_max(stars, n = 1) %>%
  select(name, description, language)

Let’s now look at project size.

df %>%
  ggplot(aes(size)) +
  geom_density(fill = "blue") +
  scale_x_log10() +
  facet_wrap(vars(language), scales = "free")

It is also skewed, but a log transform seems to be sufficient in de-skewing it.

There are repos with size 0:

df %>% filter(size == 0)

Some of these are errors since the repos are clearly there. The jd-sec-maotai does seem to no longer be there, however. Very strange.

Let’s look at created_at:

df %>%
  group_by(month = floor_date(created_at, "months"), language) %>%
  count() %>%
  ggplot(aes(month, n)) +
  geom_line() +
  facet_wrap(vars(language), scales = "free_x")

This looks very reasonable. Popular projects were created mostly around 2015-2018. Of course, since our data collection process filtered based on having the most stars, the most natural conclusion isn’t that project creations slowed down after 2018 (in fact, I would bet the opposite), but that there hasn’t been enough time for the later projects to accumulate enough stars.

Let’s turn to the number of open issues.

df %>%
  ggplot(aes(open_issues_count)) +
  geom_density() +
  scale_x_log10() +
  facet_wrap(vars(language), scales = "free")

R is different from the general programming languages in that they have more projects with less (or 0) issues.

log_df <- df %>%
  mutate(
    stars = log(stars),
    size = log1p(size),
    open_issues_count = log1p(open_issues_count)
  )

Now let’s look at variation with stars/popularity

First up: size.

df %>%
  ggplot(aes(size, stars)) +
  geom_point() +
  scale_x_log10() +
  scale_y_log10() +
  facet_wrap(vars(language), scales = "free")

There is an ever so slight positive correlations.

Let’s see what the owner type says about the number of stars.

df %>%
  ggplot(aes(stars)) +
  geom_density(aes(fill = owner), alpha = 0.5) +
  scale_x_log10() +
  facet_wrap(vars(language), scales = "free")

Repos with whose owners are organizations tend to have more stars than those with individual owners. This makes sense as organizations probably can make more expansive projects. Visually, it looks less pronounced in R, which also makes sense as R is probably the least commercial of all the languages.

df %>%
  select(stars, owner, size, language) %>%
  mutate(owner = owner == "Organization") %>%
  group_by(language) %>%
  group_map(\(x, y) list(
    owner = cor(x$owner, log(x$stars)),
    size = cor(log1p(x$size), log(x$stars)),
    lang = y$language
  )) %>%
  map(\(x) do.call(tibble, x)) %>%
  bind_rows()

df %>%
  ggplot(aes(log(size))) +
  geom_density(aes(fill = owner), alpha = 0.5) +
  facet_wrap(~language, scales = "free")

As one would expect, organizations have larger projects than users, although this is more muted in R.

Licenses

Let’s now look at licenses. We will otherize less popular licenses and keep only the top 10.

lump_lic <- df %>%
  mutate(license = license %>%
    fct_lump(n = 10) %>%
    fct_infreq() %>%
    fct_rev())

lump_lic %>%
  group_by(license, owner) %>%
  count() %>%
  ungroup() %>%
  ggplot(aes(n, license)) +
  geom_col(aes(fill = owner), position = "dodge")

We see that users are far more likely to have no license or use MIT and organizations are more likely to use more exotic licenses.

lump_lic %>%
  mutate(stars = log(stars)) %>%
  group_by(license) %>%
  summarize(
    log_stars = mean(stars),
    n = n(),
    lo = quantile(stars, 0.025),
    hi = quantile(stars, 0.975)
  ) %>%
  ggplot(aes(log_stars, license)) +
  geom_point(aes(size = n)) +
  geom_errorbarh(aes(xmin = lo, xmax = hi), height = 0.5) +
  expand_limits(xmin = 0)

License has some weak correspondence with popularity, although having an exotic license or not having a license seems to have a stronger correspondence to having low popularity.

log_df %>%
  ggplot(aes(open_issues_count, stars)) +
  geom_point() +
  facet_wrap(vars(language), scales = "free")

lm(stars ~ open_issues_count, data = log_df) %>% summary()

Call:
lm(formula = stars ~ open_issues_count, data = log_df)

Residuals:
    Min      1Q  Median      3Q     Max 
-3.5396 -0.6231 -0.0085  0.6382  5.7624 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)       5.195080   0.011810  439.89   <2e-16 ***
open_issues_count 0.383307   0.004027   95.19   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.072 on 31429 degrees of freedom
Multiple R-squared:  0.2238,    Adjusted R-squared:  0.2237 
F-statistic:  9060 on 1 and 31429 DF,  p-value: < 2.2e-16

Stars are correlated with number of issues, which seems strange. One possibility is that this could be that it is confounded by size.

This will be hard to facet by language since different languages are at different scales, so let’s just look at Python, arguably the most broad language.

log_df %>%
  filter(language == "python") %>%
  ggplot(aes(size, open_issues_count)) +
  geom_point(aes(color = stars)) +
  scale_color_gradient2(low = "blue", high = "red", midpoint = 9)

log_df %>%
  filter(language == "python") %>%
  lm(formula = stars ~ size + open_issues_count) %>%
  summary()

Call:
lm(formula = stars ~ size + open_issues_count, data = .)

Residuals:
    Min      1Q  Median      3Q     Max 
-1.8268 -0.5362 -0.1485  0.3642  4.4068 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)       5.931197   0.034671  171.07   <2e-16 ***
size              0.043931   0.003992   11.00   <2e-16 ***
open_issues_count 0.220283   0.006662   33.06   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.7736 on 6587 degrees of freedom
Multiple R-squared:  0.1819,    Adjusted R-squared:  0.1817 
F-statistic: 732.4 on 2 and 6587 DF,  p-value: < 2.2e-16

Well, it looks like even controlling for size, we still have a positive correlation between issues and stars (although far weaker).

The more likely explanation now is that popularity causes issues since the more eyes and users on your project means the more issues get unveiled.