USF Information Science Blog

It feels weird to say that my development of this package started before I even realized that the final project for this class was developing an R package, but that’s exactly how it started.

1. Package Context
2. Development Notes (Research Team and AI Disclosure)
   1. Function Debugging:
   2. Generate a synthetic testing file:
   3. Git and Version Control Errors:
3. openEvalR Package Specifics
4. openEvalR Package Usage
5. Package Link

Package Context

For a little bit of context — I currently work with Dr. Loni Hagen as her research assistant, which primarily consists of me running machine learning algorithms and statistical analyses to support her research. Our current project is looking at the Luigi Mangione case, and the frames of social media response and Sentiment. Its an ongoing project, and our first installment “Human and AI Alignment on Stance Detection: A Case Study of the United Healthcare CEO Assassination” will be presented at the FLAIRS-38 conference in May.

This project installment involved a significant amount of prompt engineering and model output evaluation using OpenAI’s API. We tested several models (GPT-4o Mini, GPT-o3 Mini, GPT 4.1, GPT 4.1 Mini) and evaluated them using several annotator agreeability metrics like Cohen’s Kappa, Qwet’s AC1, percent agreement against an established annotator Ground Truth.

These metrics are honestly fairly easy to calculate when your data is in the correct format. Packages like caret, irr, and irrCAC have accepted domain accepted functions that are trusted by the community for their accuracy and reliability, and I certainly was not going to try to recreate these functions but worse. The problem I ran into while evaluating our model results was the first sentence of this paragraph [“when your data is in the correct format“]

The thing about Open Ai’s model evaluation platform is that when you export results, you get them in a stupidly long and nested .jsonl format that was a pain to parse every time I needed to export our model results for evaluation. Honestly, I began dreading every time Dr. Hagen asked me to export our results because I knew I was about to have a 2-3 hour coding fight with these files just to parse it into an easily digestible format, such as a dataframe or a .csv.

Here is a screenshot of what our raw results looked like and the files I fought

Now, are there R and Python packages that are designed for parsing json files? Yes, there are. But even with those resources, I would be writing and rewriting huge code blocks in order to manipulate these general libraries and packages into configuring my files over and over again.

As the weeks went on, I eventually did develop coding functions that I kept on file that slowly transformed my 2-3 hour coding fights into a much more manageable 5 minute RMarkdown set up. So, when I realized that the final project for LIS4370 was developing your own R Package, I realized that I had basically already done this in my official research position and decided to go the extra step and fully refine the workflow I developed into an official R package I could publish with our research results.

Development Notes (Research Team and AI Disclosure)

Before I get into development specifics I feel the need to clarify — While this package’s functions were developed in an official research position, this package and its functions were solely developed by me and were not written by the leading Dr.s and other assistants apart of the project.

Similarly, I will disclose that this project did make use of AI tools, such as OpenAI’s ChatGPT for the following:

Function Debugging:

When strange errors were encountered during function development, I used ChatGPT to figure out what they were telling me.

Specific Example: While I was writing my cmatrix() function, I encountered an error with one of my files, where my cmatrix function was working with all other raw jsonl results, but was failing with one. R gave me this error for that file:

> print(cmatrix(df3, "item.Agreed"))
Error in map():
ℹ In index: 1.
Caused by error in confusionMatrix.default():
! the data cannot have more levels than the reference
Backtrace:
  1. base::print(cmatrix(df3, "item.Agreed"))
  2. global cmatrix(df3, "item.Agreed")
  3. purrr::map(...)
  4. purrr:::map_("list", .x, .f, ..., .progress = .progress)
  8. .f(.x[[i]], ...)
 10. caret:::confusionMatrix.default(data[[model_col]], data[[groundtruth_col]])
 11. base::stop("the data cannot have more levels than the reference")

This confused me as my function worked with all other files, and only failed on one specifically. ChatGPT gave me this insight:

This insight helped me remember a very strange situation we encountered in our research that I hadn’t realized would impact the confusionMatrix() generation the way it did — our ground truth coding used 3 levels to denote three different stances for multiple topics (-1 = Negative, 0 = Neutral, 1 = Positive), however, for one of our topics, our ground truth annotation had one topic were no Positive stances were encountered in the sample set, so one of our Ground truth columns only had 2 levels. However, the model ended up denoting one of the sample texts as positive. Meaning, our ground truth column for that topic had two levels, and the model result had three levels — Exactly what this error reported.

This insight led me to add a this line of code to my cmatrix() function:

    # This line specifically is very research specific
    model_levels <- levels(factor(data[[model_col]]))

    # Align both columns to use the same levels
    prediction <- factor(data[[model_col]], levels = model_levels)
    reference  <- factor(data[[groundtruth]], levels = model_levels)

(This was one of a couple situations that makes this package very tailored specifically to our research project, however when faced with a decision to make this package more generalizable or work best for our research, I made the decision to error on the side of our research)

Generate a synthetic testing file:

When I was finishing this package up, I had been regularly testing it with our research model results. I knew the package was working as I intended it, but I encountered a small situation when I realized I should probably provide a testing file people could use to test my package (given how specific this package it, its sort of hard to play with it without specific files).

Given the nature of our research, I was unsure if I would be allowed to share our raw results on this package’s page, partly due to privacy/GDPR concerns, and also the fact that we have not formally published this project’s GitHub.

Rather than risk it, I decided to ask Chatgpt to generate a synthetic file that mimicked the structure of our projects raw results with completely fake, topically neutral, dummy data.

Git and Version Control Errors:

When my package was finished, and I was ready to publish, I wanted to publish my package using git commands in the R terminal (I think you can just upload your directory to GitHub, but I’m pretty sure it is best practice to use git commands, I’m not sure which one we were expected to do, but I decided I wanted to use git commands). I have never used the R terminal before, and I found using it very difficult. I encountered errors pretty much every step of the way, written in lingo I have never encountered before. Chatgpt was invaluable for decoding these git command errors and helping me figure out how to publish my package from the R terminal.

I encountered errors like not establishing my inital commit, not knitting my README.md, not setting my global name and email address, not logging in to GitHub correctly. It was honestly a little bit of an embarrassing fight but I am glad I took this approach and committed to uploading my package this way as you’ll never learn if you don’t jump in and figure it out.

Here are some screenshots of my rather embarrassing fight with the R Terminal and how I utilized Chatgpt to diagnosis and address git command errors:

With that out of the way — Lets talk a little bit more about my package:

#########################################

openEvalR Package Specifics

How many functions does openEvalR have?

• My package consists of 5 functions: results(), metadata(), kappas(), qwetAC1(), and cmatrix(). All of these functions were developed during my research analysis and were constantly used by me when evaluating model results, except for metadata(). metadata() was developed as a way to try and improve the generalizability of this package and expand its function list.

How many libraries does openEvalR use?

• 10. Many of these packages come from the fact that I used tidyverse programing style (My preferred coding style). 3 of them were called due to their industry accepted functions (irr’s kappa2(), caret’s confusionMatrix(), and irrCAC’s gwet.ac1.raw()). When I developed this package, I did not set out to reinvent the wheel for those functions and retained the utilization of other packages. My focus for this package was streamlining my research’s workflow.

What type of metadata does openEvalR hold?

• My package’s metadata includes its DESCRIPTION file (package title, version, author, dependencies, and description) , its NAMESPACE file (exported functions and imported functions from other packages, generated via devtools::document() ), and a README file (package development background and usage instructions).

What type of license does openEvalR maintain?

• The CC0 license (Creative Commons Zero 1.0 Universal)

What type of classes (S3 vs. S4) and methods can be found in openEvalR? 

• Strictly S3 methodology

#########################################

openEvalR Package Usage

I added this development note to my README but I want to talk about it again here: Unfortunately, while this package was a lifesaver for me and my project, I struggle to see how other people might use this package.

The reason being, this package was designed to fit the model outputs of this projects .jsonl files, and its subsequent functions depend on the dataframe being formatted as such:

data.idx	groundtruth	model_*	*_model_output	[…]
[…]	[…]	[…]	[…]	[…]

When I began adapting my research workflow to function as a package, I was faced with the decision of if I should prioritize generalizability or my research project — for my own sake, I decided to prioritize my research project and my own peace of mind rather than the package’s generalizability.

That being said, I did make efforts to clearly document how this package expects to receive inputs so that if people did ever want to use it for their own model evaluation projects, this package could support them. I hope that this package might help people in the future, but at the very least, the functions developed for this package were invaluable for me and the statistical analysis I did for the stance detection research project, and I am very proud to be publishing this package alongside those research results.

Package Link

Here is a link to the GitHub repository I published for this package. I decided to publish a new one, rather than under my LIS4370 repository as I may be linking this package to the GitHub repository our research project eventually publishes.

openEvalR GitHub Page

This week, we were asked to outline the following in an RMarkdown:

• Create a new Markdown file using R programming.
• Focus specifically on the main functions you are developing for your final project.
• Describe each function in detail within the Markdown file. Explain the purpose, inputs, outputs, and any additional relevant information.
• Utilize Markdown formatting to present your content in a clear and organized manner.
• Include code snippets or examples within the Markdown file to illustrate how the functions work.

I will say, the resource outlining RMarkdown syntax was very much appreciated! I didn’t need it much for this assignment, but that would have been a life saver last semester when I was struggling to format some other final projects. I had to learn a alot of RMarkdown syntax by trial and error based on my CSS knowledge, so I will definently be saving that!

For my package, openEvalR, I decided to create 4 functions: results() to extract model evaluation results from evaluation reports, metadata() to extract the original dataset uploaded to OpenAI as well as the data_source_idx, kappas() to caluclate the Kappa between the model results and specified ground-truth, and cmatrix() to generate a caret::confusion matrix for each model test.

I will note that these tools are primarily being developed in order to aid the current research study I am working with (We are bringing a poster to FLAIRS 38!), but I hope that it helps someone else with parsing OpenAI’s evaluation results!

Link to my Github: Here

This week we were given a function called tukey_multiple() that contained a deliberate bug within in. The goal of this week was to debug this function using R’s debug function.

Which, can I say, learning that R has a line by line debugger has been slightly infuriating, but also very cool. For context: the first coding language I ever learned was Python, using the Thony IDE. One of my favorite functions in the Thony IDE was their line by line debugger. When I was learning coding, specifically when we started getting into function creation and recursion, the line by line debugger was my best friend and my greatest asset in coding. Since my coding education has evolved, I had not yet found a single IDE with a similar debugger as Thony….

Until now, and I am so mad it took me until now to realize it.

That being said, this week’s debugging was a little touch and go for a little bit, as I struggled to figure out exactly what was going on with this function and what it was trying to accomplish. The first bug I found was in this line:

    {
    outliers[,j] <- outliers[,j] && tukey.outlier(x[,j])
    }

The original function used &&, which only compares the first element of two vectors, so I changed it to the vectorized & operator to correctly compare all elements in the column and identify outliers row by row.

The next bug was also in this line of code, and consisted of the tukey.outlier() call. Through my research, I found inconclusive results over if this was an actual function (maybe from an older version of R) or a custom function that was not defined properly. Regardless. rather than fully remove it (and risk the whole function falling apart) I decided to write a quick dummy function that would do something similar to what I think this call was trying to achieve (identify outliers)

tukey.outlier <- function(x) {
  # Pretend outlier if value > mean + 2*sd (Dummy Function for Logic testing)
  upper <- mean(x) + 2 * sd(x)
  lower <- mean(x) - 2 * sd(x)
  return(x < lower | x > upper)
}

This dummy function returns a boolean vector that indicates whether each value in the input is an outlier — specifically, it flags values that fall more than two standard deviations above or below the mean. While this dummy function does not follow the proper outlier identification process that Tukey used (IQR), it is good enough for testing the logic behind the actual function we are debugging, tukey_multiple()

The last bug I found before I was able to run this function with no errors was the parameter allowances. This function only wanted to receive numeric columns, but wouldn’t accept single columns. To solve this issue, I added a checkpoint that ensured that main part of the function was only accepting numeric columns, and redirected the function if it was given a character or factor column.

for (j in 1:ncol(x)) {
    if (is.numeric(x[[j]])) {
      outliers[, j] <- outliers[, j] & tukey.outlier(x[[j]])
    } else {
      outliers[, j] <- FALSE 
    }
  }

Overall — now this function looks at the columns in a row, and if all the columns in this row are outliers, this function flags the rows entirely as an outlier.

For the dataset I was testing this on, I had a mix of numeric and factor columns, so all of my rows returned FALSE as outliers, so at the moment I do not see a major use for this function, so my next step would be to adapt this further to better handle factor columns or other data types, rather than defaulting to FALSE, or look at the function to try and judge how many of the columns returned TRUE outlier reports, or something else to make this a bit more useful in the real world.

However, currently, the function does run without errors, so I am going to stop here.

As always, link to my Github: here

Package: openEvalR
Title: Parsing and Extracting Tools for Open AI Evaluation JSON files
Version: 0.0.9000
Authors@R: Alina Hagen <alinahagen@usf.edu>
Description: Simplifies parsing and analyzing OpenAI evals .jsonl files in R
Depends: R (>= 4.4.2)
Imports: jsonlite, readr, tidyverse 
License: CC0
LazyData: true

Package proposal: The package that I am proposing is a parsing package that would make working with OpenAI evaluation .jsonl files much simpler.

The reason I am proposing this package is because it would actually make my life, currently, so much easier if this package existed. For a bit of context, I currently work for Dr. Hagen and will be attending a conference in 2 months presenting a study where the foundation is build on Open AI’s evaluation portal. The last 2/3 weeks every time I have had to export our evaluation results, it has been a fight to get the .jsonl files into any usable format so we can even perform our calculation. So I would like to build a package that would possibly make that process much easier, especially as I will continue to have to continue fighting these .jsonl files until further notice.

The primary functions I would like to create are: converting the .jsonl file into a dataframe, a function for extracting specific columns, a function for extracting different model batches, a summary function that summarizes the reported accuracy, and a function that reports Cohen’s Kappa for the ground truth vs the model(s)

Link to my github: here

This week were asked to picked a dataset from the Vincent Arel Bundock dataset list (fantastic resource btw!) and generate 3 visualizations from one of those sets. I decided to use a paired dataset: the Trump_tweets.csv dataset, consisting of 20761 Trump tweets from 2009 – 2017, and its assocated sentiments, sentiment_counts.csv.

Before I could produce my visualizations, I needed to load my libraries, datasets, and tweak the data format a little bit using the following code:

> library(readr)
> library(ggplot2)
> library(tidyverse)
> sentiment <- read_csv("sentiment_counts.csv")

> sentiment <- sentiment %>%
+   mutate(total = Android + iPhone) %>%
+   filter(sentiment != "none")

From there on, I was able to make the following using ggplot2:

TRUMP SENTIMENT COUNTS

ggplot(sentiment, aes(sentiment, total)) +
geom_bar(stat = "identity") +
theme_light() +
labs(x = "Sentiment", y = "Frequency",
title = "Sentiments Expressed by Trump over Twitter (2009 - 2017)")

TRUMP TIMESERIES ANALYSIS — linechart

ggplot(trump, aes(created_at, favorite_count)) +
geom_line(linewidth = 0.5) +
theme_light() +
geom_smooth(method = "lm", color = "tomato") +
labs(x = "Date", y = "Favorites", title = "Trump Tweets Favorite Frequency over the Years (2009 - 2017)")

TRUMP TIMESERIES ANALYSIS — Boxplot

ggplot(trump, aes(as.factor(year(created_at)), favorite_count)) +
  geom_boxplot() +
  theme_light()+
  labs(x = "Year", y ="Favorites", title = "Fluctuations of Trump's Tweets Favorites over the Years (2009 - 2018)")

___________________________________________________________________

Personally, I love ggplot2 — It is by far my go to tool for visualization, even when I prefer and use other languages for other tasks. For me, nothing really beats the flexibility I have learned with ggplot2 and I will usually turn to it when given any visualization tasks (Even in my technical writing class where my professor was walking everything through how to make charts in excel, I gave her a whole codebook to support the ggplot visualizations I turned in)

This is to say — I think ggplot is a great package, and I’m grateful for the tools it have given in for not only data visualization, but the customization to make my charts look exactly as I envision them. While the ones I made for this assignment were relatively basic with little customization, I have made incredibly complicated charts using this package before, such as a Sankey diagram I actually got published in a paper, or my final visualization project last semester.

As Always, link to my github: here

This week we explored String manipulation, table importing and exporting, and plyr package. I was already familiar with these processes before this week, but it was interesting to get a refresher on table importing/exporting, as I usually use readr whenever I import my datasets into RStudio.

All in all, this was a very calm week compared to last week’s S3 and S4 object lab, but it was a nice review!

As always, you can the link to my github: Here

I believe the best way to read my blog on this topic is to follow along with my compiled R script, where I used R comments to narrate as I explored the questions for this week. S3 and S4 objects, while a confusing subject, is very important for understanding object oriented programing with R.

This week, we were asked to anwser:

1. How do you tell what OO system (S3 vs. S4) an object is associated with?
2. How do you determine the base type (like integer or list) of an object?
3. What is a generic function?
4. What are the main differences between S3 and S4?
5. In your GitHub, create two examples of S3 and S4.

All of my answers to these questions can be found in the compiled report below: They can also be found on my Github in my R script

As always, the link to my GitHub: here

This week we continued our exploration of Linear Algebra and were introduced to Object Oriented Programing in R. We were given the following code to initialize the matrices A and B and asked to find A+B and A-B

We were then introduced to the diag() function and asked to create a matrix with the following values in the diagonal: 4,1,2,3

Finally, we were asked to recreate a 5×5 Matrix and were given the hint: use diag()

All in all, this week made much more sense than last week’s. This week I was able to visually track how matrix addition and subtraction worked while matrix multiplication might have well have been magic, to quote my Calculus professor. I feel a lot more comfortable with matrices after this week as compared to last week!

As always, Link to my Github: here

This week the focus was on performing mathematical computations using matrices in R. We were given the following code:

We were then instructed to find the determinant and inverse of both matrices, if possible, along with a couple other computations. The code that I used for this assignment is as follows:

When multiplying matrices, you need to be careful that the the rows of the first matrix match the columns of the second. This code works because we ensured that:

• Because A has 10 columns, a has 10 numbers, thus A × a works
• Because B has 100 columns, b has 100 numbers, thus B × b works

With this code, we get the following result:

I then proceeded to perform matrix by matrix multiplication, which gave the following interesting results:

A × tA remained a square 10 × 10 matrix, becuase both A and tA were 10 × 10. However, B × tB turned into a square matrix 10 × 10 matrix despite B starting as a 10 × 100 matrix and tB starting as a 100 × 10 matrix. Despite their differences of dimensions, we are able to multiple them becuase their inner dimensions (100 × 100) match. When we multiple them, the resulting matrix takes on their outer dimensions (10 × 10), which is why we get the result we got.

Next, we wanted to calculate the determinant and inverse of both matrices, if possible:

In the end, we determined that while we were able to caluclate the determinant of A becuase it was a square matrix, it had a determinant of 0 which means it does not have an inverse. On the contrast, we were not able to calculate either for B because it was not an square matrix and thus R gave an error when the det() function was used on B.

I thought while this was a very difficul assignment, it was very fulfilling once I figured out exactly what it meant to multiple matrices. I am still getting the hang of them, but they make much more sense after this assignment. Another concept that really helped me understand this is recently my Calculus professor gave us a mini lecture on Group Theory, indentity elements, and the role of inverses.

Basically, an inverse is something that undoes a member of a set and creates the identity element. So an inverse matrix is a matrix that “undoes” another matrix to create the identity element for the Matrix group, which looks like the following:

Once it clicked in my head exactly what inverse matrices were, how matrix multiplcation works, and how they fit into this idea of Group Theory, this assignment because to make alot more sense! It feels a little weird to rely on abstract mathematics to understand something like matrix multiplcation, but it makes alot more sense in my head becuase of that foundation. I’ll have to thank my Calculus professor for the improptu Group Theory lesson!

As always, you can find my code for this assignment on my github: here

This week we were given data that we needed to convert into a data frame and then create a boxplot and a histogram to visualize the data. Below is the code I used to create my data frame “hospital_data” using the data.frame() function.

I decided to use ggplot2 to create my boxplots and histogram, as I am most comfortable with that package over base R, and because I enjoy the customization options that ggplot2 offers.

For my boxplots, I decided to make three boxplots to compare the average blood pressure and doctor decision in the First, Second, and Final decision. This required me to load the ggplot2 and gridExtra package to properly present and compare my boxplots

These boxplots allowed me to see that the first doctor favored low blood pressure patients, as opposed to the external doctor and ER doctor, who deemed higher blood pressure patients to be the top priority.

Meanwhile, my histogram shows frequency of hospital visits doesn’t necessarily translate to higher priority cases, at least in terms of blood pressure cases.

As always, link to my Github: here