USF Information Science Blog

Package: openEvalR
Title: Parsing and Extracting Tools for Open AI Evaluation JSON files
Version: 0.0.9000
Authors@R: Alina Hagen <alinahagen@usf.edu>
Description: Simplifies parsing and analyzing OpenAI evals .jsonl files in R
Depends: R (>= 4.4.2)
Imports: jsonlite, readr, tidyverse 
License: CC0
LazyData: true

Package proposal: The package that I am proposing is a parsing package that would make working with OpenAI evaluation .jsonl files much simpler.

The reason I am proposing this package is because it would actually make my life, currently, so much easier if this package existed. For a bit of context, I currently work for Dr. Hagen and will be attending a conference in 2 months presenting a study where the foundation is build on Open AI’s evaluation portal. The last 2/3 weeks every time I have had to export our evaluation results, it has been a fight to get the .jsonl files into any usable format so we can even perform our calculation. So I would like to build a package that would possibly make that process much easier, especially as I will continue to have to continue fighting these .jsonl files until further notice.

The primary functions I would like to create are: converting the .jsonl file into a dataframe, a function for extracting specific columns, a function for extracting different model batches, a summary function that summarizes the reported accuracy, and a function that reports Cohen’s Kappa for the ground truth vs the model(s)

Link to my github: here