I have been tasked with fine-tuning an OpenAI model using our customer data, which primarily originates from a Drupal website. The data includes various types of information, such as news articles, blog posts, service descriptions, and more.

Extracting the data is relatively straightforward, but preparing it for fine-tuning is proving to be more challenging. OpenAI expects the data to be in the following format:

[
  {
    "prompt": "Question or context",
    "completion": "Desired response"
  }
]

Are there any tools I can use to transform the data into this format? I would particularly prefer an Elixir-based tool, if one exists. However, I’m open to suggestions for other tools or services that can help with this task.

I’ve considered using the OpenAI API to generate questions and answers based on the provided data, but I’m concerned this approach could become quite expensive.

Additionally, the data will likely need some preprocessing and normalization, as much of it is wrapped in HTML.

I’m new to this field and would appreciate any advice or recommendations on how to approach this problem.