JsonRemedy - A blazingly fast, Elixir-native JSON repair library

gtcode

JsonRemedy is an Elixir-native JSON repair library that intelligently fixes malformed JSON strings. It leverages Elixir’s binary pattern matching and functional composition for superior performance and elegant design.

Motivation

Large Language Models and other AI systems often produce almost valid JSON. JsonRemedy is designed to repair these issues while preserving the intended data structure, making it ideal for robust JSON handling in Elixir applications dealing with AI model outputs, legacy systems, or unreliable data sources.

Key Features

Elixir-Native Performance: Uses binary pattern matching for speed.
Intelligent Repair: Context-aware fixes for various JSON syntax and structural issues.
Blazing Fast: Leverages BEAM optimizations.
Detailed Logging: Optional repair action tracking.
Functional Design: Immutable, composable, and testable.
Multiple Strategies: Choose from different parsing approaches based on your needs.

Repair Capabilities

JsonRemedy can fix a wide range of issues, including:

Syntax Fixes: Missing quotes, unquoted keys, single quotes, trailing commas, missing commas.
Structure Repairs: Incomplete objects or arrays, missing colons.
Value Corrections: Boolean variants (e.g., True to true), null variants, unquoted strings.
Content Cleaning: Removes code fences (e.g., ```json), comments, and extra surrounding text.

Quick Start

Add json_remedy to your mix.exs dependencies:

def deps do
  [
    {:json_remedy, "~> 0.1.0"}
  ]
end

Basic Usage

# Repair and parse in one step
malformed_json = """
{
  name: "Alice",
  age: 30,
  active: True
}
"""

{:ok, data} = JsonRemedy.repair(malformed_json)
# => %{"name" => "Alice", "age" => 30, "active" => true}

# Get the repaired JSON string
{:ok, fixed_json} = JsonRemedy.repair_to_string(malformed_json)
# => "{\"name\":\"Alice\",\"age\":30,\"active\":true}"

How It Works: The Elixir Advantage

JsonRemedy leverages Elixir’s strengths by using hyper-optimized binary pattern matching, allowing for 10-100x faster processing than traditional character iteration and providing zero-copy binary operations. It also features composable repair functions for modularity and offers multiple parsing strategies including binary pattern matching, parser combinators, and stream processing.

Performance Benchmarks

JsonRemedy delivers exceptional performance:

4.32M ops/sec for valid JSON parsing.
90,000+ ops/sec for malformed JSON repair.
< 8KB peak memory usage for repairs.

It significantly outperforms the original Python json-repair library in speed and memory efficiency.

Real-World Use Cases

JsonRemedy is ideal for:

LLM Integration: Extracting data from AI responses that often contain malformed JSON.
Data Pipeline Healing: Processing data from external APIs or unreliable sources.
Config File Recovery: Loading configuration files that might have minor syntax errors.

Contributing

We welcome contributions! JsonRemedy is designed to be modular, testable, extensible, and performant. Check the GitHub repository for development setup instructions and guidelines for adding new repair rules.

License

JsonRemedy is released under the MIT License.

https://github.com/nshkrdotcom/json_remedy

3 comments

#library #json_repair

9 776 3

2025-06-08 06:31:35 UTC

Showing Posts 1 to 3

Show Best Posts
Show All (oldest first)
Show All (newest first)

Eiji

Hmm … I have so much mixed feelings.

It’s amazing that people realise that LLM are not perfect. It’s even better if they are trying to fix it’s issues.
However no matter how I look at it parsing an LLM output looks like a great example of a “workaround” term - are workarounds good in production?
And as always … an algorithm like that would soon replace all developers - is anyone still believing in it?

TLDR: Is it good for a typical production having in mind 2nd point?

What I mean in point 2 is … I definitely not disagree with any effort to improve things, but there are many ways to fix things. Assuming that the source is untrusted is definitely good for user input just for example for a security reasons. However assuming `LLM` would fail and trying to fix its output feels like going all around a thin lake.

It’s very hard to predict an LLM predictions, so we can fix only a limited subset of possible failures. What I want to ask is like … did you wrote it for a very small niche or you expected to deliver it for a typical production? Let’s keep in mind that many clients are … like a walls, so you can’t explain them details as they don’t care and expect the work “to be done” and “just work as expected” or something like that.

Since we have no idea of all possible cases is such approach worth in long term? For example what if LLM would generate a valid, but different format like XML? For sure there are simpler examples like subsets of JSON like YAML. This way in fact requires supporting everything we can support which even using best tools and practices may not be optimal.

I wonder how much of possible cases you catch. Do you have a list of them or plans to expand your library?

Below is a detailed Elixir Code Style Guide that incorporates the features you asked about (@type t and @enforce_keys) and other related Elixir constructs, such as structs, type specifications, module attributes, and documentation.

I smell AI-generated text. Also I recommend to not click the links as they are incorrectly pointing to Google.

With that said I took a look at your code and found many String.replace/3 calls. Don’t want to make another huge point, so I will just comment a single example:
https://github.com/nshkrdotcom/json_remedy/blob/db0780e56c72c57a981634e055943d53cdabe070/lib/json_remedy/binary_parser.ex#L121-L121

I really have no experience with invalid JSON, so correct me if I’m wrong, but this could be written in 2 different ways like:

def example("```json" <> rest) do
  rest
  |> trim_leading_whitespaces()
  |> example()
end

defp trim_leading_whitespaces(<<char::utf8, rest::binary>>) when char in @whitespace_codepoints

# or

String.trim_leading(string, ~r/```json\s*/)

The first version is of course more verbose comparing to 2nd. Both are replacing a pattern only at the beginning of the input string, so if it occurs it ends as soon as pattern ends in string and the rest of the string is used as-is.

Having this inconsistency in mind and the previous point I wonder if the code was also generated by AI and if so it lacks verification. There are so much (although good) words about code quality in mentioned markdown file, but at first sight it looks like a completely opposite?

Post #1