Building a SOAP endpoint in Phoenix

I’m researching the possibility of reviving a project that makes use of ancient SOAP technology. Since hardware is involved, a switch to a newer technology stack is not option, unfortunately.

Has anybody been successful at serving SOAP requests somehow, in a Phoenix project? I see there are some SOAP client libraries on hex.pm, but no SOAP servers.

There was a prior discussion over here, but it concluded that it’s a no-go for Elixir. I hope to find some new evidence that it is possible.

I’m curious if this could be solved by accepting the HTTP requests as usual with Plug or in a Phoenix controller. But then call out to a lower level implementation (possibly C or C++) that does the legwork of parsing the request. Then act upon that request and formulate a response in plain Elixir. And then finally encode the Elixir response by passing it once more through the low-level library that generates the required XML that can be send back by the Plug connection.
I’ve seen libraries like Apache Axis and gSOAP that theoretically could do such magic. But I’m not sure what it would take to actually create this integration.

Other nudges toward better ideas are very much appreciated.

PS: the wsdl specification I’m interested in, if it is of any help, is located here and documented here. It’s a specification for distributing Daisy talking books (audiobooks).

I’d say if the clients (hardware devices that can’t have their software changed, I take it?) do not utilize a central repository of WSDL and SOAP registries and hundreds of schemas then you’re better off just constructing the XML yourself manually. Both Erlang and Elixir have numerous libraries for this but if you struggle with it, we can help.

Ultimately it all boils down to this: WSDL / SOAP are just producing and parsing XML according to certain XML Schema files. That’s it really. Don’t be threatened. No need to shell out to other programming languages unless time is super tight.

1 Like

Yes, these are audiobook players for sight disabled people. A niche hardware thing (although these things are disappearing, as the smartphones have become very accessible too). To be inclusive, even this old technology needs to be supported.

I have the impression that parsing SOAP requests can become very complex, very fast, with its complex structure and nested namespaces… but maybe it is doable :thinking:

1 Like

XML can become abstractions nested within abstractions, but the question will be how many of those you actually need to interact with. The spec you linked looks like a managable thing to work with.

1 Like

If you have a wsdl file for the service, you might be able to use the plug from this library. soapex/lib/soapex/plugs/soap.ex at master · kim-company/soapex · GitHub. We are using this library with a wrapper around this plug to serve a production soap/wsdl api. I’d be happy to help if you have additional questions :smile:

2 Likes

Interesting! I have trouble understanding what the soapex library actually does. I see that it depends on detergent. I rejected it when I first came across detergent for some reason I don’t recall.

It’s the first time I see the use of Records.

Where would I need to “inject” the wsdl, at compile-time? I guess you don’t have an example usage lying around? :slight_smile:

Am I correct to assume this library allows you to intercept SOAP messages at the “operation” level, passing in the parsed arguments? And then helps encoding response, according to the wsdl definition? If so: great!

Sorry for the late reply. We create a custom plug that handles the parsing of the incoming soap requests and some utilities for sending a reply.

Using this plug builds a model from the wsdl at compile time, parses and validates incoming requests, and; similar to a controller; executes a function with the same name.

# Example module
defmodule YourCoolSoapService.ServicePlug do
  # Usage
  # wsdl_file: path to your wsdl file. xsd files should be relative to the files too
  # prefix: this prefix will be 
  use SoapService.Soap.Plug, wsdl_file: "your/wsdl/file/here.wsdl", prefix: "abc"

  # Imagine you have an operation called playMusic with 
  # the request/response types playMusicRequest and playMusicReply
  def play_music(conn, abc_playMusicRequest() = params) do
    # play the music
    case play_the_music(params) do
      {:ok, result} -> soap_resp(conn, abc_playMusicResponse())
      {:error, error} -> client_fault(conn, "That was your fault")
    end
  end
end
# Generic soap error
defmodule SoapService.SoapFaultError do
  defexception [:detail, :message]
end

defmodule SoapService.Soap.Plug do
  defmacro __using__(config) do
    quote location: :keep do
      # extracts the records from the wsdl
      use Erlsom.Records, unquote(config)

      # Expose utilities
      import SoapService.Soap.Plug
      alias SoapService.SoapFaultError

      # extracts records for the wsdl terms
      for record <- [:wsdl, :"soap:detail", :"soap:Fault", :"soap:Body", :"soap:Header", :"soap:Envelope"] do
        record_name = Atom.to_string(record) |> String.replace(":", "") |> String.to_atom()
        Record.defrecord(record_name, record, Record.extract(record, from_lib: "detergent/include/detergent.hrl"))
      end

      def init(opts), do: opts

      def call(conn, _opts) do
        # read the body, parse it and validate it against the schema
        {:ok, body, conn} = Plug.Conn.read_body(conn)
        # erlsom_model defined at compiletime by using Erlsom.Records
        model = erlsom_model()

        case :erlsom.scan(body, model) do
          # Valid wsdl operation
          {:ok, result, _rest} ->
            soap_body = soapEnvelope(result, :Body)

            # Extract the requested operation
            case soapBody(soap_body, :choice) do
              [soap_op] ->
                action = record_to_action(soap_op)

                # Apply the matching function for the requested operation
                try do
                  apply(__MODULE__, action, [conn, soap_op])
                rescue
                  e in SoapFaultError ->
                    client_fault(conn, e.message, e.detail)

                  e ->
                    require Logger
                    Logger.error(Exception.format(:error, e, __STACKTRACE__))
                    client_fault(conn, e.message)
                end

              _ ->
                client_fault(conn, "No such operation")
            end

          {:error, error} ->
            client_fault(conn, error)
        end
      end

      # Helper function that wraps the response with the proper soap terms
      def soap_resp(conn, response) do
        envelope = soapEnvelope(Header: soapHeader(), Body: soapBody(choice: [response]))
        SoapService.Soap.Plug.soap_resp(conn, envelope, erlsom_model())
      end

      # Helper function to build non-raising error
      def client_fault(conn, message, detail \\ nil) do
        detail =
          if detail do
            soap_detail("#any": [detail])
          else
            :undefined
          end

        fault =
          soapFault(
            faultcode: {:qname, 'http://schemas.xmlsoap.org/soap/envelope/', 'Client', 'soap', ''},
            faultstring: to_charlist(message),
            detail: detail
          )

        soap_resp(conn, fault)
      end
    end
  end

  # Transform the result into a proper soap response and send it
  def soap_resp(conn, envelope, model) do
    {:ok, xml} = :erlsom.write(envelope, model)

    conn
    |> Plug.Conn.put_resp_content_type("text/xml")
    |> Plug.Conn.send_resp(200, :erlsom_ucs.to_utf8(xml))
    |> Plug.Conn.halt()
  end

  def record_to_action(tuple) do
    tuple
    |> elem(0)
    |> Atom.to_string()
    |> remove_prefix()
    |> Macro.underscore()
    |> String.to_existing_atom()
  end

  defp remove_prefix(string) do
    if new_string = do_remove_prefix(string) do
      new_string
    else
      string
    end
  end

  defp do_remove_prefix(""), do: nil
  defp do_remove_prefix(":" <> rest), do: rest
  defp do_remove_prefix(<<_, rest::binary>>), do: do_remove_prefix(rest)
end
2 Likes

Thanks for the detailed reply!

I’m a bit confused. The sample code you’re sharing does not rely on soapex. It’s directly using detergent and erlsom. Is this the way you’re setting things up in your application too?

I’m getting this error:

module Erlsom.Records is not loaded and could not be found

I’m not very familiar with integrating an Erlang library into an Elixir project, so maybe I’m missing something here. Adding :erlsom as an application maybe, somewhere in the mix file?

The module Erlsom.Records is actually part of the soapex library: soapex/lib/erlsom/records.ex at master · kim-company/soapex · GitHub

1 Like

That helped :slight_smile: I’m understanding the pieces better now.

I’m bumping into the issue that using my specific wsdl (and referenced xsd’s) results in this error:

...
defining P_subject to P:subject with [any_attribs: :undefined, "#any": :undefined]
defining P_text to P:text with [any_attribs: :undefined, text: :undefined]
defining P_timeOffset to P:timeOffset with [any_attribs: :undefined, time_offset: :undefined]
defining P_title to P:title with [any_attribs: :undefined, "#any": :undefined]
defining P_title to P:title with [any_attribs: :undefined, text: :undefined, audio: :undefined]

== Compilation error in file lib/wsdl_test_web/plugs/test_plug.ex ==
** (ArgumentError) cannot define record :P_title because a definition P_title/0 already exists
    (elixir 1.16.0) lib/record.ex:291: Record.error_on_duplicate_record/2
    (elixir 1.16.0) lib/record.ex:312: Record.__record__/5
    lib/wsdl_test_web/plugs/test_plug.ex:3: anonymous fn/2 in :elixir_compiler_4.__MODULE__/1
    (elixir 1.16.0) lib/enum.ex:2528: Enum."-reduce/3-lists^foldl/2-0-"/3
    lib/wsdl_test_web/plugs/test_plug.ex:3: (module)

I turned on some logging to get the defining P_... lines. I guess there is a problem with overlap in the names that some xsd’s use. Is there a way to keep these separate? I don’t know where that prefix P is coming from.

I see these namespaces in the erlsom model:

[
   {:ns, ~c"http://purl.org/dc/elements/1.1/", ~c"P", :qualified},
   {:ns, ~c"http://schemas.xmlsoap.org/soap/envelope/", ~c"soap", :unqualified},
   {:ns, ~c"http://www.daisy.org/ns/daisy-online/", ~c"abc", :qualified},
   {:ns, ~c"http://www.daisy.org/z3986/2005/bookmark/", ~c"P", :qualified},
   {:ns, ~c"http://www.w3.org/2001/XMLSchema", ~c"xsd", :qualified},
   {:ns, ~c"http://www.w3.org/XML/1998/namespace", ~c"P", :unqualified}
 ]

abc is the prefix I passed in, the P prefix seems to be some kind of default?

Hmm. I see where it happens in detergent:

The todo gives it away: “using the same prefix for all XSDS makes no sense”

I had some time to experiment with soapex and its building blocks (detergent and erlsom). As mentioned above, I have some problems for my specific case:

  • Identical element names, but in different namespaces, collide when defining records for them (I’m quite confident this can be fixed somehow).
  • I found out that some of the SOAP clients I’ll serve don’t respect the sequence order of input parameters (elements in <xs:sequence> groups define an explicit order, while the <xs:all> grouping allows any order of elements) . Existing (Java based, JAX-RS) implementations of the service seem to be able to cope with these naughty clients, so I’ll have to find a way to deal with that too (I have little control over the clients and how well-behaving they are). Erlsom is strict about it, so this would be an additional hurdle.

So, I’m back at the drawing board, thinking about how to parse a SOAP message manually, but without being strict. It seems to me that Erlsom, the underlying xml parser of soapex and detergent, is very schema-driven. I’ve learned a bit about Records. It seems quite heavy for my relatively simple situation. The SOAP operations and their inputs and outputs are not very complicated. So I’d suspect that other parsing methods, not relying on a compile-time schema and corresponding Record definitions, would work as well. Maybe with the added benefit that I can be less strict about some aspects (like ordering, as mentioned before).

One thing that still complicates things is namespaces. Clients can alias these namespaces however they like. The only xml parsing library I know that supports namespaces is sweet_xml. XPath expressions can then be written in a way that is decoupled from which aliases are chosen. Are there other options? I’ve learned that saxy does not support namespaces natively.

Who would’ve thought that I’d be digging into XML parsing strategies in year and age… :man_shrugging:

But do you really need the namespaces? Are they important in how do you respond to the clients?

The requests come in with the namespace. So I have to parse it somehow. No dodging that requirement.