gRPC: SSLV3_ALERT_BAD_CERTIFICATE with Elixir client to Python server, even though Python to Python works fine with the same certificate

I’m trying to call Python functions from Elixir using Python’s official grpc module.

Elixir side I’m using elixir-grpc.

Since I’lll be calling across the open internet, I need SSL/TLS security. So I created certificates:

openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 365 -nodes -subj "/CN=xx.xx.xx.168" -addext "subjectAltName = IP:xx.xx.xx.168"

The python call to python works fine, in both insecure and TLS/SSL versions. But the Elixir call to Python, which works fine with no TLS/SSL, gives me an SSLV3_ALERT_BAD_CERTIFICATE error on the server side if I enable TLS/SSL, even though they’re using exactly the same certificates.

Python server:

import asyncio
import logging

import grpc
from hellostreamingworld_pb2 import HelloReply
from hellostreamingworld_pb2 import HelloRequest
from hellostreamingworld_pb2_grpc import MultiGreeterServicer
from hellostreamingworld_pb2_grpc import add_MultiGreeterServicer_to_server

NUMBER_OF_REPLY = 10


class Greeter(MultiGreeterServicer):

    def __init__(self):
        self.my_number = 0
        asyncio.create_task(self.do_stuff_regularly())

    async def do_stuff_regularly(self):
        while True:
            await asyncio.sleep(10)
            self.my_number -= 1
            print(f"my_number: {self.my_number}")

    async def sayHello(
        self, request: HelloRequest, context: grpc.aio.ServicerContext
    ) -> HelloReply:
        logging.info("Serving sayHello request %s", request)
        for i in range(self.my_number, self.my_number + NUMBER_OF_REPLY):
            yield HelloReply(message=f"Hello number {i}, {request.name}!")
        self.my_number += NUMBER_OF_REPLY


async def serve() -> None:

    with open('server.key', 'rb') as f:
        private_key = f.read()
    with open('server.crt', 'rb') as f:
        certificate_chain = f.read()

    # Create server credentials
    server_credentials = grpc.ssl_server_credentials(((private_key, certificate_chain),))


    server = grpc.aio.server()
    add_MultiGreeterServicer_to_server(Greeter(), server)
    #listen_addr = "[::]:50051"
    listen_addr = "xx.xx.xx.168:50051"
    server.add_secure_port(listen_addr, server_credentials) # secure uses the cert
    #server.add_insecure_port(listen_addr) # insecure
    logging.info("Starting server on %s", listen_addr)
    await server.start()
    await server.wait_for_termination()


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    asyncio.run(serve())

Python client:

import asyncio
import logging

import grpc
import hellostreamingworld_pb2
import hellostreamingworld_pb2_grpc


async def run() -> None:
    with open('server.crt', 'rb') as f:
        trusted_certs = f.read()

    # Create client credentials
    credentials = grpc.ssl_channel_credentials(root_certificates=trusted_certs)

    async with grpc.aio.secure_channel("xx.xx.xx.168:50051", credentials) as channel:
        stub = hellostreamingworld_pb2_grpc.MultiGreeterStub(channel)

        # Read from an async generator
        async for response in stub.sayHello(
            hellostreamingworld_pb2.HelloRequest(name="you")
        ):
            print(
                "Greeter client received from async generator: "
                + response.message
            )

        # Direct read from the stub
        hello_stream = stub.sayHello(
            hellostreamingworld_pb2.HelloRequest(name="you")
        )
        while True:
            response = await hello_stream.read()
            if response == grpc.aio.EOF:
                break
            print(
                "Greeter client received from direct read: " + response.message
            )


if __name__ == "__main__":
    logging.basicConfig()
    asyncio.run(run())

Elixir:

iex(1)> cred = GRPC.Credential.new(ssl: [cacertfile: Path.expand("../../server.crt")])
%GRPC.Credential{
  ssl: [
    cacertfile: "/home/tbrowne/code/official_grpc_example/examples/python/hellostreamingworld/server.crt"
  ]
}
iex(2)> {:ok, channel} = GRPC.Stub.connect("xx.xx.xx.168:50051", adapter: GRPC.Client.Adapters.Gun, cred: cred)

15:53:52.155 [notice] TLS :client: In state :wait_cert_cr at ssl_handshake.erl:2172 generated CLIENT ALERT: Fatal - Bad Certificate


15:53:53.187 [notice] TLS :client: In state :wait_cert_cr at ssl_handshake.erl:2172 generated CLIENT ALERT: Fatal - Bad Certificate


15:53:55.127 [notice] TLS :client: In state :wait_cert_cr at ssl_handshake.erl:2172 generated CLIENT ALERT: Fatal - Bad Certificate

Is there something funky going on with Erlang/Elixir certificate files that I’m not aware of? I seem to remember having had some uphill a few months ago with Erlang’s certificate format which might be being used underneath? Worth mentioning that I get the same problem whether I use Mint or Gun as the transport.

This is honestly not an indicator. Erlang is pretty strict in regards to the format of certificates.

I don’t know if something changed, but from what I remember the bad certificate error is related to the certificate being in the wrong format or missing some mandatory fields, but once again I am not entirely sure as the SSLV3 things seem to be freshly added as I didn’t see them when I was working with ssl a year ago.

There are a few things you can do:

  1. Don’t generate your own self-signed certificates but do it like everybody does it in the industry, point a domain to your server and use something like letsencrypt to generate fully valid certificates;
  2. Decode a certificate that erlang thinks its valid and cross reference the fields with your self-signed certificate, maybe you are missing some options or the format is incorrect.

I don’t think any http (or similar) clients from elixir or erlang are using something else but the erlang ssl library for doing the actual network requests. Since that library handles certificate validation, there is no way around it.

2 Likes

Yeah used letsencrypt, got proper cert with an associated domain, again works with Python → Python, but with Elixir → Python I now get a TLSV1_ALERT_UNKNOWN_CA error. I’ll probably reach out to the elixir-grpc maintainers.

You can do that, but this is unrelated to the grpc library, so they will not be able to help you wtih anything.

This is pretty interesting, looks like the python server doesn’t return the full chain of certificates. The easiest way to debug this is to serve https traffic using the certificate you just created from with a nginx reverse proxy + your python server in http mode, because that is a well-known working setup. If that works, then this will confirm the fact that whatever https library that grpc server from python uses, it’s not up to the latest security standards.

What I haven’t done yet is Elixir ↔ Elixir. So I’ll test that with SSL/TLS first and if I can get that working then I can try to debug the python side.