yajwt Error when decoding jwt

I am trying to decode a jwt from Cognito, I retrieved the public key from the client portal and it looked like this:

{
	"keys": [{
		"kid": "1234example=",
		"alg": "RS256",
		"kty": "RSA",
		"e": "AQAB",
		"n": "1234567890",
		"use": "sig"
	}, {
		"kid": "5678example=",
		"alg": "RS256",
		"kty": "RSA",
		"e": "AQAB",
		"n": "987654321",
		"use": "sig"
	}]
}

Using yajwt I tried to verify the jwt signature and got this error:

{:error, {:iss, "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_ay8FtC5WW"}}

Decode code:

JWT.verify(token, %{
  alg: "RS256",
  key: JWT.Algorithm.RsaUtil.public_key(key)
})

The key is pem format and parse to RSA using the JWT package (yajwt)

The error does not make any sense to me, it just throws an error and returns the iss and the aud plus the exp if the token has expired. At this point I have worked through invalid signature errors and invalid algorithm errors.

Has anyone encountered this error before?