Odd problem connecting to IEx session

So, for some reason, on our production box, we started can no longer connect to the app on the back end. (The app is running fine, serving pages, etc.). We can log in to the server, and see that it’s running, e.g. ps -ef | grep iex etc.

Normally, we just run a script that looks like this:

#!/bin/bash
# 
# A simple wrapper to facilitate connecting to an elixir
# node that was started with `elixir-start`
# 
# Usage:
# node-attach
# 

# exit if any any command fails
set -e

# establish the name and cookie (if available)
name="console@$(hostname -I)"
cookie="${COOKIE:-nanobox}"
remote="${HOSTNAME//./-}@$(hostname -I)"

iex \
  --name $name \
  --cookie $cookie \
  --remsh $remote

Normally, it works like a charm. For some reason, it stopped working.

You can also connect by hand, by starting an iex session with the appropriate name & cookie, then doing a Node.connect (which works!).

But the problem is that when I do a Control-G, instead of switching sessions, it sends the character ^G to iex.

Likewise, the up arrow sends a ^[[A, tab sends an actual tab, etc.

The other odd thing is if I type erl I just get

Eshell V9.1.5  (abort with ^G)
1>

Whereas on the box that’s working (and my local box) I get

Erlang/OTP 20 [erts-9.1.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [kernel-poll:false]

Eshell V9.1.5  (abort with ^G)
1> 

I am deploying with nanobox, and am hoping they have some ideas, but just wanted to post here if anyone else had come across something like this before.

Thanks in advance!

ps also, bash works fine, e.g tab for auto-complete, up/down arrow keys have history, etc. Only when I jump into IEx do the keys start behaving oddly.

Sounds like your terminal configuration got messed up (not an erlang/elixir issue). You can try just running reset to reset the general terminal settings in the shell but that may or may not work depending on whatever terminal emulator you use, which may also have it’s own terminal resetting for it’s side of settings. Or just entirely kill all the terminals and shells you are connecting to and re-build them. :slight_smile:

However, it sounds like it’s seeing your connection as a pipe instead of a terminal, hence why it’s sending the codes instead of interpreting them.

It happens to me all the time though over a certain laggy SSH connection I deal with (of which thankfully just reset fixes mine).

Yeah I was hoping it would be something like that!

reset did indeed reset, but no dice. IEX still think up arrow is coming from a pipe. Also tried things like TERM=vt100 that kind of thing.

Thanks for the idea!

1 Like

I think you are on to something.

I think for some reason, erlang/iex thinks it’s getting input from a pipe or a non-standard terminal, as opposed to a regular terminal.

If I type erl just to get an erlang shell, auto-completion also doesn’t work.

Yep that’s pretty common with terminal issues, it’ll happen to every program you run that depends on testing the environment to determine if it is running interactive or not. :slight_smile:

It seems to be actually more complicated that than.

As crazy as this sounds, it appears that they compiled Erlang with a missing or broken ncurses library. There is a bug with the erlang shell that it won’t let you get a remote node if the TERM setting doesn’t match one of the values when Erlang was compiled.

You can see this bug here:

https://bugs.erlang.org/browse/ERL-162

and here:

https://stackoverflow.com/questions/40146155/erlang-remote-shell-not-working . There is a problem when TERM is absent or is not known to ncurses which Erlang is built against, making remote shell connection fail silently.

Basically they need to re-compile Erlang (both 20.1.7 and 20.2.2) with a proper ncurses

2 Likes

Wow… who messed up that compile?! Was it ErlangSolutions? I’ve not seen them mess up with such a build before… (I run ErlangSolutions builds on my ‘prod’ servers for over a decade without issues, and on dev I usually compile things myself)…

1 Like