Some users on reddit say that I might have been hacked.
After reading through this whole thread and seeing the non-Postgres activity showing under the Postgres user I strongly believe you were hacked. Something CPU intensive is running on your box in an obfuscated way, something like blockchain mining.
Throw away the droplet and start a new one!
7 Likes
Oh man yeah definitely agree with @sorentwo. How is your box secured? If you use ssh it shouldnāt be possible to hack.
The fact that the command name is obfuscated as a random string definitely sounds very suspicious. We focused on the queries, but the hint was already in the first screenshot. Good lesson about making small steps, checking assumptions, and not trying to outsmart the problem 
If you rebuild the droplet, make sure to double check that the postgres ports are not reachable from outside. Itās a bit suspicious that the process runs with the postgres user, I wonder if the attack vector was somehow related to Postgres.
1 Like
Reminds me of this
https://www.imperva.com/blog/deep-dive-database-attacks-scarlett-johanssons-picture-used-for-crypto-mining-on-postgre-database/
4 Likes
Wow, fascinating read!
Amazing read.
1 Like
- I ssh as non-default root user into the droplet, but I think i never disabled password login and also never disabled root user.
- Few weeks back I had enabled postgres ports in ufw and asked postgres to listen to all ip addresses because I wanted to access prod DB in my local pgadmin client. Yes, sounds embarrasing now. Btw, I was always using non-default user for DB with a very very bad password, dont know how the hacker guessed my DB user though.
- Ever since I started my droplet a few months ago, my app log showed strange traffic coming to non-existing routes in my app e.g. /wp-admin.php, /admin/form.php etc. I never payed attention.
1 Like
Yes, will do that right away.
That will always, always, always happen to anything thatās accessible over the internet. Now, obviously, no hacker got you by getting your Elixir app to run PHP, but itās just a sign of how overwhelmingly aggressive hackers are.
As another sign, check your SSH logs. You will see login attempts to common user names with common passwords, continuously, probably starting minutes after your server went up.
2 Likes
I think using a SSH tunnel would be more secure.
This way you dont need to expose you database to the internet
5 Likes
I just viewed ssh logs and my mind is blown. Now I realise how lightly I have taken server security. From now on, what precautions on ssh security should I take ? apart from, disabling root user, disabling password authentication ?
1 Like
I will read up on how to do that. I prefer using a GUI on client to access prod DB than pqsl prompt.
Iām not an ops guy, but I know that the company Iām working for does add fail2ban to about every (virtual) machine we deploy.
It does not make your server secure per se, but by recognizing and blocking malicious IPs on the firewall level, it will take load away from your application.
Additionally, as you already said, root user is beeing disabled for SSH access completely.
All ports closed, except for 22 and those needed for the public facing services. So usually 22, 80, 443.
iptable rules are written in a way, that they allow access to internal services only for IP addresses from a given ātrustedā IP range, eg. the privat subnet in the datacenter.
No SSH login via password, not even very strong ones.
Carefully written rules in the /etc/sudoers.
Individual services might have their own rules to make them secure.
5 Likes
I follow these 2 articles how to secure the server after initial setup:
Sometimes I run ssh on a different port. Not sure if itās really needed with public key login, but gives an extra peace of mind 
3 Likes
Canāt thank you enough for the points.
Ops team reported this doesnāt change much. They experimented with that and it made operating that server slightly harder, while suspicious requests have been roughly the sameā¦
Amazing, isnāt it? What others said, particularly key-based auth for SSH.
I also have a rule that for replication to an off-site PG cluster, I do not allow an inbound connection to the master. The master initiates a reverse SSH tunnel.
And above all, parameterized queriesādonāt take user input and interpolate it into SQL queries. Of course, normal uses of Ecto give you that for free. But if you run into a case where you have to manually build SQL fragments, be careful.
Thereās no need to use psql if you choose to use an SSH tunnel (just in case you assumed that because psql is used in this tutorial. You can choose whatever client you like, you just need to configure it to connect to localhost instead of the remote host.
1 Like
What GUI clientt do you use?
Most of them have the option to connec through ssh tunnel.
