LLMs will not replace us, but this might (lab-grown HI (human intelligence))

Call me a skeptic, but I don’t think that clickbait will be replacing humans anytime soon.

EDIT: Shucks… They edited the title, and now I look like a fool!

If you make a thinking human brain, how is that different from someone seriously multi-handicapped? It would be alive, human, forced to play doom on repeat, and they want to reintroduce slavery presumably in data centers… I don’t think this is the way.

I didn’t edit the title, the moderator did.

Me neither. But one thing is for sure - if/when they do grow a super-human human-intelligence out of this, we’ll finally get the answer to one of the oldest questions - Is there such thing as a soul?

Whether we personally like it or not, my take is they are going to do it. China will 100% and the US will because otherwise China would be the only one doing it.

Also, I don’t think the ultimate plan is to have it (just) play Doom.

How so? It won’t prove anything regardless of how building human-intelligence goes. Soul by definition is in some other plane, it’s not a material thing and no amount of research happening in ‘material world’ will prove or disprove it’s existence.
Personally I don’t believe in supernatural and think that there’s no soul, because there’s no reason to believe otherwise (other than beliefs and religions that someone taught you and were not derived from first principles), but that’s kind of off-topic.
You can have self-awareness / consciousness without soul, maybe that’s what you were more getting at.

Just to better understand your angle: Does AGI (although still an abstract concept) qualify for “self-awareness”?

I hate this “<New tech name> will REPLACE YOU” rhetoric. It sounds more like spreading fear, instead of advertising tech.

Wasn’t my intention to spread FUD, but rather to attract attention to what I personally find far more plausible to be (some day) capable of autonomously developing software than LLMs.

I guess the exact line is a bit blurry and I think that proving self-awareness is also a fools errand to some extent, because how are you going to tell apart self-awareness from something perfectly pretending to be self-aware (I’m not even sure if there’s difference between the two).
Because of this dilemma I think we can say that yes, it would have to be classified as self-aware (unless we could imply from the underlying mechanism that it’s just play-pretend simulation like LLMs).
So if we were to create AGI by trying to recreate human brain (let’s say we run perfect simulation of our neurons) then we’d have to assume it’s self-aware. That probably doesn’t fully flesh out the topic though, it might depend on how exactly the AGI would be implemented (AGI itself might not always imply self-awareness is what I’m getting at).

I believe that self-awareness must (among other “more established” criteria) meet the following two conditions:

1. Arriving to the notion of self without being taught of it.
2. Genuinely failing to grasp (feel) the possibility of an absence of self (as in before birth and after death).

I don’t think this is feasible in the foreseeable future. Nevertheless, my point here being if we manage to engineer a “perfect” human brain and it turns out a complete human mind (with self’-awareness and all the goodies that come along), then I believe we’ll be able to conclude that there is no such thing as a soul. However, if we achieve that and yet we only get a super-(logically) intelligent human computer, then we’ll have something to puzzle ourselves with.