Will all this AI investment pay off? by Sergey Alexashenko raises an interesting point:
For investors to make money “because AGI”, a few things have to happen:
(...)
7. AGI can’t result in such a drastic change in society that the concept of money as we know it disappears.
of course this isn't a problem specific to AI and AGI - systems try to preserve their own relevance. the question is whether we, as a society, even want this and in what way it may hold back the next iteration of societal systems. what if we stop maximizing for wealth, either due to desire or necessity, and focus on a different value, like happiness and quality of life or minimize the worst effects of the climate catastrophe?
AGI developed by investors will be bred with the expectation of furthering the economic system that enabled it, but - like human children - it's impossible to say whether they'll really adopt their "parental" value system instead of developing their own.