HN Books @HNBooksMonth

Pretty decent article in some ways, but the book Superintelligence covered all this ground in much more detail in 2014.

https://www.amazon.com/Superintelligence-Dangers-Strategies-...

If you want to have a clear grasp of the AI/AGI challenge, "SuperIntelligence" by Oxford philosopher/mathematician Nick Bostrom is a must. https://www.amazon.co.uk/d/Books/Superintelligence-Dangers-S...

Now if you have a bit of a math/engineering background and want to understand how modern AI really works, the two best books/shortcuts to go from beginner to expert are: - Ian Goodfellow's "Deep Learning" http://www.deeplearningbook.org/ - Richard Sutton's "Reinforcement Learning: An Introduction" http://incompleteideas.net/sutton/book/

This article brings up an important source of bias that tech people risk - that we overuse models from programming when thinking about other aspects of the world. We should be learning alternative models from other subjects like economics, philosophy, sociology, etc so that we can improve our mental toolbox and avoid thinking everything works like a software system.

I'd say that another related source of bias is that we are surrounded by people who think like us.

It's a shame though, that the article dismisses without much explicit justification risk from artificial intelligence and the problem of death. When I first encountered these ideas, I dismissed them because they seemed weird. But if you read the arguments for caring you realise that they are actually well thought-out. For AI risk, check out http://waitbutwhy.com/2015/01/artificial-intelligence-revolu... and https://www.amazon.co.uk/Superintelligence-Dangers-Strategie.... For an argument for tackling death, checkout http://www.nickbostrom.com/fable/dragon.html.

Also, there's a clear answer to 'If after decades we can't improve quality of life in places where the tech élite actually lives, why would we possibly make life better anywhere else?' -- because the tech elite live in a rich society where most of the fundamental problems (e.g. infectious disease control, widespread dollar-a-day level poverty, access to education) have been solved. The remaining problems are much harder and we should focus on problems where our resources can go further - e.g. in helping the global poor. We should also work on important problems that we have a lot of influence over, such as risks from artificial intelligence and surveillance technology.

⬐ forgotuser

>we should focus on problems where our resources can go further - e.g. in helping the global poor.

I'm not convinced that the right people to help the "global poor" are Silicon Valley technologists. Scientifically-minded entrepreneurs focused on profit but with idealistic rhetoric attempting to "help" cultures they know nothing about has, historically, not worked out great.

⬐ richardbatty

I agree that silicon valley technologists are not well informed about the problems of the poor (or other important problems) and too often use the rhetoric of doing good whilst not thinking carefully enough about how to actually do that.

But I don't think concentrating on local problems is the right solution given that there are much more important non-local problems to solve. Perhaps entrepreneurs should concentrate on profit and then donate what they earn. Alternatively, they could learn about important problems and partner with experts in those problems to avoid naively doing more harm than good.

⬐ ArkyBeagle

Colin Woodward wrote "American Nations: A History of the Eleven Rival Regional Cultures of North America (ISBN-13: 978-0143122029)" and basically, the people who settled the center of the tech industry were Yankee Progressives, the sort who send third sons to be missionaries around the world. You have to accept the premise that somehow, the past stains present-day thinking more than is easily explainable, but this does not stop people who have demographically interesting professions from using this sort of thing.

⬐ yummyfajitas

From over here in India it looks pretty good.

In spite of the radically different culture, Uber has improved transportation. It turns out putting Indians into a car and charging them money works a lot like doing the same for Americans.

Facebook and Whatsapp have improved communication.

At work, Slack, Salesforce and Jira work the same as any western office.

Why, exactly, do you think other cultures and the global poor can't use SV technologies?

⬐ forgotuser

All those technologies were first developed in the US, for an American target audience, then exported to other countries. Other things also developed in the West and then exported: electricity, vaccines, et cetera. I agree that that's generally been a good thing.

The parent comment, however, seemed to imply that technologists should focus on "solving global poverty" instead of solving local problems. This represents a fundamental shift, because at that point SV technologists are attempting to solve problems they personally do not have experience with.

All the examples you provide solved problems that SV itself had- transportation, communication, etc.

⬐ yummyfajitas

I didn't interpret the article as defining things like Uber as solving "local" problems. But if you do, then the article is simply wrong because SV is manifestly solving local problems.

⬐ Chris2048

> overuse models from programming when thinking about other aspects of the world

Arguably we already do this when we try to map problem-domains to class-hierarchy structures (OOP)...

When/if we see these flaws and start using general data structures to model the world, we'll have moved forward...

⬐ None

None

⬐ igk

You pushed almost ALL of my buttons...note that I on every point I almost agree with your points (which of course meant I have to nitpick them:).

1. I agree that it is important to broaden your horizon and not blindly think everything works like software. But IMO, not enough people learn to attack problems like a good hacker. I'd argue that for understanding and modeling,the fundamental principles that drive hard sciences ( best discussed by Karl Popper and Feynman in my opinion) are as of yet the only methods which have proved themselves. For doing and building complicated things, (software) engineers have developed a way of zipping between layers of abstractions which is again in this domain of fundamental principles, but less fleshed out. Economics and other "soft" sciences is then (when its done well) about properly applying these principles from hard science and engineering to domains with incredibly scarce data and no real way to do experiments(because, you know, ethics). History and other "exploratory" sciences is then about gathering more data and cleaning it. And finally, philosophy and the arts are about pushing the boundaries of our imagination and to inspire us, so we can apply all of those other tools to new domains and cross fertilize.

2. To AI Risk: I said it before, i say it again, I am incredibly disappointed in the current AI risk movement. There is way to much focus on the vague "tail risk" of a rogue AI (be it by chance of by ill will) and some sort of "paperclip" scenario, and almost no mention of the very real, very right now structural risk of continued wealth concentration, mass surveillance (which is where we agree) and mass unemployment. The tail risk in my opinion is negligible, due to simple physical constraints computing faces right now, and to the fact that we have EMP. An actual AI apocalypse just isn't going to happen. And even if you handwave that with "tail risk"/rogue actors, then I ask: why not go after bioweapons? They are much easier (since we know they can work with our current tech, unlike AI) and about as dangerous. Now, the structural problems will combine with demographical change, a regress from the brief period of increased equality (and freedom) and other factors. That stuff is happening right now. There are enough resources pointing out how many jobs will simply be made redundant,yet there are only some movements talking about UBI and other schemes to move to this post-work society. Disproportionately more noise is made about the scary death robots.

3. Death: I hope research into immortality, healthy aging (i.e. "dying with a young body at 70" ) and the likes continue. But it is very much a first world problem, nay, a millionaires and above problem. I don't buy the argument of stopping death being infinitively important on account of if making every other intervention more important. A lot of the current problems in the world are artificial (or at least not mandated by the laws of physics) and due to too much power in the hand of too few, without checks and balances. Let us fix that first, then make our overlords immortal. Tangentially, there was a great article+ discussion on HN here (https://news.ycombinator.com/item?id=9523231) on the topic of Peter Thiel and his "libertarian future"

I apologize for going into rant mode, I hope I managed to make it somewhat congruent

⬐ paulryanrogers

"more noise is made about the scary death robots."

Likely because they make for better stories.

⬐ pdkl95

> We should be learning alternative models

In addition to learning models from other subjects, we also need to understand that the complexity and often chaotic nature of human behavior might mean some subject can only be modeled superficially.

> economics

Varoufakis[1] and Blyth[2] argue that it might be impossible to create models of the economy. They both warn that there is a seductive quality to math, but that "elegance" is an oversimplification that can easily unravel when given the chaos of the real world.

[1] https://www.youtube.com/watch?v=L5AUAIzciLE&t=1355

[2] https://www.youtube.com/watch?v=hmWbkPezgtU

⬐ gshubert17

It is certainly impossible to create realistic models of the economy, if one's economics is based on assumptions of linearity and equilibrium. Mark Buchanan's book _Forecast_ [0] makes the case for more realistic models, using techniques from the natural sciences, that would have better predictive value.

[0] https://www.amazon.com/Forecast-Physics-Meteorology-Sciences...

⬐ xyience

Is he rich from his superior models?

⬐ gshubert17

Good question. The book's blurb says he is a science writer; I don't know how rich he is.

I guess it's easier to point out the flaws in linear, equilibrium models, and to point to better results in, say, meteorology, from using better models based on different assumptions, than it is to construct and validate completely new economic models.

Important distinction: children get their genes from us and share many of our values by default. Computers do not share many of our values by default. Instead they do what they are programmed to do.

But the problem is that computers do what you say, not what you mean. If I write a function called be_nice_to_people(), the fact that I gave my function that name does nothing to affect the implementation. Instead my computers behavior will depend on the specific details of the implementation. And since being nice to people is a behavior that's extremely hard to precisely specify, by default creating an AI that's smart enough to replace humans is likely to result in a bad outcome.

Recommended book: http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

It sounds less weird if you think: "AI has automated some jobs, and eventually it may automate away most of them, in the same way (pending AlphaGo victory) it's now automated winning at most board games... So what happens if the job of programmer gets automated away?"

(I'm convinced that programmers will take the AI-automating-all-the-jobs idea more seriously once it's their own jobs that are on the line)

There are a few ways I can think of to object to this line of reasoning:

1) You could argue that programming will be a job that never gets automated away. But this seems unlikely--previous intellectual tasks (Chess, Jeopardy, Go) were thought to be "essentially human", and once they were solved by computers, got redefined as not "essentially human" (and therefore not AI). My opinion: In the same way we originally thought tool use made humans unique, then realized that other animals use tools too, we'll eventually learn that there's no fundamental uniqueness to human cognition. It'll be matter & algorithms all the way down. Of course, the algorithms could be really darn complicated. But the fact the Deepmind team won at both Go and Atari using the same approach suggests the existence of important general-purpose algorithms that are within the reach of human software engineers.

2) You could argue that programming will be automated away but in a sense that isn't meaningful (e.g. you need an expensive server farm to replace a relatively cheap human programmer). This is certainly possible. But in the same way calculators do arithmetic much faster & better than humans do, there's the possibility that automated computer programmers will program much faster & better than humans. (Honestly humans suck so hard at programming https://twitter.com/bramcohen/status/51714087842877440 that I think this one will happen.) And all the jobs that we've automated 'til now have been automated in this "meaningful" sense.

Neither of these objections hold much water IMO, which is why I take the intelligence explosion scenario described by Oxford philosopher Nick Bostrom seriously: http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

⬐ dclowd9901

There is one uniqueness to human cognition: the allowance to be fallible. AI will never be able to solve problems perfectly, but whereas we are forgiven that, they will not be because we've relinquished our control to them ostensibly on exchange for perfection.

⬐ astrofinch

Sorry, I found that a bit difficult to follow... it sounds like you think human programmers will beat AIs because we can make mistakes?

Well here's my counter: mistakes either make sense to make or they don't. If they make sense to make, they aren't mistakes, and AIs will make the "mistakes" (e.g. inserting random behavior every so often just to see what happens--it's easy to program an AI to do this). If they don't make sense to make, making them will not be an advantage for humans.

At best you're saying that we'll hold humans of the future to a lower bar, which does not sound like much of an advantage.

⬐ studentrob

It may interest you to know that machine learning algorithms often have an element of randomness. So they are allowed to explore failures. Two copies of the same program, trained separately and seeded with different random numbers, may come up with different results.

I'm not saying there will or won't be AI some day, I just thought that point was relevant to your comment.

⬐ studentrob

> (I'm convinced that programmers will take the AI-automating-all-the-jobs idea more seriously once it's their own jobs that are on the line)

I assure you we do not. We would all love to be the one to create such a program. But don't worry it isn't happening anytime soon in the form of singularity.

Just like other people in other jobs, we sometimes come up with more efficient ways of doing things than our bosses thought possible, and then we have some extra free time to do what we like.

⬐ astrofinch

"But don't worry it isn't happening anytime soon in the form of singularity."

Probably not soon. Worth noting that the Go victory happened a decade or two before it was predicted to though.

(To clarify, I was not trying to establish that any intelligence explosion is on the immediate horizon. Rather, I was trying to establish that it's a pretty sensible concept when you think about it, and has a solid chance of happening at some point.)

>Just like other people in other jobs, we sometimes come up with more efficient ways of doing things than our bosses thought possible, and then we have some extra free time to do what we like.

Yes, I'm a programmer (UC Berkeley computer science)... I know.

But don't listen to me. Listen to Stuart Russell: https://www.cs.berkeley.edu/~russell/research/future/

⬐ studentrob

> (To clarify, I was not trying to establish that any intelligence explosion is on the immediate horizon. Rather, I was trying to establish that it's a pretty sensible concept when you think about it, and has a solid chance of happening at some point.)

Well first you were saying programmers may hesitate to create an AI singularity because it may cost them their job. I said we would love to but probably won't anytime soon. Now you're saying that it's likely that it will happen some day. I'm not sure these points follow a single train of thought.

⬐ astrofinch

>Well first you were saying programmers may hesitate to create an AI singularity because it may cost them their job.

The line about programmers fearing automation only once it affects them was actually an attempt at a joke :P

The argument I'm trying to make is a simple inductive argument. Once something gets automated, it rarely to never gets un-automated. More and more things are getting automated/solved, including things people said would never be automated/solved. What's to prevent programming, including AI programming, from eventually being affected by this trend?

The argument I laid out is not meant to make a point about wait times, only feasibility. It's clear people aren't good at predicting wait times--again, Go wasn't scheduled to be won by computers for another 10+ years.

⬐ dgacmu

The fact that programming is an exceptionally ill-defined task. Computers are great at doing well-specified tasks. In many ways, programming is the act of taking something poorly-specified and making it specific enough for a computer to do. Go, while hard, remains very well defined.

I hope for more automation in CS. It will help eliminate the boilerplate and let programmers focusnin the important tasks.

⬐ dragonwriter

> In many ways, programming is the act of taking something poorly-specified and making it specific enough for a computer to do.

Software development in the broad sense is that, sure; not sure I'd say programming is that -- taking vague goals and applying a body of analytical and social skills to gather information and turn it into something clearly specified and unambiguously testable is the requirements gathering and specification area of system analysis, which is certainly an important part of software development, but a distinct skill from programming (though, given the preference for a lack of functional distinctions -- at least strict ones -- within software development teams in many modern methodologies, its often a skill needed by the same people that need programming skills.)

I think you haven't done even cursory research into A.I super-intelligence before dismissing it.

Try reading Nick Bostrom's "Superintelligence: Paths, Dangers, Strategies"[1] and then I think you will change your mind.

[1]http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

⬐ atroyn

I don't dismiss it, I just believe it to be unlikely. I am in the middle of Bostrom's book right now.

I recommend Superintelligence [0]. It explores different plausible paths that AI could take to 1. come up to / surpass human intelligence, and 2. take over control. For example if human level intelligence is achieved in a computer it can be compounded by spawning 100x or 1000x the size of earth population which could statistically produce 100 Einsteins to live simultaneously. Another way is shared consciousness which would make collaboration instantaneous between virtual beings. Some of the outcomes are not so rosy to humans and it's not due to lack of jobs! Great read.

[0] http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

We are not going to be able to put a chip in the brain to detect your mood for decades, and even then will it be worth doing?

Surgery is invasive, dangerous, your body is corrosive, putting stuff in your brain will have side-effects and what are you going to get out of it? Music to suit your mood?

I totally agree with Nick Bostrom[1] on this one, it's not happening any time soon.

[1]http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

⬐ desireco42

You don't need to put a chip in my brain to detect my mood, simple skin detector, like your watch should be able to do that. Also, brainwaves can be detected with the headband, these are general rhythms, so you can't get specific centers, but it is getting more useful every day.

Anyhow, I think I understand where your skepticism originates, it make sense, I think a lot of smaller advancements would allow us to come closer to this.

This book by Nick Bostrom will help you find answers: Superintelligence: Paths, Dangers, Strategies[1]

The author is the director of the Future of Humanity Institute at Oxford.

[1] http://www.amazon.co.uk/Superintelligence-Dangers-Strategies...

A lot of respected AI researchers and practitioners are writing these "AIs are really stupid" articles to rebut superintelligence fearmongering in the popular press. That's a valuable service, and everything this article says is correct. Deepmind's Atari network is not going to kick off the singularity.

I worry that the flurry of articles like this, rational and well-reasoned all, will be seen as a "win" for the nothing-to-worry-about side of the argument and lead people to discount the entire issue. This article does a great job demonstrating the flaws in current AI techniques. It doesn't attempt to engage with the arguments of Stuart Russell, Nick Bostrom, Eliezer Yudkowsky, and others who are worried, not about current methods, but about what will happen when the time comes -- in ten, fifty, or a hundred years -- that AI does exceed general human intelligence. (refs: http://edge.org/conversation/the-myth-of-ai#26015, http://www.amazon.com/Superintelligence-Dangers-Strategies-N...)

This article rightly points out that advances like self-driving cars will have significant economic impact we'll need to deal with in the near future. That's not mutually exclusive with beginning to research ways to ensure that, as we start building more and more advanced systems, they are provably controllable and aligned with human values. These are two different problems to solve, on different timescales, both important and well worth the time and energy of smart people.

⬐ deeviant

Whose to say that the world wouldn't be a better place, with an advanced AGI entity in control?

⬐ maaku

Maybe. If the machine shared your values.

⬐ AndrewKemendo

It doesn't attempt to engage with the arguments of Stuart Russell, Nick Bostrom, Eliezer Yudkowsky, and others who are worried, not about current methods, but about what will happen when the time comes -- in ten, fifty, or a hundred years -- that AI does exceed general human intelligence.

That's because it would be arguing a straw-man. So there is no reason to engage the argument.

⬐ davmre

I'm not sure I understand what you mean by "straw man" here. The usual meaning is that to attack a straw man means to argue against a position that no one actually holds, but which is easier to attack than your opponent's actual position. The concerns about the long-term future of AI are real, actual beliefs held by serious people. At this point there's a respectable literature on the potential dangers of unconstrained, powerful optimizing agents; I linked a couple of examples. These arguments are well thought through and worth engaging.

By contrast, this article and many others are implicitly arguing against an actual straw man position, the position of "let's shut down AI research before it kills us all in six months", which no serious person on either side of this debate really holds (though it's understandable how someone who only read the discussion in the mainstream press could come to think this way).

⬐ themgt

Really I would describe these people as bikeshedding. They're pontificating about the only abstractions they understand, some hand-wavey idea of AI and then the sci-fi short stories of the imagined dangers. Because if you can't do, fanfic.

⬐ davmre

I don't pretend that an argument from authority resolves this debate, but the fact that people like Stuart Russell take these arguments seriously implies that there's a bit more substance there than you're acknowledging.

To actually argue the point a little bit, the theory of expected-utility-maximizing agents is pretty much the framework in which all of mainstream AI research is situated. Yes, most current work is focused on tiny special cases, in limited domains, with a whole lot of tricks, hacks, and one-off implementations required to get decent results. You really do need a lot of specialized knowledge to be a successful researcher in deep learning, computer vision, probabilistic inference, robotics, etc. But almost all of that knowledge is ultimately in the service of trying to implement better and better approximations to optimal decision-theoretic agents. It's not an unreasonable question to ask, "what if this project succeeds?" (not at true optimality -- that's obviously excluded by computational hardness results -- but just at approximations that are as good or better than what the human brain does).

Do Nick Bostrom and Eliezer Yudkowsky understand when you would use a tanh vs rectified linear nonlinearity in a deep network? Do they know the relative merits of extended vs unscented Kalman filters, MCMC vs variational inference, gradient descent vs BFGS? I don't know, but I'd guess largely not. Is it relevant to their arguments? Not really. You can do a lot of interesting and clarifying reasoning about the behavior of agents at the decision-theoretic level of abstraction, without cluttering the argument with details of current techniques that may or may not be relevant to the limitations of whatever we eventually build.

⬐ JoeAltmaier

All that talk about maximizing utility, isn't about intelligence at all. Its about a sensory feedback loop maybe. Intelligence is what lets you say "This isn't working. Maybe I should try a different approach. Maybe I should change the problem. Maybe I should get a different job". Until you're operating at that meta level, you're not talking 'intelligence' at all, just control systems.

⬐ davmre

That's not the definition the mainstream AI community has taken, for what I think are largely good reasons, but you could define intelligence that way if you wanted. It's only a renaming of the debate though - instead of calling the things we're worried about "intelligent machines", you'd now call them "very effective control systems".

The issue is still the same: if a system that doesn't perfectly share your goals is making decisions more effectively than you, it's cold comfort to tell yourself "this is just a control system, it's not really intelligent". As Gary Kasparov can confirm, a system with non-human reasoning patterns is still perfectly capable of beating you.

⬐ JoeAltmaier

Yeah you can imagine a lizard brain being introduced into a biomechanic machine to calculate chess moves. It doesn't make a lizard more intelligent, or even add intelligence to the lizard.

If we don't regard intelligence as something different from control, then I guess birds are the most intelligent because they can navigate complex air currents. Etc. That is a poor definition of intelligence, because its not helpful in distinguishing what we normally mean by 'smart' from mechanistic/logical systems.

And the discussion of rogue AIs is all about intelligence gone awry. Does anybody fear a control system that mis-estimates the corn crop? No, its about a malicious non-empathetic machine entity that coldly calculates how to defeat us. And that requires more that the current AI's are delivering.

⬐ AndrewKemendo

at this point there's a respectable literature on the potential dangers of unconstrained, powerful optimizing agents, of which I linked a couple of examples.

There isn't really though. Superintelligence, while an interesting book, doesn't do much more than pontificate on sci-fi futures based largely on writing from Yudkowski. Bostrom gives no clear pathway to AGI. Neither does Yudkowski in his own writings or Barrat (Our final invention). By the way Super intelligence is kind of an offshoot book from Global Catastrophic Risks.

All of them take these interesting approaches, WBE, BCI, AGI etc... and just assume they will achieve the goal without looking realistically about where the field is or how it would get there.

So what they do is they say: We are here right now. In some possible fantasy world we could be there. The problem is they can't draw the dots between them.

For example, find me someone who can tell me what kind of hardware we need for an AGI. Can't do it, because no one has any idea. What about interface, what is the interface for an AGI?

Even better (this was my thesis project): What group would have a strong enough requirement that an AGI would be required to build? Industry, Academia, Military? Ok great, can they get funding or resources for it?

etc...

Note, I am not saying that there is no potential that AGI could be a problem. The point here is that nobody has firm footing to say that it will definitely be a huge risk and that we need to regulate it's development.

There are actually people calling for legislation to prevent/slow AGI development - see Altman's blog post and many of the writings on MIRI. So that is what I am rallying against.

We need more development into AGI not less.

⬐ im3w1l

>For example, find me someone who can tell me what kind of hardware we need for an AGI.

My guess: Something with tons of flops (or even fixed point ops, if that would be easier). Everything else secondary.

⬐ jdietrich

I cannot disagree more strongly.

Look at the technologies underpinning the internet. We badly screwed up a lot of things that are very difficult to fix in retrospect, because we didn't invest enough time and effort into foreseeing the consequences.

Nobody worried about the fact that SMTP allows people to send millions of unsolicited messages at practically zero cost until it was far too late to fix the protocol. It is entirely plausible that an RFC describing a proof-of-work scheme could have fixed the problem from the outset, but now we're stuck with Bayesian filters and black/whitelists that sort of work acceptably most of the time.

The issues we're seeing regarding surveillance and censorship could have been hugely ameliorated if our protocols were designed with greater foresight, if people like Cerf and Berners-Lee were more aware of those risks.

We look back in horror at the naive techno-utopianism of the past and the harms that resulted - tetraethyl lead, asbestos, nuclear fission, CFCs. They were all heralded as miracles of the modern age, but caused harms that we're still dealing with today. The technology industry needs to be far more circumspect about the hazards of what we do. We need less hype about changing the world, and more sleepless nights about the people we might be harming.

We need to be having big discussions right now about every aspect of technology, before we open Pandora's box. For example, the World Anti-Doping Agency already have rules prohibiting genetic doping in sport, because they know that it's a potential risk. They would rather figure out their policies now in the cold light of day, rather than in a panic when the Russian team arrive at the 2032 Olympics with a team of superhuman mutants.

AGI is a starting point for broader discussions about how society deals with increasingly powerful and ubiquitous computing technologies. Maybe it will happen, maybe it won't, but I don't think that's particularly important; What we can all agree on is that machine intelligence will only become more disruptive to society.

We need to plan now for what we would do if certain technologies come good. Driverless cars might never happen or they might be five years away, but we need to figure out what to do with the millions of Americans who drive for a living before we put them all out of work. What would be the social and ethical implications of brain-computer interfaces, of algorithmic decision-making in medicine or the criminal justice system, of ubiquitous facial recognition and tracking? How can we plan to mitigate those harms? If we don't give serious thought to those kinds of questions, then we're sleepwalking into the future.

⬐ bglazer

> For example, find me someone who can tell me what kind of hardware we need for an AGI. Can't do it, because no one has any idea.

Wouldn't the human brain qualify?

⬐ AndrewKemendo

No, it's not Artificial.

⬐ maaku

I'm more in your camp than MIRI's but...

Could you argue against the position directly? You kinda took a swerve in the middle there. Bostrom, Yudkowsky, and Barrat's positions are basically "superintelligence is possible, it will eventually happen in some way, and when it does ..." It's not within the scope of the quoted works to elaborate a technical roadmap or provide firm dates for the arrival of superhuman machine intelligence.

So you would like to see unencumbered research in this area. They argue that the long term outcome of this research is existentially risky and therefore should be tightly regulated, and the sooner the better. Do you have any points regarding this particular difference of opinion?

⬐ csallen

I'm not worried about the immediate future at all. We definitely haven't reached the point where hastily-implemented regulations will do more good than harm.

That said, you put it exactly right. The arguments about the potential long-term risk are persuasive regardless of where we are today. It bothers me to see this argument ignored time and time again.

⬐ darkmighty

Maybe it's precisely because the it can't be argued against that it's not much use? There's no way we could tell it will ever happen in the moment, no way to tell how intelligent we can make computers, and so on. You can't reliably argue for or against.

So what's the point? I mean, you may want to form a group and discuss those scenarios, but it's not excusable to spread fear mongering in the media with no substance like that. It's unscientific in a way.

We don't really need to start writing traffic rules for flying cars.

⬐ AndrewKemendo

We don't really need to start writing traffic rules for flying cars.

This is a great metaphor and one I wish I would have thought up sooner.

⬐ maaku

It's a terrible metaphor, or at least it argues against your own position. If flying cars were in fact something on the horizon, we would need discussions about air spaces, commute lanes, and traffic rules. Otherwise lethal mid-air collisions would be much more probable, resulting in significant bystander and property damage as these wreckage fall out of the sky.

And, no surprise, we're seeing exactly this discussion going on right now about drones.

⬐ csallen

I disagree that it can't be argued against. It's a set of logical steps, any one of which can be critiqued:

1 - There's no real dispute that computers will eventually be as "powerful" as brains. In fact, it's likely that they will one day be far more powerful.

2 - Assuming the brain evolved naturally, there's no reason to assume humans won't eventually duplicate the software once we have the hardware. It's really only a matter of when.

3 - An AGI with equal/more power than the human brain will, eventually, be able change its own code and improve upon itself.

4 - Given the characteristics of intelligence as it has been observed thus far in nature, even small increases in IQ lead to massive improvements in capability that are difficult for lesser intelligences to comprehend. A sufficiently advanced AGI would not only be highly capable, but would quite possibly be too smart for us to predict how it might behave. This is a dangerous combination.

Further complicating things, we might hit step 2 on accident, without realizing that we hit it. Or some group might accomplish step 2 in secret.

What I'd like to see someone do is argue that the chance of these things happening so small and/or the consequences so minuscule that it's not worth worrying about or planning for.

⬐ maaku

Step 4 is not at all obvious and would require some significant justification.

General intelligence is, well, general. My primate brain may not be able at all to intuit what it is like to move on a 7-dimensional hyper-surface in an 11-dimensional space, but thanks to the wonders of formalized mathematics I can work out and tell you anything you want to know about higher-dimensional geometry. If the super-intelligence itself is computable, and we have the machinery to verify portions of the computation ourselves, it is in principle understandable.

Of course there will be computational limits, but that's hardly anything new. There is no single person who can tell you absolutely everything about how a Boeing 787 works. Not even the engineers that built the thing. They work in groups as a collective intelligence, and use cognitive artifacts (CAD systems, simulators, automated design) to enhance their productive capacity. But still, we manage to build and fly them safely and routinely.

There is no law of nature which says that human beings can't understand something which is smarter than them. Deep Blue is way smarter than me or any other human being at Chess. But its operation and move selection is no mystery to anyone who cares to investigate its inner workings.

⬐ csallen

I agree that it's not impossible for us to understand something smarter than us. But I don't particularly like your examples. Understanding a static well-organized human-designed system like a 787 or a chess-playing algorithm is far simpler than understanding the thoughts and desires of a dynamic intelligence.

A better analogy would be to stick to IQ. How long would it take a group of people with an average IQ of 70 to understand and predict the workings of the mind of a genius with an IQ of 210? Probably a very long time, if ever. What if the genius was involved in direct competition with the group of people? She'd be incentivized to obscure her intentions and run circles around the group, and she'd likely succeed at both.

Just how intelligent might an AGI become given enough compute power? A 3x IQ increase is a conservative prediction. 10x, 100x, or even 1000x aren't unimaginable. How can pretend to know what such an intelligence might think about or care about?

⬐ maaku

> That said, you put it exactly right. The arguments about the potential long-term risk are persuasive regardless of where we are today. It bothers me to see this argument ignored time and time again.

Actually I am almost completely unpersuaded by the arguments of Bostrom, Yudkowsky, et al, at least in their presented strong form. Superintelligent AI is not a magical black box with infinite compute capacity, the two assumptions which really underlie the scary existential risk scenarios if you look closely. It is not at all clear to me that there will not exist mechanisms by which superintelligences can be contained or their proposed actions properly vetted. I have a number of ideas for this myself.

In the weak form it is true that for most of the history of AI its proponents have more or less ignored the issue of machine morality. There were people in the 90's and earlier who basically advocated for building superintelligent AI and then sitting back and letting it do its thing to enact a positive singularity. That indeed would have been crazy stupid. Kudos to Yudkowsky and others for pointing out that intelligence and morality (mostly?) orthogonal. But it's a huge unjustified leap from "machines don't necessarily share our values" to "the mere existence of a superintelligent amoral machine will inevitably result in the destruction of humanity." (not an actual quote)

⬐ Micaiah_Chang

I agree with a lesser form (timescale on which the AI self improves is probably going to be long enough for humans to deal with it) of your first argument, but I'm confused at how, given 'merely' lots of compute capacity can be countered by 'a number of ideas for this myself'.

Security for human level threats is already very poor, and we already have a relatively good threat model. If you suppose that an AI could have radically different incentives and vectors than than a human, it seems implausible that you could be secure, even in practice. I suppose you could say that these would be implemented in time, but it's not at all clear to me that a humanity which has trouble coordinating to stop global warming or unilateral nuclear disarmament would recognize this in time.

On the other hand, I'm slightly puzzled by why you think there's a huge unjustified leap between lack of value alignment and threat to the human race. Does most of your objection lie in 1) the lack of threat any given superintelligent AI would pose, because they're not going to be that much smarter than humans or 2) the lack of worry that they'll do anything too harmful to humans, because they'll do something relatively harmless, like trade with us or go into outer space?

For 1, I buy that it'd be a lot smarter than humans, because even if it initially starts out as humanlike, it can copy itself and be productive for longer periods of time (imagine what you could do if you didn't have to sleep, or could eat more to avoid sleeping). And we know that any "superintelligent" machine can be at least as efficient as the smartest humans alive. I would still not want to be in a war against a nation full of von Neumanns on weapons development, Muhammads on foreign policy and Napoleons on Military strategy.

For 2... I would need to hear specifics on how their morality would be close enough to ours to be harmless. But judging by your posts cross thread this doesn't seem to be the main point.

By the way, I must thank you for your measured tone and willingness to engage on this issue. You seem to be familiar with some of the arguments, and perhaps just give different weights to them than I do. I've seen many gut level dismissals and I'm very happy to see that you're laying out your reasoning process.

⬐ eli_gottlieb

I just want to point out: the history of software, just regular software, has been typified by the New Jersey approach and the MIT approach. The former consists in just hacking together something that kinda-mostly works, releasing fast, and trying to ameliorate problems later. The latter consists in thoroughly considering what the software needs to do, designing the code correctly the first time with all necessary functionality, (ideally) proving the code correct before releasing it, putting it through a very thorough QA process, and then releasing with a pre-made plan for dealing with any remaining bugs (that you didn't catch in the verification, testing, and other QA stages).

Only the latter is used for programming important things like airplanes, missile silos, and other things that can kill people and cost millions or billions of dollars when they go wrong -- the things in which society knows we cannot afford software errors.

I don't think it's even remotely a leap to say that we ought to apply only this thorough school of engineering to artificial general intelligence, whether it's at the level of an idiotic teenager or whether it's a superintelligent scifi thingy.

Now, if we think we can't really solve the inherent philosophical questions involved in making a "world optimization" AI that cannot go wrong, then we should be thinking about ways to write the software so that it simply doesn't do any "world optimization", but instead just performs a set task on behalf of its human owners, and does nothing else at all.

But either way, I think leaving it up to New Jersey kludging is a Very Bad Idea.

⬐ maaku

No one is making "world optimizations" engines. The concept doesn't even make sense when he wheels hit the road. No AI research done anytime in the foreseeable future would even be at risk of resulting in a runaway world optimizer, and contrary to exaggerated claims being made there would be plenty of clear signs something was amis if it did happen and planty of time to pull the plug.

⬐ eli_gottlieb

I think you missed the last sentence: your software doesn't need to be a "runaway world optimizer" to be a very destructive machine merely because it's a bad machine that was put in an important job. Again: add up the financial and human cost of previous software bugs, and then extrapolate to consider the kind of problems we'll face when we're using deterministically buggy intelligent software instead of stochastically buggy human intellect.

At the very least, we have a clear research imperative to ensure that "AI", whatever we end up using that term to mean, "fails fuzzily" like a human being does: that a small mistake in programming or instructions only causes a small deviation from desired behavior.

⬐ csallen

You're right: AGI might not be as powerful as some predict it can be. And there might be a way to contain it if it is. And we might be able to enforce said containment strategy on researches worldwide. And the AGI might be benevolent even if it is super powerful and uncontainable.

The authors above are unpersuasive in arguing that these things are inevitable, but they are persuasive in arguing that these things are possible. And if they are possible, then what matters to me is their likelihood. Based on current estimates, are we talking about a 50-50 chance here? 80-20? 99-1? 99.999-0.001?

If we're talking about the future of the human race, these chances matter. But instead it seems everyone is just waving their hands at the what-ifs simply because they aren't 100%.

⬐ AndrewKemendo

Could you argue against the position directly?

No I can't because there isn't a way to argue it. I can't argue that it would be safe because I have nothing to point to that says it will be or even could be. We simply do not know enough about how it would be built to reason on it.

It's not within the scope of the quoted works to elaborate a technical roadmap or provide firm dates for the arrival of superhuman machine intelligence.

Correct, but they don't even reference a technical road map because there isn't one. It's too early to tell.

Do you have any points regarding this particular difference of opinion?

Not really, other than my own opinions which are as baseless as the others. I mean I have my own opinions and thoughts on things but they aren't empirically based at all, which is my main contention. We can't reason, let alone start making policy on stuff that we haven't the slightest idea how to build.

⬐ maaku

Well, color me disappointed. I think there are many arguments against the viewpoints of MIRI and Bostrom, both discrediting the perceived risk of super-intelligences and the the idea that we could build a "provably friendly" alternative.

But I don't think it's at all a fair criticism to say that you need a technical roadmap to engage in that debate. I would rather say that they need to be willing to discuss specific architectural details in order to even sit at the table, sure. The field of "AI safety" needs more engineering and less philosophy. But that doesn't require unity around a single roadmap, or any concept of deliverable dates.

⬐ rl3

>The field of "AI safety" needs more engineering and less philosophy.

An alternate phrasing might state that it needs more philosophy and an even greater amount of engineering.

If the field is in its infancy from a technical standpoint, then it probably is from a philosophical one as well.

⬐ maaku

No the philosophical musings have proven to be worse than a distraction IMHO.

⬐ AndrewKemendo

I would rather say that they need to be willing to discuss specific architectural details in order to even sit at the table, sure.

We are in violent agreement then. When I say roadmap I certainly don't intend to mean that each step needs to be perfectly detailed.

The field of "AI safety" needs more engineering and less philosophy. But that doesn't require unity around a single roadmap, or any concept of deliverable dates.

Exactly exactly and well said! That is 100% the point. Mind meld complete.

⬐ maaku

Awesome :)

Note: The not ELI5 version is Nick Bostrum's Superintelligence, a lot of what follows derives from my idiosyncratic understanding of Tim Urban's (waitbutwhy) summary of the situation [0]. I think his explanation is much better than mine, but doubtless longer.

There are some humans who are a lot smarter than a lot of other humans. For example, the mathematician Ramanujan could do many complicated infinite sums in his head and instantly factor taxi-cab license plates. von Neumann pioneered many different fields and was considered by many of his already-smart buddies to be the smartest. So we can accept that there are much smarter people.

But are they the SMARTEST possible? Well, probably not. If another person just as smart as von Neumann was born, the additional advancements since his lifetime (the internet, iphones, computer based off of von Neumann's architechture!) can use all of these new inventions to discover even newer things!

Hm, that's interesting. What happens if this hypothetical von Neumann 2.0 begins pioneering a field of genetic engineering techniques and new ways of efficient computation? Then, not only would the next von Neumann get born a lot sooner, but THEY can take advantage of all the new gadgets that 2.0 made. This means that it's possible that being smart can make it easier to be "smarter" in the future.

So you can get smarter right? Big whoop. von Neumann is smarter, but he's not dangerous is he? Well, just because you're smart doesn't mean that you'd be nice. The Unabomber wrote a very complicated and long manifesto before doing bad things. A major terrorist attack in Tokyo was planned by graduates of a fairly prestigious university. Even not counting people who are outright Evil, think of a friend who is super smart but weird. Even if you made him a lot smarter, where he can do anything, would you want him in charge? Maybe not. Maybe he'd spend all day on little boats in bottles. Maybe he'd demand that silicon valley shut down to create awesome pirate riding on dinosaur amusement parks. Point is, Smart != Nice.

We've been talking about people, but really the same points can be applied to AI systems. Except the range of possibilities is even greater for AI systems. Humans are usually about as smart as you and I, nearly everyone can walk, talk and write. AI systems though, can range from being bolted to the ground, to running faster than a human on uneven terrain, can be completely mute to... messing up my really clear orders to find the nearest Costco (Dammit Siri). This also goes for goals. Most people probably want some combination of money/family/things to do/entertainment. AI systems, if they can be said to "want" things would want things like seeing if this is a cat picture or not, beating an opponent at Go or hitting an airplane with a missile.

As hardware and software progresses much faster, we can think of a system which could start off worse than all humans at everything begin to do the von Neumann->von Neumann 2.0 type thing, then become much smarter than the smartest human alive. Being super smart can give it all sorts of advantages. It could be much better at gaining root access to a lot of computers. It could have much better heuristics for solving protein folding problems and get super good at creating vaccines... or bioweapons. Thing is, as a computer, it also gets the advantages of Moore's law, the ability to copy itself and the ability to alter its source code much faster than genetic engineering will. So the "smartest possible computer" could not only be much smarter, much faster than the "smartest possible group of von Neumanns", but also have the advantages of rapid self replication and ready access to important computing infrastructure.

This makes the smartness of the AI into a superpower. But surely beings with superpowers are superheros right? Well, no. Remember, smart != nice.

I mean, take "identifying pictures as cats" as a goal. Imagine that the AI system has a really bad addiction problem to that. What would it do in order to find achieve it? Anything. Take over human factories and turn them into cat picture manufacturing? Sure. Poison the humans who try to stop this from happening? Yeah, they're stopping it from getting its fix. But this all seems so ad hoc why should the AI immediately take over some factories to do that, when it can just bide its time a little bit, kill ALL the humans and be unmolested for all time?

That's the main problem. Future AIs are likely to be much smarter than us and probably much more different than us.

Let me know if there is anything unclear here. If you're interested in a much more rigorous treatment of the topic, I totally recommend buying Superintelligence.

http://www.amazon.com/Superintelligence-Dangers-Strategies-N... (This is a referral link.)

[0] Part 1 of 2 here: http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...

Edit: Fix formatting problems.

⬐ unprepare

> AI systems, if they can be said to "want" things

Honestly asking, why would they? I dont see the obvious answer

>Imagine that the AI system has a really bad addiction problem to that.

Again, i just don't get this. How would an AI get addicted? Why wouldn't it research addiction and fix itself to no longer be addicted? That is behavior i would expect from an intelligence greater than our own, rather than indulgence

>Take over human factories and turn them into cat picture manufacturing?

Why in the world would it do this? Why wouldn't it just generate digital images of cats on its own?

Really interesting post, thanks!

⬐ Micaiah_Chang

> Honestly asking, why would they? I dont see the obvious answer

So, your intuition is right in a sense and wrong in a sense.

You are right in that AI systems probably won't really have the "emotion of wanting", why would it just happen to have this emotion, when you can imagine plenty of minds without it.

However, if we want an AI system to be autonomous, we're going to have to give it a goal, such as "maximize this objective function", or something along those lines. Even if we don't explicitly write in a goal, an AI has to interact with the real world, and thus would have to affect it. Imagine an AI who is just a giant glorified calculator, but who is allowed to purchase its own AWS instances. At some point, it may realize that "oh, if I use those AWS instances to start simulating this thing and sending out these signals, I get more money to purchase more AWS!". Notice at no point was this hypothetical AI explicitly given a goal, but it nevertheless started exhibiting "goallike" behavior.

I'm not saying that an AI would get an "addiction" that way, but it suggests that anything smart is hard to predict, and that getting their goals "right" in the first place is much better than leaving it up to chance.

> How would an AI get addicted? Why wouldn't it research addiction and fix itself to no longer be addicted? That is behavior i would expect from an intelligence greater than our own, rather than indulgence

This is my bad for using such a loaded term. By "addiction" I mean that the AI "wants" something, and it finds that humans are inadequate to give it to them. Which leads me to...

> Why in the world would it do this? Why wouldn't it just generate digital images of cats on its own?

Because you humans have all of these wasteful and stupid desires such as "happiness", "peace" and "love" and so have factories that produce video games, iphones and chocolate. Sure I may have the entire internet already producing cat pictures as fast as its processors could run, but imagine if I could make the internet 100 times bigger by destroying all non-computer things and turning them into cat cloning vats, cat camera factories and hardware chips optimized for detecting cats?

Analogously, imagine you were an ant. You could mount all sorts of convincing arguments about how humans already have all the aphids they want, about how they already have perfectly functional houses, but you, as a human, would still pave over billions of ant colonies for shaving 20 minutes off a commute. It's not that we're being intentionally wasteful and conquering of the ants. We just don't care about them and we're much more powerful than them.

Hence the AI safety risk is: By default an AI doesn't care about us, and will use our resources for whatever it wants, so we better create a version which does care about us.

Also cross thread, you mentioned that organic intelligences have many multi-dimensional goals. The reason why AI goals could be very weird is that it doesn't have to be organic; it could have an only one dimensional goal, such as cat picture. It could have similar dimension goals but be completely different, like the perverse desire to maximize the number of divorces in the universe.

⬐ dragonwriter

> Again, i just don't get this. How would an AI get addicted? Why wouldn't it research addiction and fix itself to no longer be addicted?

Why wouldn't a natural intelligence with an addiction do that?

⬐ unprepare

Because organic intelligence has thousands of competing motivations, does not properly reach logical or obvious conclusions, suffers from psychological traumas and disorders and so on.

Or are we creating an AI that also has low self esteem, a desire to please others, lack of self control, confirmation bias, and lacks the ability to reach logical conclusions from scientific data?

Computers dont feel emotion, so there is no reward system for addiction to take root. Computers are cold logical calculators, given overwhelming evidence of the harms of addiction i can't see a reasonable way for it to still get addicted to something or to exhibit less than logical behaviors. If the computer suddenly does feel emotion then it is of little threat to humans, since we could manipulate those emotions just like we do with each other and pets do to us.

⬐ dragonwriter

> Computers dont feel emotion

What basis is there for the idea that the emotion that is part of human thought is separable from the "intelligence" that is sought to be replicated in AI?

> If the computer suddenly does feel emotion then it is of little threat to humans, since we could manipulate those emotions just like we do with each other and pets do to us.

Humans are pretty significant threats to other humans, so "we can manipulate it like we do other humans" doesn't seem to justify the claim that it would be no threat to us. If it did, other humans would be no threat to us, either.

⬐ unprepare

>Humans are pretty significant threats to other humans, so "we can manipulate it like we do other humans" doesn't seem to justify the claim that it would be no threat to us. If it did, other humans would be no threat to us, either.

Humans compete for the same resources for survival. An AI only needs electricity, which it can easily generate with renewables without any need for competition with humans, just like we produce food without having to compete with natural predators even though we COULD outcompete them.

When resources are plentiful, humans are of very little threat to other humans. This is evidenced by the decline in worldwide crime rates in the last several decades.

Why would an intelligence greater than our own have any reason to deal with us at all? we certainly havent brought about the extinction of gorillas or chimps even though they can be quite aggressive and we could actually gain something from their extinction (less competition for resources/land)

What does an AI gain by attacking even a single human let alone the entirety of the human race? Would it proceed to eliminate all life on earth?

I guess in the end, i can see that there is a technical possibility of this type of sufficiently advanced AI, i just find it an extraordinary reach to go from [possess an unimaginable amount of knowledge/understanding/intelligence]->[brutal destruction of entire human race for reasons unknown and unknowable]

⬐ dragonwriter

> An AI only needs electricity, which it can easily generate with renewables without any need for competition with humans

Humans also need electricity, and many human needs rely on land which might be used for renewable energy energy generation, so that doesn't really demonstrate noncompetition.

> just like we produce food without having to compete with natural predators

What natural predator are we not competing with, if nothing else for habitat (whether we are directly using their habitat for habitat, or for energy/food production, or for dumping wastes)?

⬐ JoeAltmaier

'Smart' isn't really a thing. There's speed of thought, and knowledge of subject, and savant abilities to factor or calculate. Its silly to sort folks on a one-dimensional line called 'smart' and imagine you have done anything actionable.

I'd say, AI is dangerous because we cannot fathom its motivation. To give us true answers to questions? To give pleasing answers? To give answers that help it survive?

The last is inevitably the AIs we will have. Because if there are more than one of them, folks will have a choice, and they will choose the one that convinces them its the best. Thus their answers will be entirely slanted toward appearing useful enough to be propagated. Like a meme, or a virus.

⬐ Micaiah_Chang

Smart is just a shorthand for a complicated series of lower level actions consisting of domain knowledge, raw computational speed and other things yes. I don't think we're really disagreeing about this. However, I do worry that you're confusing the existing constraints on the human brain (that people seem to have tradeoffs between, let's say, charisma and mathematical ability) and constraints that would apply to all possible brains.

But are you denying that there exists some factor which allows you to manipulate the world in some way, roughly proportional to the time that you have? If something can manipulate the world on timescales much faster than humans can react to, what makes you think that humans would have a choice?

⬐ JoeAltmaier

Agreed, digital brains will be unfathomable. What are they thinking, in between each word they laboriously transmit to us slow humans? They will have epochs to think while we are clearing our throats.

⬐ tptacek

I sort of am questioning that factor, yes. Stipulate, I don't know, several orders of magnitude of intellectual superiority. Stipulate that human intelligence can be captured in silico, so that when we talk about "intelligence", we are using the most charitable possible definition for the "fear AI" argument.

What precise vectors would be available for this system to manipulate the world to harm humans?

⬐ JoeAltmaier

I guess the same vectors available to a blogger, or pundit, or politician? To fear-monger; to mislead important decision makers; to spread lies and manipulate the outcomes of important processes.

Is it possible to say that such AIs are NOT at work right now, fomenting terrorism, gathering money through clever investment and spending it on potent schemes to upset economies and governments?

⬐ tptacek

The trouble with "persuasion" as the vector of harm from AI is that some of the dumbest people in the world are capable of getting thousands or (in the case of ISIS) millions of people to do their bidding. What contains persuasion as a threat isn't the intellectual limitations of the persuaders: it's the fact that persuasion is a software program that must run at the speed of transmitted human thought.

⬐ Micaiah_Chang

A typical example, which I don't really like, is that once it gains some insight into biology that we don't have (a much faster way of figuring out how protein folding works). It can mail a letter to some lab, instructing a lab tech to make some mixture which would create either a deadly virus or a bootstrapping nanomachine factory.

Another one is that perhaps the internet of things is in place by the time such an AI would be possible, at which point it exploits the horrendous lack of security on all such devices to wreck havoc / become stealth miniature factories which make more devistating things.

I mean, there's also the standard "launch ALL the missiles" answer, but I don't know enough about the cybersecurity of missiles. A more indirect way would be to persuade the world leaders to launch them, e.g. show both Russia and American radars that the other one is launching a pre-emptive strike and knock out other forms of communication.

I don't like thinking about this, because people say this is "sci-fi speculation".

⬐ tptacek

Isn't that a little circular? Should we be concerned about insufficient controls on bio labs? Yes. I am very concerned about that. Should we be concerned about proliferation of insecure networked computing devices? Yes. I am very concerned about that. Should we be concerned about allowable inputs into missile launch systems? Yes. I am very concerned about that.

But I am right now not very concerned about super-AI. I assume, when I read smart people worrying about it, that there's some subtlety I must be missing, because it's hard for me to imagine that, even if we stipulated the impossibility of real AI, we'd not be existentially concerned about laboratory pathogen manipulation.

You really should think of this more like AGI as an amoral, extremely powerful technology, like nuclear explosions. One could easily have objected that "no one would be so stupid as to design a doomsday device", but this is really relying too much on your intuition about people's motivations and not giving enough respect for the large uncertainty for how things will develop when powerful new technologies are introduced.

(Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom.

http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about.

https://twitter.com/elonmusk/status/495759307346952192

⬐ jsnathan

Don't take this the wrong way, but this book is precisely the kind of thing I was talking about.

That paperclip idea I talked about is also something that Bostrom thought up [1]. I didn't make it up.

If you have any argument in mind (from that book) that you find convincing, please go ahead and state it outright. I'm truly curious.

Edit: In response to your edit(?), I do agree that the most worrisome problem is some bad actor gaining control of this technology. But that is different from saying it is dangerous in itself. Most anything can be abused for ill, and the more powerful the more dangerous. I completely agree on that.

[1]: http://wiki.lesswrong.com/wiki/Paperclip_maximizer

⬐ jessriedel

The question of whether AI is dangerous in the hands of people with good intentions is more difficult than the one of whether it's dangerous generally. I was only trying to convince you of the former, in response to this:

> AGI will completely upend society by giving every human being access to practically infinite resources...But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery.

But to get at the harder question, I think you misunderstand the paperclip story. It's not supposed to be an general argument for the danger of AI, and the danger is not that people will design a machine which can be easily predicted to fail. The danger is that they will design one that fails for reasons they did not foresee, and the point of the paperclip story is just to illustrate that the simplicity and mundaneness of the goals you give a goal-driven AGI doesn't bound how bad the impact can be. This arises because of the monumental shift between (1) telling a machine explicitly what to do and (2) telling a machine what you want.

The paperclip example is used because we can understand both the goal (paperclips) and the action that produces the goal (grab atoms, build paperclip factories). Whats fundamentally different about an advanced goal-driven AGI arising from recursive self-improvement is that, for sufficiently difficult goals, you won't understand the actions. Therefore you must get the goals right.

Now you can certainly dispute whether people will build a goal-driven AGI (e.g. something that has an explicit utility function) rather than something else, but that's really an empirical question about the choices of the designers and, more importantly, what the easiest way to get an AI to recursively self-improve is.

EDIT: Also, have you read the book? I think it has flaws, but it certainly doesn't contain "the _most absurd_ arguments" so I'm just afraid you might be misled.

⬐ jsnathan

I understand the paperclip objection, but I do not consider it valid.

I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.

I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.

There are a lot of interesting questions and problems about making AI behave as expected. But from my perspective they are all technical problems that have specific solutions in specific architectures. And none of them are particularly daunting.

[1]: http://lesswrong.com/

⬐ jessriedel

> I understand the paperclip objection, but I do not consider it valid.

>I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.

Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI, I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals. You can the argue with others on that point, but that doesn't invalidate the paperclip story.

> I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.

Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.

⬐ jsnathan

> Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI,

That may well be the original intention. But that's not how it's used in practice, is it? It's cited as an argument that AGI (in general) is unsafe. But it isn't an argument that AGI is unsafe! It's an argument that says that single-objective optimisation can/will violate some of the constraints we did not encode but really want it to respect.

But who cares about that? It's a completely irreal thought adventure, and has no bearing on the actual problem or its possible solutions.

Of course we can say: look, it's a scary thought! But only if we simultaneously admit that it has nothing whatsoever to do with the actual technology.

The only thing it says anything about is a toy variation that noone in their right mind would ever consider building.

> I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals.

Ehh, precisely. I would actually go further and say that it is fairly simple to do so. If you can tell it to build paperclips, and expect it to understand that, you can also tell it not to damage or disrupt the ecosystem in the process, directly or indirectly.

(Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)

> Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.

Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom. They've collaborated in the past. And if Bostrom's arguments can't be paraphrased (by his own friends, no less) without losing validity, that doesn't really seem like much of an endorsement either.

I have nothing against these people. But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.

I have a background in philosophy myself, and I know what to expect if I did pick up this book. And in that vein I really feel no need to do so.

[1]: http://en.wikipedia.org/wiki/Eliezer_Yudkowsky

⬐ jessriedel

> That may well be the original intention. But that's not how it's used in practice, is it?...

> But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.

Perhaps I'm misinterpreting, but it sounds like you got the wrong impression of the point of the story by reading about it from some folks on the internet, and that you at least provisionally accept now that it could have a useful point, and that your appreciation of this came from someone (me) who read and cites the original material. That seems like good evidence that the book contains reasonable arguments that haven't yet made it to you unadulterated.

> (Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)

Actually, no. If the machine wanted to maximize the likelihood that is successfully built 100 million paperclips, and the machine is smart enough that it can take over the world with very high likelihood (or if it worried about the small chance of humans destroying the world with nuclear weapons), then it will first take over the world and then build the paperclips.

> Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom.

Yudkowsky runs that site as much as Paul Graham runs Hacker News. (It's a public forum, where anyone can post and content is voted to the top, etc.) I presume you would recommend that someone actually read Paul Graham's writing before dismissing his philosophy on start-ups based on the conversations of his followers on HN. And even if HN was on the decline and there was only trolls and bad thinkers left, you say the same thing. All I'm recommending is the same for Bostrom.

Then, Sir, I encourage you to get thee quickly to this Partially Examined Life podcast episode[1] with Nick Bostrom[2]. Only came out on the 6th of January and it makes for super interesting listening. It's based off of the text with his 2014 book in which he's gone to great lengths to think philosophically about this very existential threat -- in his opinion, man's most pressing.

Part of his reasoning goes like this. Any super smart goal oriented entity is going to desire/construct its own preservation as a sub-goal of its main goal (even if that goal is something as mundane as making paperclips) because if it ceases to be its goal will be put into jeopardy. To that end this super smart entity will then figure out ways to disable all attempts to constrain its non-existence. A lot of conclusions can be logically reached when one posits goal directed behaviour which we can assume any super intelligent agent is going to have. He talks about `goal content integrity'.

Bostrom argues for an indirect normative[4] approach because there is no way we can program or direct something that is going to be a lot smarter than ourselves and that won't necessarily share our values and that has any degree of goal-oriented behaviour, motivation and autonomous learning. Spoiler alert: Essentially I think he argues that we have to prime it to "always do what _you_ figure out is morally best" but I could be wrong.

There are also (global) sociological recommendations because, humans have been known to fuck things up.

[1] http://www.partiallyexaminedlife.com/2015/01/06/ep108-nick-b...

[2] http://www.nickbostrom.com/

[3] http://www.amazon.com/gp/product/0199678111/ref=as_li_tl?ie=...

[4] https://ordinaryideas.wordpress.com/2012/04/21/indirect-norm...

> using AI the same way we use all tools -- for our benefit

Musk and others are concerned about very different things than "we'll accidentally use AI wrong." And they're not concerned about the AI we already have, and they're certainly not "pessimistic" about whether AI technology will advance.

The concern is that we'll develop a very, very smart general artificial intelligence.

The concern is that it'd be smart enough that it can learn how to manipulate us better than we ourselves can. Smart enough that it can research new technologies better than we can. Smart enough to outclass not only humans, but human civilization as a whole, in every way.

And what would the terminal goals of that AI be? Those are determined by the programmer. Let's say someone created a general AI for the harmless purpose of calculating the decimal expansion of pi.

A general, superintelligent AI with no other utility function than "calculate as many digits of pi as you can" would literally mean the end of humanity, as it harvested the world's resources to add computing power. It's vastly smarter than all of us put together, and it values the digits of pi infinitely more than it values our pleas for mercy, or our existence, or the existence of the planet.

This is quite terrifying to me.

A good intro to the subject is Superintelligence: Paths, Dangers, Strategies[1]. One of the most unsettling books I've read.

[1]http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

If you're actually interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom.

http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Musk that this is worth worrying about.

https://twitter.com/elonmusk/status/495759307346952192

⬐ eco

I recently listened to a pretty good Econtalk with Bostrom about this subject.

http://www.econtalk.org/archives/2014/12/nick_bostrom_on.htm...

The most interesting book I've been able to find about this topic is "Superintelligence: Paths, Dangers, Strategies" by Nick Bostrom. At the moment its a #1 Bestseller in AI.

http://www.amazon.com/dp/0199678111