I think the Effective Altruism movement really belies its own values and cause with the fact that one of its own funds is for supporting:
> organizations that work on improving long-term outcomes for humanity. Grants will likely go to organizations that seek to reduce global catastrophic risks, especially those relating to advanced artificial intelligence. [1]
Yes, an argument can be made that it's important to fund prevention of global catastrophes, as while they're unlikely compared to the immediate threat of malaria, they'll cause much greater damage, thus increasing risk. However, to consider artificial intelligence to be a potential global catastrophe at all, let alone the single one requiring extra funding, is mostly unfounded. We currently can barely even define what the actual risk is, let alone how to mitigate it.
It's one thing to walk past homeless people in my city and not give them money, because I know that money could much more easily and effectively safe a life in malaria-ridden parts of the world. I think it's absolutely morally repugnant to walk past them and not give them money, because instead you're paying people to sit around thinking about AI.
I used to feel similarly until I realized that the EA movement isn't a hierarchical organization: it's just a bunch of totally separate orgs who have a common philosophy about how to do good in the world.
Sure, OpenPhil funds AI research. But Givewell (who, last I checked, share the same office) has in their Top Charities list, only those that work in poor countries
Agreed. I've loosely held an EA-like philosophy for about a decade and I think that OpenPhil orientation towards AI is pretty disappointing.
I account for my time in terms of things like number of people saved from blindness or death due to malaria, and I definitely do not count future simulated persons as worthy of any of the same concern as actual humans who exist today.
I watched a debate involving William Macaskill last summer and he poses the hypothetical question:
"You are outside a burning building and are told that inside one room is a child and inside another is a painting by Picasso. You can save one of them. To do the most good, which do you choose?"
The point he's trying to illustrate is that, if you knew for certain that you could turn around and sell the Picaso for millions and use that money to purchase malaria bed nets, the expected number of lives saved by using the Picaso could be hundreds, and so there's a moral dilemma present.
Like many hypothetical questions, this one feels a bit "off" or "unrealistic", but if you don't get hung up on the oddities, I think one can sense the essence of his question, and it reminded me of the point you're making here as well as a responder's question asking you why you think the way you do.
I do think these questions are hard for us to wrap our heads around -- how to value high probability immediacy against somewhat uncertain non-proximal/non-immediate things that might be "much higher value". Part of my human brain goes splat when I try to weigh these things.
In terms of the moral dilemma with the painting, I do have quite a bit of sympathy for the argument that one should do what they feel will produce the most good, which might be to save the painting and purchase malaria nets. My father on the other hand seemed to believe that to be absolutely morally wrong, which seems to be siding with your sentiments. Practically speaking, I think I'd almost certainly save the child's life, because one's human impulses would be so strong that they would override any high-and-lofty-rationality, and one wouldn't have time anyway to do deep analysis. But the question in a hypothetical sense does seem quite valid and hard.
> But the question in a hypothetical sense does seem quite valid and hard.
I appreciate this sentiment, and tend to think likewise. However, the situation is a hypothetical. In the end, perhaps most important is the practical decisions we make, which is almost never situations like the ones you describe above, but more like "what cause should I donate to"? In that sense, it might be hard to discern between different causes in the Givewell top lists, but picking either of those at random is probably a good heuristic that already beats a fairly widespread heuristic of just giving to something like Make-A-Wish, if you're starting from the point of donating €x to a charity.
However, to consider artificial intelligence to be a potential global catastrophe at all, let alone the single one requiring extra funding, is mostly unfounded. We currently can barely even define what the actual risk is, let alone how to mitigate it.
Although I happen agree with you personally, I don't think we should commit the fallacy of assuming that because their position seems absurd to us that it comes from a place of bias or ignorance.
To the contrary, when I listen to Holden and other EA leaders talk, it's clear they've spent way more time thinking about this stuff than I have. He is and they are thoughtful and humble about exactly the questions you pose: How much should we weight "known good" good done today vs "potential good" done in the future, how confident should we be in our ability to predict the future, etc.
As one example, Open Phil has hired historians to conduct research on how well people have (in the past) been able to predict the future, precisely to inform their thinking in this regard.
Holden is also open about how his thoughts on the importance of AI alignment have changed over time.
Again, we can disagree with him (as I do). But we absolutely should not claim that because we disagree, his position must be out of a place of thoughtlessness or bias.
It's perfectly reasonable for them to hold that view. The unfortunate thing is to insist that it's the most rational and/or correct view. To say that they're biased isn't an insult. We all have biases, and it's useful to recognize and admit them.
>I don't think we should commit the fallacy of assuming that because their position seems absurd to us that it comes from a place of bias or ignorance.
I had a really smart person talk about AI and how to deal with it. His conclusion was a gigantic let down. He was out of his element, hes an economist, but his conclusion was-
Either AI is going to be peaceful, or its going to kill us and there is nothing we can do to stop it.
Maybe most civilizations end like this, but why not look for third options?
My comment is going to be unpopular, and I'll admit my bias upfront: I think Yudkowsky is a crank, and is neither a psychology nor an AI expert (he's a self-proclaimed expert, but he's not actually engaged in academic research on AI, because of reasons).
His "experiment" is hard to control or reproduce, its goals are ill-defined, its results are hidden (really, which sort of experiment hides its results and merely asks us to have faith the result was positive?). He makes a lot of unwarranted assumptions, like "a transhuman mind will likely be able to convince a human mind" (why? where is the scientific or psychological evidence that a superior mind must necessarily be able to convince inferior minds of arbitrary things? This is a huge, unwarranted assumption right there).
This kind of psychological experiments -- because this is what they really are, rather than about AI -- are really hard to conduct properly, its results hard to interpret and difficult to reproduce even for subject matter experts, which Yudkowsky isn't. This one looks like it was designed by an amateur who happens to be a fan of sci-fi.
I am aware of one reproduction of the experiment, the goals seem pretty darn explicit, and its results are public. He has stated the rules of engagement, and has said that he did it "the hard way". If nothing else, one should at least be confident that Yudkowsky is honest.
His claim that "a transhuman mind will likely be able to convince a human mind" is what his experiment demonstrates, not what is assumes, and frankly it is absurd to make it sound like he has not repeatedly given justifications for the statement.
What actual misinterpretations or other issues are you worried about?
- What reproduction? What would you consider a successful reproduction, for that matter? If I told you I reenacted the experiment at home with a friend, would you consider this a reproduction? Someone saying they reproduced it on the internet would convince you? What are your standards of quality?
- What is the goal of the experiment? Is the goal "show that a transhuman AI can convince a human gatekeeper to set it free"? Or is it actually "show that a huthatman can talk another human into performing a task", or even "an internet (semi)celebrity can convince a like-minded person into saying they would perform a task of very low real-world stakes". How would you tell each of these goals apart?
- The results are most definitely not public. What is public is what Yudkowsky claims the results were, but since the transcripts are secret and there are no witnesses, how do we know they are true (or even not assuming dishonesty or advanced crankiness, how can we tell if they are flawed?). Would you believe me if I told you I have a raygun that miniaturizes people, that I have tested it at home and it works, and that I have a (very small) group of people who will tell you what I say is true? No, I cannot show you the raygun or the miniaturized people, but I can tell you it was a success!
- "A transhuman mind will likely be able to convince a human mind" is what is stated as truth in the fictional conversation at the top of the AI-box experiment web page. Yudkowsky has repeatedly provided "justifications", but these are unscientific and unreasonable.
Yudkowsky claims that because a person can convince another person of claiming they would perform a task (setting an hypothetical AI free), that then a "transhuman" mind is likely to convince a human gatekeeper. The logical disconnect is huge. First, that people can convince other people of things is no big revelation. Unfortunately, it doesn't follow that because some people can convince other people of some things in certain scenarios, then people can universally convince other people of arbitrary things in every context. Worse, we don't even know what a "transhuman" mind would be like; assuming it means "faster thoughts" (a random assumption), why would more thoughts per minute translate into higher convincing capacity? Is it true, for that matter, that higher intelligence translates into higher ability to convince others of stuff?
----
Another example of methodological flaws: in both runs of the experiment, the participants seem to be selected from a pool of people fascinated by this kind of questions and who would be open to suggestion that a "transhuman" mind can convince them of stuff. Let's look at them:
First participant: Nathan Russell. Introduces himself as
> "I'm a sophomore CS major, with a strong interest in transhumanism, and just found this list."
Then shows interests in a similar experiment and considers how it could be designed. Note that the list itself, SL4, is for people interested in the "Singularity". Enough said.
Second participant: David McFadzean. Correctly claims the first experiment is not proof of anything, and is willing to take part in a second experiment. Later Yudkowsky describes him like this:
> "David McFadzean has been an Extropian for considerably longer than I have - he maintains extropy.org's server, in fact - and currently works on Peter Voss's A2I2 project."
The mentioned website still exists and it has something to do with a Transhumanist Institute. I start to see a pattern here.
His total is 3 for 3. I do not know how to explain these results without either taking them to be honest attempts at a fair experiment or by assuming those involved colluded. I find the latter absurd, given my priors about the honesty of members of LessWrong (Yudkoswky in particular, though he wasn't involved in the reproduction).
> If I told you I reenacted the experiment at home with a friend, would you consider this a reproduction? Someone saying they reproduced it on the internet would convince you?
It is not so simple. I would want evidence that you and your friend were smart and had a decent understanding of the domain, and that your friend was in a similar state of unbelief about the plausibility of being convinced. I would want a statement that it was a serious attempt at doing things "the hard way" and true to the experiment, on both sides, lest you get [1]. Of course I would want the standard rules, or a reasonable modification publicly stated, in addition.
To show that "I can't imagine anything that even a transhuman could say to me which would change [my mind]" is not evidence, and should not be treated as such. To provide evidence that "humans are not secure systems".
You say "very low stakes", but Yudkowsky convinced someone who had offered a $5000 handicap. That hardly seems like a trivial quantity.
> [maybe it's all a lie]
You have to be very cynical to take this worldview.
> we don't even know what a "transhuman" mind would be like
The experiment is under assumption of a true singularity, ergo. nigh-unlimited intelligence. I can discuss what outcomes I think are likely for AI development, or which are merely plausible, but the experiment is about one particular hypothetical, so that would be a different conversation.
> the participants seem to be selected from a pool of people fascinated by this kind of questions and who would be open to suggestion that a "transhuman" mind can convince them of stuff
I am unconvinced that this experiment would work if the gatekeepers did not have an understanding of the topic; they are meant to play a gatekeeper, after all. A person who considers the singularity plausible but thinks an AI box is effective seems like the perfect control should the singularity happen and people want to figure out whether to AI box it.
But that's just it: I simply don't think Yudkowsky or any of the sort of people who would be enthusiastic about the sci-fi theories on SL4, or host extropy.org, or believe in Roko's Basilisk, or read Harry Potter fanfic and find it philosophically insightful, have a decent understanding of the AI domain. Everything about him and his followers smacks of fringe cultists completely outside mainstream research.
I don't think the chosen participants have a particularly deep understanding of the domain, they just think they do (because that's what defines Singularity believers, LessWrong readers, and people who believe they are hyper "rational" and that this is some kind of superpower). I think they understand AI no more than a Star Wars fan understands space travel.
Sure, I don't particularly care if you or anyone else wants to disengage from LessWrong-esque ideas because they sound weird. I only entered this discussion because it sounded like you might have had an actual argument.
Yudkowsky's original intention to not release the dialog was to prevent people from saying "I wouldn't have been swayed by that, therefore AI escaping is impossible!". Even if we grant the first part of the sentence, that an AI escaping is impossible doesn't follow. It's very much possible, and strong evidence is that only human-level intelligences have escaped the same situation.
> and strong evidence is that only human-level intelligences have escaped the same situation.
I don't know if this is the same situation, but laboratory beagles, farmed mink, etc show that non-humans can pursuade humans to release them from cages.
An AI will eventually be much smarter than a human, so it doesn't matter how the human succeeds - it's enough to know that a human has succeeded even once.
I read some of the dialog and the user being the gatekeeper talks openly about his inability to socialize that caused him to receive special social treatment until 10th grade.
What if there were 2 gatekeepers, north korean style?
You have to find every group of humans trying to build an AI, and either convince or force them not to do so.
This seems hard, given:
- (a) The cost of starting an AI venture is minimal (perfectly within reach for a small group or a smart individual with $10 million or less for salaries plus cloud computing expenses). It's a lot harder to keep tabs on small and non-obvious activity like this than, say, the enormous centrifuge facilities needed to refine uranium for nuclear weapons.
- (b) The personal and societal economic, and other, benefits of advancing the state of the art in AI are potentially enormous. People will be motivated to try, whether their goal is to line their pockets, to better understand what intelligence is by advancing the state of the art of trying to put it into machines, or to make the world a better place.
- (c) The problem domain is not well understood, so it's possible to stumble into a dangerous design by accident.
It seems like only some extreme dystopian scenario could halt the progress of technology to the point where an AI-related disaster becomes impossible.
For example, a global war or disaster that destroys civilization's logistical and technological bases, kills much of the world population and forces the survivors to focus mainly on survival. Or an ideological revolution that sees all the nerds of the world lined up against a wall and shot, and those who remain alive become culturally permanently uninterested in advancing technology to avoid the same fate. Or a world-dominating tyrannical government that keeps a close watch on every expert programmer and computer system in the world, monitoring to make sure unauthorized AI experiments don't take place.
Even that might not be enough, because if any of the human population survives, even the biggest global catastrophe, cultural revolution or tyrannical world empire's effects probably won't last more than a few thousand years.
I'm not so sure. Even prosaic, current AI, can be very unpredictable. See this blog post http://aiweirdness.com/post/172894792687/when-algorithms-sur... , which is a somewhat more causal summary of this paper "The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities" https://arxiv.org/abs/1803.03453v2
The authors of the paper explain a bunch of anecdotes when they wanted the AI to do one thing, but the measure they told it to optimise for wasn't actually what they intended, and so unexpected things happened.
That's an amusing read, and sure, AI can do things we don't expect. I think we're more on the order of a bridge failure and less on the order of Skynet here, though. To end up in that kind of nightmare scenario we have to entrust some AI with capabilities we could just elect not to.
Note that GiveWell is sticking to its original mission of evaluating charities based on scientific studies and isn't recommending AI funding.
However I'll push back on this. If it's okay for the government to fund basic scientific research where the benefits are long-term and unpredictable, why not private individuals? Or are you saying research shouldn't be funded at all?
I think it's fine for private individuals to do it (though I disagree with their choice), but not to claim that it's evidence-based Effective Altruism while doing so - it is an emotionally-led value judgment, just like donating to my local dog shelter would be (which I understand others may disgree with). That said, I do recognise now that effectivealtruism.org does not represent the entirety of the movement; I have no fault with what GiveWell are doing.
Of course, there'll never be evidence about the effectiveness of research into existential risks until it's too late. So there's no way to objectively allocate funding between, say, AI safety and disease pandemics.
While I think it's important to fund such research, it's a good point that funding orgs should either be evidence-based or not. Funding some things based on evidence and other things based on hunches might be an unstable combination. Once hunches are allowed, they could easily drift into funding things where the evidence says it's ineffective, but it gets overruled.
Since AI can kill 7,000,000,000 people, You could probably toss a non-zero probability on it. Any math people know how to model this?
Even with a bunch of different variations for the probability, you could understand why AI is so dangerous. potentially.
There are probably lots of non zero probability events like getting hit by a rogue quasar. The difference is we cannot stop the quasar and that humans are required to discover AI.
I dont give money to other charities, I spend money on mine. We dont have any employees, just volunteers. I spent a total of 50$ on the LLC paperwork and 8$/mo for a website host. Everything else has went directly to the kids/families we are teaching.
For estimating the likelihood of AGI, you can start with estimates from people in the field. Some say 0%, others say 80% chance in 50 years. Any way averaging of these produces a risk high enough to justify serious funding into safety research.
Your assumption seems to be that either it is a pure emotional value judgement (like donating to a dog shelter because you felt like it) or a perfectly objective judgement (ie. meta-reviews of randomised controlled trials). Perhaps that's a false dichotomy and there's a difference between supporting a cause based that is now supported by a significant body of philosophical and technical literature and just donating to whatever you feel like? There's nothing wrong with the later, just that you are kidding yourself if you think it is likely to be effective.
Effective Altruism is not a single monolithic organization and I think you really go too far to assume all/most advocates think AI research is more important than feeding the homeless.
Personally, I wholeheartedly subscribe to effective altruism and also don't think AI research is a good use of my funds. All of my donations go to malaria relief.
Effective altruism is about principles, not particular causes.
Isn’t this just a statement that you disagree either with the relative probability of an AI-related disaster or the relative degree of harm it could cause?
It seems harsh to say it’s “morally repugnant” when in the end you’re just saying the way you assign probabilities and degrees of negative outcomes would lead you to invest in a different portfolio of charities or efforts than what this other group would do.
(I’m not arguing for or against the correctness of a belief that AI poses attention- and funding-worthy threats. Only that these other people motivated to allocate resources to it have studied the problem in great detail and their sincere belief after looking into it is that it is worth being part of the overall portfolio of charity investment, and this really would (in their sincere judgment) mitigate big-scale harm, no different than fundamental research into global warming or drug-resistant bacteria, regardless of whether lay people can more easily envision the types of harm from those other threats.
> Isn’t this just a statement that you disagree either with the relative probability of an AI-related disaster or the relative degree of harm it could cause?
Yes, but the point of this movement is that they're supposed to be evidenced-based and maximising ROI. There is evidence for the existence and threat of global warming and drug-resistant bacterial. When you move away from charitable efforts whose effectiveness we can directly measure, you're not doing EA any more, just regular emotional/value-based charity.
I do have different values and feelings about such non-evidence-based charity, and I think theirs are morally repugnant, especially when they claim to be doing EA.
That's a fair point about moving into areas in which the effectiveness is hard to measure. However, even apart from AI threats, I'd argue that there are many areas like this that pose potentially great harm. For example, I think that we ought to spend more effort to prevent regulatory capture in corporate-friendly legislation. But whether we can find a metric we'd all agree on to target this from an empirical point of view is a very difficult question.
It reminds me of Gilb's Law, "Anything can be measured in a way that is superior to not measuring it at all."
So if your sincere beliefs were such that the threat posed is high enough, and total harm would be astronomical, then you might believe that accepting a present-day metric that has a lot of variance and which is hotly debated might still be ok. Empirical progress along that rough metric might be more valuable, still in an expected value sense, than progress in other possibly less-harmful areas even if they have more clearly defined metrics.
Again, not arguing for or against, just trying to represent why someone might be sincere about investing in AI safety and why, under their particular beliefs and preferences, it could still be rational from an EA point of view, despite more ambiguity.
There are always risks that a certain model's loss function does not correctly correspond to the goals they wish to optimize towards, or that there is a large amount of idiosyncratic noise in the observation of the loss criteria. But you can still account for the risks of these sources of error in an overall method that is still empirical.
It's like calling someone morally repugnant because they agreed to give you money equal to the sum of two and two and then gave you three dollars, insisting that the math actually does come out that way. If they honestly do believe that, they are simply mistaken, not morally repugnant.
The founder of Open Philanthropy claims to have sought out the most effective uses of charity money in the entire world, and it happens to include paying his roommate and brother-in-law to think about AI.
The big problem with this decision isn't that it's mistaken, it's that it's corrupt. He should have excluded things that were in his self-interest from consideration, even if he honestly, mistakenly believes they are the best thing anyone can do with their money.
This would be an argument about corruption or disingenuous claims by a specific person involved with the charity movement, and could definitely heighten skepticism about other involved parties.
But it would not be a criticism of the general idea of risk-return optimization applied to charity, and would not exclude a rational participant from belief that allocating some capital towards AI research was part of an optimized approach.
The original comment made it seem like the generic idea of choosing to invest in AI safety as part of the EA framework was, in spirit, intrinsically "morally repugnant."
The specific repugnant actions of one party would not necessarily support that, anymore than say a pro football player's racist comments would imply that all of football is inherently morally repugnant.
It would seem valuable to separate and distinguish vitriol directed at the specific suspicious actions of one person from generic and wide-sweeping criticism of an entire framework that person happens to be associated with, especially when the framework itself attempts to be value-neutral conditioned on one's beliefs after looking at some evidence.
If you're starting to talk about relative values and how one person values different things than another, who could disagree? But then the argument for effective altruism starts to crumble.
Huh? How does that affect the argument for effective altruism (which is essentially just mean-variance optimization applied to charity, and says nothing about what expected value you ought to believe about any given charity, nor what personal tolerance for risk you should have, apart from summaries of how certain other groups have come to believe about those topics)?
The original comment was the one that brought up "morally repugnant" choices in investing. That's what brought in relative value judgments. I was trying to ask how it differs from simply disagree with someone else's assessment of the evidence.
If Person A evaluated the evidence and sincerely believe you should give $3 to X and $2 to Y, based on empirical outcome optimization, they are using the EA framework.
Person B might look at the same evidence and believe you should give $5 to X and $0 to Y, but that hardly makes Person A "morally repugnant" (which might be Person B's unrelated moral judgment) nor does it make their choices fall outside the scope of the EA framework of decision-making.
As far as people disagreeing with posterior distribution over the goal-maximizing choices after seeing the same evidence, this would ideally fall under something like Aumann's Agreement Theorem (of course with the exception that people are not fully rational, Bayesian agents).
Given that, there is some expectation that if both parties are really rational and have the same value of goals, then they ought to come to the same conclusions given the same evidence.
But if they don't have the same goals (e.g. maybe you just happen to care more about animal welfare than me), then the EA framework doesn't say anything about you and I having the same investment priorities.
Really, popular EA press is mostly just saying, "Look, we take value judgment X on issues A, B, and C. If you agree and you start from the same value judgment we do, then based on the following evidence, we believe it's optimal to invest in foo, etc."
It seems like you're saying that it's not possible for two different people, *both making decisions in the effective altruism framework" could come to different beliefs about what to invest money into.
But there's no part of EA that requires that for two agents with different value judgments at the start.
It's not being "simply mistaken", it's a calculated belief. Someone might think it's a moral act to fund, say, ecoterrorists to bomb the headquarters of a major oil company, believing that the loss of life is worth it for helping the world by destablising the company. We might disagree, and consider that highly immoral.
That's an extreme example of course, and in this case the worst that can happen is some loss of money, rather than loss of lives, but I hope it illustrates my point: them choosing to pay people to think about AI, rather than giving that money to other causes, is a moral choice with which I disagree.
IMO, the problem with AI risk funding isn't that it is 'wrong', per se. Instead it is that it is very speculative and not grounder in real world data, or measurable benefits that can be achieved TODAY.
The whole point of effective altruism is to be very ground in evidence based causes. IE, if I spend 10K on this cause, then it will definitely save X lived by next year.
AI risks can't be reduced to these kinds of numbers just yet.
It could still be true, we just don't have any evidence yet to prove it though.
I would say they can be reduced to numbers like that today, it's just that the implied error bars around the numbers would be huge.
Then in the optimization problem of determining what to invest donations into, the amount of gain from putting money into AI research would be correspondingly penalized by the amount of uncertainty (this is known as mean-variance optimization).
It's very much like a choice to invest in a solid, fundamental company which you know will give you 5% return next quarter, or invest in a very uncertain start-up which might give you 1000% return next quarter or might give you -50%.
The start-up investment might have a higher expected value, but also a much higher risk. At that point, the way to distinguish between whether you prefer to invest in the "sure bet" company or the "risky and ambiguous" start-up would be down to your personal tolerance for risk.
It could be perfectly rational to invest in the start-up in that situation. If you personally have a high risk tolerance. If you don't, then the start-up would look like a crazy, speculative bet with huge downside.
So some people might look at AI and say, "We don't have a good idea what the capabilities will be in ~50 years. So there is a huge risk that my charity donations will be wasted because in 50 years we realize we never needed to worry about this problem. Or we might use current AI research to thwart an unimaginable global crisis."
That person then looks into various sources of thought on putting actual numbers and actual error bars, the best we can, onto the problem, say by reading the Global Catastrophic Risk book, or stuff by Bostrom, or consulting surveys of current practitioners' estimates.
At the end of that process, it could totally end up being the rational course of action for that person to still invest in AI charities. Yes, their estimate for the expected value of the donation might have huge error bars around it, but under their particular beliefs about risk-return trade-offs, that might not cause their optimized decision to change.
Even as a massive sceptic of Cyberpunk AI (and for that matter, the efficacy of preventing it by setting up charities to do AI research), I think people are perfectly entitled to spend their money on it if it's what they're interested in. I also think it's plausible some such programmes might create extremely commercially valuable byproducts even though they don't lead anywhere useful directly, much like the Apollo missions.
Trouble is, if you invest enormous amounts of time and energy in arguing that a general belief stuff might work isn't good enough for philanthropy and even writing "why we don't recommend" articles about specific charities dispensing aid that don't deprive enough people of it to have a control group for RCTs, you deserve every criticism you get when you then funnel foundation money to very well-funded AI startups with unfalsifiable solutions to a purely hypothetical problem, even if you didn't have close personal relationships with people working for them. Even though I consider some EA analysis to be good and well reasoned (and they're certainly not the only people making evidence based bearish arguments about, say, microfinance) it's a little difficult to see how AI research could possibly pass a decision-making heuristic so supposedly rigorous that it writes off sanitation as an area to invest in because the estimations of diarrhea reduction aren't blinded. Ultimately, EAs have their cognitive biases just like anyone else, and I'm more sympathetic than they are to aid organizations' view that conducting RCTs is difficult and expensive and they don't want people suffering in control groups to quantify how well common sense healthcare interventions work, and disagree that such organizations should be held to higher standards of analytical rigour than AI researchers hoping to be Sarah Connor.
I think your reply is interesting, as it exactly the sort of thing that Effective Altruism talks about.
The example you gave is endemic of the way we think of charity precisely because it deals with both emotion and things we can individually see. Effective Altruism aims to make charity about things that don't directly affect us, that we can't see, and find areas that where the "bang per buck" is high.
Seattle spent $195 million on the homelessness according to this: https://www.seattletimes.com/seattle-news/homeless/how-much-..., to help a group that this advocacy organisation http://www.homelessinfo.org/what_we_do/one_night_count/ puts at 10,000 people. I wonder how a person can rationally look at that - $195 million at ~$19.5K per person and think they can make a dent. Compare that to https://www.givedirectly.org/basic-income and doubling the poorest of the poor's daily income by giving them $1 a day for 12 years. These are people I can't see and don't have to think about, yet for less than $5000 I can provide someone with security and an income for over a decade. If I give $70 a week for 12 years, I can take 10 people out of abject poverty for 12 years. That's fractionally above my coffee budget.
That's the question Effective Altruism wants people to ask themselves. How can I, with the amount of money I have, do the most tangible good in the world, and what is an equation that helps me decide.
Far from being "absolutely morally repugnant" to walk past a homeless person and preference things we can't see, I think it is morally honest to look at the risks humanity faces, and preference things that are under funded, where $1 can go far. That takes several forms, both immediate - like the Give Directly example - versus long term, where risks that could wipe out our species are considered. I think a strong argument can be made that, for minimal expense, it would be possible to have an affect on a potentially species destroying technology like AI, and that the money spent there is more likely to do good than adding that to an already well funded area.
That's catastrophic risk, minimal expense vs human suffering and high cost. How you solve for the equilibrium there.
> However, to consider artificial intelligence to be a potential global catastrophe at all, let alone the single one requiring extra funding, is mostly unfounded.
As another commenter points out, EA is not a single unified group, but rather a disparate set of people who are united by the belief that we should put money where it has the largest effect. Some, but not all, of them are convinced by the arguments in favor of doing AI alignment research. If you aren't, then don't donate money for that, donate it to what you think is a more effective cause!
I'm curious, though, how people come to the conclusion that AI risk is nonsense? Is it a gut reaction, or does it come from thinking or reading about the problem? Every popsci article I've seen on the topic has been atrocious, so kudos if you thought "nonsense!" after reading such an article.
I assume that investing in AI risk is nonsense for one simple reason: I have yet to see a single study showing that a single undesired behavior has been stopped or even slowed down.
DDOS attacks are as stupid as you can get from an intelligence point of view, and yet not a single proponent for AI risk has come up with a way to stop even a single one AFAIK. I haven't heard either of a single military program that has been thwarted, and those computers are killing people right now. I also doubt that they could get a Pentagon official to stop doing anything.
Until the AI risk community show that they can do anything other than talk, no matter how small, I'll remain skeptic.
I'm confused. The research in question is "how can we make sure that a superintelligence would be morally aligned with humans". Why would dealing with DDOS attacks would be the right first step? It seems completely unrelated.
Because (IMHO) if they want to morally align a superintelligence, the first step would be to either morally align a dumb intelligence or to show that you can convince the people building this intelligence to steer it in this direction.
If they do neither, they risk either coming up with a plan that doesn't work (because it was not tested) or that no one cares about.
i don't think it's nonsense exactly. i think it merits some funding, and we'll probably be glad we did it even if it turns out there never was a risk of an unfriendly superintelligence.
but I have misgivings. in particular, the idea that marginal charitable dollars ought to be used to fund philosophers to think abstract thoughts is very counterintuitive!
(every argument that it's THE MOST IMPORTANT THING IN THE WORLD seems to have the same form. first give me a bunch of weird historical and metaphysical assumptions, then make me admit I'd assign them a finite probability of being true, then multiply by a kajillion future simulated lives or whatever.)
most of the philosophers who research this stuff, and most of the people involved in Effective Altruism, are part of the same niche subculture. i don't think this is a grift, I think they are quite honest in their convictions. but it makes me go "hmmmmm".
AI risk has a profile that is very, very difficult to deal with rationally. By that I don't just mean "think about rationally rather than emotionally", but that it is difficult to even use rational tools to analyze it. It is what most people would consider a very, very small probability of what is arguably the worst possible outcome (i.e., considered from a strictly materialist perspectiv, there are outcomes worse that "the total extinction of life on Earth"!). Tiny magnitude probabilities of huge magnitude disasters are mathematically unstable; very small shifts in the value of the probability, and relatively small changes in the log value of the disaster's size, result in radically different scenarios and correct risk mitigations.
It's even worse if you try to draw the probability distribution of something like "given strong AI will be acheived within the next 100 years, what is the distribution of the resulting likely outcomes?" In this case, the uncertainties are so large that, again, people can come to radically different conclusions even assuming both sides are being generally rational. It asks people trying to draw that distribution to consider what the most likely outcome is of at least a radically super-intelligent being is, and more likely, an arbitrarily large community of radically super-intelligent beings. Who can seriously claim to have an accurate probability distribution of that? If we understood radically super-intelligent beings, we'd already be radically super-intelligent beings, so we are not good at modeling them almost by definition.
But certainly by observation; I am opaque in many ways even to my 7 and 10 year old children, and as with most humans, they are already exceptionally intelligent as "intelligent systems" go. (A 5 year old of normal intelligence is already exceptionally intelligent as intelligent systems go.) The idea that I could model something that was even my clone otherwise but operating at a hundred times the speed is already absurd; add an actual increase in intelligence to it and I stand no chance. Our only commonality will be those things imposed on us by the universe; i.e., I can be confident it must consume some negentropy to survive, etc. But any higher-level actions would be impossible for me to model.
People who think intelligence is unlike to emerge quickly are unlikely to consider it a serious risk. People who think it's unlikely for an intelligent being to augment itself, then use its augmentations to augment itself, in an explosion of uncontrollable intelligence, will not consider AI risk a very likely outcome. And it's not as if it's an irrational or impossible outcome; for all we know, while there may be low-hanging fruit above human intelligence it is absolutely entirely possible that O(work to increase intelligence) > O(ability to perform intelligence-increasing work as intelligence goes up). After all, I can't help but look up in the sky and observe that there does not seem to be a near-lightspeed expanding bubble of computronium coming my way, per the most dramatic fears of the singularity. We could end up with something very intelligent, even dangerously intelligent, but not end up with us waking up one day to a digital god among us.
The precise distribution of outcomes is not actually all that important for just figuring out whether the research is valuable. Alignment research would be valuable if it moved a 1% probability mass from worst-possible-world to human-extinction, a 1% probability mass from human-extinction to human-survival, or a 1% probability mass from human-survival to human-flourishing. As long as you can convince yourself that there is a probability of things being far from optimal in some sense at some time in a way that funding research now could nontrivially affect, you have a justification.
Remember that people were created by evolution, and evolution is dumb. Nonetheless, the returns on moderate deltas in intelligence are very strongly selected for, and major changes have happened with small perturbations in evolutionary history. This gradient is so sharp that the same species that produces perfectly healthy 80 IQ individuals also produces Feynman and Euler, the latter so mathematically productive that the Wikipedia page listing things named after him says that "In an effort to avoid naming everything after Euler, some discoveries and theorems are attributed to the first person to have proved them after Euler."
Note that necessarily society develops as soon as intelligence reaches the point at which is is achievable, not with some evolutionarily significant delay, so you cannot judge the gradient by looking at the cap on intelligence of existing species; we are the necessarily the smartest, because we are first. Rather, you have to look at those behind us, and there it seems the gradient is extremely steep; our closest intellectual competitors can not so much as write a single coherent sentence.
The probable conclusion is that human intelligence is more than sufficient to build effective general intelligence, and improvements to intelligence are likely extremely steep on the slope to superintelligence, even ignoring the nine orders of magnitude improvement you get for free by merely running in silicon.
> I think it's absolutely morally repugnant to walk past them and not give them money, because instead you're paying people to sit around thinking about AI.
This is not a new argument. Sister Mary Jucunda, a Zambian nun, wrote to a NASA scientist (Ernst Stuhlinger) in the early 70's with a similar critique of funding for space research.
> About 400 years ago, there lived a count in a small town in Germany. He was one of the benign counts, and he gave a large part of his income to the poor in his town. This was much appreciated, because poverty was abundant during medieval times, and there were epidemics of the plague which ravaged the country frequently. One day, the count met a strange man. He had a workbench and little laboratory in his house, and he labored hard during the daytime so that he could afford a few hours every evening to work in his laboratory. He ground small lenses from pieces of glass; he mounted the lenses in tubes, and he used these gadgets to look at very small objects. The count was particularly fascinated by the tiny creatures that could be observed with the strong magnification, and which he had never seen before. He invited the man to move with his laboratory to the castle, to become a member of the count's household, and to devote henceforth all his time to the development and perfection of his optical gadgets as a special employee of the count.
> The townspeople, however, became angry when they realized that the count was wasting his money, as they thought, on a stunt without purpose. "We are suffering from this plague," they said, "while he is paying that man for a useless hobby!" But the count remained firm. "I give you as much as I can afford," he said, "but I will also support this man and his work, because I know that someday something will come out of it!"
> Indeed, something very good came out of this work, and also out of similar work done by others at other places: the microscope. It is well known that the microscope has contributed more than any other invention to the progress of medicine, and that the elimination of the plague and many other contagious diseases from most parts of the world is largely a result of studies which the microscope made possible.
I think there are many parallels between AI/ML and the microscope, and I think safety research is a very reasonable inquisitive lens for developing these new technologies.
Well, fighting poverty also grows the economy and number of brains available for research in the future. Today we are bringing people out of poverty for good, much charity isn't money in a bottomless pit.
The pit has a bottom as evident by the very realistic goal to end extreme poverty by 2030.
Still basic research is important, because it has such a long tail. Just saying both approaches have validity and will bring massive progress.
That's a rather condescending reply, lecturing her like a schoolchild. Funding space research is one thing, but spending $billions on poor designs like the Shuttle is another and on expensive American lifestyles is another.
The reply begins with a sincere "...First, however, I would like to express my great admiration for you...".
The denoted intent of the letter is certainly not condescending.
Perhaps the content is what makes the letter condescending?
The letter, if written today, might come off as condescending. It states many obvious truths, such as the observation that space development provides valuable tools to Zambian nuns. Today, this is obvious [1], and to point out such obvious facts to an expert borders on condescension.
But GPS wasn't at all an obvious implication of NASA funding in 1970!
If the letter comes off as condescending today, it's only because time (and science funding) has turned the impossible into the pedestrian.
For one, so far everyone predicting doom about AI has been a layman a subject. Maybe you aren't aware, but this is a field that people do PhDs and get professorships in. Noone respected agrees with the doom-sayers.
> For one, so far everyone predicting doom about AI has been a layman a subject.
This is a myth. It was arguably true 5-10 years ago, but concern with AI safety is not a fringe position even among the highest levels of AI researchers now.
You can read through list of grants granted by the Future of Life institute for AI Safety research, almost all of which are to researchers associated with respected universities, not laymen, here: https://futureoflife.org/ai-safety-research/
> Maybe you aren't aware, but this is a field that people do PhDs and get professorships in.
I'm aware of my own existence, thanks ;-)
> Noone respected agrees with the doom-sayers
But many, many, many respected AI researchers (who I've talked to about this) certainly do agree that AI/ML safety/robustness are important topics of inquiry.
Including researchers at all the top CS departments, at Deepmind, at OpenAI, etc.
Consider critiquing the concrete projects that are funded with AI safety money. Very little of that money flows to research about "preventing malicious superintelligence" or whatever strawman you have in mind. And even projects that consider those questions also consider many much more near-term AI safety questions.
There is a difference between AI/ML safety/robustness (as in ensuring self driving cars don't behave erratically given unexpected inputs), and the skynet predictions.
> Consider critiquing the concrete projects that are funded with AI safety money. Very little of that money flows to research about "preventing malicious superintelligence" or whatever strawman you have in mind. And even projects that consider those questions also consider many much more near-term AI safety questions.
But this ("preventing malicious superintelligence") is EXACTLY what we are talking about. The comment in the OP was quoting the following
> Grants will likely go to organizations that seek to reduce global catastrophic risks, especially those relating to advanced artificial intelligence.
It's one thing to walk past homeless people in my city and not give them money, because I know that money could much more easily and effectively safe a life in malaria-ridden parts of the world.
The homeless in America live a life that's close to as miserable as the poor anywhere. One of the problem of most charities is bureaucracy eating up funds. Despite a lot of claims by bureaucracy, people who are homeless can often spend the money better on themselves than a bureaucracy.
I don't think there's any particularly good not to give to the homeless if you feel like it and have the money.
Not to defend not giving money to the homeless, but isn't the calculation more complex than that? It's not just about how much money the bureaucracy eats, it's also how effectively the remaining portion is used, and I would think that would include actual resources delivered to people.
More simply put, which helps more people and in a larger amount, $10 in the US or $5 in some extremely poor part of the world, where food and services may be much cheaper?
I don't know the answer, but I think it's not as simple as "half is taken by bureaucracy so I shouldn't give to a bureaucracy".
Not to defend not giving money to the homeless, but isn't the calculation more complex than that? It's not just about how much money the bureaucracy eats, it's also how effectively the remaining portion is used, and I would think that would include actual resources delivered to people.
Certainly, altogether the calculations are extremely complicated. There's whether a given organization knows what resources to provide, whether the provided resources are going to be used even if they are otherwise "right", etc.
With all the complexity and uncertainty, directly giving to people seems like one entirely legitimate approach since it guarantees that people get resources, not that other necessary are problematic but certainly other approaches deserve certainty. Direct giving at least has "what you see is what you get".
Then I'll defend it. The money would be much better utilized donated to programs that try to target homelessness as a whole. The old "they'll just spend it on drugs/alcohol/etc" cliche is actually backed by studies: https://www.theatlantic.com/business/archive/2011/03/should-...
Some points from the article:
"We choose to donate money based on the level of perceived need. Beggars known this, so there is an incentive on their part to exaggerate their need, by either lying about their circumstances or letting their appearance visibly deteriorate rather than seek help."
"If you travel to a poor city, for example, you'll find swarms of beggars by touristy locations. If the tourists become more generous, the local beggars don't get richer, they only multiply."
Further, a controversial issue is that of fake homeless people, which undoubtedly exist. Some say the easiest way to tell is if they'll accept food instead of money. From what From what I remember of someone's experience relayed in a reddit thread (which I'll admit has the potential to be exaggerated or unreliable) the majority of homeless people who said they needed "money for food" wouldn't accept food or would throw it away as soon as they thought the giver wasn't looking, indicating they were actually just after money.
If you feel like you absolutely must give directly to homeless people, carry around wrapped protein bars or something similar. Encouraging panhandling is just exacerbating the problems.
For some anecdotes, the last time I went to a convention in Baltimore, I heard stories about multiple people being assaulted or verbally harassed by panhandlers. In one case the provocation was that the person "didn't give them enough." The area is full of career-beggars who have no interest in actually improving themselves, like this one guy who pretends to be a youth baseball coach to collect fake donations every year. I try to stay indoors as much as possible but they even come inside the hotel lobbies to try to scam people sometimes. My experience and hearing those assault stories made me wonder how many of the people on this site decrying homeless-deterrent architecture have actually been in an affected area before.
I really included that to cut off possible misinterpretations of my point before they began so I didn't have to waste time on counterpoints to some argument other than the one I was making. That said, tangents with interesting info dumps is one of my favorite parts of HN, so thanks. :)
To provide the counter argument I imagine the Effective Altruism folks may give (not that I necessarily support this position):
A number of specialist AIs could potentially eliminate 50%+ of all jobs in the next 50 years (a generation). What does society look like when we don't have the safety nets to support a massive unemployed population? What could it mean for geo-politics if the rich countries population is simply rich because they have the most advanced algorithms, but most people don't work? We're talking about massive social unrest worldwide potentially - capitalism cannot necessarily survive this paradigm shift and since the fall of the Berlin Wall the world has no other social organization in modern times, unless we'd like to revert to totalitarian rule.
This all presumes of course, that the first AGI they turn on doesn't just turn the universe into a pile of paperclips (ducks)
You absolutely don't need an app to give homeless individuals money. But when I was homeless, I found that it was vastly more helpful for me if someone gave me a few bucks than if I had to stand in line for hours, fill out reams of paper, etc to try to get some assistance. I am happy to encourage any model that fosters a little more direct giving of this sort.
I am also looking to create a pilot program to try to help homeless individuals begin to develop an online income from the street as I did. Charity helps you survive another day, but you need an income of your own to get your life back and many programs seem to actively be against the idea of homeless people trying to work for a living. People seem to see it as cruel to expect them to. But earned income is the only real path out of dire poverty.
Fair enough. I also find it unfortunate that so many people seem to see themselves as freelance interrogators of the homeless asking about what they will do with the money they're given.
Yes, thank you for saying that. The problem runs deeper than you may realize. Even very well meaning people would walk up to me and ask me "Are you homeless? What's your name?" They didn't volunteer their name, address or socioeconomic status when asking such questions.
They were trying to ascertain if I would need help. A much better question format that isn't so problematic would be along the lines of "I have X (clothes, blankets, whatever). Would you like to have that?" If someone isn't actually homeless but would be happy to have a free blanket, what do you care? Maybe that blanket will help prevent them from becoming homeless.
The blanket example reminds me of all the handwringing about people "misusing" LLINs as a fishing aid, as though fighting malaria is more important than not starving.
This study investigated the extent of bed net misuse in fishing villages.
Wow. There is something incredibly fucked up about funding a study into bed net misuse rather than a study on the dire need for more help with meeting basic necessities like adequate nourishment as evidenced by so-called bed net misuse.
Or, you know, the sarcastic reply: If you die of starvation, I guess you no longer need to worry about details like malaria.
This seems like a good study that will save lives.
Using malarial nets to dry fish means not using them to prevent malaria.
Ok, good chance the answer is "give more nets". How are you supposed to know that without this study?
The insecticides might also be fish poison, this could lead to a collapse of an important food source. A study like this could help surface that before it becomes a disaster.
My problem is not with them trying to determine what is being done with bed nets and why. My problem is with the framing. It is incredibly judgy and it is the kind of language that goes along with policies that boil down to "the beatings shall continue until morale improves."
Such language tends to point to an agenda and to an underlying hostile attitude towards the population supposedly being served. I was homeless for a few years. A lot of homeless programs are actively hostile to homeless individuals. This helped sharpen my existing tendency to be critical of such details.
If you think how it is framed doesn't matter, perhaps we can discuss some choice words for you or your profession or your demographic and see if you still think details of that ilk don't matter. Hint: When you say it matters if it is done to you, but it is irrelevant when done to some downtrodden group receiving "assistance" (often of the "Please stop helping me!" variety), then you are prejudiced and the existence of this prejudice out in the world is likely one of the root causes of the group in question being downtrodden and unable to make their lives work.
It reminds me of an anecdote about CS Lewis. He is supposed to have given a beggar some change, and he friend asked him "what did you do that for? He's just going to spend it on booze." And Lewis is supposed to have replied "well, that's all I was going to use it for anyway."
The painful truth is that it is morally repugnant to donate. I hate saying this, by the way. If you had any experience with outreach, You would understand it is actually quite destructive to give them money, and it is much better to donate to, for example, Union Gospel Mission, a place in Seattle that does excellent homeless outreach. The city in fact contracts out to them for their services.
> The painful truth is that it is morally repugnant to donate. I hate saying this, by the way.
I hate that you said it without explaining what you meant. :P
Seriously, when you say "it is morally repugnant to donate" and "it is much better to donate to X" in the same paragraph, I am not sure what to think about it. Do you think that donating to X is still morally repugnant, although a bit less than donating to other causes? Or that donating can either be wrong or right depending on who you donate to (i.e. how the money is going to be used)? Because the latter seems like something that Effective Altruists would totally agree with.
Sorry, my comment was quite unclear. I meant it is dangerous to street people to give the money directly. It is much better to give it to you in the Stabley Schomann organization that knows how to deal with their problems.
> organizations that work on improving long-term outcomes for humanity. Grants will likely go to organizations that seek to reduce global catastrophic risks, especially those relating to advanced artificial intelligence. [1]
Yes, an argument can be made that it's important to fund prevention of global catastrophes, as while they're unlikely compared to the immediate threat of malaria, they'll cause much greater damage, thus increasing risk. However, to consider artificial intelligence to be a potential global catastrophe at all, let alone the single one requiring extra funding, is mostly unfounded. We currently can barely even define what the actual risk is, let alone how to mitigate it.
It's one thing to walk past homeless people in my city and not give them money, because I know that money could much more easily and effectively safe a life in malaria-ridden parts of the world. I think it's absolutely morally repugnant to walk past them and not give them money, because instead you're paying people to sit around thinking about AI.
[1] https://app.effectivealtruism.org/funds/far-future