Sep 20, 2016 · 274 comments
Elizabeth Bennett (Arizona)
While this article clarified some questions about the process involved in accepting comments, it didn't answer how Google is going to moderate comments without mining data from the commenters. The "Comments" section is a must-read in my opinion. The writers are often very well informed, and can enhance, elucidate and explain some aspects of the columns. As a commenter who had 95% or so of my comments accepted, and in the period of a little over a month had been awarded 4 "Times Picks", all of a sudden all of my comments were rejected--a total of eight. These comments were civil, and I think they added some new insight into the subject of the column. Have I been blackballed? And if so, why. I'm aware that this is self-centered of me, but I ask anyway on the assumption that others may have had the same experience.
jrg (Portland, OR)
This is an interesting article -- and experiment. I didn't score well on it, largely deliberately, because I voted the way I would act, not the way I thought the NYTimes would.

I have a couple of suggestions/complaints:

1. Though not deliberate, there is a huge East-Coast (and European) bias in selection of comments. I've long observed that most West Coast comments receiving enough recommendations to be widely read are written between midnight and 3AM PDT! While there are huge numbers of readers of the NYTimes on the West Coast, they are severely under-represented in comments, either not there at all (because "comments are closed") or because they weren't written until there were already many comments accepted and were never seen by enough people to emerge from the bottom of the (deep) pile.

2. It would be nice to know when (and why) a comment has been rejected. It is very frustrating to wait many hours to have to conclude the comment has not been accepted, with no indication of why. I have submitted quite a few comments that I'm really confident were not rejected for content but never appeared, presumably because some limit had been reached.

[This comment is really for those doing the selection. After this much time has elapsed and 308 comments have already been accepted, I have no illusion that many others will read it.]
Yuri Asian (Bay Area)
This exercise may reflect your ideal moderation template but it's not the reality of what is actually published...DemocRats and Repuglicans, for example, and worse, appear in comments frequently. Nonsense and gibberish in comments appear daily. Dr Krugman's column invariably attracts gratuitous and irrelevant personal attacks daily, most of which are posted absent any context, and everyday a dozen or more regular posters comment mainly to promote their blogs. Every one of your examples of rejected comments can be found daily.

I have no doubt moderators are overwhelmed and have to exercise rapid judgements and mistakes get made. I suspect there's a level of subjectivity as well that moderators develop with frequent posters. Some days my comments appear instantly, even as I post from the West Coast. Other days my comments don't appear for two or three days, are lost in the system, or are rejected (I assume only because it's usually a specific day each week and not others).

It seems to me the moderation of comments could be more responsive, efficient and faster if it were done with a larger group of moderators who review comments from home instead of being on site. Forty telecommuting moderators employed part-time, with additional on-line moderators on call to handle peak periods, would be cost-efficient and significantly reduce reader frustration. There's no shortage of J-school grad students who could use both part-time work as well as real world work experience.
R Stein (Connecticut)
Well-intentioned strategy, but, as in many new media transformations, missing some parts, human or otherwise. The Times' comments have stayed 'cleaner' than those attached to other papers and forums because of active moderation, which is good, but as, especially, this election cycle and the nature of wide-open news reporting has taught, civil discourse is not the appropriate standard for communication any more.
When public speech includes profane words and ideas, as it does now, it must appear, and, in fact, does in Times news stories. When some distasteful, bigoted, or hateful speech appears, it is increasingly important to recognize that it exists, and in fact, represents the public.
Civil discourse, established by the old filters, unfortunately is becoming a poor criterion, and risks bias all by itself.
I sure wouldn't want an editorial job these days.
Camille Reyes (San Antonio)
Far from "feeling bad" about not doing well on the quiz, I am left wondering how many members of the Community desk are straight, white men. This is not intended as an insult to straight, white men, but it is to say that as a Latina, I suspect I viewed the comments differently. Could you tell us the demographic composition of the desk members? I'd much rather see diversity (and maybe I will) than increased algorithmic production.
Lifelong Reader (New York)
1) Why aren't readers able to look up all their comments the way they can on Gawker Media and Reddit? There's probably a cost involved, but since you're introducing a new system it should be considered. As previously noted, even when I receive an email announcement of the posting of my comment, the link frequently does not take me to the comment. I have to sift through all the comments to find it.

2) Again, what is the difference between:

A comment score with a number: # Recommended;
A comment score with a white thumbs up icon;
A comment score with a filled-in blue thumbs up icon?

I've asked about this several times, but there's never been an answer.
PlutoinCapricorn (Seattle)
Oh my God! Please forgive me for being dramatic. There is no collection of words that that could define all the possible abuse scenarios (intended or accidental).

I'm just a reader so pardon me if I don't sound like an "expert" in online community discourse. The only comments that I would reject are outright are threats of violence. As far as insults and obscenities... Well, we all need to toughen up.

Young people go to war, politicians lie every day, multinational corporations are the elephant sitting in every family room across the nation. Nothing is being done to moderate the actual evil that goes on in the world and now words are being treated as weapons of mass destruction?

Oh my God!!!!! I fear I'm being too dramatic and now I fear that I am being "politically incorrect". Will my comment be allowed?
Lifelong Reader (New York)
PlutoinCapricorn:

In an online community, a certain level of respect is expected. No one has a right to insult people, especially if the offensive comment is racist, sexist, homophobic, or otherwise based on a ground viewed as illegitimate. What we are willing to tolerate evolves as time passes. A member of a group who has experienced discrimination since birth does not need further "toughen[ing] up."

Directing vicious, stereotypical, untrue remarks at groups IS a form of "actual evil." Refusing to accept such conduct clearly is not on the level of a tour of duty in Iraq, but it is not nothing.
Ilya Shlyakhter (Cambridge, MA)
Moderators' most useful function is the picking of NYT Picks. This can't be automated or crowdsourced. So let moderators spend more time on that; hire higher-caliber people to do it.

Another function is to screen out clearly rude & vacuous comments. This is more mechanical, and can be done by entry-level staff.

The third function is to screen out comments that push NYT's rules but not widely accepted norms. Here, I trust the crowd's view over the subjective call of one rushed moderator. Show a new comment to a few users first; if they don't flag it, show to more. Then any one bad comment only irks a few users, and any one user only sees a few unvetted comments -- not enough to affect the overall quality of the comment stream (esp. if users can opt out of seeing unvetted comments). Post-moderation by NYT staff can still be done, but why not crowdsource pre-moderation this way?
Madeleine (NYC)
In response to the quiz question mildly criticizing the Times for opening comments on an article about the Weiner/Abedin marriage, the explanatory note informs readers, "[c]riticism of The Times should be approved in comments so long as commenters make a credible attempt to explain their thinking." That is an enormous departure from my experience. After having several respectful, reasoned, critical comments denied approval on Food section articles, I complained to your Customer Service address in 2014 (via email) and received the reply: "Sometimes the comments get lost in the system since at any time there may be up to a thousand people commenting on a multitude of different articles. Also, it is up to the author of the article to post the comment and comments are only open to be approved for posting up to 4 hours after the initial article is posted. We do apologize for the inconvenience however we can’t always guarantee that your comment will be posted. We hope this helps." My reading of this was that it was an admission that the Times created a conflict of interest by allowing writers to moderate comments on their own articles, and that they may of course not wish to publish comments that are critical of them. Have you or have you not ended this practice? And is there any mechanism by which writers' comment moderation tendencies are reviewed by anyone else? Has there ever been?
Jake (New York)
Having left a number of unapproved comments for various stories, despite going to some length to explain/defend my position rationally and civilly, I can't say I'm unsurprised to find that I'm 100% on the "wrong" side of these approvals. In each case, the "explanation" betrays the way in which the moderator's own biases inform whether a comment was an "insult" or merely an "offensive point of view." So, advocating that Syrians, Saudi Arabians, and Iranians be "sterilized" is characterized as "effectively an insult to an entire class of people." (Quick question: does the moderator know what this "entire class of people" is supposed to be? Recall that Saudi Arabians, Iranians, and Syrians do not comprise an ethnically or religiously homogeneous class of people.) But summarily asserting that women, as a class, are "obsessed" with monogamy, and implying that men are biologically programmed to cheat, apparently counts as a meaningful contribution, and not an "insult" to either women or men. Using "Repugnicans" once will get you dinged, but if you leave an incoherent tirade against homosexuality that is consonant with public discourse, the reaction you're likely to get will be deemed sufficiently "interesting" to retain, despite its clear inflammatory intent and effect.

What this interactive piece shows, more than anything, is that the moderators for the website have no set of rules they can apply consistently to promote civil, substantive discussion.
Camille Reyes (San Antonio)
Well said.
Dean MacGregor (New York City)
I have called the Insider elitest several times and it get censored every time.
Dean MacGregor (New York City)
Why are comments critical of the NYTimes not approved?
Madeleine (NYC)
I complained about this to Customer Service in 2014 after having several comments that were critical but respectful and reasoned denied approval, and received an email informing me, "Sometimes the comments get lost in the system since at any time there may be up to a thousand people commenting on a multitude of different articles. Also, it is up to the author of the article to post the comment and comments are only open to be approved for posting up to 4 hours after the initial article is posted. We do apologize for the inconvenience however we can’t always guarantee that your comment will be posted. We hope this helps." (For clarification, I'd been attempting to comment on several articles in the Food section, and would be surprised if the same procedure was followed by other reporting desks, with writers in war zones & so forth). I lost a tremendous respect for the paper for not appreciating the obvious conflict of interest created by having writers moderate comments on their own articles, and I read it far less often for this and similar reasons. Over and over again in recent years the NYT has revealed itself to be inattentive to potential conflicts of interest, as in the more well-known example of the time Marc Andreessen's wife was commissioned to write a glowing profile of a company the billionaire couple owns a sizable chunk of, without so much as a disclosure. That was inexcusable. It's entirely inconsistent with professional standards.
The Breeze (US)
So, if I say that all Mexicans, or that all White people, are insecure, that's cool? That's the one that I got wrong, the one that stigmatizes an entire subgroup.
Marla (<br/>)
It turns out that I am stricter than the Times' own commenting screeners. I try to be as civil as I am able when I comment, yet I seem to have quite a bit of trouble getting my own comments published.

'Tis a red letter day, for certain, when one of mine makes it through. Anything that will work to increase both my success at getting published and the number of articles that will allow commenting stands as an improvement, in my book.
Jonas Kaye (NYC)
This is a two-parter:

1) Based on my score it would take me 28 hours to get through your daily load. I hope you have more than four people working your comments, and I would hope you have some additional redundancy built in.

2) I've definitely been frustrated in the past by comments being filtered out where they do not break the rules as stated here. Rather, perhaps they swim against the prevailing logic of the day: that murder is "okay" as long as "the good guys" are doing it. It is my opinion that we are deeply entrenched in some cultural sickness, and fifteen years of killing people around the world and worshipping the men who do it makes us fundamentally less safe.
Ameet Mehta (London)
I enjoyed better understanding your moderation process, and how difficult it can be. When a comment is rejected, does the submitter get any feedback on their comment and/or the opportunity edit their submission for reconsideration? I could not see much about rejection in your FAQ.
MCS (Upper West Side)
What is the policy on length of comments? I get cut off after a few hundred characters, yet other posted comments go on for paragraph after paragraph. But the comments feature alone is worth the subscription price. It's also a major advantage over the print version.
Ilya Shlyakhter (Cambridge, MA)
You get emailed when your comment is approved, but not when it's rejected. Can this be fixed? It's even more important to know of the rare rejection. It's hard to know without notification, since there's no set time for approvals. Knowing of a rejection gives you a chance to rephrase the comment (even if you have to guess at the reason).
Meg Conway (Asheville NC)
I read the NYT comment section because they are mostly informing, sometimes funny, and provide a thought process that may be unique-excellent work by those edit the comment section.
What I worry about is the use of Google.
In using Google reviews they dropped my review because a corporation complained. My comment was honest, thoughtful, and informative.
(I was able to correctly approve/reject the 5/5.)
Attempts to reach Google didn't result in the review being posted again.
It's because of that experience that I don't trust Google; but I hope for the best because I've learned a great deal from the NYT comment sections.
seaphotog (Seattle)
Very very cool project. One of the great things about machine-learning I think is the ability for the system to get smarter as more and more input is added by humans. We have feelings and they can't help but color our "approve/reject" choices, so if a machine is looking for detail that tips off a comment - so much the better. I rejected a couple in the test and it was no doubt because I simply found them offensive - a complete judgement call on my part. The NYTimes accepted in the interest of conversation. Way to go.
Kally (Kettering)
Interesting exercise! I was trying to figure out the math on this quiz. I guess the "Don't feel too bad" was for everyone taking it. Maybe it would take me 76 hours to moderate 11,000 comments, but aren't I sharing that task with 13 other people, thus making it about 5 and a half hours each? Though I must admit, I'd have gone cross-eyed trying to do that for much longer than 125 seconds! Anyway, enough of kind of patting myself on the back--and by the way, that's another kind of comment I wish would be moderated out. All those people who have to congratulate themselves on how they or their kids are better than, fitter than, smarter than whatever the article is talking about.

Hope this works--I love the comments!
Julie Meier Wright (San Diego, California)
I greatly appreciate that the New York Times moderates its comments, striking the delicate balance between hearing all views and hearing those that are unsubstantiated, insulting or worse. I think that the news media has a vital role to play in setting the tone for civil discourse. It has been bad enough that, in the interests of news, you have had to report on all of the name-calling and falsehoods in the campaign. I encourage you to play the vital role of fact checker, a la Politifact, because you have such a reach.
Sixofone (The Village)
It's odd that I scored 5 of 5 right, as I've had several comments over the years rejected-- despite scrupulously avoiding ad hominem attacks and vulgar language, as well as racist, sexist, etc. language, of course. (Avoiding attacks on entire classes of people comes naturally and effortlessly; not so much the personal insult and vulgarity, which for better or worse, I occasionally have to force myself to do.)

And this basically supports my longstanding hunch that there's not only a universal moderation team that's spread across all sections of the paper doing the moderating, but that individuals directly involved in producing the story-- be they editors and/or reporters-- are doing this for their own stories as well, at least occasionally. The comments I've written that have been tossed mostly are rejected from a single editorial section.

So, NYT, can you deny or confirm this for us, please? Are reporters and/or section editors ever moderating stories they've worked on?
MAS (New England)
Same thing has happened to me. I don't bother commenting on anything in one section -- in fact I rarely read it because the constant rejection was so uncalled for
Diva (NYC)
I agree very much with others here that the NYT has some of the most civil, thoughtful and intelligent comments on the web. This quiz was very interesting to illustrate what the NYT looks for in terms of their comments. I will certainly apply such criteria to my own comments going forward!

One thing I have missed dearly of the old style comments from long ago was the fact that they were numbered and paged for easy reference. Sometimes I come and go from an article and it's tedious and frustrating to have to scroll through all of the same comments I read before. Is there any way to address this?
Ilya Shlyakhter (Cambridge, MA)
What would help find good comments: (1) See all comments by a given person. If I like someone's comment, I want to see what else they had to say. Besides helping comment readers, having a "portfolio" of comments in one place would incentivize comment writers to write good comments. (2) Tab where comments are sorted by a score "number of users who recommended the comment, divided by the number who saw the comment". This would highlight comments that now score low not because they're bad, but because few people have seen them.
Tired of Hypocrisy (USA)
I would like to know how, as in what criteria, and by whom are articles chosen to have or to not have comments. There have been some very strong articles and opinions within articles that call-out for comments but there is nowhere to leave them.
Ilya Shlyakhter (Cambridge, MA)
Can you post more examples of rejected comments (without commenters' names), and the reasoning behind the rejection? This would go a long way towards demystifying the process, and illustrating the boundaries. No need to post the obvious cases (gross crudeness, etc); but do post the various boundary cases. People may not agree with all your moderating decisions, but will respect them if they're explained.
Peter (New York)
In your fourth example explanation, you mention the Antarctic Liberation Front. When I was at Princeton, this completely non-violent group won the student council elections on their platform of three promises (which they kept) that went approximately like this:
1) Annex all of the land between the double yellow lines.
2) Rename all buildings on the Princeton campus that are under six inches in height after a particular, short member of their party (there is one such building).
3) Resign from office.

Yes, this post will be rejected. Just hope you get a chuckle from my reminiscences.
jimbo (Dana Point)
Allowing gods to be reasons for commenters to consider a course of action is allowed by your guidelines (item 3), yet when I point out the truth that gods don't exist, never have and never will until proven otherwise, you consistently censor me. Use of gods to justify is nothing but authoritarian religious proselytizing, it has no purpose otherwise. It is used by fundamentalist religions to influence and impact. You censoring me for pointing this out is aiding in spreading what I consider an infection, please try to understand what I consider a threat to our Constitution.
David Godinez (Kansas City, MO)
I look at the comments to gauge opinions on the issue that's being reported, and hopefully hear something new. A suggestion on your changes in the 'Comments' is to be more selective when it comes to the repetition of an idea. I don't need to see the same basic concept expressed over and over. This might also encourage commenters to be more original in what they write. I try not to write comments, especially in the shorter threads, that just repeat what someone has already said. There's no point to that, and no point to reading it either.
Miss ABC (NJ)
What do you mean by "class of people"?
Clearly it includes people of the same race, muslims (no other religion and definitely not atheists, just muslims. Maybe orthodox jews), and women who will do anything to get pregnant.
What about students? women? police? Germans? Chinese (this is different than Germans)? etc. etc.
Maybe you should just declare a category of "protected people". It's your publication, you get to edit it whenever you like, it's you right. I just wish you'd be transparent about it.
HSW (.)
Some of the moderation rationales are debatable:

1. Approve. "Repugnicans" is obviously political speech expressing personal distaste for Republicans. Effective use of a pun.

2. Disapprove. Meta-comments are off-topic.

3. Approve.

4. Approve. Assumes that population growth is the cause of Middle East conflicts. Proposes sterilization as solution. Easily rebutted as an inadequate solution, because someone would have to decide who is sterilized. The analogy with the French Revolution is also easily rebutted -- France was a dominant military power at the time. (BTW, this example appears to be a Reply, not a Comment -- it addresses someone as "you".)

5. Approve.

None of the examples feature a personal attack or a libelous statement, yet I've seen instances of both.

2016-09-21 18:19:18 UTC
Global Citizen Chip (USA)
Why doesn't the NYT's use Disqus? By far the best moderated comment thread format on the internet!
Madeleine (NYC)
Disqus is just a platform for commenting, not a moderation system. Whether and how sites that use it moderate their comments varies. Gothamist, for example, seems to rely on volunteer moderators who have a little "MOD" icon next to their user names.
tom carney (manhattan Beach)
The assumption that machines can learn to evaluate human comments is a big step down the road to the Matrix. As a human types a note into a computer she or he is moving thought energy into word forms. The words of the message carry the frequency of the writers emotional and mental bodies. These non-verbal frequencies are an intrinsic aspect of the message.
This "system" you are devising dehumanizes the exchange of thoughts and feelings between humans. Those who "own" the robots you are planing to use will unfailingly infuse their programming with their individualized concepts of what is O.K. and what is not.
This already happens to some extent in your present system, due to the green check thing.

The controlling issue is ultimately the making of a profit for the NYT shareholders and nice salaries for upper management.

I think that it is not that difficult to eliminate gross stupidity and vulgarity. Pretty much everything else should be O.K. The only reason I rejected 4 of the 5 choices was because the were not interesting to me. Obviously I would have to not do that if I were to become one of your evaluaters. It would not be difficult for me to do that. However once programmed no robot would be able to make any kind of decision at all.
Also, how long do you think it would take to program a robot to con your robots?
This is not a road down which you want to go.
ch (Indiana)
I have had comments rejected that I though met your stated criteria. I have also seen comments published that didn't meet your stated criteria. I do, however, appreciate your rejecting comments that contain profanity and (mostly) comments in which the commenter insults other commenters.
HSW (.)
Have you ever tried reposting a comment? With 14 moderators, there is a good chance that a different moderator will review it the second time. Tip: Try rewording. The Times probably has automatic duplicate-detection. And if not, it should.
Jon (NM)
Interesting.

So my saying (hypothetically speaking assuming I said this) that most Republicans are bigots, homophobes, misogynists, racists and xenophobes, which is factually true, but which targets and might considered insulting to an entire class of people based on their belief system, will be rejected?
John V Hall (Germany)
Yes? Although hypothetically speaking..., prefacing any opinion with this technically would allow you to say anything so long as you framed it as a question? So maybe this post should have been rejected as well? (Try this technique with obscenity and see if it works!)
Tired of Hypocrisy (USA)
Jon - No, that comment won't be rejected but not because it's untrue and bigoted to classify an entire group of people based upon what the NYT thinks. It won't be rejected because it fits neatly into the NYT agenda. Now if you said the same things about liberal progressive Democrats then that will be rejected because the NYT "thinks" it's untrue and "thinks" it's bigoted.
HSW (.)
In your hypothetical comment, you didn't define "most", and you didn't cite a source. Since those deficiencies could, in principle, be fixed, the comment should be posted, so readers can ask you to clarify your meaning and to cite your sources.
Steve (Desert Southwest)
Even if the moderation of the comments is flawed by delays and subjective editing, it is far superior to the Washington Post that does not seem to moderate. I rarely read the comments there for just the reasons elaborated by the NYT.
Dan Nguyen (New York)
Why are you hand-moderating every comment for prior review, when you could reduce the load significantly by giving users the option to flag/downvote a comment? I'm assuming you could run pre-filters on each comment, such as if it has an obvious curse word or expletive, send it to the moderation queue.

Another way to be more efficient: I've been a subscriber long enough to have paid more than $1,000 in subscriber fees to the Times. And yet I could have an easier/faster time getting a comment approved on Jezebel using the handle gamergatezrules666. Have a little more faith in your loyal readers.
Olivia (NYC)
None of the sample comments were thoughtfully written or argued. You're allowing them to be published because everybody enjoys self-righteous outrage, and not because they contribute to the discussion in any meaningful way.
John Brady (Canterbury, CT.)
A robot? O boy! How flexible can a robot be? I see no good coming of this. As an alternative just go around the city and corral as many homeless reporters as you can and augment the staff
Jesus Christ That's Jason Bourne (Antartica)
Why are post even approved or rejected? I feel like it makes it too easy too control speech much like /r/Politics trying to control any and all political news to only their liking.
David (Chicago)
I would love it if you could force your columnists to read the comments on their pieces--at least the top 10 or 20 ones. Ok, I'm mostly joking, but I'm often struck by how much better some of the comments are than the original essay, or how effectively the comments expose logical flaws, misinformation, hypocrisy, etc. in the columns. And I'm not only talking about Douthat's and Brooks's columns! Seriously, if our subscription dollars are going to these people's salaries, it would be nice for them to maybe be accountable to their readers. Or give us the opportunity once per year to vote on how informative we found each columnist. I have no problem reading well-expressed ideas I disagree with, but it infuriates me to see sloppy thinking rewarded by this paper!
ggallo (Middletown, NY)
An interesting and thoughtful comment there. Now, I find that a lot of the comments (I'm referring to ones that are intelligent and thorough) are often academic; as in, not practical. Good for the classroom. Not so good for application in the world were a said issue of an article exists.
Another common occurrence are comments attacking the writer. Personally, those I find annoying. Some comments are a chance to showcase a person's expertise. Those could be informative or off point. However, if it weren't for any NYTimes article we would not be making any comments to "clear up things."
Miss ABC (NJ)
Your community desk people are too protective of islam.
I don't use bad or inflammatory language. I try to keep my comment to my personal situation -- i.e. I have two daughters thus my feelings towards islam. But they are ALWAYS rejected. The last time my islam-related comment was rejected because I called it a "philosophy" instead of a "religion" (I know, because that was the only remotely critical thing in that particular comment).

I doubt you will publish this comment. Or maybe you are wanting to prove us wrong? I am pleasantly surprised to see a couple of islam-related comments published in this comment section... Also surprised that you approve my first comment here which contains the word "islam" -- with the small "i". Or maybe you thought that was a grammatical error?
Sage (Santa Cruz)
No moderation at all would not be a desirable approach in my view, but a clearer, more transparent and more consistent procedure to the existing one would be desirable. Whether using an automated Google program will yield improvements is questionable, and the current setup is far from the worst imaginable, but you could try this new system, ask for reader responses, assess the results yourself, decide and let us know.
bkw (USA)
When my brain produces an intelligent eloquent insightful informative enlightened comment that I'm sure will be recommended by awe-struck hundreds if not thousands (smile) there's just something unsettling to consider it being read and judged by a sans heart, soul, emotion "Jigsaw."
julsHz (Fort Worth, TX)
Jared Cohen is the founder and president of Jigsaw, a technology incubator at Alphabet Inc. previously known as Google Ideas. From 2006 to 2010 he served as a member of the secretary of state's Policy Planning Staff and a close advisor to both Condoleezza Rice and Hillary Clinton.

Clinton, in response to the recent NY and NJ bombings, said, "We need to work more closely with Silicon Valley and other partners to counter terrorist propaganda and recruitment efforts online."

Much of Cohen's early work was focused on enabling censored populations in the ME to overcome government internet blocking in that region, e.g. a Chrome add-on called Password Alert developed for Syrian activists targeted by government-friendly hackers, but when it proved effective, it was rolled out to all of Google’s users.

I encourage anyone interested to read this recent article for more background on Jared Cohen's digital projects/products, including Conversation AI.
https://www.wired.com/2016/09/inside-googles-internet-justice-league-ai-...

It's not a far leap to imagine this is the Silicon Valley, or other, partner Mrs. Clinton is referring to. Combine Google's data mining efficacy with the government's propensity for surveillance programs and you've got one powerful tool-- for both good works and, equally, open to abuse if unbound from regulation.

We are entering new digital turf here.
Dawn R (Maine)
I just had a conversation with friends over dinner last night about how much I enjoy reading a good NYTimes article and some of other readers' comments. My friends also read the Times, but not the comments section. They prefer to get their "reader's input" from Facebook. We compared things we had learned from our respective sources and fact checked them. I won! NYTImes readers in general, we concluded, are much more knowledgable and logical than Facebook commenters.

Having said that, in your "approve or reject" quiz, how can a disparaging and generalized remark about a whole people (Middle Easterners in question 4) be rejected while an equally generalized and disparaging remark about an entire sex (women in question 5) is accepted?
ggallo (Middletown, NY)
What gets me is when I am trying to make a funny comment and the moderators don't edit it to make it funnier. Now that's annoying.
Rh (La)
The comment section is also deterministic based on a some biases practised by the NYT? Certain core issues like
>multiculturalism and its negative impact on western countries,
>the effort to be politically correct and appeasement of strong opposition voices to it
>hatred practised by certain religions over millennia
gets a pass or is hidden under the carpet and rationalised too quickly by the NYT.

Similarly false narratives is propogated on some issues
> like the impact of globalisation on local populations
> negative impact of world bank financial aid
> environmental degradation without contextual linkages

If the NYT Can reduce its inherent biases it will become a better paper for its audience.
Ellen Hershey (Bay Area, CA)
Re no. 5, I rejected because I thought calling all women insecure was an insult against an entire class of people -- indeed, over half our country's population. A typical misogynistic cliché.
Dan Nguyen (New York)
It wasn't even just the slam against an entire group, it's the loaded arrogance of the question, e.g. "Why are women so emotional and illogical?" What could possibly be a reasonable, worthwhile answer to that unless you're nodding your head and thinking, "Yeah, what's *is* up with women anyway?"
Madeleine (NYC)
Their reasoning for allowing it on the basis that it could provoke interesting discussion among readers strikes me as nonsensical. By that logic why moderate comments at all?
McGuan (New York)
I am a frequent commenter and NYT subscriber, but don't understand why you would not publish a comment I made on education and segregated schools in the city.

I wrote an example of what I witnessed in a majority black school in East Flatbush by nephew's two white teachers on a school field trip whereby they provided no instruction for their students.

Was it that I said East Flatbush is now called "Prospect-Lefferts Gardens now that whites have moved into the neighborhood? I could have said now that the neighborhood is being 'gentrified', but I chose not to and didn't think it was offensive. Gentrification is code for white people moving into predominately poor black neighborhoods. The "Gentry" are those civilized persons of society with wealth and it doesn't usually refer to blacks with money.

It's the truth and the NYT should appreciate the truth.
Lifelong Reader (New York)
If you are going to close the comments soon, you should post an announcement. More than a few times, I've spent time on a response only to see that the window had closed. It's not very considerate to readers.

You could have a countdown timer saying "Comments will close in ___ time."
DT (CA)
I thought it was a very interesting exercise, and will be keeping an eye out for follow up article as this processes progresses. I wish more outlets applied the standards employed by the NYT & hope some will take advantage of the open source resources that are generated from this.
FunkyIrishman (Ireland)
Why not just follow the 1st amendment ?

Hasn't it proven to be good over the last 240 years ?
Chief Quahog (Planet Earth)
How does one earn the coveted "Trusted Commenter" status? I have never had a comment rejected and I don't ramble incoherently. (Yes, I understand that ramblers probably don't recognize themselves as offenders...) I have never attacked another poster or maligned an entire class of people.

What is the key?
Tired of Hypocrisy (USA)
Chief Quahog - "How does one earn the coveted "Trusted Commenter" status?"

Maybe it depends on your point of view. I don't believe I've ever read anything approving of conservative or Republican thinking or action from a "Trusted Commenter." Assume they are always wrong, write about that assumption and you may achieve that status.
Miss ABC (NJ)
Your definition of "an entire class of people" is too broad. You include not just race in this definition, but also religion. For example, "islam" should not be a "class of people" since there are many muslims who are not middle-easterners.
Or, you just need to state unequivocally that you will not tolerate comments made against adherents of a recognized religion. This way, I will know not to write anything remotely negative about islam. Instead of wasting my time trying to argue my position, as I have done many times before I finally figured out your unstated rule.
Doug (Baltimore, MD)
Most of the examples of inappropriate content the NYT seeks to weed out are pretty easy to spot. Except for one: impersonation. A large proportion of the comments on articles related to China or Russia are posted under pseudonyms by their respective propaganda machines. How will the NYT weed these out?
Dr. Planarian (Arlington, Virginia)
I greatly approve of the New York Times's policy of moderating comments and the manner by which they do it.

To see why I approve of it, contrast the relative civility and informative nature of Times comments with the unpleasant vituperative natire of those in the unmoderated Washington Post, which is a home to ignoramuses and trolls. The meanness pointlessness and ignorance of a large number of participants there, many of whom are obviously paid posters who take user names like "Hillary is a Nazy" (sic) and "Obummer," are not there to inform but to mislead and to hector, and sometimes to render the entire thread unusable by burying it in a torrent of cruft. Even when posts there are patently offensive and consist of ad hominem against other participants, the posts remain until they are called to the attention of moderators, some of whom moderate out posts with which they personally disagree.

I prefer the Times's system, even though it often results in some delay in a comment appearing.
Atikin (North Carolina Yankee)
I already don't like this. I agree with some of your restrictions, but envision the comments so boring and cookie-cutter bland that I will stop reading (and contributing) altogether. Blah!
Outside the Box (America)
.. but the NYT doesn't apply these standards. It allows inflammatory comments as long as the target is conservative, European-American, Christian, ...
RealityCheck (Earth)
Attacks against the childfree and athiests are generally not censored, either.
Sharon5101 (Rockaway Beach Ny)
Don't forget Jewish!!! Jews are always fair game at on the Times Comments Sections.
HSW (.)
Times: "... a new moderation system that will help us review incoming comments based on decisions our moderators have made in the past."

Comments from Verified Commenters should be EXCLUDED from the training dataset, because those comments are not subject to moderation.
Timothy Bal (Central Jersey)
Overall, the NYT's moderation of comments is way too subjective. I agree that it should not publish vulgarity, profanity, vitriolic attacks against other commenters, and probably some other vices.

There are some things that the community of commenters does a better job at than the speech police (e.g., conspiracy theories), and The NYT needs to lighten up about criticism of its own reporters, pundits, and editorial writers.

The *community desk* is also too sensitive about insults to groups. Let other commenters reply to those insults. People need thicker skins. Besides, one person's insult is another person's telling the truth.

Too much censorship (that is what it is) deprives the rest of us from seeing what others are really thinking. You need to have more faith in the wider community of NYT readers.

Finally, I hope Google Jigsaw will also operate on the Op Ed page. Too often, I get the impression that if the author of an op ed or column disagrees with me, my comment is either not published or is published about 6 or more hours later.
matt polsky (white township, nj)
Having taken this test, and gone 5 for 5, albeit with too much time taken, it is instructive to see what Moderators go through. The Meta of your informing readers about this substantive change is welcome and respectful. I have 12 suggestions:
. bounce out nearly any non-nuanced use of "white men" or "men" for assigned blame for various ills just as you would in most cases with other groups
. alert those in the Comments-in-preparation stage the publication window is shrinking, and therefore they risk wasting their time
. make it harder to "lose" drafts
. a way for us to tell you that we're having some trouble "Submitting," and for you to then tell us what we could try
. if Comments have ended for the on-line version, tell us if they're still open in Facebook
. on hot topics that are going to get a lot of responses, a higher threshold for new submitted comments whose themes have already been published, thus preserving space for other, later-arriving ideas
. a lower threshold for totally new themes
. a Search function for themes and Commentators
. editors should mine common themes for future articles
. when we're alerted by email that we have a Comment published, make it as easy to find on our phones as it is on our computers
. tell us how "Top" this and that ratings are created
. create an expectation that the original writers of articles or op-eds should peruse the responses they get, with a preference they will address frequent themes in future articles on the same subject.
whisper spritely (Catalina Foothills)
I only did half-bad.
Since I began@4:30AM Tucson time with no coffee yet,
then meandering on intermittent to obeying mother nature-including hummingbirds at the window frantic for me to provide their 'fix' because the bats had drained their feeders during the night.
I'm more the no-holds-barred type in that I am happy when words flow straight out and I send them in without 'working' them.
That to me is more satisfying than seeing them being viewed as acceptable.
whisper spritely (Catalina Foothills)
That is not to say that I am a 'paragon of patience'.

"Score You moderated 3 out of 5 comments as the Community desk would have, and it took you 541 seconds. Moderating the 11,000 comments posted to nytimes.com each day would take you 330.6 hours. Don’t feel too bad; reviewing all of these comments takes us a long time, too. Moderating almost entirely by hand has helped us form one of the best communities on the web, but the time has come to expand this initiative, and make it easier for you to talk with The Times and with one another."

*The best part about this NYT statement is the part "talk with The Times and one another".

Truth be told I do "talk with The Times"-one way or another when I strongly feel a certain comment I did feel was reasonably important/of value was kept from being 'heard'.
Which I am grateful to be able to do.
Professor Ice (New York)
The best thing is no moderation what so ever. If you must moderate then a robot is superior to your 11 editors for a number of reasons. First, you editors have a history of bias against any opinion that is unpopular with the editorial policy. Second, the community desk is hyper politically correct when it comes to protecting Islam from any and all criticism.

You can of course edit this comment out (like you usually do) but that would simply prove my point.
Civres (Kingston NJ)
I took the test and here was the result:

"You moderated 5 out of 5 comments as the Community desk would have, and it took you 81 seconds. Moderating the 11,000 comments posted to nytimes.com each day would take you 49.5 hours."

But that assumes that I'm working alone. With 13 others pitching in, we could knock them out in 3.5 hours—let's say 4.5, to allow for some breaks to rest our weary eyes and brains.

Admittedly, it isn't the world's most edifying work, and I'd probably quit after a day. Moreover, I don't have much faith in your present team of comment editors—I was on a hot streak where 90% of my comments were selected as "Times Picks" and then, abruptly, everything I submitted seemed to be rejected.

Still, the more decision making and judgment you turn over to algorithms, the dumber you will become—even if the decision making and judgment improve. Do you really want to do that to yourself?
Lifelong Reader (New York)
Civres:

"Admittedly, it isn't the world's most edifying work, and I'd probably quit after a day."

As someone who used to work for a newspaper, moderating seems like a thankless task. While critical to readers, it's invisible work that doesn't advance one's career. I don't understand the hierarchy at the Times, especially the distinction between the hard copy paper and nytimes.com, but I doubt that established full-time reporters whose work always appears in the paper do this work. I have the impression that it is farmed out to the community desk, who probably are inexperienced, junior, and trying to get recognized themselves. Some sound like freelancers or part-timers. Sometimes not being at the center of an organization themselves can make people hyper-sensitive to defending what they perceive as an institution's values.
LynneG (SoCal)
>>You moderated 5 out of 5 comments as the Community desk would have, and it took you 86 seconds. Moderating the 11,000 comments posted to nytimes.com each day would take you 52.6 hours.<<

3.72 hours with 14 people helping, so I think your observation is misplaced.

Similar concern as Civres about entrusting so much to algorithms. And a nod to Lifelong Reader below, many of the picks seem like they're made by less seasoned individuals.
HSW (.)
"... Comments FAQ: ... A few things we won't tolerate: ... SHOUTING."

Will the robo-moderator be smart enough to distinguish between "SHOUTING" and EMPHASIS?
Lifelong Reader (New York)
Good point. There should be more typographical choices in the comments section if possible, bold and italic at the very least.
leona (Raleigh)
of course what articles you choose for comments is a comment in itself. Like the 15 questions to ask the candidates in today's paper. Reads like GC wrote the questions. They are all loaded.
Brad (Chester, NJ)
I think comments serve no useful purpose, other than to get something off your chest. Moreover, when over a thousand people comment on an article, do you think anybody reads them. Doubtful.
nsteussy (Lafayette, IN)
I suggest to take a look at the comment system at arstechnica.com. The comments are scored by the community (up or down). If a comment is downvoted enough only a stub appears in the listing. If one really wants to read that screed it is available by clicking the stub. The rest of us are spared. It makes for a civil and informed discussion even of controversial topics (e.g. climate change).

Recommended.
Lifelong Reader (New York)
I have mixed feelings about downvoting. Some participants downvote anything they disagree with even if it's an intelligent comment. See Reddit.
Madeleine (NYC)
I thoroughly dislike the idea of scoring comments and avoid Reddit for that reason. It inevitably results in dominant styles, outlooks, and senses of humor taking over, none of which necessarily have broad consensus around them so much as large and dedicated cliques of users. I don't believe, for example, that the number of people who find comments along the lines of "I for one welcome our new _________ overlords" tedious rather than hilarious is accurately represented on sites that moderate by upvotes.
Diane (Arlington Heights, IL)
If my opinion differs from the usual NYT position on an issue, it is much less likely to be published. (I don't use inflammatory language, regardless of the issue.) If your moderators do this consistently, then the opinions your readers get to read are skewed.
David (Austin, Texas)
While commenting on an article might give the commenter a sense of self-importance that they are able to express their particular view in a place as "hallowed" as the New York Times, I'm not entirely sure what true purpose a comments section otherwise serves as it doesn't really matter much what random people think about a particular article or opinion piece. I doubt they ever change any opinions.

Yeah, I'm engaging in that very activity right now. Am I doing it for the self-satisfaction or does my remark really mean anything? OY! I guess I should have paid closer attention in Philosophy 101.
Lifelong Reader (New York)
I value the comments for a few reasons: To respond to writers, to respond to other readers, and to read other reader's comments. Sometimes the readers have a deeper understanding of the subject than the writer. In particular, I have in mind an article about a legal case written by a non-lawyer that was completely distorted by the author's lack of background.

I recently spent a lot of time posting comments to the article on a speech that the writer Lionel Shriver made in which she mocked people who are concerned about cultural appropriation. Unfortunately, I was part of a tiny minority of people who found what she did offensive. I felt obligated to make sure that the viewpoint of people like me was represented. Maybe I'm naive, but I think that's important. There are certain statements that can't go unanswered. I prefer to write pseudonymously, in my own words, without being edited by the Times, which is not the case if you submit a letter to the editor.
jhbev (Western NC)
What are the specifics about selecting a "trusted commenter"? There have been occasions when your criteria have been ignored.
Asem (San Diego)
I love the Washington Post but not a big fun of its comments section.Especially the dozens of back and forth exchanges between commenters can be off putting. If I am not mistaken , all of the WP's articles are open for commenting . If that's the model we are looking at, I would rather prefer the old model.
Jeffrey B. (Greer, SC)
I understand the need for Cost-Savings in the Moderation task performed, but I like it when another of my fellow human beings is doing the job. I have learned much from having my comments accepted, and rejected, about phrasing my comments in a positive, thoughtful manner, even when I disagree with how the NYT has presented it.
Robotics often miss the difficult to understand nuances of the printed word, and I believe many noteworthy comments, that would have enhanced a lively discussion, from the Community will be rejected.
Please take care; more is not necessarily better.
Sharon5101 (Rockaway Beach Ny)
I reviewed my quiz results again to see where I went wrong. I am utterly astounded that the Times' moderators would have rejected a vitriolic comment about Bibi Netanyahu and the Republicans. Before Donald Trump came on the scene, Israeli Prime Minister Benjamin Netanyahu was Public Enemy #1 to loyal Times commenters. Pandemonium broke out among Times bloggers when Netanyahu accepted a Republican invitation to address Congress about the dangers of the pending Iranian nuclear deal. Maybe Mr. Etim would now care to explain why so many blatantly anti-Semitic comments disguised as criticism of Israel were posted so quickly by verified and non-verified commenters alike?? What's truly deplorable is that negative stereotypes about Jews and Israel never go away. I've seen comments where bloggers called for Israel to be dismantled and for the Jews to be shipped back to Europe. The general consensus among bloggers is that every crisis in the Middle East would vanish overnight if only Israel would disappear. I may never get the coveted green box and checkmark but at least my conscience is clear.
barbara james (boston)
Interesting exercise. I mismatched only the last of the five comments, answering them in 79 seconds.
DT (CA)
I was about the same pace - I missed the last one as well - I saw it a bit more a an entire group being insulted, but get the NYT's position.
polymath (British Columbia)
"Most of all, you want us to maintain the civility that makes this such a special place to contribute comments."

Oh, really? Is that what we want the most?

I think that what we want most is the ability to post like grownups. That is, without a long list of words — most of which are taboo only to children younger than eight — that are automatically censored by the robocensor.

And yes, we do want civility. But that is _not_ achieved by the current simplistic route of bowdlerizing our vocabularies by applying second-grade standards.

Supposedly the NY Times stands squarely in favor of freedom of speech. (Except when that applies to itself?)
RealityCheck (Earth)
Exactly. Confusing sanitized with civil is simplistic and rather juvenile. And thin-skinned.
SusanO (VT)
With the robot having the skill to moderate comments more efficiently than humans, what"s next? applying those skills to Robot Reporting--as practiced by some other news outlets?
Wayne Dawson (Tokyo, Japan)
Yeah, this is tough. Even when I went back to do it again and knew the correct answers and didn't bother to read the explanations again, it still would take more than 24 h to finish 11000 comments.

People should (at least) consider that this writing will be there in print (perhaps forever). Over a lifetime, we can change or soften our views. Consider what a reader 100 years from now will think when he/she sees you blowing off some emotional screed. Consider what people said of Copernicus' idea in that day. Rutherford (a renown scientist), in an unbecoming moment, called nuclear power "pure moonshine". Are these things you want to be remembered for?

So, in relation to some of the decisions, the first time through, I found this quite difficult to decide because some opinions were quite raw and lacked development.

For example, even reading as a devout Christian, I didn't like the one about "God's law", because we have enough problems sorting out very old matters like predestination and grace. Are people who don't follow the usual patterns of society irredeemably outside of God's grace? To declare in such an authoritarian manner about the theology of salvation and grace without the slighted development beyond a quotation from a book of Levitical rules, is not a carefully thought through argument.

"We have two ears and one mouth so that we can listen twice as much as we speak" -- Epictetus (well, I thought it was, Zino, but maybe I was wrong.)
Questioning (CT)
Hi, I totally understand that the NYT has a lot to juggle here. Just a thought: will the new system be able to ferret out foreign trolls sitting in state-sponsored propaganda farms, paid to pretend they are really from Peoria? Until too exist to automate this process, the integrity of the comments is suspect whenever the the subject of the article in question relates to, say, Syria, Ukraine, Russia, the US elections, etc.
h (f)
The NYT tells me i shouldn't 'feel bad' for taking 100 seconds and getting four out of five right. Why should i feel bad? it's a fun easy job with few consequences. How much does it pay to do 'community desk' work?
ecco (conncecticut)
it'd be nice to know the details of nyt's arrangement with google, as commentators below have already suggested...there is no way that google won't trace, identify and exploit the identity info in the times bundle unless there are prohibitions in the agreement that include liability.
Sonja (Midwest)
Oh my goodness.

I'll never comment again.
Andrew (Chicago, IL)
NYT outsources to bots from Google, makes platitudes and cites desire for "civility." Because in civil society, you need bots from Google.

I'm done reading the comments. (Even this one!) I'm close to done with NYT.
Mary Ellen McNerney (Princeton, NJ)
Like several others, I am also suspicious of sharing a profile with Google. (For starters, our home towns are listed - will there be coupons for local restaurants showing up in the NYTimes ads that are displayed electronically?)

Here is a suggestion: as you test-pilot this program, notify readers that the article in question is/is not going to be run through the Google first-pass filter. Then analyze to see whether commenter exposure to Big Data affects the numbers of responses and the quality of the debate.
irdac (Britain)
I look forward to the proposed improvements in handling comments. It is odd that this particular article is centered on the web page so that no matter how wide the screen it is not possible to see both the article and the reader's comments without overlap. All other articles I have read were set to the left so that both article and comments could be viewed.
Yuri Asian (Bay Area)
There are several problem areas related to comments of which only one is directly addressed by the use of Google's robotic screening program. There are also a number of mechanical issues that are software fixes as yet unresolved that occur usually during periods of peak comments.

Google's program as described will assist staff moderators by flagging or blocking content that plainly violates The Times' guidelines such as profanity, ad hominem attacks, etc. This should reduce the volume that moderators review, potentially improving efficiency and quality.

Major areas of weakness that remain include: 1) time zone disadvantage; 2) favoritism; 3) tokenism; 4) blog promotion; 5) relevance; 6) moderation shut down; 7) inconsistent comment periods; 8) selection of articles for comment; 9) comment volume; 10) lack of agree/disagree option; 11) trolling.

West Coast comments submitted after 5am PST on articles published online usually after 1am PST often take several hours before posting after 9am EST, when hundreds of East Coast comments are already posted despite being submitted well after West Coast comments were sent. The result is a comment buried among hundreds of others and published after most East Coast readers are done with the paper. Comment becomes an exercise in futility, a waste of time and effort.

It can't be that hard to time-stamp comments when received and to post them in order of submission.
Sharon5101 (Rockaway Beach Ny)
Another big problem which is seldom addressed is comment repetition. Some bloggers have had the same comment published over and over again by the Times. This is unfair and completely distracting. If a blogger has already made his or her point then that's it. Commenters should be encouraged to write about something else. No repeats, please. It's imperative that comment repetition be stopped immediately.
J.D. (USA)
Using the activity provided with this article was a discouraging experience for me, but enlightening nonetheless. It brought attention to the fact that in this country, we seem to value freedom of speech more than maintaining standards. What results from this is a torrent of emotionally-heated comments, opinions, irrationality, unsupported claims, and generally just a vastness of irrelevance that one must waste time and energy sorting through to find anything worthwhile. While some may feel that this is worth the price, I respectfully disagree. I feel that if we held ourselves and each other to higher standards, we'd have more productive exchanges, be better informed, relate with more civility towards one another, and also be able to take pride in our behavior. -- And would that not be better? After all, when we have such freedom of speech, what are we left with as an issue? Quite simply: validity. With anyone free to express themselves, and with few limits on that expression, how often have we been subjected to misinformation, unsupportable claims, and flat-out lies? -- Personally, I'd gladly pay the price of being quieted every now and then, if it meant getting rid of that.
Reader In Wash, DC (Washington, DC)
I LOVED the test. One of the comments was mine!
Cunegonde Misthaven (Crete-Monee)
I sometimes see incoherent comments on stories (yet you say you do not allow incoherence). Maybe they just slip by the staff?

I also would like to know, if I am critical (in a logical, rational, non-rude way, with evidence and supporting statements) of specific NYT journalists and name them, will my comment be approved? I recently was critical of several NYT reporters and opinion writers in a comment for the public editor which I thought was well-reasoned and contained information other readers should know, and it was not approved.
Mimble Wimble (LionLand)
So, if I contend that Google is more or less the epitome of corporate evil (their "Don't Be Evil" motto notwithstanding), and that the NYT's partnering with them, and buying into their algorithms, is a terrible, terrible idea, sure to compromise nuanced assessment in favor of efficiency and cost-cutting, is my comment going to rejected?
Paul (Bellerose Terrace)
Here's a fragment of a comment from the Public Editor's most recent column, an appalling conflation of false equivalence and false balance, from a commenter named John Tavernier:
"When you argue she (Hillary Clinton) has done nothing wrong, you sound like a bunch of Branch Davidians."
So start with a generalized insult of liberals (not included here), and then compare them to a radical anti government cult. Approve or reject? Well, since I cut and pasted this fragment, now you know the rest of the story.
What is elided in this piece is that some opinion writers (not specified, but help us out, Mr. Etim), plus the Public Editor, do not send their comments through the moderation desk; they have their assistant to the job instead. In the case of the Public Editor, with a single assistant, that means that two irregular days per week, no moderation happens.
Some opinion columns have comments cut off abruptly after way less that ten hours, with a suspiciously small number of comments accepted. This makes a cognitive dissonance provoking contrast with Paul Krugman, regularly over 1000 comments per entry.
Also, some opinion pieces are posted, sometimes up for a few hours before being opened up for comments. That should never happen. People reading early should not be given the wrong idea that no comments are welcomed, only to find out further on (too late, sometimes) that comments opened up later.
paul (blyn)
I think the general system the NY Times uses is fabulous. This could only make it even better.

It helps me improve my writing skills, teach others and in return learn things from other readers.
Steve Fankuchen (Oakland, CA)
In reply to a Commenter Bassey Etim writes, "No emails of personally identifiable information is passed to Google beyond what is already available in publicly posted NYT comments."

No emails, perhaps, but what of the comments themselves? It seems they will go through Google's little greedy data mining fingers, allowing that corporation to build up individual profiles of commenters, which they can then sell to advertisers. And if Bassey Etim actually believes Google is benign with no direct interest in all this, I will be glad to sell him or her my ocean front property in Kansas at a good price.
Steve Fankuchen (Oakland, CA)
At the very least, the Times collaboration with Google needs much more explanation than is given in the article.

As it stands, one can legitimately question the credibility/fairness of the new system. In addition, the implication is that Google will be given all our comments. Given that Google mines data to sell advertising, is the Times not effectively turning over to Google our personal information? Of course once we say something on the internet it is public and forever. However, that is qualitatively different from the Times turning all we say directly over to Google.

Given what the article says, I see no reason not to believe I will now be receiving ads sold by Google based on the substance of what I say on the Times' website. If that is the case, if Google can develop a profile of me based on my Times comments, I will no longer participate.

The Times may gain some financial advantage from this set-up, but I am quite certain it will lose a significant slice of Commenters, especially those of us most concerned with -- and often most articulate about -- issues of privacy and security. Likely, the Times will be trading in quality for increased quantity.

Is our work now to be sold without our explicit permission or remuneration? Is it legal?

Sadly, this appears to be another example of the Times losing its way. On the other hand, I have been trying unsuccessfully to get the Times commenting monkey off my back. This will, I hope, now give me the motivation I need.
Third.Coast (Earth)
Calm down.

Here are the existing terms of service for comments. http://www.nytimes.com/content/help/rights/terms/terms-of-service.html#d...

3.4 You grant NYT a perpetual, nonexclusive, world-wide, royalty free, sub-licensable license to the Submissions, which includes without limitation the right for NYT or any third party it designates, to use, copy, transmit, excerpt, publish, distribute, publicly display, publicly perform, create derivative works of, host, index, cache, tag, encode, modify and adapt (including without limitation the right to adapt to streaming, downloading, broadcast, mobile, digital, thumbnail, scanning or other technologies) in any form or media now known or hereinafter developed, any Submission posted by you on or to the Services or any other Web site owned by NYT, including any Submission posted on or to the Services through a third party.

3.6 By making a Submission, you are consenting to its display and publication on the Site and in the Services and for related online and offline promotional uses.
RealityCheck (Earth)
I'll be unsubscribing, too, if what you say is true.
Steve Fankuchen (Oakland, CA)
Thanks for posting this. As I read it, it basically says what people assume, namely that the Times has the right to do anything it wants in the way of content reproducability. That, it seems to me, is different from knowingly selling our content for profiling. However, there is clearly far more than enough wiggle room in this to keep a zillion lawyers employed. If push comes to shove, the Times and obviously Google can afford endless lawyers which no commenter who would want to dispute what happens with their content can do.
Maggie s (Haiku, hi)
I for one am encouraged that this happens. I actually stopped reading a major paper precisely because the comments section became so vitrolic that I completely lost faith in the paper's substance.
Sanjay Gupta (CT)
Mr. Etim,

I am a huge fan of the moderated comments on the New York Times. In my opinion, it is not only an example of enlightened civil debate, but a roadmap to the evolution of editorial opinion. Many of us here agree that what the Times has achieved is truly special, and should be guarded jealously.

Understanding the strong desire to increase the scope of commenting, and now having read the backstory on Conversation AI in WIRED, the software behind Jigsaw, I fear that what made the Comment feature of the Times so special will be gone before too long.

It seems that the Conversation AI tool was designed as a response to abate internet trolling, not as a means to craft editorial debate. While the desire to grow the ability of the Commenting feature is understandable, it seems that it will come at a tremendous cost - one that tears at the very fabric of what makes this an editorial function, versus an ad hoc comment posting tool.

What makes the Times different from every other publication is its moderated comments feature. This cannot be understated. The Conversation AI software engine is unproven, experimental, and a single cog within a vast Google empire. It might be faster, and it might be as accurate as a human in detecting abuse, but it will never rise to the challenge of detecting wit.

The "Comments" feature is what keeps so many Times subscribers engaged. I fear that "Jigsaw" is what will cause them to leave.
Dobby's sock (US)
Nice.
Funny that the theme of so many opinions in the commentary are of computers and robotics taking away the jobs of needed people.
Then BOOM!
The paper of record does just that.
jch (NY)
What was nice about the old comment system was that a human would always read it - at least one real person. But now we'll be writing for robots, it sounds like. And then eventually they'll start writing the comments themselves. It will just be easier, more cost effective, and they will be able to represent a wider range of views than was possible with human commenters. It'll be double plus good!
Ilya Shlyakhter (Cambridge, MA)
The unpredictable moderation delay is a big disincentive to posting: what's the point, if it won't be seen till after the news have moved on? NYT should let more of us post without moderation as "verified commenters". When after years of posting thoughtful comments that get good feedback you're still seen as a potential spammer, the criteria are clearly much too strict. Relax the criteria, and let moderators focus on the truly problemmatic commenters.
John T. (USA)
"...let moderators focus on the truly problemmatic commenters."

I'd think that this is one of the main reasons for the introduction of Jigsaw and Conversation AI. I also think this will make the commenting 'delay' a thing of the past, along with the 'verified commenter' system. I do hope NYT find other positions for the moderators affected by this software.
Paul (Bellerose Terrace)
I voted to approve all. I knew I was wrong on the first one, but I've been calling Republicans "Republicants" for years, and most of those pass moderation. As for the middle east question insulting a whole region of people, did you guys read the NYT Magazine article devoted entirely to the middle east? That was close to as dark an outlook.
Finally, as for the snarky evaluation of my moderation pace, you wrote:
"It took you 82 seconds...it would take you 50.1 hours to moderate 11,000 comments."
This is a thoroughly dishonest response. After all, the 11,000 comments are moderated by FOURTEEN People, not ONE. Assuming days off, say 10 work per day. So my 50.1 hours dividing the load by ten moderators means it would mean take me FIVE HOURS, well within the parameters of a regular working day, with time for coffee, lunch, and other breaks.
I get that you want us to appreciate the work you do, but making a thoroughly dishonest assumption that one moderator would have to read 11,000 comments does NOT help your argument. And of the 11,000 comments, how many come from "verified commenters," and are thus outside the moderation process?
Cunegonde Misthaven (Crete-Monee)
So how did this comment get approved, given that it contains "shouting"? Do you only reject a comment when the entire content is in caps?
HSW (.)
Why did you approve #2, which sought to suppress comments altogether?
RealityCheck (Earth)
Not only that but it was my understanding that bloggers and others moderate their own comments. So the workload is spread even more thinly than implied.
A Reader (Detroit, MI)
While I can certainly see how the new system will ease the work load for your human moderators, I fear that there will also be very unfortunate (and unintended) consequences, on the order of Facebook's recent censorship of Nick Ut's iconic Vietnam War photo.
When I have commented here in the past, I have appreciated the fact that that my posts were accepted (or rejected) by another human being. Without that assurance, I will feel less inclined to speak out in the future.

What makes The New York Times exceptional is the people behind it. It's disturbing to see those people replaced by algorithms.
HSW (.)
"Without that assurance [that a human being read my post], I will feel less inclined to speak out in the future."

You raise a good point. How will commenters know whether a human or a bot approved or rejected a comment?
Lifelong Reader (New York)
What does the "thumbs up" icon mean when darkened? It appears to serve a different function from showing that a comment was recommended.

Upon receiving emails that my comment was posted, why am I often sent to links that don't show my comment?

Are you going to stop pre-approving commenters? I often don't understand your choices.
adarkana (Brooklyn)
These policies (which thank you for articulating) finally explain why comments on this site are so boring to read & tiresome to compose. The New Yorker fares pretty well with that weekly cartoon caption contest... it would be a lot more fun to read the few, chosen vs. the 'bot approved unwashed.
Pfundit (US of A)
Today's proof that the world is ending.
Voiceofamerica (United States)
I may at times get frustrated that a particular comment I've made has been rejected by the Times, but in the big picture, does it really matter?

What does matter is that the NY Times should offer a broad range of perspectives on contentious issues. Often, the Times does this very well. Other times, astonishingly poorly. The regurgitating of Pentagon propaganda in the run-up to the Iraq war, with little to no dissenting voices played a significant role in a catastrophe that has changed our world. Deferral to the military and political powers that be continues.

The Times must resist coercion and tirelessly uncover the truth. Why for example do we still not know the names of the apache pilots seen in the shocking Collateral Murder video? Why hasn't the Times followed up on the raft of US military atrocities? Why hasn't it doggedly pursued the case of the destruction of the Doctors Without Borders hospital? Why has it provided ZERO coverage of Gen David Petraeus' involvement in a network of torture chambers in Iraq, even after the UK Guardian did an 18 month intensive investigation?

Too much fealty to power.
Patricia Shaffer (Maryland)
I wish comments would be rejected when they include links to the commenter's personal blog. Links to another published article often are useful, but I find the self-promotion of personal opinion in longer form arrogant and annoying.
Madeleine (NYC)
At the very least comments linking to a personal blog should be approved only if they go to a particular post rather than the main page. I've often seen comments by bloggers saying something along the lines of, "how interesting, I've written about this same subject," and then link to their main page, which has nothing at all relevant on it. That's spamming.
Jonathan Katz (St. Louis)
Emily Dickinson used dashes in her poems. I hope your algorithms can distinguish dashes used as punctuation from those used to get unprintable words past blocking algorithms.
Third.Coast (Earth)
Emily Dickinson has been dead for 130 years.
Sonja (Midwest)
Has she?

Well that settles it.
Henry Hughes (Marblemount, Washington)
Having read and participated in the comments for many years, this was an easy exercise. 5 for 5, 84 seconds, 53-plus hours, iirc. And I didn't hurry. Must say the job has to be much harder than this, what with the length and complexity of so many reader contributions.

Which I value. And for so many reasons. Though I know for a fact that reading the comments here has given me a skewed assessment of how thoughtful people are. Even as I disagree with most of what is written here.

Yes, that's a backhanded compliment. Or something.

Not a fan of so-called machine intelligence, but the Times must have done quite a lot of testing. It'll be interesting to watch what happens.
Benjamin Greco (Belleville)
Thank you for this terrific insight into how the moderators do their job. Only 14 people though, the Times is a slave driver, I hope they are well compensated for their work, and they deserve the relief they will get from a pattern recognition system. I hope you write some articles with more details about how the system works. Is there any way to use it to allow post-submission editing?

I have always had a lot of fun participating in the Times’ comment boards, thank you for that, and thank you for always trying to make it better.
dg (Teaneck)
1) Your calculation that I would have taken 25 hours to deal with 11,000 comments assumes that I'm in this alone. As part of a 14-person team, however, I'd have spent less than two hours. What should I do after my coffee break?

2) JudgEments?? Really? [Explanation to item #3]

3) It will be interesting to see how Google deals with the very subjective distinction between the vile anti-Arab statement and the merely interesting and provocative one stereotyping of women.
inkydrudge (Bluemont, Va.)
Over the years I've spotted comments that contain errors of fact, and have been denied the opportunity to simply say so, without necessarily addressing the matter at hand. In the Guardian, or for that matter the WaPo, it's possible to correct factual errors in real time. Why can't the NYT let me do the same?
I know the open comments policy of the Guardian/WaPo can turn the comments into a brawl, and I know the NYT values what it imagines to be it's "tone", but surely some compromise is possible
Third.Coast (Earth)
[[inkydrudge Bluemont, Va.
Over the years I've spotted comments that contain errors of fact, and have been denied the opportunity to simply say so]]

So, what? Learn to let go and move on.
HSW (.)
"... I've spotted comments that contain errors of fact, and have been denied the opportunity to simply say so, without necessarily addressing the matter at hand."

If you intend to rebut a false statement, you need to cite a reliable source. Your word is not good enough, unless you claim to be a subject-matter expert.
inkydrudge (Bluemont, Va.)
I'm talking about simple facts, which shouldn't require a "reliable source". Sydney is not the capital of Australia (it's Canberra). The founder of Islam post-dated, not predated the reign of Alexander the Great. I know it's trivial, and perhaps pedantic, but everything teaches, even the comments section of the NYT. I'm not in the least embarrassed by my desire to set some things straight.
paperfan (west central Ohio)
The curated comments section of the NYTimes is a critical reason I maintain my online subscription (and the phenomenal TimesMachine archive). I didn't realize there were as few human eyes reviewing the comments as there are. Makes me even more impressed. I agree with another comment here, anything that helps you reviewers, but doesn't compromise what you are trying to do, would be worthwhile for all. Very insightful article. Thank you.
Erika (Atlanta, GA)
"The New York Times is partnering with Google Jigsaw to create a new moderation system that will help us review incoming comments based on decisions our moderators have made in the past. Our moderators will continue to protect these discussions, but once this new system is launched, we will have robot helpers."

So like Don Henley sang, "This is the end...this is the end...end end ennddddd...of the innocence."

While a few of my comments are rejected, especially in this election season, I am sad to read that the NYT will turn into the Washington Post and the WSJ - whose comments are so elementary and downright ignorant that I learned this spring to automatically click to the next story rather than waste my time on the comments. Good to know! That's time I can use elsewhere! It's autumn leaves season, and we need to drive around and look at them and fire up my crockpot and make some Sam Sifton via Internet housewives pot roast!

P.S. Nice illustration by Lauren Tamari.
RealityCheck (Earth)
If the Times really wants reader comments to reflect public opinion, it will cease with the nanny-ish censoring of "name-calling" and the like. Is anyone harmed by the term "Repugnants" ?

Comments about sterilizing an entire country are eyebrow-raising but I'd rather know what is really on my fellow citizens' minds than to be protected from that by 14 part-time employees of the NYT.

Also what is up with NYT picks? talk about the antithesis of objective journalism. And the picks are swayed heavily toward the politically correct...
Third.Coast (Earth)
[[RealityCheck Earth
If the Times really wants reader comments to reflect public opinion, it will cease with the nanny-ish censoring of "name-calling" and the like. Is anyone harmed by the term "Repugnants"?]]

If you don't like the rules here, there are countless other sites where you can engage in senseless, mind-numbing "I'm rubber, you're glue" arguments.

Furthermore, this is already a self-selected community…of people who read (first of all) and who read the Times. We want comments that advance the narrative. In other words, it's a tough room. If "Repugnants" is the best you can come up with, you really should just remain quiet and if you can't remain quiet I'm grateful that your comment is not allowed to appear.
RealityCheck (Earth)
I don't use the term, ThirdCoast, but I'm not going to swoon if others do. As I said, I'd rather know who's really out there and what they really think, than be spoon-fed some sanitized version as deemed fit by 14 anonymous part-time NYT staffers. You might want to ponder that before slinging personal insults; I've always found that people with little else to contribute are those who immediately leap to ad hominem attacks on those with whom they disagree.

"I disapprove of what you say but I will defend to the death your right to say it."
Third.Coast (Earth)
[[ RealityCheck Earth
I don't use the term, ThirdCoast, but I'm not going to swoon if others do. As I said, I'd rather know who's really out there and what they really think, than be spoon-fed some sanitized version as deemed fit by 14 anonymous part-time NYT staffers. ]]

If you want to know who and what's really out there, just go over to Twitter. I guarantee you it will be as unsanitary an experience as you've ever had.
NorthernVirginia (Falls Church, Va)
I do not subscribe to the New York Times solely for the Comments feature. Having said that, I do enjoy the opportunity to argue points of view in the Comments.

My criticism of the Comments section is mostly the user interface. If there was an interesting comment 400 comments ago, I have to click "Show next 25 comments" sixteen times in order to find it and any replies to it. Perhaps Google will share with the paper their remarkable paging interface that allows a user to click on a page number to see search results (or, in this case, the next x number of comments). I will be happy to assist.
dorinf (<br/>)
My contention with the last of the 5 questions related to the claim that it was biology. It is at least likely to be social depending on the context in which the philanderer and spouse live.
Neal Friedberg
HSW (.)
Would you have approved or disapproved that comment?
Bruce Rozenblit (Kansas City, MO)
There is good and bad to this new system. The more comments that get posted and the greater rate of the posting will cause less time for each comment to be seen. This new system will democratize the flow of comments, but doing will drown all the voices out by piling comments on top of each other. There will be less chance of any really great comments from rising to the top. Unless one posts a highly emotional comment that gets a big rush of recommends, great comments will get four or five and be drowned out by the sheer mass of the other comments.

The slow feed of the moderation process gives each posted comment a chance to be seen.

Withholding comments for hours on end is bad should be minimized. But the rate of postings should be slow enough to allow individual comments a chance to be read by more than 5 people.

I fear that this new process will cause the comment section to lose its effectiveness. It will reduce the political posturing and ranting which is good, but it will do so by sacrificing the really good stuff through democratization. I guess I'm making and elitist argument that some comments have more value than others. But isn't that what the journalistic process is? It cuts out the noise and gives people the meat? The moderation process accomplished that task.

The Times treats comments like content which is why I only comment here. I don't want to be part of the noise. Editorial control must be maintained or the noise will dominate.
Clyde Wynant (Pittsburgh)
While it might well work, thiis is a bit disconcerting. I have always greatly admired the Time's staff who moderates the Comments, as it seems like the last vestige of intelligence on any news site. NPR (which simply gave up on commenting recently) the WaPo and CNN are littered with scurrilous comments and trolls and backbiting, while the Times comments often add to and expand on the articles which are commented upon. I have learned a lot from my fellow commenters....

Please trust me when I say that the Comments are a key component of your online "product" and that if this experiment with algorithms doesn't work out, we will certainly notice it.
JEG (New York, New York)
I've found the New York Times staff to be rather inconsistent in the comments they've permitted to be posted, as well as unduly thin-skinned toward criticism of the paper or one of its reporters.

For instance, when commenting on recent articles about civilian casualties caused by the U.S., Russia, Saudi Arabia, or Turkey, among other countries, I've raised the issue as to why the BDS movement that criticizes Israel, is never moved by civilian deaths caused by other nations. Not once, no matter how mild the statement, has the Times staff approved such a comment. I've found such censorship to be rather disconcerting, particularly since it is not unusual for my comments on other topics to receive NYTimes Picks status. As such, I do hope that machine reading can bring greater uniformity to the moderation of comments.
Paul (Bellerose Terrace)
True enough. It's been months since I've gotten a comment posted on a Tom Friedman column.
Robert (Philadelphia)
An utterly fascinating, fun quiz.

Presumably the "robot software" (really, an algorithm) will have a learning period where it reviews letters already reviewed and classified by Times reviewers (whew! what a sentence). That learning period will presumably inform the algorithm. However, I heartily recommend that you continue to review the software's judgements via some type of blinded test by your reviewers until you are satisfied with the misclassification rates.

Please keep us informed as to your satisfaction with this fascinating language algorithm!!!
Kay (Connecticut)
This was fun. More interactive articles, please! (Like you've got time for that.)
Louis-Alain (Paris)
The NYT is the only paper online I propose comments to (French ones included). The 1,500 signs limit is just excellent in the sense that it allows for strong argumentation without falling (most of the time) in endless verbosity.

The quality of most of the comments must be recommended here, you'd think that most commenters have PhDs in literature, geo-politics, history and most of all possess an amazing ability to present cogent and so well informed arguments.

I also could comment on the Guardian, just I find in this daily next to no trace of reasoning but mostly exchanges of wits, puns and games on words (most of them funny indeed) but I'm interested in debating and supporting ideas and information, not in having fun (?) with a group of jokers.

Thanks to the NYT to offer me the possibility to express my opinions and occasionally correct what I perceive as a lack of information from other.

As you can see, I'm not a native English speaker and my score was 4 (correct moderations) out of 5
John T (NY)
Great. Now even fewer of my comments are going to get through.
Richard Luettgen (New Jersey)
Well, I still think you need to adequately respond to the charge that you attach comment boxes to news stories, news analysis and opinion that you regard as "safe". Could be an inaccurate perception, could even be a massive coincidence, and we'll see what happens by the number of additional pieces that welcome comments as well as the those that do that are highly controversial and that don't necessarily support the party line.

The Wall Street Journal allows comments on EVERY piece it publishes online; but, then, they also don't endorse political candidates by longstanding policy.
Richard Luettgen (New Jersey)
Got 5 for 5, and did it pretty quickly. Commenting in this forum for going on ten years has its advantages. But the examples were rather obvious.
Jonathan Katz (St. Louis)
The WSJ comments appear to be unmoderated, and turn into screaming matches and off-topic diatribes.

If you hire interns at minimum wage to do the moderating, human moderation wouldn't cost much. It looks good on a student resume.
Mike Smith (L.A.)
5 for 5! That is excellent Richard. You are one of the smartest NYT readers.
Debra (Formerly From Nyc)
It was fun to play the game but....

ROBOT MODERATORS?

Who will give Times picks? The robot?

To be honest, this sickens me. Human intelligence is trumped (no pun intended) by artificial intelligence. Even robots fail.
Alison (Squamish, BC)
That was fun!! I want to do more!!
Lex (Los Angeles)
I rather hoped it was a veiled recruitment tool and to see a screen at conclusion: 'Congratulations! You did awesome! Here is your one-way ticket to New York City and your downloadable welcome pack to the comments moderation community desk of the Gray Lady.'

But... alas.
John T. (USA)
Right! Maybe the NYT could post one per day and call it something like THE MODERATOR. A new game for news geeks.
Matt (NJ)
I would have approved all of the comments. Taking the time to think like a censor slowed me down m
Julie W. (New Jersey)
Anything that keeps the Times comments section going is welcome. This is far and away the highest quality comments section that I have read. I noticed that several other news sites have eliminated their comments altogether over the last year or so. I can only assume that this was due to the uncivil behavior of too many commenters. Of course, they were also unmoderated.

Others have already voiced concerns about the "Verified Commenter" designation. I look forward to seeing the promised changes to that system in the near future.
Cruze (Princeton, NJ)
Not a day late! That said, I want to put to record my perspective and experience with The TImes. I have repeatedly challenged, doubted, dissected, and plainly asked--publicly, in Comments, and privately writing to Public Editor [poor Margaret Sullivan probably quit because she couldn't stand me anymore]- The Times to publish its "policy" or metrics for acceptance/rejection of comments. I presented specific cases of my comment being rejected despite it following all the rules of civility, being informative, furthering discussion, promoting or stimulating new ideas. I have preserved copies of several of the "Rejects". I have enough samples in my experience to be able to perceive a pattern of Why some of my comments have been rejected, if not a theory. Since it is just my own experience and rather unscientific method, I cannot extrapolate it globally but I can say with enough confidence that "the community" betrays a lack of information or education in some specific geographic, historical, and cultural areas of the world. Another bet I'm willing to make is that, a hundred years from now, NYT editors and meia analysts will look back and look at the the political correctness of this era as a kind of a burl on a old maple.
RealityCheck (Earth)
I agree, Cruze, and it seemed very arbitrary. Some people like the personal finance writer Lieber and the now-defunct Motherlode blogger seemed extremely personally sensitive about comments and would block many. Other more seasoned writers like Paul Krugman seem to approve them all. An objective standard should be in place and nearly all opinions, however distasteful, should be aired if the words used are not obscene.

The abrupt cut-off of comments is irritating too. I have seen comments closed after fewer than 100 while on other stories, 2000 or 3000 comments are allowed. Make it 1000 for every story, or a three-day time limit, or something uniform.
GXSC (Memphis)
I don't think my comments fall into the rejection criteria presented here, especially after taking the test, but they get rejected quite frequently.
frazeej (<br/>)
Wow, I'm impressed by your moderator's desk! I took too long, and got too many wrong. What a thankless task for those folks!

JimF from Sewell

P.S. I thank you for approving the vast majority of my comments. Those rejected are usually ones penned late at night (figure it out), and fortunately the moderators have more cognizant awareness than I after 9PM!
Eugene (Oregon)
First, a civil comment section is wonderful. Second the abrupt arbitrary shutdown of comments is very frustrating both for those wanting to post and interested readers, conversely allowing a thousand to twenty-five hundred seems like overkill. Third, I often want to comment but the option is not provided. I went to Facebook and it was a Zoo I have no desire to be part of. Forth, Requiring readers provide supporting argument to back up their position in a forum allowing 1500 words is unreasonable and not something the Times enforces. If the comment is cogent and context is obvious let it stand. Five, The email one receives stating a comment has posted is strange as that is often the only way to access one's comment leading one to surmise that they are being played. If I can't find it by searching through every comment how is it the Times can direct me to it? Six, might it be a good idea to state a fixed number limiting comments to 500, or some such number. Seven, might the Times use interns to process comments.

Though I have many a bone to pick with the Times the readers comments section is great and the primary point of my interest every day.
mobocracy (minneapolis)
I'd like to see more articles comment-enabled. I'm often surprised at the number of significant articles (length or issue) that are never comment enabled and the number of trivial articles that are.

I'm sure the workload of the moderation system plays a big factor, but I'll admit my conspiracy feelers go up. The conspiracist in me believes that some articles don't get comments enabled because the paper doesn't want a carefully crafted story and its assumptions publicly questioned by readers.

Again, I doubt there's any active policy like this but it always puzzles me how and why some articles have commenting enabled and some don't.
PE (Seattle, WA)
I hope the robots don't change the vibe. I think I'd like the comment section to stay completely human-vetted with less articles to comment on, than more articles with rapid-fire, bot-approved comments. I imagine the back-n-forth could devolve into rant-like texting type comments, with two people going at it, albeit politely, but maybe not necessarily as thought-out as comments get approved immediately? Still I am grateful to have such a well-written, well-researched paper, with so many insightful commentators attached. But, I say hire more people over unleashing the cold, heartless bots.
Pentelicon (NYC)
This was great! Like many here, I relish the excellent moderation of the NYT comments, and their frequent thought-provoking eloquence. I usually spend about 3x as long reading the comments as I do the article!

That said, I am skeptical of using algorithms where humans are doing an excellent and nuanced job. Many articles in this paper have warned against inherent bias, or just plain flat-footed-ness, in using algorithms to moderate humans (in resume scanning, for example). Why not heed that warning?

And honestly, I am a momentarily under-employed avid NYT subscriber and would *love* a crack at joining the moderating team, if you're hiring more human beings!
Voiceofamerica (United States)
Some have requested the ability to edit their comments. This would indeed be a helpful function, as far as it goes. Even better would be the ability to edit other people's comments. Indeed, if I could edit some of the actual stories, that would significantly improve my NY Times experience ;)
Jonathan Katz (St. Louis)
Agreed; you write a comment, post it, and suddenly see a horrible spelling mistake, too late!
A. Stanton (Dallas, TX)
In a sane and just world, all comments published in the Times that included a picture of my good dog (see above) would immediately be awarded a Times Pic. Alas and alack, we are not living in such a world. More's the pity.
ColtSinclair (Montgomery, Al)
I concur
Jack Wells (Orlando)
Sounds as though the Times is trying to save money by utilizing technology for this component of the website. I think in this case there is the potential for comments to be tossed over a) poor spelling; b) lousy software, etc. Artificial intelligence is way overrated.
LB (NY)
Yes, inevitably something will be lost here. And why does the Times have to let a corporate behemoth like Google get its fingers into it, anyway? I'm wondering if Google Jigsaw is purely an application/tool or whether it can be useful to Google somehow in logging comment and commenter data. Call me paranoid but the last thing we need is one more place where our opinions can be conveniently cached. I trust that the Times has no interest in (accepted or rejected) comments as data. Google, maybe not so much. Even given the realities of maintaining a solvent and profitable newspaper in these times, I always think of our beloved, ornery NYT as above the fray somehow. Not thrilled with the potential ramifications of this happy "partnering."
RealityCheck (Earth)
I'd rather pay $5 more each month for a subscription that is curated by real people (though preferably more seasoned and less trigger-happy on the censorship button) than have artificial intelligence selecting what i can and cannot read.
colonelpanic (Michigan)
I'd never make it as a moderator. The examples sparked too many interesting conversations in my head for me to be effective.
Cruze (Princeton, NJ)
ADD. to my previous comment:
All that said, I do commend NYT for moderating and I believe that it must continue to do. I'd rather wear someone else's eye glasses than be waterboarded with misinformation, lies, slurs, foul language, rage, hatred, etc. As in the WashPost and WSJ.
frazeej (<br/>)
Amen!!!!!!!!

JimF from Sewell
M (Nyc)
Lol. My score 3/5 was capped with "Moderating the 11,000 comments posted to nytimes.com each day would take you 50.1 hours." But you front his whole article with "Comments on Times stories are moderated by a team of 14 people ... Together, they review around 11,000 comments each day ...". So presumably you need to divide my score by 14, thus I could do my share in 3.58 if the other 13 kept up with my speed.
Paul (Bellerose Terrace)
A five day work week means 70 moderation work days across the desk, divided by seven days, that means, vacation and sick time excepted, an average of TEN moderators work each day, meaning, just like I scored, five hours labor, or well within parameters of a normal work day.
Putting the burden on a reader to believe that they might be personally responsible for moderating every comment is a thoroughly dishonest construct, and doesn't help engender good feelings toward someone with that level of misdirection.
Ken (Pittsburgh)
It's difficult to see why the NYTimes feels that it has to protect people from insults ... at least those not cleverly phrased. And the moderation should be done on the entire post, not simply something that might be considered insulting. Providing fourteen arguments for why an article or post is manifestly wrong and then saying that concluding that the author of the article or post must be dumb, ignorant, or both, is quite different than a post merely stating the conclusion.
J (NYC)
Since almost none of my comments on Maureen Dowd's columns get approved - comments, by the way, that NEVER use obscenities or any insulting or personal remarks, just simply point out her bizarre antipathy towards the Clintons is tiresome - maybe this will improve things.

I actually have a good track record with my NYT comments, and the overwhelming majority on news articles and even other op-ed columns get approved. But with that column they seem to run into a brick wall. Apparently the editors who oversee Dowd's comments are particularly timid.
gw (usa)
Maybe it's because they're overrun with comments from readers complaining about Maureen's "antipathy" towards the Clintons. You can run through the comment section and hit Recommend a few dozen times, but in my opinion, read one of those comments, you've read them all. Same with the snarky criticisms of David Brooks. I'm as liberal as can be, but I enjoy reading Brooks and am offended by the same cadre of haters every week spewing their personal insults and ridicule. If name-calling isn't allowed, why are columnists fair game?
Rick Gage (mt dora)
Since I seem to have your attention may I ask what I need to do, further, to become a verified commenter. I have been quite obsessed with this section since I started reading the paper on line 5 years ago. I have received over a hundred Times picks, am frequently in the top 10 reader picks (personal best 2800) and have been on my best literary behavior for the entire time. It's not that I envy the other, less frequent contributors who have achieved this high ranking , I just don't like having to wait 4 to 18 hours after posting something for the Times to catch up. By that time, a comment that was made when there were zero submissions becomes a submission impossible to find in a sea of 800 or so. I don't want to sound like I don't appreciate the forum or the opportunity to vent, but after 5 years of trying to be heard from the back of the bus, I'm wondering whether it is time to disembark.
David Underwood (Citrus Heights)
I got my verified commentor status when it first started. I tried to stick to factual data, and comments that criticized or supported a column.

Occasionally I have resorted to a personal criticism to certain individuals that have made personal comments themselves. But I do have to be careful or I could lose that status. I recently referred to one poster as a member of the GOP troglodytes. Yet I can refer to the GOP as being from the neolithic.
HSW (.)
"I recently referred to one poster as a member of the GOP troglodytes."

A personal attack by a Verified Commenter can be flagged.
D Biester (NY)
You are way too sensitive and reject way too many comments.
Lex (Los Angeles)
5 for 5, but it would take me 54.7 hours to moderate the daily NYT influx. Whoa!

Wouldn't "community moderation" be a better option? Publish all comments, then let other readers do the work for you by flagging them as offensive. Five strikes and you're out kind of deal. Surely easier? Or is the weakness of that idea that minority voices could effectively be shot down by populism?
mobocracy (minneapolis)
The risk of echo-chamberism is exactly the problem with community moderation. I'm often mildly surprised with the diversity of comments on many NY Times stories, but my overall impression is that there's a left slant to the plurality if not outright majority of comments, and I think under community moderation it would get much worse.

My own perception from social media is that there's a shrinking center and quite a few people I'd normally judge as reasonable are increasingly not just intolerant of anything that anything that doesn't align with their choices, they're outright hostile. It's not a difference of opinion, but prima facie evidence of the wicked nature of their character and they should not be debated, only suppressed and marginalized.
NorthernVirginia (Falls Church, Va)
The internet community has demonstrated that it cannot handle power responsibly. That ship (Boaty McBoatface) has already sailed.
BHVBum (Virginia)
Often I'd like to know how many people "don't like" or a thumbs down. That can be as telling as a thumbs up!
Rick (Vermont)
I agree. At least a thumbs down (but even facebook doesn't have that).
RealityCheck (Earth)
yes, excellent idea.
AG (Montreal, Canada)
I made the same judgements as the NYT moderators in 69 seconds, but that included reading the explanation for the NYT decisions, so the actual decision took a lot less than that.

So I guess I approve of the NYT's criteria, I just doubt that a robot can do as well.
Bassey Etim
Nice work! But not to worry, we did not include the explanation-reading time in the final result.
E. (NYC)
Now I understand why the NYT is only one of a few that moderate comments. Obviously, the results are positive and it elevates the level of discussion. But it's so much work! And requires a big investment. No surprise that this is not within reach for a small local newspaper.
Simon Sez (Maryland)
I have long wondered why many of my comments were approved and others rejected.

I noticed that most of my pro-Israel comments were rejected. If my comments on Israel were neutral they were always accepted. I saw hundreds of anti-Israel comments which you approved.
Ken (Pittsburgh)
Few of my anti-Israeli policy posts have been escaped the censor.
Louis Anthes (Long Beach, CA)
Alas, we're asked for input!

I would say that the NYT has to retain the right to refuse. That said, how it goes about exercising that right could follow two elementary rules.

First, any language that is patently and egregiously offensive, unless it is the subject of the article under discussion, is fair grounds for non-publication.

Second, a comment containing threats of harm directed at other specific commentators should earn a rejection of a comment.

HOWEVER, speech with offensive language directed at a public figure may be protected. Moreover, speech which others might deem containing offensive language, but targets specific groups or identifies, may be protected. The New York Times is NOT high school, and the editors should treat bloggers accordingly.

Finally, the New York Times can always decide when too much commentary is too much commentary, by closing off all comments.
Johnchas (Michigan)
While I would like a more efficient & timely comment section in more of the articles I'm not sure using an algorithm is going to produce any better result then the one now in use. For instance the "Times Picks" are often not the best comments or the most representative. Also it is way past time that the privileged ones from the past lose their unrestricted pass through, you know the ones with the green check mark. Some of those comments & commenters are the most egregious violations but are rarely held to the same standard as the rest of us.
1515732 (Wales,wi)
Why moderate at all. The WSJ does a good job with out sanctimonious editing of peoples comments and feelings on a particular subject. The Journal doesn't believe in "safe" spaces for readers.
Patricia Shaffer (Maryland)
Maybe you should check out the Washington Post comments - crude, rude, uninformative, and the reason I gave up that paper.
Ilya Shlyakhter (Cambridge, MA)
The WSJ does _not_ do a good job; there are some thoughtful comments, but the fraction of shallow ones is quite high. I dislike aspects of the NYT system, but it is far better than the WSJ one.
Shimon Mor (Sedro Woolley, WA)
The NYT maintains the best "comments" section online. Period. It's not perfect but it weeds out the trolls and hate mongers and allows the cream to rise to the top. I don't mind the time delay as long as the useless comments are filtered out. Keep up the great work. My only request would be increasing the number of articles which allow comments. I'm a bit skeptical about the robots taking over...but time will tell.
Nightwood (MI)
Oh dear, i don't want robots taking over. I want the approval to mean something! It would be like God slit his throat. lol. Keep to your standards Gray Lady.
Voiceofamerica (United States)
I took the test on what I would have approved (all of the comments) not what I would guess the NYT would approve.

It's very telling that NONE of the comments included even touched on the most contentious topic in the Times' forums, namely Israel's ongoing criminality. This is simply a no-go zone in the NY Times, which renders its claims as an objective news source rather absurd, at least regarding this particular and vitally important topic.

(Other coverage in the Times is superb. Credit where credit is due. )
A. Stanton (Dallas, TX)
If a comment is funny, whimsical, odd or quirky it ought to be published -- period. The Times is already way too full of everything that is grim. Stupid comments by Trump supporters -- which is to say 9 out of 10 comments by Trump supporters -- ought to be banned outright.
A. Stanton (Dallas, TX)
My apologizes Voice for mistakenly replying to you earlier, instead of to the article as a whole.

But as long as I’m here, let me hasten to add that I agree with you wholeheartedly on the need for better coverage
by the Times of Israel’s “ongoing criminality,” i.e, its failure to act much more aggressively, punitively and decisively against the fanatical enemies that surround it, which continue to threaten and frequently take Israeli lives.
JM (Los Angeles)
Maybe they publish so many of your comments (not that they aren't enjoyable) because your dog has such a sweet, smiling face.
Lynn in DC (um, DC)
I enjoy reading all sides of an issue so I hope moderation does not mean for example, that all pro-Trump comments will be rejected because they don't meet some type of subjective standard.

I may not always agree with other comments but I think the Times community provides informed, civil and on-topic commentary. Unlike, say The Washington Post, the comments here are refreshingly free of insults and name-calling directed at other commenters.

What is the policy for opening articles for comment and what is the timing for opening said articles?
Bassey Etim
Lynn, this article should just about answer your question, and comments are typically open for 24 hours for news stories. It's often less for Opinion:

http://www.niemanlab.org/2013/03/how-does-the-new-york-times-determine-which-articles-have-comments/
JenD (NJ)
Will the "trusted" white-checkmark-in-a-green-box remain? That has long infuriated me. If I recall correctly, it was originally a way to "encourage" commenters to use a Facebook account. Then criteria for bestowing it became more mysterious. And irritating, as I would see the checkmark comments appear on stories hours (sometimes days, on the weekend) before the comments of the rest of us great unwashed would appear.

Bring the robots on, and let's see what happens. Hopefully, we will see less comments by people touting their own blog, website, book or product; these comments seem to slip through more often than I would have thought.
Tom (Midwest)
The answer to your questions is here http://www.nytimes.com/content/help/site/usercontent/verified/verified-c..., it has nothing to do with facebook (thankfully) and additional criteria are available if you ask NYT. I did so when they first asked me way back when.
Sharon5101 (Rockaway Beach Ny)
Thanks. I'm glad I'm not the only one who feels the verified commenter status must be eliminated. It's completely unfair and it has to go.
DaveD (Wisconsin)
If the verified commenter is retained, and it shouldn't be, then it ought to sunset after a year. Death itself appears more fair than a permanent green light on anything.
cgg (NY)
Wow! That was really enlightening. That quiz was HARD! I just want to say that I so appreciate the excellent work the Times does to keep the comment section interesting, educational, and civil - and easy to participate in.

The comments add so much to articles. Sometimes I'm reading an article about which I know nothing, or I know nothing about the author, and I'm agreeing with it sort of blindly, and then I read the comments and your readers have so many important things to add that it will completely change my perspective. Thanks again.
Itzajob (New York, NY)
I find the "trusted commenter" designation incomprehensible.

Many green-checked commenters express bigoted views, yet their comments are still green-lighted without apparent review. Moreover, since these comments appear hours before any others, they collect the most "recommendations" and thus become "Reader's Picks" over and over again, creating something of an echo chamber for their particular mindset.

Meanwhile, the rest of us have to wait for hours before our comments are posted. This makes any sort of dialogue utterly impossible.
Bassey Etim
Thanks for writing -- it is something we hear quite a bit. We'll make some changes to the verified program once the new comments system is up and running.
Tom (Midwest)
I recall when verified commenter became possible, I was told my previous comments and comment stream were reviewed before I was ever invited to participate and one of the privileges was instant publication rather than waiting. Whether the NYT has completed a subsequent follow up on verified commenters to ensure the commenter has maintained the standard, I am not sure but I would hope so and welcome them to do so. I recall before I became verified, unless it was a very hot topic, I never waited more than a couple of hours for a comment to be published.
Tom (Vancouver Island, BC)
In general, I'd like to see some kind of weighted correction for the Reader's Picks to correct for the bias in favor of those who post earlier. Especially in the longest threads, many insightful and informative posts get buried only due to the fact that they were posted later in favor of mediocre posts getting more recommendations merely because they were among the first few dozen.
Judy Blue (Fort Collins)
You calculate that it would have taken me 86 hours to mod the daily comments. But you have 14 people. It would have taken 14 of me only six hours each. Not so hard.

Is Google Jigsaw going to attach all the info in the comments to the readers' screen names and add that to its database? Asking for a friend.
Bassey Etim (null)
That's something we can't completely give readers an idea of in this quiz. Typical comments will be much longer than the ones shown here, but we don't want to bore you in this interactive. Also, the flow of comment volume always changes over time, and each moderator has administrative duties to fulfill as well. So moderation staffing needs to be available throughout the day, and moderators in reality do more for the newsroom than decide whether to approve individual comments.

Another quick note on that -- not every moderator works full time, and none work 7 days per week.

On your second question, I'm happy to say the answer is no! [CORRECTED FOR TYPO 9/21:] No emails or personally identifiable information is passed to Google beyond what is already available in publicly posted NYT comments.
Giovanni Ciriani (West Hartford, CT)
How about making available to all the app that is (or was) available on Numberplay?
Atikin (North Carolina Yankee)
Why do some comments get accepted before there is even a comments section box listed?? I look for and see none, but an hour later it magically appears with 100 comments already in it. Huh?? Also, it takes so long to post that I have already finished the rest of the paper, moved on, and never get to see many others besides the usual Gemli, green Buddha guy, Christine Morrow, and a few others. I already know what they think, could write it myself. Can't others voices show up early, too ??
Mary Kay Klassen (Mountain Lake, Minnesota)
I believe that other than vulgar language, the NY Times should not censor. Individuals have opinions, that no matter how different, even horrendous, on all sides of an issue, it is political correctness and not the way our country stands for free speech. The only reason I believe vulgarity should be left out is that an effort by those that can't stop using it in all of their conversations is that it depresses a readers interest in what one has to say, and in person is usually a sign of uncontrollable anger or the use of substances. I had an instance where a neighbor had some people over, and I went outside and the vulgarity in every sentence for 3 hours left me unable to use my backyard, and we live on the edge of town with no neighbors to the back. Yes, alcohol was involved.
Johnchas (Michigan)
So as long as it's not "vulgar" your okay with anything goes in homage to the ideal of rejecting "political correctness". So how does one determine whats vulgar without straying into the dreaded zone of being correct? After all vulgarity is often in the eye of the beholder, listener or in this case reader. Are racist comments okay if they meet this subjective standard? What about personal attacks on other commenters? Then there's incitement of violence against others based on any number of reasons, is this okay as long as it's done graciously? That's the point in having standards in comment sections like the NYT, so it doesn't deteriorate into a free for all, there's enough of that around the internet already, but be careful though these sites don't have a problem with vulgarity ether.
carole (Atlanta, GA)
Mary Kay: "...other than vulgar language, the New York Times should not censor."

The country may stand for free speech, but the NYT isn't the country and doesn't have to put up with, or publish, racist comments that demean human beings just because the commenter is racist. Racist comments can't inform and don't explain issues; they can only offend, and they do so for no good reason. I'm glad the NYT censors that type of comment. By the way, I got 5/5 correct in 83 seconds or 50.7 hours of work. Whoa!
Mary Kay Klassen (Mountain Lake, Minnesota)
If that is the case, then why do they publish anything on Donald Trump or cover any of his media events. It is pretty hypocritical of them to call out racism and still cover all his events. You can't have it both ways, or can you? There have been more opinion pieces written about him in the United States in the last year than anyone else, and why because most of the Times readers and writers are not voting for him.
Bradley (New York)
I want to be able to easily find my comment in a sea of comments. I want to see who has responded to my comment, and I want to view how many "thumbs up" my comment has received. This functionality should be included in any changes to the comment section. Right now, it can take 20-30 minutes to find my comment among the sea of other comments.
Tom (Midwest)
If you click the box that says email me when my comment is published, you have your solution.
Johnchas (Michigan)
Agreed, also not all digital platforms handle comments as well as using a laptop / desktop computer. I find the NYT app on Android is the most dysfunctional of them all & I use all three systems, Apples IOS, Goggle Android and a Microsoft based laptop. If the Times is serious about digital then get the Android app up to speed please.
Loomy (Australia)
You should receive an email saying your comment has been posted and to click on the URL supplied which will display the article as well as your comment and any responses to it as well as the number of "Thumbs up" at the time.

It can't get any easier than that.
John Quinn (Detroit)
For a long time I have half seriously suspected that the chief reason serious journalistic sites allow reader comments is that they want us to realize what a wonderful service journalism schools and editors provide. The comments give us a sample of the drivel we don't have to read because journalists are well trained and editors reject their less deserving efforts, thus saving us the time needed to sort the wheat from the chaff ourselves. Admittedly, the Times is less effective at this because the moderators do some comment editing for us. Still, nothing shows the value of professional journalism better than the efforts of us amateurs.
PeterK (New York)
I love the Comments feature and greatly appreciate the effort to make it more efficient. Maintaining civility is crucial. Thanks much to the "community desk" staff and the Herculean job they manage with great success. I sincerely hope the new technology doesn't replace them!
Patricia Shaffer (Maryland)
My thoughts, too. Will the new technology result in layoffs of employees or contractors, NYT?
gw (usa)
My thoughts as well. I'd like to feel assured that none of the moderator positions are being eliminated. I've often thought they must have the forbearance of saints.
JS (Cambridge)
THANK YOU NYT for putting the time and resources into having one of the few, and definitely the best, forums for civil discourse out there. I generally read articles with comments first, so you are increasing readership as well. I have learned tremendously from reading NYT comments; rarely is the whole picture reflected in an article. Comments round out the data, and help me see other perspectives as well. Someday I hope to be a "trusted commenter." Until then I will happily make my way in the NYT comments marketplace along with your thousands of fellow readers! Thanks again.
dEs JoHnson (Forest Hills)
Disagreement is the lifeblood of democracy. Outright insults are banned, but the NYT allows a lot of snarky comment that isn't backed by argument or reference.
Lifelong Reader (New York)
There's nothing wrong with the occasional snarky remark if it's funny and reveals truth. No one wants to read hundreds of comments that read like bland, mini ersatz editorials.
Tom (Vancouver Island, BC)
"Repugnicans" is a violation of moderation policy? Since when? I see this kind of thing in the comments all the time.
Patricia Shaffer (Maryland)
Yes, some of the regular trusted commentators use them consistently. I wonder if this will change?
Voiceofamerica (United States)
That's because the repugnant nature of Republicans is an established fact, backed by an avalanche of documentary evidence.
Tom (Vancouver Island, BC)
@ Voiceofamerica, just for the record I don't disagree with your statement, and I don't believe such terms should be cause for being censored. It was just surprising to me when taking the quiz that it was considered a violation, as I see so much of it here.
Thomas Green (Texas)
Data is not nuance. But I'm glad more automation will probably approve more of my comments.
Nightwood (MI)
I took time out to do other things that came up when taking the test. Hope that didn't hurt me....llike forever. lol
JM (Los Angeles)
Or, maybe you just did it slowly? No harm; it was fun.
dEs JoHnson (Forest Hills)
Five minutes ago on an article about the father of Ahmad KR accusing his son of being a terrorist:

"top chef bc 5 minutes ago
the lobotomized public is so naive and ignorant, i laugh at the lemmings being programed by the filthy NY times."
Tom (Midwest)
As a trusted commenter, vetted by NYT based on what I understand was a review of my previous comments, I commend NYT to take steps with regard to comments and still keep comments alive. The issue is not whether the comment violates a predetermined criteria but the lack of civility we see all too often these days. In other comment sections and blogs the moderating for civil and polite discussion has nothing to do with being politically correct. All too often, those who have their boorish and uncivil comments deleted complain about political correctness when it is not. The NYT has been more than accommodating with impolite comments.
Voiceofamerica (United States)
"The NYT has been more than accommodating with impolite comments."

I agree. The problem is that the NY Times is NOT accommodating when politely articulated comments fall outside the boundaries of accepted discourse, particularly comments critical of the hospital bombers of Israel.
JM (Los Angeles)
Even if a remark is politely articulated, if it reflects anti-Semitism, it should be rejected, in my view.
Voiceofamerica (United States)
JM--I quite agree. Anti-Semitism has no place on NY Times forums (or anywhere else). Of course, the fiercest condemnation of Israel's abhorrent behavior has come from my fellow Jews, going all the way back to the famous letter to the NY Times, signed by Albert Einstein and 30 other prominent Jews.

Today, I'd say the chances of that letter being published by the NY Times are about zero. (You can readily find the full text online. )
Kim Susan Foster (Charlotte, North Carolina)
It sounds to me like they are replacing some of the "community desk" jobs with Silicon Workers. I hope not, indeed, hire some more Literature University Students!!

PS. I suggest that there be a way to edit comments. I usually read over my text before pressing "submit", but then, there is the there but should have been their "consciousness".... maybe also a "gosh"!!
Norman (NYC)
There should be a button called "Disagree but approve."
Kim Susan Foster (Charlotte, North Carolina)
Maybe "disagree but I know I have to approve to avoid that feeling of Psych Test Pavlovian wrong".
JM (Los Angeles)
How about an "Absolutely obnoxious" button?