When a university decides to bring a controversial speaker to campus, the students have a right to express their disagreement with the decision. I think that everyone can agree with this. Where the agreement breaks down is (1) whether students should do this and (2) when we recognize that there are many different ways to express disagreement and that some of them involve silencing tactics (i.e., deliberate attempts to prevent the speaker from expressing himself or herself).
Students who are outraged at the prospect of a speaker they dislike coming to campus can express themselves. They can let the university know they do not agree with the choice of speaker. They can boycott the speech or gather outside the venue to show that there is opposition on campus to the speaker's views. Granted, I don't generally think that students in such a situation should do these things because they are depriving themselves of the opportunity to encounter diverse viewpoints, which is often conducive to growth. But I am more concerned with attempts to silence or disrupt the speaker or persuade the university to rescind the invitation. Just because some students object to the speaker does not mean nobody else should have the opportunity to hear the speaker.
For me, the key is that my exercise of my right to free expression should stop short of attempting to interfere with your rights. I can make my opposition to your views known, but I should not attempt to stop you from speaking or disrupt your attempt to do so.
Organized boycotts of a speech are fine; attempts to persuade the university to rescind the invitation are problematic. Let the speech take place and then expose it for what it is. Use it as an opportunity to teach people where the views expressed fall short. Holding a protest outside the venue is fine; interrupting the speech with bullhorns inside the venue is unacceptable. Again, I can express myself, but I should not try to stop you from being able to do the same.
Suppose an atheist/skeptic/humanist conference invited a speaker who was a racist, young earth creationist, and Holocaust denier all-in-one. Would I want her at the conference? Of course not. And I might take the time to explain why. Or I might attend the speech and take copious notes so I could provide an effective rebuttal at a later date. I might instead chose to stand outside the venue holding a sign that provided a different viewpoint. What I would not do is attempt to stop the speaker from speaking or interrupt her speech.
But doesn't that mean I am supporting her views? Nope. It means nothing of the sort. It means that no matter how vile I find her views, I value her right to express them. It means that I recognize that the only way free expression has any meaning is when it protects speech with which I strongly disagree.
Now I hope you will understand why I did not sign the petition asking the Bay Area Science Festival to remove Rebecca Watson from the list of speakers following her participation in the doxing of a woman on Twitter. I disagree strongly with Ms. Watson's behavior, but I'm not going to advocate the sort of silencing tactics I've been condemning here when they are committed by her group. It is not okay when we do it either.