What does Baldwin mean by "black people, mainly, look down or look up but do not look at each other, not at you, and white people, mainly, look away"?

Baldwin is addressing the way in which race has created a hierarchy whose strictures are obeyed by both whites and blacks. 

Black people "look up" at those within the race who are of a higher economic class and/or whose skin is of a lighter hue. They "look down" on those who are poorer or darker. Black men might be inclined to "look down" on black women; heterosexuals may also "look down" on homosexuals.

Racism, sexism, homophobia, and classism, which often coincide, have dissuaded efforts at black unity and even the basic acknowledgement that another black person's life, however degraded or different, is still important and invaluable. If members of the community were able "to look at each other," as fellow individuals, this understanding would be obvious. The "not at you" is a reference to Baldwin's nephew, to whom the letter in The Fire Next Time is addressed.

Whites, he argues, "look away," unable to see the presence of black people at all. To look away from someone is to refuse to acknowledge their existence. In his impassioned argument, Baldwin spares no white American—neither conservatives nor liberals—from his accusation that they do not think that black lives matter as much as their own; he also says that their concerns over rioting and unrest at the time were not in response to conditions in ghettos, but instead a reaction to possible intrusion onto their property. When one's first response, in face of another's pain, is to talk about oneself, there is not only no concern for the other, there is no sense that he is there at all.