Civil rights advocates, students and school officials in Southern California said Wednesday reports that the Justice Department could be taking legal action against universities over affirmative action is alarming.