In a society made up of only a few types of people, your hero isn't one of them. An Indian in a Western town, an alien in a sci-fi game of human marines, or a half-orc in a party of elves, dwarves, and humans are all examples of outsiders. Locals are likely to raise prices on the Outsider, ignore pleas for help, and generally treat him as if he's of a lower class than the rest of their society.

In addition to the roleplaying effects above, your hero's Charisma suffers a -2 modifier among all but his own people.

Savage Worlds Deluxe Edition p. 32