This is almost certainly the short story "The Fittest" by J. T. McIntosh (a pseudonym for James Macgregor).

It matches your description in almost every single detail, with one significant inversion that is a very common type of memory transposition.

Let's break down the plot points:

• Protagonist wakes from suspended animation: The story's protagonist, an army sergeant named Dave, is put into experimental "cold sleep" and wakes up about 17 years later.

• Facility is abandoned/looted: He awakens in the underground military bunker where the experiment took place. It is empty, and he has to force his way out. He gathers supplies (canned food, a rifle, etc.) before leaving.

• World is a war zone: He emerges to a desolate, ruined landscape. Cities are destroyed, and nature is reclaiming the land.

• Finds a young kid, terrified and trapped: This is a key scene. He finds a young boy (Timmy) who is pinned by a fallen beam in a ruined building. The boy is terrified of him.

• Helps the kid and gains trust: Dave carefully frees the boy, gives him some of his food, and earns his trust.

• Meets the family: The boy leads him to the cellar where his mother (Jean) and baby sister are hiding. The mother is also terrified of him at first, until Timmy explains that Dave saved him.

• The mother explains what happened: Jean, the mother, is old enough to remember the world before "The Breakup." She tells Dave that there was no foreign war. Society collapsed from within due to riots, chaos, and starvation.

• The enemy approaches: The family lives in constant fear of roving gangs they call "packs." They hear a pack approaching.

• The Twist/Reveal: This is where your memory is almost perfect, just inverted. Dave, a career military man, prepares to defend the family. He asks Jean what the "pack" members look like. She is hesitant, but finally tells him:

  "They're black. All of them."

In the story, the protagonist is white, and the reveal is that the roving gangs that destroyed society are all black. Your memory has swapped the roles, remembering the protagonist as being black and the mother's line as "They look just like you." The core concept of the twist—that the societal collapse was a race war and the enemy is defined by their race—is identical.

This story was originally published in The Magazine of Fantasy & Science Fiction in 1955 but was very widely anthologized in paperbacks throughout the 60s and 70s, which is almost certainly where you would have read it. It appeared in collections like The Best from F&SF, Fifth Series.