Tens of thousands of people died at Sachsenhausen concentration camp
Nathan Gasch said when Martin Hartmann moved into the retirement complex four years ago he noticed a picture of his neighbour in an SS uniform on the wall.
Mr Gasch says he was shocked but that he never reported the incident.
However, investigators from the justice department were on Mr Hartmann's trail and this week the US expelled him.
Romanian-born Mr Hartmann, 88, was forced to move back to Germany last month and stripped of his US citizenship.
The truth about his past was uncovered in a two-year investigation by the Department of Justice's Office of Special Investigations (OSI), a department created in 1979 to pursue war criminals.
'Must have known'
During World War II Mr Hartmann worked as a member of the SS Death's Head Guard Battalion at Germany's Sachsenhausen concentration camp.
Tens of thousands of prisoners died at the camp where inmates were used for forced labour and subjected to medical experiments and torture.
Mr Hartmann concealed his past when he moved to the US in 1955 and later applied for US citizenship in 1961. Even now his wife Ellen claims that he could not have known the true nature of the camp at Sachsenhausen.
However, OSI officials say it is impossible that he did not know.
"Martin Hartmann and other members of the SS Death's Head Guard Battalion were indispensable accomplices in the brutal crimes committed in the Nazi concentration camp system," Eli Rosenbaum, head of the OSI, was quoted by Israeli daily newspaper Haaretz as saying.
OSI officials told the Arizona Republic newspaper that Mr Hartmann volunteered for the role in 1943, rather than being sent into combat, and that at no time did he apply for a transfer as camp guards were allowed to do.
Last month, Mr Hartmann admitted to having served as an armed SS guard and having personally assisted in Nazi persecution, and agreed to leave the US and never return.
His sudden departure has stunned fellow residents at the Leisure World retirement complex in Mesa, Arizona, none more so than Mr Gasch who even spent time as a prisoner in the Sachsenhausen camp - though not at the same time Mr Hartmann was there.
Speaking of the incident with the picture Mr Gasch, who comes from Poland, told the Associated Press news agency that at the time he simply "walked out of the room".
"Maybe I was too childish," he said. "I figured we were living in a community here. I just let it go."
Mr Gasch still bears the tattooed number on his left arm which marked him out as a concentration camp prisoner six decades before.
"They must have seen I had my number," Mr Gasch said, adding that though he was shocked about Mr Hartmann being forced to leave he is not sorry.
"He was one of them," he said.