Ngozi Fulani was questioned about her background by Lady Susan Hussey, Prince William's godmother, at Buckingham Palace on Tuesday. The late Queen's lady-in-waiting has since resigned.
Ms Fulani likened the conversation with Lady Hussey, 83, to "an interrogation".
The palace described the remarks as "unacceptable and deeply regrettable".
And a spokesperson for Prince William said "racism has no place in our society". The incident has overshadowed the Prince and Princess of Wales's visit to the US to hand out awards for his Earthshot Prize.
Ms Fulani was a guest at the reception representing domestic violence charity Sistah Space when she described Lady Hussey moving her hair to see her name badge, and then challenging her repeatedly to explain where she was from.
Speaking to BBC Radio 4's Today programme, Ms Fulani said: "It was like an interrogation. I guess the only way I can explain it, she's determined: 'Where are you from? Where are your people from?'"
Ms Fulani also rebuffed suggestions that Lady Hussey's remarks had anything to do with her age.
"Let us be clear what this is. I've heard so many suggestions it's about her age and stuff like that, and I think that's kind of a disrespect - an ageism kind of thing.
"I have to really question how this can happen in a space that's supposed to protect women against all kinds of violence.
"Although it's not physical violence - it is an abuse."
Ms Fulani continued: "If you invite people to an event, against domestic abuse, and there are people there from different demographics, I don't see the relevance of whether I'm British or not British.
"I'm very proud of my African heritage. This is like the Windrush thing to me. You're trying to make me unwelcome in my own space."
She added she felt like she was being asked to "denounce my British citizenship".
Rishi Sunak said the UK must "continually learn the lessons" from acts of racism. The prime minister said it "wouldn't be right" for him to comment on the incident at the palace, adding "they've acknowledged what's happened and made an apology for it".
Asked about his own experiences, Mr Sunak said that while he had experienced racism, the country had made "incredible progress".
"But the job is never done," he added, "and that's why whenever we see it we must confront it and it's right that we continually learn the lessons and move to a better future."
In its statement on Wednesday, Buckingham Palace said: "We take this incident extremely seriously and have investigated immediately to establish the full details.
"In this instance, unacceptable and deeply regrettable comments have been made. We have reached out to Ngozi Fulani on this matter, and are inviting her to discuss all elements of her experience in person if she wishes.
"In the meantime, the individual concerned would like to express her profound apologies for the hurt caused and has stepped aside from her honorary role with immediate effect.
"All members of the household are being reminded of the diversity and inclusivity policies which they are required to uphold at all times."
A spokesperson for Prince William said "racism has no place in our society", adding: "The comments were unacceptable, and it is right that the individual has stepped aside with immediate effect".
Ms Fulani later told BBC News: "Now, violence isn't always physical, it can be verbal.
"And that felt like violence to me, and when you put your hand in my hair like I'm not even a person - you can just do what you want and say what you want - I don't want to be in your presence."
In an interview on ITV's Good Morning Britain, Ms Fulani also said Buckingham Palace had not contacted her about the incident, insisting she would accept an invitation to discuss it with them.
"See, what we're about is positive results, so absolutely, I think a discussion should be held," she said.
Ms Fulani, who was representing her charity which supports women of African and Caribbean heritage across the UK who have faced domestic and sexual abuse, was one of 300 guests at the event where the Queen Consort, Camilla, had warned of a "global pandemic of violence against women".
BBC News has approached Lady Hussey for comment through Buckingham Palace.
Lady SH: Where are you from?
Me: Sistah Space.
SH: No, where do you come from?
Me: We're based in Hackney.
SH: No, what part of Africa are you from?
Me: I don't know, they didn't leave any records.
SH: Well, you must know where you're from, I spent time in France. Where are you from?
Me: Here, the UK.
SH: No, but what nationality are you?
Me: I am born here and am British.
SH: No, but where do you really come from, where do your people come from?
Me: 'My people', lady, what is this?
SH: Oh I can see I am going to have a challenge getting you to say where you're from. When did you first come here?
Me: Lady! I am a British national, my parents came here in the 50s when...
SH: Oh, I knew we'd get there in the end, you're Caribbean!
Me: No lady, I am of African heritage, Caribbean descent and British nationality.
SH: Oh so you're from...
Watch: Rishi Sunak reflects on his own experiences of racism amid royal race row