Ray J was seated near Whitney Houston‘s family at last night’s 2012 Billboard Music Awards, but Whitney’s loved ones weren’t happy about it and almost had him physically removed before the segment in which Houston posthumously received the Millennium Award.

TMZ reports that Pat Houston, Whitney’s manager and sister-in-law, believed Ray J was a bad influence on Whitney and was upset to see him at the awards ceremony. She asked security to remove him, but he refused. Pat recruited several police officers but was told that physically moving Ray J would cause a scene that would certainly be caught on tape. Not wanting to ruin Whitney’s moment, Pat backed down and let Ray J keep his seat.

When they took the stage to accept the Millennium Award, Pat and Whitney’s daughter Bobbi Kristina gave no indication there had been any drama. Both thanked the singer’s fans, and Bobbi Kristina added, “I’m just blessed to have been in such an incredible woman’s life and there will never be another one, ever. And I thank you, all of you, fans, everyone, thank you so much.”

TMZ says that Ray J, who was romantically involved with Whitney, complained to friends, “Pat ruined what was a great opportunity to honor the woman who I loved.”

Watch Whitney Houston’s Tribute at the 2012 Billboard Music Awards