Papa John's founder John Schnatter has resigned as chairman of the board.
The company made the announcement late Wednesday, hours after Schnatter apologized for using a racial slur during a conference call in May.
Forbes said Schnatter used the N-word during a media training exercise. When asked how he would distance himself from racist groups, Schnatter reportedly complained that Colonel Sanders never faced a backlash for using the word.
In a statement released by Louisville, Kentucky-based Papa John's, Schnatter said reports attributing use of "inappropriate and hurtful" language to him were true.
"Regardless of the context, I apologize," the statement says.
The University of Louisville also said Wednesday that Schnatter resigned from its board of trustees, effective immediately.
Schnatter stepped down as CEO last year after blaming slowing sales growth on the outcry surrounding football players kneeling during the national anthem. He remains chairman of the company he started when he turned a broom closet at his father's bar into a pizza spot.
The National Football League also ended its partnership with Papa John's and struck a deal with Pizza Hut to make it the official pizza of the league.
Papa John's shares fell nearly 5 percent Wednesday after the report, closing at $48.33.
There are more than 3,400 Papa John's restaurants in the United States and Canada, including 40 in New Jersey.
NJ Advance Media staff writer Jeff Goldman contributed to this report.