(NewsNation) — Responding to pledges by Kevin McCarthy to remove him from House committees, Rep. Eric Swalwell said the Republican leader is not ‘up to the job’ for speaker of the House when the new Congress convenes in January.

Republicans regained control of the House during the midterm elections, and despite challenges from some in his party, McCarthy is expected to become the new speaker. He recently told Fox News he would make a good on a promise to bar Swalwell, Ilhan Omar and Adam Schiff from some committees.

Swalwell, a Democrat from California who sits on the Intelligence Committee, called it political retaliation.

“I think I can make the case that he’s going after me in retaliation for our caucus voting to kick Marjorie Taylor Greene off the (Jan. 6 investigative) committee,” Swalwell said. “If you think anything that I have said or Adam Schiff has said or Ilhan Omar has said is comparable to Marjorie Taylor Greene posting on social media about killing Nancy Pelosi, then you just don’t have the facts.”

The nine-member committee investigating the Jan. 6, 2021 riots at the U.S. Capitol has only two Republicans: Adam Kinzinger of Illinois and Liz Cheney of Wyoming. McCarthy had selected five Republicans to sit on the committee, but he pulled all of them when Nancy Pelosi rejected two of the picks. Since then, Republicans have called the committee a partisan witch hunt.

McCarthy has argued Swalwell’s association with a Chinese spy, Schiff’s “lies” and Omar’s “antisemitism” disqualify them all from having certain committee assignments.

“One thing I said from the very beginning, Eric Swalwell cannot get a security clearance in the public sector,” McCarthy said Nov. 20 on Fox News. “Why would we ever give him a security clearance and the secrets to America? So, I will not allow him” to serve on the the House Intelligence Committee.

Schiff and Swalwell serve on the intelligence committee, while Omar sits on the foreign affairs committee. McCarthy said that would change come January.

That is, if he gets enough votes to become speaker. House Republicans reelected him as caucus leader earlier this month by a vote of 188-31. He’ll need 218 to win the speakership, and with Republicans in a position to hold just 222 seats, most of those 31 votes will need to flip in McCarthy’s favor.

“I think his biggest problem right is that to even become speaker, he’s gonna have to strike a drug deal with the likes of Matt Gaetz and other people in his conference,” Swalwell said.

McCarthy and Republicans are now also dealing with fallout from a meeting between former President Donald Trump and Nick Fuentes, a white nationalist who denies the Holocaust happened. Fuentes dined with Trump and Ye, the rapper formerly known as Kanye West, last week at Mar-a-Lago.

Referring to McCarthy as “KKKevinMcCarthy” in a tweet, Swallwell criticized the Republican leader for not commenting on the meeting, unlike others in his party. Former Vice President Mike Pence told NewsNation that Trump should apologize for hosting the pair.

“If you cannot denounce white nationalism or white supremacy when its leader goes to your former president’s home and has dinner, how are you going to lead this country with so much on the line?” Swalwell said. “(McCarthy) doesn’t like me, he’s gone at me, he’s gone after my family … and I never go looking for a fight, but I will punch back twice as hard in my own defense when someone does that. I just don’t think he’s up for the job.”