Scott Foley will never forget the day that Shonda Rhimes broke the news to him that he no longer would be on Scandal.

Relax: Foley’s Jake Ballard isn’t going anywhere when the ABC drama returns Thursday night after a two-month hiatus. Instead, the actor was recalling the moment four years ago when ABC decided he was the wrong actor to play “gladiator” Stephen Finch (portrayed by Henry Ian Cusick in the first season), even though Rhimes had already promised him the part.

“I had read the Scandal pilot script very early on in the process and really wanted to be a part of it,” says Foley. “I went through the process and ABC didn’t think I was the right choice for it. It was probably one of the hardest phone calls that Shonda’s had to make, to me at least. Saying, ‘Hey, I know I said you had this role…but you don’t have this role any more!’”

Of course, Rhimes and Foley had the last laugh, as the actor ended up on Scandal anyway, in a much juicer, higher-profile role, while Cusick quietly disappeared from the series after Season 1. “Thank God I didn’t get it!” Foley says now. “Shonda, true to her word, got me on the show and found a great role and has really turned this character, with the ‘meet cute’ that Olivia and Jake had in a coffee shop, into a really important, viable part of the story. …Everything always works out!”

ABC’s scandalous Scandal behavior wasn’t the only juicy revelation that Foley shared with TV & Not TV as he prepared for the show’s midseason return on Jan. 29:

His wife hates Scandal spoilers. Lots of people press Foley for Scandal intel, but not his wife, Marika. “This is the first job I’ve ever had where I can’t run lines with her. I can’t tell her anything. She won’t let me! And it’s hard, because I have a lot of lines,” says Foley. “Shonda does a great job, unlike other shows, of giving you things to say and really strong, tough dialogue, and I have, for 12 years now, the night before, always been able to run my lines with my wife. I can’t do that on this. She doesn’t want to know anything. She gets mad when I say, ‘Babe, just this once!’ She won’t do it.”

He’s addicted to HGTV — and Downton Abbey. The passion that Scandal fans have for his show is rivaled only by Foley’s devotion to all things HGTV. “I’m a big HGTV watcher,” he says. “I have my cable box turned to HGTV so when my cable turns on, it’s on HGTV. Property Brothers, Fixer Upper, Income Property, House Hunters, Love It or List It…I’m pretty handy!”

The actor is also savoring the chance to catch up on the new season of Downton Abbey with his wife, Marika. “Downton Abbey’s back, and I’m very excited,” he says. “Given our schedule, we’ll usually sit down and watch two or three episodes at a time. We’ll probably watch it all in a week and a half. We like to hold out, though, because once it’s gone, you’ve got to wait another year. It’s like Orange is the New Black!”

Raising three kids is overwhelming. As the father of a two-month-old newborn, son Konrad, Foley realizes he may be in over his head now that he has three kids, all under 5. “I only have three, but I recently read that someone with four kids said, ‘Imagine drowning, and then someone throwing you a baby,’” he says. “And I thought, that’s right! I kind of feel like I’m drowning right now!”

He doesn’t miss filming love scenes with a pregnant Kerry Washington. Shooting Scandal last season proved to be a challenge for the cast and crew, who had to disguise the fact that star Kerry Washington was pregnant in real life — but not on the show. “It was so strange,” says Foley. “They did such a good job last year of not really having to deal with it. Look, it was obviously a presence in every scene we did, but the only times I really thought about it was the love scenes we would have, which made it a little awkward.”

But not any more: “Having her back in full force this year, and having her around and able, it’s a blessing for the show,” says Foley.

Check back Thursday, when Foley teases Scandal’s big return, and Jake’s frantic search for the now-missing Olivia Pope.