The East Lothian side, previously managed by Harvey, sit second-top of the East of Scotland Football League premier division and are in second place in group I.
Harvey only managed for a short spell at his home-town club after leaving due to a heated debate with a committee member.
Speaking to the Herald, he said: “To be honest, it sounds absolutely crazy, but the players did okay on Saturday.
Nicky Hogarth: Former Graeme High pupil signs for Falkirk on loan from Premier League newcomers Nottingham Forest
Falkirk set to bring in goalkeeper ahead of Peterhead match
McGuffie talks style of play and mentality ahead of Peterhead clash
Syngenta: Finlay could be playing 'four or five' levels above the EoS second division, says boss
'Things did conspire against us' - Boss John McGlynn on Falkirk's second-half display
“They stuck to the game-plan and they defended okay at times in the match, but once again individual errors cost us.
“Every single time we made an error, they punished us for it, but on a whole we actually defended pretty well.
“They had something like six shots and they scored with five of them.
“It wasn’t like we were conceding chance after chance.
“We are being punished, but when you are down at the bottom of the league and you are struggling for form, it just feels like you don’t get any breaks at all, which is so frustrating.
“In regards to the performance as a whole, I thought my players responded well and did better than they have in recent weeks.”
He praised his players for their response to his criticism after last weekend’s 5-2 league defeat in a relegation six-pointer against Newtongrange Star.
After that match, he said they had let the supporters of the club down with some of the performances they have put in over this campaign.
In the league, they sit second-bottom on just 20 points from 28 matches.
“Tranent are a very good side as well who will be looking to win the league they are in,” he said.
“Last weekend I was critical of the players and I had qualms about doing that, but I did because it hasn’t been good enough, and they did so much more right on Saturday that bodes well for our upcoming matches.
“The errors we are making are down to individual moments, and it’s hard to legislate for that.”