Hi MurphyShaw,
It does sound a bit like an air leak. If you havent already done this,try taking a tin of wd40 or similar and spray it around all the gaskets, inlet to head, carb to inlet, brake servo banjo bolt (if fitted) and listen for a change in the engine revs. If they rise, then its drawing the fluid into the mixture.
There are a couple of things you might also need to consider.
As the OP said, have you changed the manifold gasket?
What sort of inlet manifold have you fitted - If you have cut the old inlet/exhaust cast type as some do, then the casting joint can come very close to where its cut?
Make sure that the flanges where the new inlet and exhaust meet are the same thickness as with having only one stud and the thick washer to tighten both can be a problem.
The needle will definitely need looked at and the car properly tuned and set up.
I,m in the middle of the same problem and have had to look at all of the above. Unfortunately, i dont have the bench mark of knowing that the car was ok before i started as mine is a complete rebuild.