Certainly a lot of people are worried.
This is the much circulated petition for keeping the Fed independent against the backlash from the Congress (the highlights are mine).
Open Letter to Congress and the Executive Branch
Amidst the debate over systemic regulation, the independence of U.S. monetary policy is at risk. We urge Congress and the Executive Branch to reaffirm their support for and defend the independence of the Federal Reserve System as a foundation of U.S. economic stability. There are three specific risks that must be contained.
First, central bank independence has been shown to be essential for controlling inflation. Sooner or later, the Fed will have to scale back its current unprecedented monetary accommodation. When the Federal Reserve judges it time to begin tightening monetary conditions, it must be allowed to do so without interference.
Second, lender of last resort decisions should not be politicized.
Finally, calls to alter the structure or personnel selection of the Federal Reserve System easily could backfire by raising inflation expectations and borrowing costs and dimming prospects for recovery. The democratic legitimacy of the Federal Reserve System is well established by its legal mandate and by the existing appointments process. Frequent communication with the public and testimony before Congress ensure Fed accountability.
If the Federal Reserve is given new responsibilities every effort must be made to avoid compromising its ability to manage monetary policy as it sees fit.