Social Security Administration is the right office to call.
As a H-1, you should select not to pay FICA in the very beginning if you were not sure that you would stay in this country long enough to be qualified for the benefits.
Now, it might be too late and too difficult to get a refund.
You are not automatically qualified for the retirement benefit unless certain criterions are met.
