Although the types of records that qualify as "public" depend upon the jurisdiction in which you reside, certain personal records in the United States are considered part of the public domain, which means anyone can access them at any time for any reason. Public records can range from applications for building permits, to birth and death certificates, to arrest warrants. Whatever the case, there's no way to prevent this information from being seen, so the next-best thing is to make sure the records are accurate. If they're not, you can attempt to have them changed.
Look over the record in question for the contact information of the corporation, individual and/or agency who authored it. For example, if you applied for a permit to build a deck and the construction company misrepresented the value of the proposed deck, you'll need to contact the construction company as well as the local government agency with whom you filed the permit request.
Call, email or visit the pertinent parties to request that they correct the information, keeping in mind that you also may need to take action. For example, if you notice that a warrant for your arrest in regards to unpaid tickets is still posted, you'll need to visit the police department (with your payment receipt) and request that the warrant be removed. Provide the clerk with your receipt, making sure she sees the date it was paid, and request her to remove the warrant. If, on the other hand, you want to amend your birth certificate (your legal name, for example), you'll need to contact your birth state's Office of Vital Records, fill out an official "Record Change" form, pay any required fees and wait for a new birth certificate to be sent to you, which will reflect its change in the public record.
Give the public records system time to update your records to reflect their current status. Government agencies --- especially on the state level --- have thousands of records to process every week, and it's possible that it may take months to get your records up to date, even if you've already rectified the underlying problem. For example, correcting a birth record in Illinois can 25 to 30 weeks.