I've run into an issue when provisioning a server to AWS.

We don't use the Puppet/CA proxy functionality for this particular environment, but the SSH provisioning assumes that one is specified and will fail otherwise, after which Foreman responds to the API called that it's performing a rollback, an actual rollback isn't performed however.
Line from the log where this happens:

Rolling back due to a problem: [Configure instance euweawlt-andras-e6e0c632-9d52-4559-89aa-326cfb8824d4.domain.com via SSH 2003 failed [#<Host::Managed id: 27620, name: "euweawlt-andras-e6e0c632-9d52-4559-89aa-326cfb8824d...", ip: "x.x.x.x", last_compile: nil, last_freshcheck: nil, last_report: nil, updated_at: "2015-02-17 14:06:55", source_file_id: nil, created_at: "2015-02-17 14:06:55", mac: nil, root_pass: nil, serial: nil, puppet_status: 0, domain_id: 1, architecture_id: 1, operatingsystem_id: 19, environment_id: 3, subnet_id: nil, ptable_id: nil, medium_id: nil, build: false, comment: nil, disk: nil, installed_at: nil, model_id: nil, hostgroup_id: 121, owner_id: 9, owner_type: "User", enabled: true, puppet_ca_proxy_id: nil, managed: true, use_image: nil, image_file: nil, uuid: "i-9327a975", compute_resource_id: 7, puppet_proxy_id: nil, certname: nil, image_id: 109, organization_id: nil, location_id: nil, type: "Host::Managed", compute_profile_id: nil, otp: nil, realm_id: nil, provision_method: nil, primary_interface: nil, grub_pass: "">, :setSSHProvision]]

Thanks to help from gwmngilfen on IRC it's been narrowed down to this line in app/models/concerns/orchestration/ssh_provision.rb:88
respond_to?(:initialize_puppetca,true) && initialize_puppetca && delAutosign if puppetca?
Commenting that out solves the issue and everything works fine.

I'm guessing this isn't specific to AWS but I haven't had the chance to test it on anything else yet.