Migrating from Chef to Cinc
Intro
Many users already in the Chef ecosystem may have reason to migrate from Chef Infra Server to Cinc server. Thankfully, this is a relatively painless process and all cookbooks, users, data bags, roles, environments, etc. can be retained during the migration process. This blog post documents the migration process from Chef Infra Server to Cinc Server in Red Hat Enterprise Linux 8.
Cinc Server Setup
In order to create your new Cinc Server, stand up a new server on your OS of choice by following the steps detailed at Server Installation.
Once the Cinc server has been configured, you must configure the admin user:
cinc-server-ctl user-create #{username} #{first_name} #{last_name} #{email} #{password} --filename #{output_file_name}
for example…
cinc-server-ctl user-create cincadmin cinc admin cincadmin@dev.local thisisasecret --filename /tmp/cincadmin.pem
Workstation Setup
Backup / Restore operations are performed via a 3rd machine rather than on the source or destination servers. This machine will be used as a workstation on which Chef object-based backups can be created. I deployed an RHEL8 server for this purpose with enough disk space to store the backup. For reference purposes, our server with 17 users, 63 environments, 481 roles, and 539 cookbooks (each with many versions) consumed 9.2GB for the backup. This is a relatively small environment - others may require far more storage to create the backup.
After deployment of the workstation, the following tasks were completed:
Install Cinc Workstation, knife-ec-backup, knife-tidy, and pre-reqs
Use the commands below to install the required tools:
curl -L https://omnitruck.cinc.sh/install.sh | sudo bash -s -- -P cinc-workstation -v 22
sudo yum -y install gcc postgresql-devel
chef gem install knife-ec-backup
chef gem install knife-tidy
The following files were then created on the server:
# /chef_backups/conf/knife_src_server.rb (Source Server Knife Config)
log_level :info
log_location STDOUT
client_key '/chef_backups/conf/knife_src.pem'
chef_server_url 'https://chef.domain.local'
node_name 'berks'
# /chef_backups/conf/knife_dst_server.rb (Destination Server Knife Config)
log_level :info
log_location STDOUT
client_key '/chef_backups/conf/knife_dst.pem'
chef_server_url 'https://cinc.domain.local'
node_name 'cincadmin'
ssl_verify_mode :verify_none
Copy the Destination Server client_key
Copy /tmp/cincadmin.pem
from the Cinc Server to /chef_backups/conf/knife_dst.pem
on the Workstation.
Copy the Source Server WebUI Private Key
Copy /etc/opscode/webui_priv.pem
from the source Chef Infra Server to /chef_backups/conf/webui_priv_src.pem
on the Workstation.
Destination Server WebUI Private Key
Copy /etc/cinc-project/webui_priv.pem
from the destination Chef Infra Server to /chef_backups/conf/webui_priv_dst.pem
on the Workstation.
Source Certificate Trust
Run the following command to trust SSL certificates for the source Chef server.
knife ssl -c knife_src_server.rb fetch
I was not able to do this for the destination server in my environment since it was configured to use the same FQDN as my source server and DNS was still pointing at the source server. If you are in the same position, then SSL checks for the destination server can be disabled in /chef_backups/conf/knife_dst_server.rb
by setting ssl_verify_mode
to :verify_none
.
Backing Up Chef Infra Server
OK, assuming everything above has gone well, it is now time to backup your Chef server! From your Workstation, run the command below to start the backup:
knife ec -c /chef_backups/conf/knife_src_server.rb backup /chef_backups/ --webui-key /chef_backups/conf/webui_priv_src.pem --concurrency 10
Set concurrency to whatever your source server can handle. I have seen some people recommend 5, but we had enough overhead to go with 10 and honestly, I could have probably gone higher. This took several hours the first time. Subsequent backup runs seem to capture the deltas and took 30-45 mins each time.
Knife Tidy
Theoretically, you should be able to use knife tidy
at this point to remove unused cookbook versions, clients, etc, however this didn’t work in my testing and frankly, I didn’t feel comfortable doing this at the same time as the server migration. See the knife-tidy git repo for more information on this utility.
Data Restoration
Once the backup is done, run the following command to perform a dry-run of the backup to the destination server. The dry-run will still create the Chef Organization on the destination server, but will not copy nodes, clients, roles, environments, acls, cookbooks, etc.
knife ec -c /chef_backups/conf/knife_dst_server.rb restore /chef_backups/ --webui-key /chef_backups/conf/webui_priv_dst.pem --dry-run
Once that has been done, run the following command on the destination Cinc server to associate the cincadmin
user with the newly created organization:
cinc-server-ctl org-user-add myOrgName cincadmin
Now you can restore the data to the destination server:
knife ec -c /chef_backups/conf/knife_dst_server.rb restore /chef_backups/ --webui-key /chef_backups/conf/webui_priv_dst.pem
Local Testing
Cinc offers a build in method for running automated tests against the server. The following was done on the Cinc server:
- If DNS does not yet resolve the intended FQDN to the IP address of the Cinc server, then update
/etc/hosts
so that your desired FQDN resolves to the IP address of the Cinc server. - Run
cinc-server-ctl test
The following output was observed:
Pending: (Failures listed here are expected and do not affect your suite's status)
1) opscode-account user association user not in org can be invited to the org by an admin when the inviting admin is removed from the org, invites issued by that admin cannot be accepted
# Known failure: passes w/ 200 b/c no USAG cleanup performed for deleted user
# ./spec/api/account/account_association_spec.rb:662
2) opscode-account user association user not in org can be invited to the org by an admin when the inviting admin is removed from the system, invites issued by that admin can't by accepted
# Known failure: passes w/ 200 b/c no USAG or other group cleanup performed for deleted user
# ./spec/api/account/account_association_spec.rb:671
Finished in 54.85 seconds (files took 3.85 seconds to load)
173 examples, 0 failures, 2 pending
In our case, I built our new Cinc server using a cookbook on our Chef server, so I was able to run chef-client
on the Cinc server to ensure that the server was able to execute a Chef client run against itself (remember, we previously updated /etc/hosts
to make this possible).
Remote Testing
The following remote tests were performed to ensure other existing clients were able to connect and execute their Chef runs against the Cinc server…
- If necessary, update
/etc/hosts
on an existing machine so that the FQDN of the Cinc server resolves to the IP address of the Cinc server. - If the FQDN of your Cinc server is different than that of the existing Chef server, then update client.rb and knife.rb accordingly.
- Run chef-client
- Run the following to ensure the Cinc server saw the test client run:
knife status --run-list
To ensure our pipelines for Roles, Environments, Cookbooks, etc were still working, I temporarily modified the jobs to update /etc/hosts
so that chef.domain.local resolved to the Cinc server IP, and then executed each job to ensure they were successfully able to create and update their respective objects.
I then executed knife
commands with the option -c /chef_backups/conf/knife_dst_server.rb
to ensure I could see the updates on the destination Cinc server.
For example…
# knife role show myUpdatedRole -c /chef_backups/conf/knife_dst_server.rb
# knife cookbook show myUpdatedCookbook -c /chef_backups/conf/knife_dst_server.rb
# knife environment show myUpdatedEnvironment -c /chef_backups/conf/knife_dst_server.rb
Examining the New Environment
The following commands can be used to examine the new environment and ensure it matches the source environment. If you receive errors regarding the cincadmin
user not having access to the org, just re-run the following command on the destination server:
cinc-server-ctl org-user-add orgName cincadmin
List all cookbooks (and versions)
knife cookbook list -c /chef_backups/conf/knife_dst_server.rb
List all nodes
knife node list -c /chef_backups/conf/knife_dst_server.rb
Show details for a single node
knife node show myNodeName -c /chef_backups/conf/knife_dst_server.rb
List all nodes and their last checkin date
knife status -c /chef_backups/conf/knife_dst_server.rb
Get last checkin date for a specific node
knife status "hostname:anyExistingServerName" -c /chef_backups/conf/knife_dst_server.rb
Node Migration
Migrating nodes is a simple operation since the newly built server has retained client keys for each node. If the new Cinc server has the same FQDN as the Chef server that it is replacing then all that needs to be done is a DNS update so that the FQDN now resolved to the IP address of the Cinc server. If the Cinc server has a new FQDN, then you must update client.rb and knife.rb on each client machine to perform the migration.
References
See the following links for more information on the above process: