Keeping Private config files Private in Git

Keeping Private config files Private in Git

Problem: You are working on a project with several other people, each one of you using radically different test environments. The project uses a configuration file that each developer, and eventually each user has to customize. You also want to keep a 'standard' general purpose config file with sane defaults in the source repository.

This is very commmon when developing web applications. For example, a web app that sends out mail notifications may need to use your private SMTP server. You may not feel like telling the whole world where to find it once they hack into your private network.

Solution: Just don't commit that file to the source repository of course :)

Better Solution: People are clumsy and sometimes forget to ignore it. Furthermore, changes to the config file upstream will make practically every new patch conflict with your own working tree. This becomes hell to manage, unless you are truly awesome. Or you use Git. Git is truly awesome.

Furthermore, by using these sane development practices, you'll see other benefits from them. Git makes some very advanced tricks easy, and will make development easier for everyone on your team.

One thing i recommend for git newbies is to use a graphical browser to just view all the changes each step of the way. I like Qgit personally. This exercise is done through the command line, but for some people it also helps to be able to visualize what is going on.

For starters, this requires a bit of preconfiguration. When coding on FAS, i have two main branches i work with, 'master' and 'loupz'. 'master..loupz' (git-ese for all the patches that are in loupz, but not in master) contains one single patch. This patch fixes the configuration files to use my own environment. It contains a few passwords and some information about my local network topology that i just don't want to share. It looks like this:

yankee@koan ~/Projekten/FAS2 $ git log master..loupz
commit cc4990566df6cfd67707892d81c826b898aff5cd
Author: Yaakov M. Nemoy 
Date:   Tue Jul 22 15:23:16 2008 +0200

    My modded fas.cfg
    
    This commit should never be seen in public.  If it does, yell at me.

Once this is set up, it's time to start hacking. In putting together this example, i forgot to 'pre-setup' my hacking session, so i will use git-stash to skip over the 'edit' steps.

yankee@koan ~/Projekten/FAS2 $ git stash
Saved working directory and index state "WIP on loupz: 048419f... My modded fas.cfg"
(To restore them type "git stash apply")
HEAD is now at 048419f My modded fas.cfg

The first thing to do is to create a new branch for all the future development. Since we were hacking on fasclient today, i named it accordingly. This new branch has to branch from loupz, and not master, because loupz has the working configuration.

yankee@koan ~/Projekten/FAS2 $ git checkout -b fasclient loupz
Switched to a new branch "fasclient"

Then I do my hacking, or in this case, just run git-stash apply.

yankee@koan ~/Projekten/FAS2 $ git stash apply
# On branch fasclient
# Changed but not updated:
#   (use "git add ..." to update what will be committed)
#
# modified:   fas/model/fasmodel.py
# modified:   fas/user.py
#
no changes added to commit (use "git add" and/or "git commit -a")

Then it's time to add code from the working tree to the index, the staging area, so we can commit them to the repository. I like git-add --interactive, because it gives me an extra level of verification as to exactly what is being committed.

yankee@koan ~/Projekten/FAS2 $ git add --interactive
           staged     unstaged path
  1:    unchanged       +11/-0 fas/model/fasmodel.py
  2:    unchanged       +19/-0 fas/user.py

*** Commands ***
  1: [s]tatus   2: [u]pdate   3: [r]evert   4: [a]dd untracked
  5: [p]atch   6: [d]iff   7: [q]uit   8: [h]elp
What now> 5

Snip!

The rest of this part is like watching sausage being made ;) . Next we commit our changes.

yankee@koan ~/Projekten/FAS2 $ git commit
Created commit 53dc510: An alternative way to get the data to fasclient as efficiently as possible.
 2 files changed, 29 insertions(+), 0 deletions(-)

If you're following along in Qgit, you'll see that the commit history has become a bit forked between stash, loupz, fasclient, etc... Since we don't need the stuff in the stash anymore, let's just clear it.

yankee@koan ~/Projekten/FAS2 $ git stash clear

So now there should be a string of patches on top of loupz that have been tested with the private configuration file. We know the code works. If we were to call git-push now, the troublesome patch would be included in the push. Here's the fun part. We're going to use the awesomeness of git to rewrite the commit history. This is called rebasing. In this instance, we're going to rebase loupz..fasclient, namely all the patches that are in the fasclient branch, but not in loupz, and rebase them onto master. The command is simple:

yankee@koan ~/Projekten/FAS2 $ git rebase --onto master loupz fasclient
Already on "fasclient"
First, rewinding head to replay your work on top of it...
HEAD is now at 6c583ab Adding some legal voodoo and mumbo jumbo.
Applying An alternative way to get the data to fasclient as efficiently as possible.
warning: squelched 2 whitespace errors
warning: 7 lines add whitespace errors.

To double check this, check git-log

yankee@koan ~/Projekten/FAS2 $ git log
commit 71f41ede6350b481bba358ed612a6cc93fad7a78
Author: Yaakov M. Nemoy 
Date:   Sun Sep 28 22:40:25 2008 -0400

    An alternative way to get the data to fasclient as efficiently as possibl
    
    Ricky and I are trying to figure this out, this commit is one option.

commit 6c583ab7a95f28f3871360a34f233a6454a46e53
Author: Yaakov M. Nemoy 
Date:   Sun Sep 14 20:22:30 2008 -0400

    Adding some legal voodoo and mumbo jumbo.

commit 4adc06b3f4dfd7cfd2adf14a1d990afe1bddbe15
Author: Yaakov M. Nemoy 
Date:   Sun Sep 14 20:17:45 2008 -0400

    Adds help and some documentation to the user

Graphically, you should see that there are now two diverging branches, fasclient and loupz. We now need to merge master to fasclient. This is simple.

yankee@koan ~/Projekten/FAS2 $ git checkout master
Switched to branch "master"
yankee@koan ~/Projekten/FAS2 $ git merge fasclient
Updating 6c583ab..71f41ed
Fast forward
 fas/model/fasmodel.py |   10 ++++++++++
 fas/user.py           |   19 +++++++++++++++++++
 2 files changed, 29 insertions(+), 0 deletions(-)
yankee@koan ~/Projekten/FAS2 $ git branch -d fasclient
Deleted branch fasclient.

Note: we also deleted fasclient because it's no longer needed.

Time to push upstream.

yankee@koan ~/Projekten/FAS2 $ git push 
To ssh://ynemoy@git.fedorahosted.org/git/fas.git
 ! [rejected]        master -> master (non-fast forward)
error: failed to push some refs to 'ssh://ynemoy@git.fedorahosted.org/git/fas.git'

It turns out that our local repository was not completely up to date. This is not a big deal, because rebase can help us solve this problem as well. We only need to use a more simple incantation of git-rebase. We simply want to take all the patches that are in our local 'master' branch, but are not on the origin/master branch upstream, and have them tacked on the end.

yankee@koan ~/Projekten/FAS2 $ git rebase origin/master
Current branch master is up to date.

Oops, we need to fetch first ;)

yankee@koan ~/Projekten/FAS2 $ git fetch
remote: Counting objects: 61, done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 46 (delta 31), reused 0 (delta 0)
Unpacking objects: 100% (46/46), done.
From ssh://ynemoy@git.fedorahosted.org/git/fas
   6c583ab..cc20015  master     -> origin/master
yankee@koan ~/Projekten/FAS2 $ git rebase origin/master
First, rewinding head to replay your work on top of it...
HEAD is now at cc20015 Initial fas_client method.
Applying An alternative way to get the data to fasclient as efficiently as possible.
warning: squelched 2 whitespace errors
warning: 7 lines add whitespace errors.

Finally, we can just push all these changes upstream.

yankee@koan ~/Projekten/FAS2 $ git push
Counting objects: 11, done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.19 KiB, done.
Total 6 (delta 4), reused 0 (delta 0)
To ssh://ynemoy@git.fedorahosted.org/git/fas.git
   cc20015..3df1588  master -> master

If you view this graphically, you will see that the branch with the private configuration is horribly out of date. We need rebase one more time. But this time, we need to check out the branch first.

yankee@koan ~/Projekten/FAS2 $ git checkout loupz
Switched to branch "loupz"
yankee@koan ~/Projekten/FAS2 $ git rebase master
First, rewinding head to replay your work on top of it...
HEAD is now at 3df1588 An alternative way to get the data to fasclient as efficiently as possible.
Applying My modded fas.cfg

And that does it!

This method provides three benefits. First, it makes it very easy to maintain a private configuration without conflicting with other developers on your team. Secondly, it makes it very easy to ensure that there are no 'floating bits' that are uncommitted on your working tree. The only thing that isn't handled by Git are the bits you are actively working on. You can keep each branch separate and organised. Thirdly, once you have to merge with upstream, you can do so cleanly, without a nagging 'merge' commit. This will help you maintain a clean and professional looking log to development.

To make the last point very clear, try to find two git repositories, one with 'merge' commits, and one without, and view it in Qgit. You will notice the repo with 'merge' commits will fork frequently any time two developers work simultaneously. It can sometimes be hard to follow. In contrast, look at the other repo. Development goes in a single linear line. It only forks when a developer or team has to work on a radically different feature. It is much clearer why there are forks in this second repo.