How to move a project from Google Code to GitHub

Google Code is shutting down, so you need to move your projects to a new hosting site.

This article tells you how to export a project from Google Code to GitHub.  Google Code provides an "Export to GitHub" button at the top of your project pages.  That button uploads the repository to GitHub (first converting it to Git if needed), but it doesn't always work; even when it does, there is a lot more to moving an entire project.  This blog posting takes you through the extra steps that you need to perform.  Some of these steps ought to be done by the "export to GitHub" button, but the export tool is incomplete.  Others are necessarily manual steps.

1. Warn your teammates that you are going to move the repository; give them a chance to push their changes or do other cleanup.  Ask them not to make any more changes to their clone, ever.
2. I suffered authentication problems when I clicked the "Export to GitHub" button in parallel for multiple projects in different browser tabs.  Click it for one project; go through GitHub authentication, etc.; and then you can click the "Export to GitHub" button for the next project.
3. The "Export to GitHub" button won't work if your repository is too big or if it ever contained a file larger than 100MB.  In this case, you need to do the conversion by hand.
1. Create the new repository on GitHub
2. Run the following:
   cd DIRFORGITCLONE
   git init
   fast-export/hg-fast-export.sh -r HGCLONE
   git checkout HEAD
   git remote add origin git@github.com:USERNAME/PROJECTNAME.git
   git push --set-upstream origin master
3. If the push failed because your repository contains files that are too large, use the BFG Repo-Cleaner to remove them, then redo the push.  Sample command lines:
   bfg --strip-blobs-bigger-than 100M
   git gc --aggressive --prune=now
4. Migrate the issues:  see https://code.google.com/p/support-tools/wiki/IssueExporterTool .  I didn't have a problem with the throttling; issues seemed to be uploaded faster than one per second.
4. Redo the repository export if you didn't do it manually in the previous step.
   The reason is that often, the tool run by the "Export to GitHub" button says that repository conversion was successful, but in fact some history is lost or munged (especially around merges and file renames).  One way to identify problems is to run  git log --graph  and look at the end of the output.  There should be just one root, which is from the beginning of the history; if there are multiple roots or a root that is not at the beginning of the history, then you must regenerate the history from the Mercurial repository.  However, I recommend that you always do so, just in case.
   Run the following commands:
   cd ~
   git clone https://github.com/frej/fast-export.git
   REPO=myrepositoryname
   cd REPOSITORY_PARENT
   rm -rf $REPO
   git init $REPO
   cd $REPO
   ~/fast-export/hg-fast-export.sh -r HGCLONE --force  -A ~/authornames.txt
   git checkout HEAD
   git remote add origin git@github.com:mygitusername/$REPO
   git push --force --set-upstream origin master
5. The GitHub repository appeared in your personal GitHub account.  If you collaborate with other people on the project, move the repository to an organization's account by browsing to "Settings > Transfer ownership".
6. Rename your old clone of the Google Code repository, and clone the new GitHub repository.
   If you manage your clones using a program such as mvc, then update its control file, such as ~/.mvc-checkouts.
7. If the old Google Code repository used Mercurial:
1. convert the ignore file from Mercurial to Git:
   git mv .hgignore .gitignore
2. Remove any occurrence of syntax: glob, and convert any regex patterns in the .gitignore file into globs.  Convert other patterns as necessary.  Take care because git interprets patterns differently depending on whether they contain a slash ("/") or not.
3. Build the project and run tests to create temporary files that should be ignored.
4. Run git status to ensure that they are ignored.
5. git commit -m "Rename .hgignore -> .gitignore" .hgignore .gitignore
8. At top of the GitHub page for the project (not the GitHub Pages page):
1. Click the red "Stop ignoring" button
2. Add text for the description field (from Google Code description)
3. Add a link to a real homepage for the project.
   If it doesn't have one already, then use http://USERNAME.github.io/PROJECTNAME/ (this is a better choice than the wiki), and create this homepage per the next step.
9. Create a homepage for the project, if it doesn't have one already.
   A good way to do this is to create a GitHub Pages (github.io) homepage at http://USERNAME.github.io/PROJECTNAME/:
   git checkout --orphan gh-pages
   git rm -rf .
   rm .gitignore
   Create index.html
   git add index.html
   git commit -a -m "First GitHub Pages commit"
   git push --set-upstream origin gh-pages
   Browse to http://USERNAME.github.io/PROJECTNAME/ to verify the content.
10. If your project doesn't arlready have a README file, then create a README file (or maybe README.md), which will show up at the bottom of your GitHub page.  If you are creating the README purely for the benefit of people browsing on GitHub, and your real documentation appears elsewhere, then your README will be short and will redirect people to the project's real homepage.  Your project ought to include two different types of documentation:  for users and for contributors.  These often appear in different places but should be easy to find.
    You should also include a LICENSE file.  Otherwise, nobody (who is under the control of a legal department) will be able to use your software.
11. If your wiki is intended as developer-written documentation, then move all wiki pages to GitHub pages or to the main repository: see https://github.com/morgant/finishGoogleCodeGitHubWikiMigration .  (The instructions are hard to follow; read them carefully!)
    Only keep the pages in a wiki if you expect users to edit it.
    The import process creates a "wiki" branch.  If this branch is useless (especially if it contains only one file or is redundant with the project's homepage or GitHub Pages page), then delete it by clicking the appropriate trashcan icon at https://github.com/USERNAME/PROJECTNAME/branches
12. Look up the permissions on Google Code at https://code.google.com/p/PROJECTNAME/adminMembers and give corresponding permissions on GitHub at https://github.com/USERNAME/PROJECTNAME/settings/collaboration .  Users in the Owners group don't get notifications, so put all of those people also in some other group.
13. Convert post-commit hooks to GitHub; see "Post-Commit URL" and "Post-Commit Authentication Key" at https://code.google.com/p/PROJECTNAME/adminSource and add them at https://github.com/USERNAME/PROJECTNAME/settings/hooks/new .
14. Add email notifications at https://github.com/USERNAME/PROJECTNAME/settings/hooks/new?service=email corresponding to the "Activity notifications" at https://code.google.com/p/PROJECTNAME/adminSource .
15. Copy files in the Google Code "Downloads" section to elsewhere.
16. The project's documentation and buildfiles probably refer to Google Code.  Update all such mentions.
1. Update direct references to files in the repository:
   preplace "https?://PROJECTNAME.googlecode.com/hg/"   "https://raw.githubusercontent.com/USERNAME/PROJECTNAME/master/"
2. Seach for more occurrences of googlecode.com and code.google.com, and edit each one manually.
3. If the old Google Code repository used Mercurial, search for occurrences of hg, and edit each one manually.
17. Write up the change of repositories for your changelog or release notes.
18. Run tests, then commit and push your changes to GitHub.  For example:
    git commit -a -m "Update references from Google Code to GitHub and from Mercurial (Hg) to Git"
19. If the push created or changed your project's homepage, run a link checker to check for broken links:
    plume-lib/bin/checklink -q -r `grep -v '^#' plume-lib/bin/checklink-args.txt` MYURL
20. Update your continuous integration server so that it refers to the new repository location.
21. If you have a README.html or similar HTML file in your Google Code repository, then external links to it may exist.  In the old Google Code repository, edit it to add a line such as
      <meta http-equiv="Refresh" content="1;URL=http://USERNAME.github.io/PROJECTNAME/" />
    to the header, and commit and push to Google Code.
22. Set up forwarding ("project moved") at Google Code, at https://code.google.com/p/PROJECTNAME/adminAdvanced .  Forward to the project's homepage or new GitHub Pages homepage.
    If you ever need to temporarily undo the forwarding, go to https://code.google.com/p/PROJECTNAME/adminAdvanced .  Google Code may or may not remember the forwarding URL; you will need it when you re-enable forwarding.
23. Tell anyone who might have cloned the old Google Code repository to rename their clone, not use it any more, and instead clone and use the new GitHub repository.

If you have any corrections or additions to this guide, let me know.

Declarative specification of FSM-inference algorithms

The paper “Using declarative specification to improve the understanding, extensibility, and comparison of model-inference algorithms” recently appeared in IEEE Transactions on Software Engineering.  The paper is by Ivan Beschastnikh, Yuriy Brun, Jenny Abrahamson, me, and Arvind Krishnamurthy.  The title is a mouthful, because the paper accomplishes quite a bit, but I'll try to break it down here.

The models that the paper speaks of are finite state machines (FSMs), like this one that expresses some legal sequences of commands that a client program might issue to a mail program:

It is useful to have a FSM that expresses the behaviors of a system.  Unfortunately, the system designers may not have written one down, or the one they wrote may not be accurate.  An alternative is to infer a FSM from observed system executions.  This is such a good idea that many researchers have published such model-inference algorithms.

Computer scientists usually write algorithms as pseudocode, and model-inference algorithms are no exception.  Pseudocode is familiar to programmers and enables the inference algorithm to be implemented, but it is not helpful in understanding the inference algorithm.  The extensive proofs in any algorithm textbook show the limitations of reasoning about pseudocode.

The paper proposes InvariMint, an approach to specify model-inference algorithms declaratively.  The algorithm designer specifies what properties of the input log the algorithm should preserve, rather than how the algorithm works in terms of programming-language statements.  For instance, the algorithm designer might say, "If event A is always followed by event B in the log, then the algorithm should output a model in which event A must always be followed by event B."  This circumscribes what types of generalization that the algorithm is allowed to do.  The algorithm designer doesn't have to figure out how to encode this in a programming language nor to verify that the inference implementation is correct.

Here is how InvariMint works, in a nutshell:

• At design time, the algorithm designer decides what sorts of properties the inference algorithm should express about a trace.  The designer generalizes these properties to patterns and specifies them as FSMs.
• At run time, given an execution trace (a log), the algorithm mines matches for each of those patterns from the trace.  The algorithm intersects all the mined FSMs to obtain the final inferred model for the program.

We applied the InvariMint declarative approach to two model-inference algorithms that had previously been specified procedurally.  Specifying them declaratively with InvariMint (1) leads to new fundamental insights and better understanding of existing algorithms, (2) simplifies creation of new algorithms, including hybrids that combine or extend existing algorithms, and (3) makes it easy to compare and contrast previously published algorithms.  Furthermore, the InvariMint-generated algorithms were significantly faster than the original procedural versions of the algorithms.

You can read the paper here.  The InvariMint implementation is distributed along with Synoptic.

NSF GRFs for Pavel Panchekha and Doug Woos

I am delighted that two of my students — Pavel Panchekha and Doug Woos — have won NSF graduate fellowships this year.

Pavel applies his mathematical background to problems in compilers and verification.  Pavel is co-advised by Zach Tatlock.

Doug works in the intersection of systems, networks, and programming languages.  Doug is co-advised by Tom Anderson.

Both Doug and Pavel are authors of our forthcoming PLDI paper on verifying distributed system implementations.

Overall, UW CSE garnered 9 NSF GRFs — more than CMU, MIT, or Stanford — and only one less than Berkeley, which has a larger student pool than UW.  UW's success is mostly because our students are terrific, but also because we advocate fiercely for their success.

Are mutants a valid substitute for real faults in software testing?

My colleagues and I won an ACM Distinguished Paper award for our paper “Are mutants a valid substitute for real faults in software testing?".

Both practitioners and researchers need to evaluate the quality of test suites — for example, researchers want to know whether a new testing technique improves a test suite.  The true measure of a test suite's quality is how many real faults it detects.  The set of real faults is typically unknown, so the state-of-the-art approach is to measure how many artificial faults, called mutants, a test suite detects.  Hundreds of research papers make the assumption that if a test suite detects more mutants, then it will detect more real faults as well.  Amazingly, no one knows whether this assumption is true!  Or, no one did until our research.

The paper reports on extensive experimentation that shows that mutant detection is the best available proxy for test suite quality.  We also showed how mutation analysis can be improved and identified its fundamental limitations that prevent it from perfectly predicting real fault detection.  Our analysis accounts for confounding factors such as code coverage.  In addition to these experimental results, the real faults and test suites we assembled can be used in future testing research.

The paper was presented on November 20 at FSE, one of the two top software engineering conferences.  The paper was authored by René Just, Darioush Jalali, and Michael Ernst of UW CSE, Laura Inozemtseva and Reid Holmes (a former postdoc at UW) of the University of Waterloo, and Gordon Fraser of the University of Sheffield.  This is the second ACM Distinguished Paper award this year for René, Gordon, and me.

You can read the paper at http://homes.cs.washington.edu/~mernst/pubs/mutation-effectiveness-fse2014.pdf.  You can obtain the tools and experimental data at http://mutation-testing.org and http://defects4j.org.

UW CSE at ISSTA

René Just and Michael Ernst of UW CSE, along with their colleague Gordon Fraser of the University of Sheffield, have been awarded an ACM Distinguished Paper award for their paper “Efficient mutation analysis by propagating and partitioning infected execution states”.  This paper will be presented on July 25 at ISSTA, the premier conference in software testing and analysis.  The paper speeds up mutation analysis by 40% over the previous state of the art.  Mutation analysis is widely used in testing research, because it is the most precise known approach to measure the quality of a test suite.  This is Michael Ernst's 6th ACM Distinguished Paper award, not to mention other best paper awards he has received.  The photo shows Ernst and Just flanking the conference chairs Corina Pasareanu and Darko Marinov.

UW CSE is well-represented elsewhere in the conference, with 2 other technical papers and 3 tool demos.  One other paper is “Empirically revisiting the test independence assumption” by Sai Zhang, Darioush Jalali, Jochen Wuttke, Kıvanç Muşlu, Wing Lam, Michael D. Ernst, and David Notkin, which shows how and when tests cases depend on one another, so that a test fails depending on what other test cases are run before it.  Another paper is “A type system for format strings” by Konstantin Weitz, Gene Kim, Siwakorn Srisakaokul, and Michael D. Ernst, which shows how to verify correct use of format routines such as printf.

ISSTA 2014 papers

My research group had 3 papers accepted to ISSTA, the International Symposium on Software Testing and Analysis.  Here are brief descriptions of them.

“Empirically revisiting the test independence assumption”

by Sai Zhang, Darioush Jalali, Jochen Wuttke, Kıvanç Muşlu, Wing Lam, Michael D. Ernst, and David Notkin.

Two tests are independent (non-interfering) if running one test does not change whether the other test fails or succeeds.  If tests are dependent, then reordering them (as in test prioritization or test selection) could cause a previously-passing test to suddenly start failing, even though the program has not changed.

Both practitioners and researchers generally ignore the problem of test dependence:  they assume that test cases are independent.  We developed presents algorithms and tools for detecting dependent tests.  Our paper shows that test dependence does occur in real-world test suites, and it shows that test dependence can cause both false alarms and failed alarms in prioritized (reordered) test suites.

You can also read the abstract.

“Efficient mutation analysis by propagating and partitioning infected execution states”

by René Just, Michael D. Ernst, and Gordon Fraser.
Abstract:

Given two test suites, mutation analysis ranks one of them as better than the other.  Mutation analysis is notoriously compute-intensive, because it runs many variants of the program over each test suite.  This paper reduces the cost of mutation analysis by 40%, by eliminating redundant executions that reveal no information.

The technique works by first running an instrumented version of the program over the test suite.  This pre-processing step more than pays for itself by enabling other executions to be skipped:

• mutated expressions that produce the same value as the original program expression did,
• mutanted expressions that produce a different value, but a containing expression produces the same value as it did in the original program, and
• different mutants that all compute the same value (all but one of which can be skipped).

You can also read the abstract.

“A type system for format strings”

by Konstantin Weitz, Gene Kim, Siwakorn Srisakaokul, and Michael D. Ernst.

Most programming languages support format strings, but their use is error-prone. Using the wrong format string syntax or passing the wrong number or type of arguments leads to unintelligible text output, program crashes, or security vulnerabilities.

We have designed and implemented a type system that indicates erroneous usage, or statically guarantees that calls to these APIs will never fail.  The annotation burden for the user of our type system is low, and it found over 100 bugs in open-source projects.

You can also read the abstract.

Reading the papers

The papers are not publicly available yet, because we are editing our submissions in response to suggestions from the referees.  (Whether or not a paper is accepted, it is always a pleasure to receive feedback from experts who have read your paper and thought about ways to improve the research.)  The camera-ready deadline is June 6.  In the meanwhile, if you are interested in a paper and are willing to provide feedback, I would be happy to send you a preprint; just ask.

ACM SIGSOFT Impact Paper Award

Jake Cockrell, Bill Griswold, David Notkin, and I received the 2013 SIGSOFT Impact Paper Award for our paper Dynamically Discovering Likely Program Invariants to Support Program Evolution, which appeared in ICSE '99.

The award citation reads:

This paper initiated a revolutionary and important line of research that is still very active and relevant today: extracting models of running software and reasoning about its behavior. The approach it defined has significantly influenced both the state of the art and the state of the practice. In addition, the paper described Daikon, a tool that implements this approach, and which continues to be vital and useful today, some 15 years later.

The award includes a $1000 honorarium, which we donated to the David Notkin Endowed Graduate Fellowship in Computer Science & Engineering.

This paper was the foundation of my PhD dissertation.

Later, a journal version of the paper was published in TSE in 2001.  If you read the journal version, feel free to skip sections 5-8, which are not as interesting as the other parts of the journal paper.

This work received other awards including being selected for expedited journal publication (this was the precursor to the ACM Distinguished Paper Award, which was introduced in 2002), honorable mention in the ACM doctoral dissertation competition (the first honorable mention that had been awarded in 7 years), and best software engineering dissertation of the prior three years (awarded at ICSM 2001).

However, I feel that a retrospective award such as the SIGSOFT Impact Paper Award is much more meaningful than any of those awards that were presented at the time of publication.  It's very difficult to determine the importance of a piece of research when it first appears.  The perspective of time reveals whether the paper was foundational or merely clever.

I tend to dislike best-paper awards (and proxy awards such as nomination or expedited journal publication), because they are so random and therefore not very meaningful.  (Before I offend my colleagues who are reading this, let me clarify that the randomness or meaninglessness applies to the times I have been so honored, at ICSE 1999, ICSE 2000, VMCAI 2003, ESEC/FSE 2003, ICSE 2004, ASE 2005, ISSTA 2006, ASE 2006, ICSE 2007, ESEC/FSE 2007, ASE 2007, ISSTA 2008, ISSTA 2009, ECOOP 2011, and ESEC/FSE 2011.  But the honors accorded to your papers, dear reader, were surely well-deserved recognition of research that is clearly better than any other research presented at that conference.)  That's why I don't generally list these on my webpages, for example.

Despite my opinion of how meaningful a best-paper award often is, I do support conferring them.  The reason is that these are useful to graduate students and to junior faculty members:  the awards buttress those young scientists' c.v.s and therefore their careers.