Converting CGEN from CVS to git

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Converting CGEN from CVS to git

Andrew Burgess
I would like to discuss the idea of converting CGEN from CVS to git.

Motivation
==========

I have been working with CGEN off and on for the last 8 years, and
though I haven't posted any fixes upstream in that time, I do have a
set of fixes locally that I would like to try an upstream over the
next year.

Obviously, moving from CVS to git isn't a requirement for working on
this project, but hopefully we can all agree that moving out of CVS
makes development easier, and git seems like the obvious choice.

Summary
=======

I've done some initial work on a git conversion.  My converted
repository can be found here:

  [hidden email]:T-J-Teru/cgen.git

And the scripts I use to perform the conversion, along with a detailed
README that lists all of the validation I have done on the conversion
(summary below) can be found here:

  [hidden email]:T-J-Teru/cgen-2-git.git

History of the CGEN Repository
==============================

I think most people reading this will already know this history.  But,
by laying out my understanding you'll be able to correct me if it
turns out what I think I know is actually wrong.

In the past many tools hosted on sourceware.org existed in a single
large CVS repository and made use of CVS's module system to allow a
subset of subdirectories to be checked out using a single name.

So, GDB existed in this repository, and a request to checkout 'gdb'
would give you the top-level directory, the gdb subdirectory, and also
the bfd subdirectory, the libiberty subdirectory, the opcodes
subdirectory, and so on.

The binutils project was also in this CVS repository, checking out
'binutils' would give you the same top-level directory as for 'gdb',
but then the gas, ld, bfd, libiberty, opcodes, etc subdirectories.
Notice that some subdirectories are common to both projects, while
others are specific to a single project.

The CGEN project was also in this CVS repository, its main components
are in the 'cgen' subdirectory, but a number of other subdirectories
are also included in a 'cgen' checkout.

If a user had checked out the binutils project, and then wanted to
regenerate the CGEN components, they could simply ask CVS to checkout
the 'cgen' project in the same location, and any additional
subdirectories required for the 'cgen' project would be fetched from the
server.

Then binutils and gdb moved from CVS to git.

We now have the binutils and gdb project in a single shared git
repository, leaving CGEN behind in CVS.

So, until a few months ago if a user wanted to rebuild the CGEN
components the only way was to checkout binutils-gdb from git, then
checkout CGEN from cvs to a separate location and copy the cgen
subdirectory from the cvs checkout into the git repository.  After
this the user can continue to use CGEN.

A few months ago I improved on this by changing binutils-gdb so that
it could make use of CGEN in a separate location, avoiding the need to
copy the cgen subdirectory around.  This was done in this patch series:

  https://sourceware.org/ml/gdb-patches/2018-11/msg00061.html

Now the user only need checkout binutils-gdb to one location, checkout
CGEN to a separate location, configure binutils-gdb to know where CGEN
is after which they can regenerate the CGEN parts of binutils-gdb as
needed.

It is now possible to maintain a clean separation between binutils-gdb
and CGEN during development.

Conversion Design
=================

The observation that in order to regenerate the CGEN components, the
only part of the cgen CVS module that needed to be copied into the
binutils-gdb was the cgen/ subdirectory was the reasoning behind one
big choice I made, that is:

    Only the cgen subdirectory of the current cgen CVS module would be
    included in the new git repository.  Everything else is unneeded
    clutter, and should be discarded.

As CGEN is not very active right now I made the choice to discard
almost all branches when converting.  My assumption is that the old
CVS module will remain available in read only mode, so if an
historical branch is needed then it can always be recovered later.
The only branches I kept are the main development branch (now master)
and the 1.1 release branch (called cgen-1_1-branch).

Conversion Status
=================

After the conversion there are only two branches in the CGEN git
repository, 'cgen-1_1-branch' and 'master'.

The only tags are 'cgen-1-0', 'cgen-1-1', and a set of tags matching
the pattern 'cgen-snapshot-*'.

The 'cgen-1-1' tag was added during the conversion, and matches what
was released as CGEN version 1.1.  This is simply the tip of the
cgen-1_1-branch.

Other than cgen-1-1 (which doesn't exist in CVS) every branch and tag
has been manually checked out in CVS and git and compared, and in all
cases the content matched.

The release tags cgen-1-0 and cgen-1-1 have been compared to the
contents of the release tar files, no differences were found.

I have then used the latest version from CVS, and the HEAD of master
from git to regenerate all of the CGEN created files in the
binutils-gdb/opcodes subdirectory, and in the binutils-gdb/sim
subdirectory for targets that use CGEN.  In all cases the files
regenerated from CVS and git were identical.

Additionally, I checked out the sid project and confirmed, again using
the latest from CVS and HEAD of master from git, that the regenerated
files matched.

I would be interested to hear any additional ideas for testing that I
should perform.

Other Infrastructure
====================

Until June 2018, a new snapshot tag was generated every month.  After
June this stopped.  I don't have visibility of the script performing
this tagging, but this script would need to be updated to work with a
git repository (if we wanted the tagging back again).

Is there any other infrastructure that would need to be updated around
CGEN?  I am happy to help with (or do) script rewrites if that is
required.

Next Steps
==========

What I think needs to happen next:

 + Interested people review the conversion and give feedback.
 + I update based on feedback and repeat until we're all happy.
 + I and/or others update any support scripts that are identified.
 + Create a git repository on sourceware.org and import conversion.
 + Switch cgen part of old CVS repository to read only mode.

The feedback and update depends on people reviewing and giving feedback.

I don't have permissions to create a git repository on sourceware, or
change the admin settings on CVS, nor am I sure who to approach.  When
we get that far any advice would be great.

Finally, is there anything else I can or should do to help make this happen?

Thanks,
Andrew
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Jose E. Marchesi

    I would like to discuss the idea of converting CGEN from CVS to git.
   
    Motivation
    ==========
   
    I have been working with CGEN off and on for the last 8 years, and
    though I haven't posted any fixes upstream in that time, I do have a
    set of fixes locally that I would like to try an upstream over the
    next year.
   
    Obviously, moving from CVS to git isn't a requirement for working on
    this project, but hopefully we can all agree that moving out of CVS
    makes development easier, and git seems like the obvious choice.
   

Yay!  That would be great, moving the project to git.

I am also using CGEN, for a new target, and have also hacks I will want
to upstream at some point.

   
    The observation that in order to regenerate the CGEN components, the
    only part of the cgen CVS module that needed to be copied into the
    binutils-gdb was the cgen/ subdirectory was the reasoning behind one
    big choice I made, that is:
   
        Only the cgen subdirectory of the current cgen CVS module would be
        included in the new git repository.  Everything else is unneeded
        clutter, and should be discarded.

Makes full sense.
   
    I don't have permissions to create a git repository on sourceware, or
    change the admin settings on CVS, nor am I sure who to approach.  When
    we get that far any advice would be great.

The [hidden email] may be able to help?
   
    Finally, is there anything else I can or should do to help make this happen?

Thanks for working on this.
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Frank Ch. Eigler
In reply to this post by Andrew Burgess

andrew.burgess wrote:

> I would like to discuss the idea of converting CGEN from CVS to git.

No objections here, glad to see someone is still interested!  Agreed
that old branches are disposable, that a straight main branch cvs-to-git
conversion is sufficient.  Any objections to just doing it?

It could be nice if someone (tm) were to pursue merging cgen into
binutils proper, but that is a separate issue.

- FChE
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Jose E. Marchesi
   
    > I would like to discuss the idea of converting CGEN from CVS to git.
   
    No objections here, glad to see someone is still interested!  Agreed
    that old branches are disposable, that a straight main branch cvs-to-git
    conversion is sufficient.  Any objections to just doing it?
   
    It could be nice if someone (tm) were to pursue merging cgen into
    binutils proper, but that is a separate issue.
   
What work would be required to achieve that, other than adding the cgen
subdirectory to the binutils repo?

As far as I can see, CGEN is officially still part of binutils, and you
and Doug are listed as the domain maintainers in binutils/MAINTAINERS.
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Frank Ch. Eigler
Hi -

>     It could be nice if someone (tm) were to pursue merging cgen into
>     binutils proper, but that is a separate issue.
>    
> What work would be required to achieve that, other than adding the cgen
> subdirectory to the binutils repo?  [...]

Probably some internal RH chatter about signing over copyright to the
FSF, assuming that binutils still requires this.

- FChE
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Andrew Burgess
In reply to this post by Frank Ch. Eigler
* Frank Ch. Eigler <[hidden email]> [2019-02-03 11:29:58 -0500]:

>
> andrew.burgess wrote:
>
> > I would like to discuss the idea of converting CGEN from CVS to git.
>
> No objections here, glad to see someone is still interested!  Agreed
> that old branches are disposable, that a straight main branch cvs-to-git
> conversion is sufficient.  Any objections to just doing it?
>
> It could be nice if someone (tm) were to pursue merging cgen into
> binutils proper, but that is a separate issue.

Yes we _could_ add CGEN to binutils (assuming we could resolve the FSF
copyright assignment), but should we?

I know right now, binutils/gdb are the only real users, but that
doesn't have to always be the case, ideally CGEN could be used for
more than just generating parts of binutils/gdb, and in that case
having the tools jammed into binutils-gdb would be a little odd.

Could we not start in a separate repository, then if we find that
there is some problem that would be made easier by having the source
in bintuils-gdb we can do that later?

Part of my motivation for patching binutils-gdb to not require CGEN
in-tree was a vision of a future where CGEN would exist as a true
standalone utility that was simply used by binutils (like gcc, flex,
etc).

Thanks,
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Frank Ch. Eigler
Hi -

> [...]
> Could we not start in a separate repository, then if we find that
> there is some problem that would be made easier by having the source
> in bintuils-gdb we can do that later?
>
> Part of my motivation for patching binutils-gdb to not require CGEN
> in-tree was a vision of a future where CGEN would exist as a true
> standalone utility that was simply used by binutils (like gcc, flex,
> etc).

We've dropped the ball on this thread.  I'd be happy to look over
someone else's cvs-git conversion for this (src/cgen + src/cpu?),
or try my own.  There are no blockers to having it on sourceware.

- FChE
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Andrew Burgess
* Frank Ch. Eigler <[hidden email]> [2019-03-03 17:38:40 -0500]:

> Hi -
>
> > [...]
> > Could we not start in a separate repository, then if we find that
> > there is some problem that would be made easier by having the source
> > in bintuils-gdb we can do that later?
> >
> > Part of my motivation for patching binutils-gdb to not require CGEN
> > in-tree was a vision of a future where CGEN would exist as a true
> > standalone utility that was simply used by binutils (like gcc, flex,
> > etc).
>
> We've dropped the ball on this thread.  I'd be happy to look over
> someone else's cvs-git conversion for this (src/cgen + src/cpu?),
> or try my own.  There are no blockers to having it on sourceware.

The original thread included links to my attempt at getting this done:

The mail is here, this describes the process I went through and
some of the choices I made.

  https://www.sourceware.org/ml/cgen/2019-q1/msg00000.html

The link to my conversion is here:

  [hidden email]:T-J-Teru/cgen.git

And the scripts I used to make the conversion (if you wanted to build
on them at all) are here:

  [hidden email]:T-J-Teru/cgen-2-git.git

I would value any feedback you have.

I hadn't forgotten this thread, February got pretty busy for me, but I
had scheduled some time this month to try and move this forward.

I look forward to your feedback.

Thanks,
Andrew
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Andrew Burgess
* Andrew Burgess <[hidden email]> [2019-03-06 13:28:31 +0000]:

> * Frank Ch. Eigler <[hidden email]> [2019-03-03 17:38:40 -0500]:
>
> > Hi -
> >
> > > [...]
> > > Could we not start in a separate repository, then if we find that
> > > there is some problem that would be made easier by having the source
> > > in bintuils-gdb we can do that later?
> > >
> > > Part of my motivation for patching binutils-gdb to not require CGEN
> > > in-tree was a vision of a future where CGEN would exist as a true
> > > standalone utility that was simply used by binutils (like gcc, flex,
> > > etc).
> >
> > We've dropped the ball on this thread.  I'd be happy to look over
> > someone else's cvs-git conversion for this (src/cgen + src/cpu?),
> > or try my own.  There are no blockers to having it on sourceware.
>
> The original thread included links to my attempt at getting this done:
>
> The mail is here, this describes the process I went through and
> some of the choices I made.
>
>   https://www.sourceware.org/ml/cgen/2019-q1/msg00000.html
>
> The link to my conversion is here:
>
>   [hidden email]:T-J-Teru/cgen.git
>
> And the scripts I used to make the conversion (if you wanted to build
> on them at all) are here:
>
>   [hidden email]:T-J-Teru/cgen-2-git.git
>
> I would value any feedback you have.
>
> I hadn't forgotten this thread, February got pretty busy for me, but I
> had scheduled some time this month to try and move this forward.
>
> I look forward to your feedback.

Hi Frank,

Did you get a chance to look through the conversion?

How would you feel if I reached out to the sourceware overseers list
to get an empty git repository setup (I guess that's the right place
to ask)?

Having the repository setup doesn't mean we're committed to a specific
conversion (or even converting at all), but having the repository
setup will probably take a while, so maybe we should get that ball
rolling.  Then once we agree on a specific conversion we can push it.

Thanks for your time,

Andrew

Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Frank Ch. Eigler
Hi, Andrew -

> > The mail is here, this describes the process I went through and
> > some of the choices I made.
> >   https://www.sourceware.org/ml/cgen/2019-q1/msg00000.html
> > The link to my conversion is here:
> >   [hidden email]:T-J-Teru/cgen.git
> >   [hidden email]:T-J-Teru/cgen-2-git.git

OK, those look OK.  With respect to email addresses, retaining
<[hidden email]> as a fallback is fine.

> How would you feel if I reached out to the sourceware overseers list
> to get an empty git repository setup (I guess that's the right place
> to ask)?

I set up this for us/you now, and have given you cgen group membership.
You should be able to commit your work to ssh://sourceware.org/git/cgen.git/
and take it forward.  Thanks for taking the initiative!


- FChE
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Andrew Burgess
* Frank Ch. Eigler <[hidden email]> [2019-03-27 16:09:46 -0400]:

> Hi, Andrew -
>
> > > The mail is here, this describes the process I went through and
> > > some of the choices I made.
> > >   https://www.sourceware.org/ml/cgen/2019-q1/msg00000.html
> > > The link to my conversion is here:
> > >   [hidden email]:T-J-Teru/cgen.git
> > >   [hidden email]:T-J-Teru/cgen-2-git.git
>
> OK, those look OK.  With respect to email addresses, retaining
> <[hidden email]> as a fallback is fine.

I hadn't originally realised / remembered that [hidden email]
was a valid email address, so I mostly just hunted back through the
mailing lists looking for the last email that each user used.

Do you think it might be better if I just used the sourceware email
for anyone who hasn't been seen in say, the last 3 months?

>
> > How would you feel if I reached out to the sourceware overseers list
> > to get an empty git repository setup (I guess that's the right place
> > to ask)?
>
> I set up this for us/you now, and have given you cgen group membership.
> You should be able to commit your work to ssh://sourceware.org/git/cgen.git/
> and take it forward.  Thanks for taking the initiative!

Thanks for doing this.  I know have a little time available again, so
I'm hoping to get this sorted soon.

Once I know how you feel on the email question, I'll do a final
conversion and push it up.

Thanks,
Andrew
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Frank Ch. Eigler
Hi -

> > OK, those look OK.  With respect to email addresses, retaining
> > <[hidden email]> as a fallback is fine.
>
> I hadn't originally realised / remembered that [hidden email]
> was a valid email address, so I mostly just hunted back through the
> mailing lists looking for the last email that each user used.
>
> Do you think it might be better if I just used the sourceware email
> for anyone who hasn't been seen in say, the last 3 months?

Sure.  Given the inactivity in the cgen repo, that's everyone, no? :-)
Another source of mailmap supplemental info is the in-progress gcc
svn-git conversion.  But we'll have human names and -a- plausible
email address for everyone, which should be good enough.

- FChE
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Andrew Burgess
* Frank Ch. Eigler <[hidden email]> [2019-04-21 12:42:14 -0400]:

> Hi -
>
> > > OK, those look OK.  With respect to email addresses, retaining
> > > <[hidden email]> as a fallback is fine.
> >
> > I hadn't originally realised / remembered that [hidden email]
> > was a valid email address, so I mostly just hunted back through the
> > mailing lists looking for the last email that each user used.
> >
> > Do you think it might be better if I just used the sourceware email
> > for anyone who hasn't been seen in say, the last 3 months?
>
> Sure.  Given the inactivity in the cgen repo, that's everyone, no? :-)
> Another source of mailmap supplemental info is the in-progress gcc
> svn-git conversion.  But we'll have human names and -a- plausible
> email address for everyone, which should be good enough.

Ooops, I'm on a bunch of the toolchain related mailing lists, I meant,
looking for folk over all of those lists.  I will go ahead and do that
as I remember having to look quite far back to find addresses for some
people.

Thanks,
Andrew
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Andrew Burgess
In reply to this post by Frank Ch. Eigler
* Frank Ch. Eigler <[hidden email]> [2019-03-27 16:09:46 -0400]:

> Hi, Andrew -
>
> > > The mail is here, this describes the process I went through and
> > > some of the choices I made.
> > >   https://www.sourceware.org/ml/cgen/2019-q1/msg00000.html
> > > The link to my conversion is here:
> > >   [hidden email]:T-J-Teru/cgen.git
> > >   [hidden email]:T-J-Teru/cgen-2-git.git
>
> OK, those look OK.  With respect to email addresses, retaining
> <[hidden email]> as a fallback is fine.
>
> > How would you feel if I reached out to the sourceware overseers list
> > to get an empty git repository setup (I guess that's the right place
> > to ask)?
>
> I set up this for us/you now, and have given you cgen group membership.
> You should be able to commit your work to ssh://sourceware.org/git/cgen.git/
> and take it forward.  Thanks for taking the initiative!

I have now pushed both the master and 1.1 branch (the only remaining
branches) to the new git repository.

This was based on a conversion I just performed, but as the last
change I see in CVS was Alan's update of the copyright dates in
January, this new conversion is mostly unchanged from my original
proposal - I did update some of the email addresses used.

What I think needs to happen next:

  - Maybe someone can confirm that they can checkout the new
    repository, and even better test it.  Instructions for how I've
    been testing this against binutils-gdb are below.  Then,

  - If this is possible it would be great to make the old CVS
    read-only so nobody accidentally commits anything there.  I don't
    have permissions to do anything like this, finally

  - Update the web pages to point at the new git repository.  I have a
    patch for this below, and I _might_ even be able to commit it.  I
    wont know until I try I guess.

Thanks,
Andrew

---

How to test with binutils-gdb
=============================

There are two methods for using cgen with binutils-gdb, I call these
in-tree and out-of-tree.

The in-tree is the old-school way, and should probably be discouraged;
this involves checking out cgen into the binutils-gdb source tree,
something like:

  mkdir -p cgen-test/build
  cd cgen-test
  git clone git://sourceware.org/git/binutils-gdb.git
  cd binutils-gdb
  git clone git://sourceware.org/git/cgen.git
  cd ../build
  ../binutils-gdb/configure --enable-cgen-maint=yes --enable-targets=all
  make all-opcodes all-sim

This should work fine, with the only drawback being that the cgen
repository is living in the middle of our binutils-gdb checkout.

The (IMHO) better way to make use of cgen is out-of-tree; this
involves checking out cgen into a separate directory then telling
binutils-gdb where to find it at configure time, something like this:

  mkdir -p cgen-test/build
  cd cgen-test
  git clone git://sourceware.org/git/binutils-gdb.git
  git clone git://sourceware.org/git/cgen.git
  cd build
  ../binutils-gdb/configure --enable-cgen-maint=$(cd ../cgen && pwd) --enable-targets=all
  make all-opcodes all-sim

This should give identical results to the first method, but with the
benefit that binutils-gdb and cgen are now separate source trees.

If anyone sees anything unexpected then please let me know.

---

Below is the patch for the htdocs CVS repository for review:

-

Update repository details from CVS to git

Updates the details of how to find the repository from CVS to git.
I've also removed reference to the snapshots as these no longer seem
to be generated.

2019-04-21  Andrew Burgess  <[hidden email]>

        * index.html: Update repository details from CVS to git.

Index: index.html
===================================================================
RCS file: /cvs/cgen/htdocs/index.html,v
retrieving revision 1.27
diff -u -p -r1.27 index.html
--- index.html 29 Aug 2011 17:34:25 -0000 1.27
+++ index.html 21 Apr 2019 21:05:28 -0000
@@ -68,28 +68,26 @@ connecting.
 
 <p>You can access the development source tree a couple of different ways.
 <dl>
-<dt><b>Anonymous CVS read-only access</b></dt>
+<dt><b>Read-only git access</b></dt>
 <dd>
   <pre>
-  cvs -z 9 -d :pserver:[hidden email]:/cvs/src login
-  {enter "anoncvs" as the password}
-  cvs -z 9 -d :pserver:[hidden email]:/cvs/src co cgen
+    git clone git://sourceware.org/git/cgen.git
   </pre>
 </dd>
 
-<dt><b>Read-only web-based CVS access</b></dt>
+<dt><b>Read-write git access (CGEN developers only)</b></dt>
 <dd>
-  You can browse the source code using the <a
-  href="/cgi-bin/cvsweb.cgi/src/cgen?cvsroot=cgen">cvsweb
-  interface</a>.
+  <pre>
+    git clone ssh://sourceware.org/git/cgen.git
+  </pre>
 </dd>
 
-<dt><b>CVS snapshots</b></dt>
+<dt><b>Read-only web-based git access</b></dt>
 <dd>
-  Periodically, snapshots of a complete anonymous CVS working tree are
-  <a href="ftp://sourceware.org/pub/cgen/snapshots/">archived here</a>.
+  You can browse the source code using the <a
+  href="https://sourceware.org/git/gitweb.cgi?p=cgen.git">gitweb
+  interface</a>.
 </dd>
-
 </dl>
 
 <h2>Bug reports and patches</h2>
Reply | Threaded
Open this post in threaded view
|

Re: Converting CGEN from CVS to git

Jose E. Marchesi
   
   
      - Maybe someone can confirm that they can checkout the new
        repository, and even better test it.  Instructions for how I've
        been testing this against binutils-gdb are below.  Then,

Works for me.
I tried with both the in-tree and out-of-tree approaches.
   
      - If this is possible it would be great to make the old CVS
        read-only so nobody accidentally commits anything there.  I don't
        have permissions to do anything like this, finally
   
      - Update the web pages to point at the new git repository.  I have a
        patch for this below, and I _might_ even be able to commit it.  I
        wont know until I try I guess.

+1