The COD project starting... ============================================================================= From: "armellebail " Mailing-List: list sdpd@yahoogroups.com; contact sdpd-owner@yahoogroups.com Date: Wed, 12 Feb 2003 19:24:13 -0000 Subject: Re: FW: [sdpd] More than 100 SDPD papers in 2002, in fact... > Well, it is not as bad as it seems, it is much worse. Not as worse as ICDD a few years ago, with 60000 entries instead of 300000 ;-). Moreover, the www.cristal.org staff is quite small (one manager) and is really a "not for profit organization". I would welcome any gift of 180000 entries from CSD and 50000 from ICSD. The problem is that it is not really apparent in these databases if a powder structure is the result of a Rietveld refinement starting from an isostructural compound or if it is a true SDPD. The ICDD old version even did not provided the information if the crystal structure was determined or not. When crystallographers will be tired to buy all these fragmented databases, maybe they will do something. A role for IUCr ? If not, a role for whom ? At least the SDPD-Database is free access. Not an excuse for incompleteness, though. Best, Armel ===================================================================== From: "Dr. Michael Berndt" Mailing-List: list sdpd@yahoogroups.com; contact sdpd-owner@yahoogroups.com Date: Fri, 14 Feb 2003 14:26:31 +0100 Subject: [sdpd] Open crystallographic database - a role for whom? Hello, > When crystallographers will be tired to buy all these fragmented > databases, maybe they will do something. A role for IUCr ? > If not, a role for whom ? Well, imagine, one of the database 'not-for-profit' organizations would start to 'accumulate' other (complementary) databases. The result would be a monopole and the situation would become worse. But what if crystallographers work together to establish a public domain database with all relevant crystallographic data? This would not only overcome the current situation with 'fragmented' databases, it would also prevent for becoming dependent from monopolists. What would be needed? 1. A small team of engaged scientists with some experience in database and software design to coordinate the project. 2. The authors (i.e. the scientific community = YOU) who provides the project with database entries (note, that if you have'nt sold your experimental results exclusively, you are free to distribute the data to such a database, even if they have already been part of a publication - and a lot of good data have never been published). 3. Free software a) for maintaining the database, b) for data evaluation and calculation of derived data (e.g. calculated powder pattern from crystal structures for search-match purposes), c) for browsing and retrieval We are not in the same situation as decades before when the well-known databases (ICSD, CSD, PDF) started. Today we have the Internet, fast computers, and a big pool of free available software. The question is: Do we have enough scientists who are willing to cooperate? Looking forward to your comments Best wishes Michael ------------------------------------------ Dr. Michael BERNDT Crystal Impact GbR Postfach 1251 D-53002 Bonn Germany Phone: +49 - 228 - 9813643 Fax: +49 - 228 - 9813644 Email: berndt@crystalimpact.com ------------------------------------------ ======================================================================= From: Armel Le Bail Mailing-List: list sdpd@yahoogroups.com; contact sdpd-owner@yahoogroups.com Date: Fri, 14 Feb 2003 15:11:40 +0100 Subject: Re: [sdpd] Open crystallographic database - a role for whom? >The question is: Do we have enough scientists who are willing to >cooperate? You have at least one, me. Tired enough to to the main job (writing AND reviewing papers) and obtaining nothing or peanuts in return. But Crystal Impact is a "(not) for profit" organization as well, or am I wrong ? The Pauling file is not especially free access ? Open access literature is developing fast. Yes, why not a free global database (no discrimination of inorganic/organic) of atomic coordinates (even including coordinates of structures determined in Iraq) on the model of the protein database and of the ArXiv preprint server with mirrors everywhere in the world in universities. The whole database would occupy less than 10% of my current computer hard disc. Just one problem : crystal structure atomic coordinates come mainly from chemists having not yet really understood the interest of the deposition of their papers at preprint servers. They would be too lazy for sending data, I guess. Moreover, conflicts of interest exist in the IUCr community largely sponsored by "not for profit" organizations building databases. On the other hand, probably > 95% of 30000 crystallographers having determined the >300000 known structures are still alive. They would have full right to make additions to the open database... Just have to convince them ;-). Best, Armel PS - OK, send the bomb on me now, Donald R.. Am I not one of these ungrateful french people ? ======================================================================== Date: Fri, 14 Feb 2003 15:16:26 -0800 Subject: Re: [sdpd] Open crystallographic database - a role for whom? From: Luca Lutterotti To: Armel Le Bail Dear Armel, I can also participate. I probably don't want to spend all my life on Maud. I will open the source as soon as I get something stable, and then I will try to get involved in something else. Actually some times ago, speaking with another colleague, we were thinking about another possible open source project in which we needed to have a general database for materials, structures etc. and we decided we should start from there (he was already thinking to a general idea/structure; he is nearly an expert in that field). A network database, open, platform independent, but we didn't think in such a great manner for something also open to the contribution of others from the web. I like this idea of an open database (both for the database, program etc. and for the content) where the people can contribute on the net. Let say I can put my effort as a programmer. Actually I am not good at all as organizer (I have difficulty to barely manage my desk, I even don't think for my life) or to coordinate it, but if someone else is taking care of this I would like just to do work. Probably the Java experience and a little bit of database/distributed programming I have my help. I know also Lachlan will probably have some new spare time and new resources in the future for something interesting. It will probably not too difficult to involve him and he is very good on such stuff. My only concern is about these pseudo "non-profit" organizations. Do we really need one? Is crystalImpact a non-profit one? Best regards and cheers from the Golden Gate (I know you can look at it from the web, but from the window of my office it has a different perspective; the only problem is that there are some anti-French going around here ;-)) ), Luca Lutterotti Temporary address (until 15 August 2003): Department of Earth and Planetary Science University of California at Berkeley Berkeley CA 94720-4767 phone: (510) 642 7431 (Prof. Wenk), fax: (510) 643 9980 Dipartimento di Ingegneria dei Materiali, Universita' di Trento, via Mesiano 77, 38050 Trento, Italy e-mail address : Luca.Lutterotti@ing.unitn.it Home page : Mirror: Phone number :+39-0461-88-2414 Fax : +39-0461-88-1977 ------------------------------------------------------ ============================================================================== From: "Dr. Michael Berndt" Mailing-List: list sdpd@yahoogroups.com; contact sdpd-owner@yahoogroups.com Date: Mon, 17 Feb 2003 15:44:22 +0100 Subject: AW: [sdpd] Open crystallographic database - a role for whom? > But Crystal Impact is a "(not) for profit" organization as well, or am I > wrong ? The Pauling file is not especially free access ? Crystal Impact is a profit organization - no doubt. So you may ask what would be our interest in a free database. 1. We do not produce databases. The Pauling File is not a product of Crystal Impact. We developed the browsing and retrieval software of the CD-ROM version of the Binaries Edition on behalf and in collaboration with MPDS. That's all. The product is distributed exclusively by ASM International and we don't have any influence on pricing or access conditions. 2. On the other hand most of our customers are using crystallographic databases. For the success of our products it is necessary, that such databases are available. In long terms a free standardized global database would certainly be of our interest. That's why we would support such a project as well as any improvement of the existing databases. Best Michael ------------------------------------------ Dr. Michael BERNDT Crystal Impact GbR Postfach 1251 D-53002 Bonn Germany Phone: +49 - 228 - 9813643 Fax: +49 - 228 - 9813644 Email: berndt@crystalimpact.com ------------------------------------------ ========================================================================= Hi Michael, Luca & Lachlan, I am trying to write some of my "naive" ideas about databases and will send this to you soon for comments - just to see if we may agree on a project and if something can be done on an International serious scale. Best, Armel Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel +33 2 43 83 33 47 Fax +33 2 43 83 35 06 Email alb@cristal.org Web http//cristal.org/ ========================================================================== Hi Michael, Luca & Lachlan, Here is the draft. The idea is to make first a maximum noise on the Internet about the possibility to create that COD (Crystallography Open Database), telling clearly what it will be and what it will not be. The text is presented in a humoristic way like a quiz. Fast written before I change my mind... You may not agree with the series of A-answers that try to define the COD. We should first agree on something. Then if a consensus is obtained, we could try to extend the number of people agreeing to the text and who would like to cooperate actively and sign it. The next step would be to present that signed text on the Internet and ask for more signatures and helpers. A fully automated system is absolutely essential for a free database. We should have only a very minimum job to do in maintaining it. On the other hand, the COD should be convincing enough, and not pursuable in law. I must say that I am only able to develop very simple databases and have no real skills about MySQL and PHP which seem the best choice (?). Moreover, like Luca, I am not a great organizer. I could contribute to that COD at least by the accomodation of a mirror on a local server, and by adding CIF files (of my own crystal structures ~100 and probably from my lab production ~500). Best regards ! Armel ========================================================================== Hi Michael, Luca & Lachlan, If you have not seen it yet, download that recent paper about the American Mineralogist Database http//www.minsocam.org/MSA/ammin/toc/abstracts/2003_Abstracts/Jan03_Abstracts/Downs_p247_03.pdf MySQL + PHP... Armel ================================================================================ Hello, I have read your paper in American Mineralogist 88 (2003) 247-250 and visited the database Web pages. Impressive. One silly question is the software part for MySQL / PHP available, or some modules ? Best regards, Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel +33 2 43 83 33 47 Fax +33 2 43 83 35 06 Email alb@cristal.org Web http//cristal.org/ ============================================================================= Date: Thu, 20 Feb 2003 09:30:44 -0500 To: Armel Le Bail From: Bob Downs Subject: Re: Database Hi Armel: You are asking an original question. I can't think of any reason not to give it out. Can I first ask you why you want to see it? We have a paper coming out about the creation of the database that I am attaching as a preprint to this email. Bob ___________________________________ Dr Robert T. Downs on sabbatical at: Center for the Study of Matter at Extreme Conditions University Park Campus, Bldg. VH150 Florida International University Miami, FL 33199 Tel: (305) 348-3442 Fax: (305) 348-3070 Cell Phone: (520) 990-7367 _____________________________________ Permanent address: Dept of Geosciences 530 Gould-Simpson Building University of Arizona Tucson Arizona 85721-0077 Cell Phone: (520) 990-7367 phone:(520) 626-8092 fax: (520) 621-2672 e-mail: downs@geo.arizona.edu url: ===================================================================== Hi Bob, To be clear, I am involved in a project of crystallography open database. A kind of preprint server (you probably know ArXiv) for crystal structures with atomic coordinates. Free access... Your server is exacly what we wanted to do. So why reinventing the wheel ? Just like the ArXiv server, there should be no manual control, only automatic control on CIF files. The upload should be automated. I had another silly question why limiting the database to the American and Canadian Mineralogist data (+ European Journal of Mineralogy) ? Yes, I know you would walk on the feet of the commercial ICSD, but they copy your data as well... A complete mineralogical database of atomic coordinates with free access would be wonderful. Best regards, Armel Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel +33 2 43 83 33 47 Fax +33 2 43 83 35 06 Email alb@cristal.org Web http//cristal.org/ =========================================================================== Date: Thu, 20 Feb 2003 11:01:07 -0500 To: Armel Le Bail From: Bob Downs Subject: Re: Database Hi Armel: Let me tell you my plans and current status. First of all, the data sources. My funding is from NSF, and they have told me that they would like to fund this project for a long time. I am told that I must have written agreements with the journals that published the data in order to include it in the database. I don't really appreciate the legal parts of this, but the requirement comes with the money. So, as of today, I have written agreements with Am Min, Can Min, European J Min, Acta Cryst, and Springer-Verlag. The Acta Cryst and Springer-Verlag agreements have just been finalized so there is no mention of them at the website. I once considered copying ICSD and I tested this with the Am Min data. However, almost every dataset from ICSD had errors in them. Some errors were small and insignificant, but others are major. I am certain that you are aware of this. So I now have a team of undergraduate students that I pay $10/hour to enter the data by hand. They use my software, XtalDraw, to check the data. I double-check everything that they do. (This part is a lot of work, but it is the part that seems to be the most important aspect of the funding). Many of the published data have errors in them, I try to figure them out and fix them. If I can't figure them out, then I contact the authors, if I can, and resolve the problems. In some cases the authors cannot figure out the problems either, so I do not include that dataset. The datasets at the website are not verbatim from the papers, but are corrected datasets. The message from NSF is that science has not done a good job at making their data availble to the public, so they are currently spending a lot of money of "information technology", and this database is exactly what they want us to be doing. I have plans for the next cycle of funding. I will have a search/match program for powder diffraction. It is already made, and it works, but the interface is not quite perfect. It will only include patterns computed from the database. Here is the link if you want to try it out, Please don't pass this on to anyone yet. I also plan to link each dataset with a pdf of the original article. Acta Cryst first suggested this, they would like to do that, and I am waiting for Am Min to tell me if I can do it with their papers. I have not talked to the other journals yet, because I want a firm base of Acta Cryst and Am Min to force their hands. The link will be via a button next to the dataset. One other plan is to link additional data to the pages. As an example see These books were made by Richard Bideaux and he donated them to MSA. I will somehow put the two parts together. These 3 things will wait for the next round of funding. Right now I am concentrating on getting the data. You make the comment, "So why reinventing the wheel ?" I looked at your webpage and see that we are overlapping in some areas, but you are doing some very differnt and interesting things as well. Are you interested in joining forces? By the way, I can send you a copy of the code, but it may take a couple of days to get it together. I have a computer science grad student looking after it, back in Arizona, and there are communication delays because I am in Florida right now. Bob Dr Robert T. Downs on sabbatical at: Center for the Study of Matter at Extreme Conditions University Park Campus, Bldg. VH150 Florida International University Miami, FL 33199 Tel: (305) 348-3442 Fax: (305) 348-3070 Cell Phone: (520) 990-7367 _____________________________________ Permanent address: Dept of Geosciences 530 Gould-Simpson Building University of Arizona Tucson Arizona 85721-0077 Cell Phone: (520) 990-7367 phone:(520) 626-8092 fax: (520) 621-2672 e-mail: downs@geo.arizona.edu url: ==================================================================== To Michel, Lachlan and Luca Hi, No news from you... Frightened by those "rebel" ideas about databases ? I have contacted the author of the American Mineralogist Database. Below is the answer. I have bought a book about MySQL and PHP... May be I will obtain the Am. Miner. software... Let us see. Best, Armel X-Sender downs@pop.geo.arizona.edu X-Mailer QUALCOMM Windows Eudora Version 5.1 Date Thu, 20 Feb 2003 093044 -0500 To Armel Le Bail From Bob Downs Subject Re Database Hi Armel You are asking an original question. I can't think of any reason not to give it out. Can I first ask you why you want to see it? We have a paper coming out about the creation of the database that I am attaching as a preprint to this email. Bob At 0914 AM 2/20/2003, you wrote Hello, I have read your paper in American Mineralogist 88 (2003) 247-250 and visited the database Web pages. Impressive. One silly question is the software part for MySQL / PHP available, or some modules ? Best regards, Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel +33 2 43 83 33 47 Fax +33 2 43 83 35 06 Email alb@cristal.org Web http//cristal.org/ _____________________________________ Dr Robert T. Downs on sabbatical at Center for the Study of Matter at Extreme Conditions University Park Campus, Bldg. VH150 Florida International University Miami, FL 33199 Tel (305) 348-3442 Fax (305) 348-3070 Cell Phone (520) 990-7367 _____________________________________ Permanent address Dept of Geosciences 530 Gould-Simpson Building University of Arizona Tucson Arizona 85721-0077 Cell Phone (520) 990-7367 phone(520) 626-8092 fax (520) 621-2672 e-mail downs@geo.arizona.edu url http//www.geo.arizona.edu/xtal/group/index.php3?page=home ======================================================================= To Michael, Lachlan and Luca Hi again, Seems that NSF is in the right direction... Armel X-Sender downs@pop.geo.arizona.edu Date Thu, 20 Feb 2003 110107 -0500 To Armel Le Bail From Bob Downs Subject Re Database Hi Armel Let me tell you my plans and current status. First of all, the data sources. My funding is from NSF, and they have told me that they would like to fund this project for a long time. I am told that I must have written agreements with the journals that published the data in order to include it in the database. I don't really appreciate the legal parts of this, but the requirement comes with the money. So, as of today, I have written agreements with Am Min, Can Min, European J Min, Acta Cryst, and Springer-Verlag. The Acta Cryst and Springer-Verlag agreements have just been finalized so there is no mention of them at the website. I once considered copying ICSD and I tested this with the Am Min data. However, almost every dataset from ICSD had errors in them. Some errors were small and insignificant, but others are major. I am certain that you are aware of this. So I now have a team of undergraduate students that I pay $10/hour to enter the data by hand. They use my software, XtalDraw, to check the data. I double-check everything that they do. (This part is a lot of work, but it is the part that seems to be the most important aspect of the funding). Many of the published data have errors in them, I try to figure them out and fix them. If I can't figure them out, then I contact the authors, if I can, and resolve the problems. In some cases the authors cannot figure out the problems either, so I do not include that dataset. The datasets at the website are not verbatim from the papers, but are corrected datasets. The message from NSF is that science has not done a good job at making their data availble to the public, so they are currently spending a lot of money of "information technology", and this database is exactly what they want us to be doing. I have plans for the next cycle of funding. I will have a search/match program for powder diffraction. It is already made, and it works, but the interface is not quite perfect. It will only include patterns computed from the database. I also plan to link each dataset with a pdf of the original article. Acta Cryst first suggested this, they would like to do that, and I am waiting for Am Min to tell me if I can do it with their papers. I have not talked to the other journals yet, because I want a firm base of Acta Cryst and Am Min to force their hands. The link will be via a button next to the dataset. One other plan is to link additional data to the pages. As an example see http//www.minsocam.org/MSA/Handbook/ These books were made by Richard Bideaux and he donated them to MSA. I will somehow put the two parts together. These 3 things will wait for the next round of funding. Right now I am concentrating on getting the data. You make the comment, "So why reinventing the wheel ?" I looked at your webpage and see that we are overlapping in some areas, but you are doing some very differnt and interesting things as well. Are you interested in joining forces? By the way, I can send you a copy of the code, but it may take a couple of days to get it together. I have a computer science grad student looking after it, back in Arizona, and there are communication delays because I am in Florida right now. Bob ======================================================================= Date: Fri, 21 Feb 2003 09:00:35 -0500 To: Armel Le Bail From: Bob Downs Subject: Re: Database Hi Armel: So to be honest, I do not know much about ArXiv. I went to their website but they do not have any sort of statement of purpose there, that tells me what they are up to. Can you clarify it for me? What I saw was a number of preprints there. Do the published articles g et posted? How is this done? Do the journals object? Can you post a copy of a pdf from a journal without their permission? I wonder if an author can legally submit a pdf reprint to a site such as this? If so, then this offers a unique way to build a library of reprints from other journals. Bob _____________________________________ Dr Robert T. Downs on sabbatical at: Center for the Study of Matter at Extreme Conditions University Park Campus, Bldg. VH150 Florida International University Miami, FL 33199 Tel: (305) 348-3442 Fax: (305) 348-3070 Cell Phone: (520) 990-7367 _____________________________________ Permanent address: Dept of Geosciences 530 Gould-Simpson Building University of Arizona Tucson Arizona 85721-0077 Cell Phone: (520) 990-7367 phone:(520) 626-8092 fax: (520) 621-2672 e-mail: downs@geo.arizona.edu url: ======================================================================== Date: Fri, 21 Feb 2003 10:29:29 -0500 To: Armel Le Bail From: Bob Downs Subject: Re: Database Hi Armel: I understand now. So what you want to do is create a repository of data that may never have been published. I am willing and happy to put this data on my site. Especially if you are willing to create the mechanism for submittal. We could post data in the same format as what I currently have, and provide a link to a fuller set of tables and text, as the author may desire. A reference to a paper that may actually exist at some later time can be added to the dataset. This makes some sense. What do you think? Bob At 09:56 AM 2/21/2003, you wrote: Hi Bob, >So to be honest, I do not know much about ArXiv. I went to their website >but they do not have any sort of statement of purpose there, that >tells me what they are up to. Can you clarify it for me? What I saw >was a number of preprints there. Do the published articles get posted? >How is this done? Do the journals object? Can you post a copy of a pdf >from a journal without their permission? I wonder if an author can >legally submit a pdf reprint to a site such as this? If so, then this >offers a unique way to build a library of reprints from other journals. What is ArXiv is clarified at : Some conference texts (UNESCO, etc) are there. ArXiv started in 1991, it is NSF funded. It is a repository of preprints, that could be published if accepted. Some journals object, others not (most Physics journals do not object). As ArXiv is mainly devoted to Physics, Elsevier has built his own preprint server - meaning that Elsevier does not object... That Chemistry Preprint Server has only 600 papers since too years, mainly in physical chemistry... Seems that chemists have not the same reasons for letting know fast their results as physicists. You shall not post a final revised version of a manuscript once published. That is all the difference. You own the preprint since it was not treated by any editor yet, the editors own the published revised (if any revision) manuscript if you give them the copyright. Note that I used the Elsevier chemistry preprint server for uploading once a paper rejected by an Elsevier journal. That paper is rated 4 stars at their preprint server ;-). An excellent paper in my opinion... Just too innovative maybe. So, you may see the interest for that COD (Crystallography Open Database) which would include (if crystallographers want) the atomic coordinates before submission to publication. The coordinates would be available 3-12 months or even 24 months before being included in the CSD or ICSD databases. This is something these databases directors should have made (I agree with NSF that "science has not done a good job at making their data availble to the public". CSD and ICSD have made the choice of asking to any user a small money contribution which is not small in many countries. CSD has now a staff close to 50. They do not move at all in the Internet direction, for offering free access. Why not to force them a bit with that COD ? Note that many structures determined were never published and could be added there. Why to write any paper on an apparently uninteresting compound ? A reference to a set of coordinates in the COD could well replace a reference to a published paper. Best, Armel _____________________________________ Dr Robert T. Downs on sabbatical at: Center for the Study of Matter at Extreme Conditions University Park Campus, Bldg. VH150 Florida International University Miami, FL 33199 Tel: (305) 348-3442 Fax: (305) 348-3070 Cell Phone: (520) 990-7367 _____________________________________ Permanent address: Dept of Geosciences 530 Gould-Simpson Building University of Arizona Tucson Arizona 85721-0077 Cell Phone: (520) 990-7367 phone:(520) 626-8092 fax: (520) 621-2672 e-mail: downs@geo.arizona.edu url: ===================================================================== Hi Bob, >Are you interested in joining forces? Sure ! You have the whole force and we having nothing yet in fact. Only intentions. The small group here includes lachlan Cranswick Michael Berndt Luca Lutterotti and me. Armel =================================================================== Hi Bob, Yes this makes sense. It is worth trying I guess. Not only for data that may never have been published, for any data which individuals may decide to send because they have solved the structure and want to share the results for free because they think that knowledge should be open. No data should be accepted without a name (person or lab) so as the database user would know if he can trust these not refereed results. The COD (Crystallography Open Database) would be a repository for not refereed results, or not yet refereed results, or even also refereed if the data depositors decide to deposit their already published structures, considering that the editors are not the owners of these atomic coordinates, nor the CSD and ICSD databases... Creating the mechanism for submittal should not be very complex, I guess. Not referred science is not as "secured" as the peer-reviewed science. But we are adult crystallographers, aren't we ? A discussion about that started at the SDPD mailing list. http//groups.yahoo.com/group/sdpd/ Best, Armel ====================================================================== To Michael, Lachlan, Luca and Bob Hello, I have to make the COD project going on. So here is a draft of website at which you can already upload data http//sdpd.univ-lemans.fr/cod/ I have made some graphisms. Hope you will appreciate the opening box illustrating the concept of free access database of atomic coordinates. There is yet only one basic PHP script for upload. But it should be modified in order to add directly the data inside of the database. And the search part is not working. I have donwloaded some php scripts from the American Mineralogist Crystal Structure Database. But of course, without the MySQL database part, it is hard to understand it all. Have a look... See the PDF file. It is time to make changes, to disagree, possibly to retire from the project. I have changed my old server (OmniHttpd) by Apache. But this is new for me so that I have not yet arranged the Perl scripts to work again - and not sure about security. But the system is now compliant with MySQL (though my progress in MySQL are still limited). Probably a main site will have to be selected for uploading data. I do not care if it is not at Le Mans (CCP14 , Crystal Impact ?) provided the Open Access Spirit is kept for ever ;-). The project may well be disclosed to the public before the search part is working. This would show if uploads occur. Best regards, Armel PS - I will be absent for a few days, till next monday... and will do nothing without your agreement. Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel +33 2 43 83 33 47 Fax +33 2 43 83 35 06 Email alb@cristal.org Web http//cristal.org/ ===================================================================== Date: Wed, 26 Feb 2003 10:47:51 -0500 To: Armel Le Bail From: Bob Downs Subject: Re: The COD Cc: "Luca Lutterotti lachlan Cranswick \"Dr. Michael Berndt\"" , hareesh@cs.arizona.edu Hi Armel, You are moving fast, I will have to accelerate. I have asked my web programmer, Hareesh, to put all the code together in a single zipped file and send it to you. He has already been warned to prepare for that, so hopefully you will recieve it soon. I will be open and honest here. We started this project not knowing what we were doing, and we are stumbling our way along. I hope that the AMCSD is offering a good website that is fast and easy, but I really don't know. I don't get a lot of feedback from users. Is it fast? It is for me, but how about over in Europe? I am not certain how our code will respond to a great growth in data, we fix parts of the code as is necessary. For instance, we did not handle simultaneous, multiple hits in earlier versions. When you examine the code that Hareesh sends to you, please feel free to make suggestions that can improve it. I am certain that there is plenty of opportunity to improve it. That said, I am quite excited by what you are proposing, and I will do all that I can to bring it about. Let me address some of the things in your database quiz. I almost feel like it was written for me. The quiz could use an opening paragraph that describes your concept of the open database. It can be figured out by the end, but I see no reason not to state it clearly, up front. The AMCSD is in its infancy. It was really just started Sept 2001, and we started to gather Am Min papers then. It has not grown fast because nothing has been put on the web before it was thoroughly checked. (I am certain that errors still exist, but not many.) I am now considering moving faster. I think that we can get all the Acta Cryst data up there in a month or so, but it will not have been thoroughly checked. I will put a line in the datafile that states that this data has not been thoroughly checked, but the database will grow fast once that is done. We will probably have to work hard on the database code at that point in order to speed it up, but who knows. Anyways, I don't see a problem with adding all the data in the world to this web site. You talk about why IUCr does not have a good interface to access the data published in Acta Cryst. Let me show you a portion of an email from Peter Strickland, ( editor of Acta Cryst). Dear Bob Thank you for your message. We would be interested in cooperating with your crystal structure database for minerals. We are happy to grant you permission to include the crystal structures of any minerals that have been published in our journals. We will also be willing to allow you to download CIFs from our web site (for database building) for any articles for which we make the CIFs available. In recognition of this permission, we would ask the following: (1) If your database provides links to original journal articles in the future for any other journals, that links are provided to our journal articles (we can provide you with details of how to do this). (2) That we are sent any CIFs you create that are for structures that have been published in our journals (we will then make these available in association with the original article). (3) That we investigate linking between bibliographic information in our journal articles and the structural information in your database. I would be grateful if you could confirm that you agree to the above conditions. Finally, having had a quick look at your web site, I have noticed that the CIFs that you provide are not fully compliant with the CIF Core Dictionary. We would be willing to provide advice on ways of improving the portability of the CIFs you supply; Brian McMahon (bm@iucr.org), our Research and Development Officer, would be very happy to correspond with you about this. In a later email he gave permission to the complete set of data, not just mineralogical. I interpret item (3) to indicate that IUCr is interested in a database of their structures. I see the possibility of linking our database to their papers in the line above the references that has the cif,pdf box etc in it (on their web pages), as well as to other data related to the contents of the paper you may be looking at. Anyways, I think that if we put all of Acta Cryst data in this database, then they have what they want. I picture the data published in journals and those deposited through the COD to be treated in the same way within the database. A search for chemistry or mineral name, for instance, brings up hits, without regard to source. Hareesh and I are currently working on a system to link retrieved data to the papers that contained the data, if available. This idea came from Strictland's item (1). I am currently waiting for the president of the Miineralogical Society of America (the society that publishes Am Min) to decide if we can do the same with Am Min data. I will ask NSF for money to make pdf's of old journal articles for this purpose. Anyways, a similar link can be provided for data entered through the COD. Not only does someone submit the crystallographic table, but they also submit any manuscript that they want to go along with it. Then when a hit to the data is done, the user has the option to examine any text that the author might want to include. I will provide you with an example of this later on. (Hareesh, please make an example). If the papers are available from Am Min, Acta Cryst and COD, then the other journals will be compelled to follow. Lastly, can I ask you to please not use the olivine example that you are using, because it has negative Biso values, and that is so strange that it may preoccupy some readers (including me). Pick a datset that is more "normal". Okay, thats it for now. We are putting alot on our plates, and it seems to be fun. Bob _____________________________________ Dr Robert T. Downs on sabbatical at: Center for the Study of Matter at Extreme Conditions University Park Campus, Bldg. VH150 Florida International University Miami, FL 33199 Tel: (305) 348-3442 Fax: (305) 348-3070 Cell Phone: (520) 990-7367 _____________________________________ Permanent address: Dept of Geosciences 530 Gould-Simpson Building University of Arizona Tucson Arizona 85721-0077 Cell Phone: (520) 990-7367 phone:(520) 626-8092 fax: (520) 621-2672 e-mail: downs@geo.arizona.edu url: ===================================================================== Date: Fri, 28 Feb 2003 09:30:59 -0500 To: Armel Le Bail From: Bob Downs Subject: Re: The COD Cc: "Luca Lutterotti lachlan Cranswick \"Dr. Michael Berndt\"" Hi Armel: Here is a prototype of the modifications to our database for viewing an associated document. We now have agreements with Acta Cryst and Am Min to link directly to papers from the data. I am not going to make this public till the next funding cycle however. I see this as an avenue towards letting the authors of COD put additional info up with the crystallographic data. How does it look? Bob _____________________________________ Dr Robert T. Downs on sabbatical at: Center for the Study of Matter at Extreme Conditions University Park Campus, Bldg. VH150 Florida International University Miami, FL 33199 Tel: (305) 348-3442 Fax: (305) 348-3070 Cell Phone: (520) 990-7367 _____________________________________ Permanent address: Dept of Geosciences 530 Gould-Simpson Building University of Arizona Tucson Arizona 85721-0077 Cell Phone: (520) 990-7367 phone:(520) 626-8092 fax: (520) 621-2672 e-mail: downs@geo.arizona.edu url: ===================================================================== Date: Fri, 28 Feb 2003 16:42:41 -0700 (MST) From: Rajan Hareesh To: cc: Subject: Amcsdb source code files! hi Armel, I am Prof. Robert Down's Web Programmer, Hareesh. I was told that you were working on a project,similar to the "American Crystal Structure Database" for searching the Crystallography database. I have attached a zipped file containing all the source code neccassary for the working of the database.The code was written primarily using Php and MySQL along with some Javascript.Hope you find it useful!Please let me know if you need any more details on it. Goodluck on ur project! hareesh. amcsd_files.zip ===================================================================== Hello Hareesh, Thanks a lot for the scripts, and congratulation for your excellent work. More details would be a reduced database with the MySQL part files (say 5-10 entries), and detailed explanations allowing to see how it is implemented exactly. Armel =========================================================================== Hello Bob, Yes, from Europe (at least from my fast Internet access), the AMCSD is giving answers quickly enough. To be honest, before to be able to improve Hareesh's code, I will have to progress a bit in PHP and MySQL... I have added an opening paragraph to the database quiz, and selected another example from the AMCSD with more realistic thermal parameters. About adding PDF files in the COD, why not ? This would be possible in the preprint sense, if the journal in which the publication is intended allows for preprints distribution prior to publication. Or the PDF may contain additional informations that will not be published in the same form. I have already modified the upload form in order to suggest to add such PDF files - but I have put a 200Ko limit in size. The accomodation of a CIF + PDF database would be extremely hard-disk consuming if > 100000 entries were added ! Acta Cryst is certainly interested in links towards their database of PDF files, but I think that this will be open only to the online journals subscribers, otherwise the link will propose to buy articles... Are you seriously thinking to succeed in proposing access for free to the PDF papers published in the Journal of the Mineralogical Society of America ? >If the papers are available from Am Min, Acta Cryst and COD, then the >other journals will be compelled to follow. The example works fine - but I think that you will not be allowed ;-). Old journals will never authorize free access, unless some miracle. It could be time to announce the COD on the Internet already, even if not functioning yet, and see what happens ? We can expect some negative reactions from the commercial databases... and some positive reactions from individuals. The text of the announce could be short Subject Crystallography Open Database =================================================== Dear Crystallographers, A project of Crystallography Open Database (COD), accomodating crystal structure atomic coordinates prior to their publication, is under development. It is intended to give faster access to the latest structure determinations, for free. Its development and success depends completely on your contribution, either by data download or/and by giving help in software improvements. Visit the COD project Web pages for more details and a crystallography database(s) quiz http//sdpd.univ-lemans.fr/cod/ The advisory board Michael Berndt, Robert T. Downs, Lachlan M.D. Cranswick, Armel Le Bail, Luca Lutterotti, Hareesh Rajan ================================================== Your opinion ? Best regards, Armel Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel +33 2 43 83 33 47 Fax +33 2 43 83 35 06 Email alb@cristal.org Web http//cristal.org/ ========================================================================= Date: Mon, 03 Mar 2003 14:11:25 -0500 To: Armel Le Bail From: Bob Downs Subject: Re: The COD Hi Armel: Thanks for the note. It is not clear to me what your immediate goals are regarding Hareesh's code. I am glad that you want to spend the time on it, cause more heads will create a faster database. But, just in case we have a misunderstanding, we will put the COD stuff on the database, you don't have to write the code to do that. However, I can imagine that things can go faster or more easily if you are involved, but it is not a requirement. Before, when I said that you have to do the code, I was referring to the COD upload stuff. Anyways, I may be misunderstadning your goals completely here, (who knows?). Regarding the MSA journals. I have emailed the president of the society twice, and he has not yet responded to my message. So I called the Executive Director, Dr. J. Alexander Speer. He really is the one who runs the society. He agrees to give us free access to all articles 3 years old and older, with a subscriber link to the newer articles. They are modifying their current interface to the newer articles in order to bring up the free abstract if the user cannot, or does not want to proceed to the article. I will ask for NSF money to make pdf's out of the even older articles and these will, of course, be free too. If you are at a subsribing institution then the articles should pop up without a problem. I would really like to see completely free access to the articles, but this is better than nothing. Perhaps we should test the COD a little bit more before advertising it. I have some data that I could put up there. Tell me when it is ready and I'll give it a shot. Otherwise we are only advertising our intentions. By the way, where to you picture sending this notice? Bob _____________________________________ Dr Robert T. Downs on sabbatical at: Center for the Study of Matter at Extreme Conditions University Park Campus, Bldg. VH150 Florida International University Miami, FL 33199 Tel: (305) 348-3442 Fax: (305) 348-3070 Cell Phone: (520) 990-7367 _____________________________________ Permanent address: Dept of Geosciences 530 Gould-Simpson Building University of Arizona Tucson Arizona 85721-0077 Cell Phone: (520) 990-7367 phone:(520) 626-8092 fax: (520) 621-2672 e-mail: downs@geo.arizona.edu url: ===================================================================== Hello Bob, Writing a minimal code for upload is already done, and it works http//sdpd.univ-lemans.fr/cod/upload.html You may try to upload files. It is voluntarily short (no identification which has to be inside of the CIF or other data file), in order to not discourage and bore people... Previous experiences of such free open databases failed because no entry was send at all but mine (see for instance a database of raw powder patterns at http//sdpd.univ-lemans.fr/powbase/ ). So, I think that the COD will start extremely slowly, that is why it can already be advertised (sending the announce to various crystallography mailing lists - Rietveld, SDPD - to newsgroups - sci.techniques.xtallography, etc - to the IUCr "What's New Web page", to some IUCr Newsletters in more details, etc). Seeing some uploads occuring would be encouraging to work more on the COD project. This is not the same here as your option to get the data from journals you are sure to get data, the COD imay not work at all... depending on the crystallographers goodwill. I will have time next to improve the upload code which should be able to import the essential data of the CIF files directly into the various tables of the database, on the fly (transforming SHELX files into CIF - there is a software for that). So I need to know the complete structure of the database (the MySQL part, the table names, their content, the directory tree, etc, this is why I asked for a reduced small set, 5-10 entries, of your database) for writing the code. But may be I should try to build the COD in my own way (I do not think that some lists of the AMCSD would be useful in the COD - I mean the list of mineral names, and list of authors). I understand that you are putting the COD stuff on the database, that's great. Some day I will have to install it on my server, and so I will need the complete database material (or do you wish to accomodate it in Arizona ?) and probably some mirrors would have to be installed elsewhere in the world (in the hypothesis that COD is a success...). Waiting for large data uploads, I would add the probably few entries by hand or by building a code treating the uploaded files automatically, each day, that code finally evolving into a fully automatic system, hopefully (just like you at the very beginning of your project, it is difficult to predict the exact final COD state). Best regards, Armel =========================================================================== From: Armel Le Bail Mailing-List: list sci-le-mans@yahoogroupes.fr; contact sci-le-mans-owner@yahoogroupes.fr Date: Tue, 04 Mar 2003 08:29:56 +0100 Subject: [sci-le-mans] Cherche compétences Hi, Pour un projet de base de données cristallographique, cherche expert en PHP/MySQL/Apache pour collaborer bénévolement (pour la gloire) au projet COD : A+ Armel ===================================================================== From: "Dr. Michael Berndt" To: "Armel Le Bail" Cc: Subject: AW: The COD Date: Tue, 4 Mar 2003 11:44:12 +0100 Dear Armel, sorry for the silence and thank you for keeping me informed about that project. Yes, feel free to use my name and put it on the advisory board list. I agree with Lachlan, that most people feel too time limited for contributions. But I think they will do it, if they are convinced that its worth to do it. I think its the right time to do something. My hint (in the sdpd posting) about future monopoles is not based on fiction. While I hoped in the past, that the Pauling File would become an upcoming alternative to the ICDD file, MPDS and ICDD meanwhile agreed in a long-term collaboration. The result will be an expanded PDF file (with atomic coordinates), i.e. a stronger monopole on powder patterns and in addition a serious competition to the ICSD. So my feeling is: now or never :-) Michael -----Ursprungliche Nachricht----- Von: Armel Le Bail [] Gesendet: Dienstag, 4. Marz 2003 09:55 An: Dr. Michael Berndt Betreff: The COD Hi Michael, Not a word from you about the COD. Do you give me permission to put your name in the advisory board list and on the bizarre quiz text, and to send the announce for the COD birth ? What is your feeling about that project finally ? Armel ===================================================================== To: sci-le-mans@yahoogroupes.fr From: Daniel Chateigner Mailing-List: list sci-le-mans@yahoogroupes.fr; contact sci-le-mans-owner@yahoogroupes.fr Date: Tue, 04 Mar 2003 13:59:00 +0000 Subject: Re: [sci-le-mans] Cherche compétences et oui, je veux bien, éminent ça me fais vibrer d'avance ... daniel >Si tu as lu le fichier PDF et son "Database(s) Quiz" et que tu >adhéres à ces idées de libre accès aux connaissances >produites par le secteur académique, puis-je t'ajouter à la >liste des membres éminents du Comité Consultatif du projet >COD ? Plus on est de fous... > >A+ >Armel > ===================================================================== From: "L. Cranswick" Subject: Re: The COD To: alb@cristal.org (Armel Le Bail) Date: Tue, 4 Mar 2003 15:14:24 +0000 (GMT) Hi Armel, Sorry - in Oxford today. I have no objections - but it would mainly be the time limited issue (plus got a work permit for Canada yesterday - so am now in the process of moving). But is the NSF funding a rival that makes this superfluous? That is the impression I got? Lachlan. > > Hi Lachlan, > > Not a word from you about the COD. Do you give me > permission to put your name in the advisory board list > and on the bizarre quiz text, and to send the announce > for the COD birth ? > > What is your feeling about that project finally ? > > Armel > > -- ----------------------- Lachlan M. D. Cranswick Collaborative Computational Project No 14 (CCP14) for Single Crystal and Powder Diffraction Birkbeck University of London and Daresbury Synchrotron Laboratory Postal Address: CCP14 - School of Crystallography, Birkbeck College, Malet Street, Bloomsbury, WC1E 7HX, London, UK Tel: (+44) 020 7631 6850 Fax: (+44) 020 7631 6803 E-mail: l.m.d.cranswick@dl.ac.uk Room: B091 WWW: ===================================================================== To: sdpd@yahoogroups.com Cc: rietveld_l@ill.fr From: Armel Le Bail Mailing-List: list sdpd@yahoogroups.com; contact sdpd-owner@yahoogroups.com Date: Wed, 05 Mar 2003 09:01:14 +0100 Subject: [sdpd] Crystallography Open Database Dear Crystallographers, A project of Crystallography Open Database (COD), accomodating crystal structure atomic coordinates prior to their publication, is under development. It is intended to give faster access to the latest structure determinations, openly. Its development and success depends completely on your contributions, either by data download or/and by giving help in software improvements. Visit the COD project Web pages for more details and a crystallography database(s) quiz (PDF file): Thanks for your future help, the COD is yours, it is the right time to do something for an open database controlled by crystallographers, now or never ! The advisory board (wishing to enlarge) : Michael Berndt, Daniel Chateigner, Robert T. Downs, Lachlan M.D. Cranswick, Armel Le Bail, Luca Lutterotti, Hareesh Rajan ===================================================================== Date: Wed, 5 Mar 2003 10:11:14 +0100 From: ptitjean@ccr.jussieu.fr (Michel PETITJEAN) To: cod@cristal.org Subject: a propos du COD X-Antivirus: scanned by sophie at shiva.jussieu.fr To: cod@cristal.org Chers Collegues, Je viens d'apprendre l'existence du projet COD. Je ne suis pas producteur de donnees, mais utilisateur occasionnel de CSD et PDB. A l'heure de l'Europe, est-il bien raisonnable de developper un produit concurrent de CSD, qui devrait avoisiner les 300.000 structures, et qui offre un service acceptable pour un prix acceptable egalement ? Pour PDB, l'acces est gratuit, mais base a Rutgers. Est-ce que COD doit egalement contenir des proteines ? Concernant le format, il me semble que vous utilisez du CIF, qui n'est pas le meilleur type de format pour une base de donnees. De plus, l'utilisateur a devant lui plethore de formats: cela en fera un de plus a connaitre s'il veut extraire des fichiers de reponses pour les relire ensuite, et cela suppose que les specifications du format soient publiques et rigoureuses (eventuellement les routines de lecture/ecriture devbraient etre publiques). Qui finance le projet COD ? Je ne suis pas contre le projet COD, qui aurait pour merite de briser un (ou deux) monopoles. Mais, ayant pas mal travaille sur des bases de donnees chimiques, je considere que COD est un projet plus difficile qu'il n'en a l'air. Exemple, comme il serait bon d'avoir la formule chimique du compose cristallise (ou allez-vous vous contenter du RN ?), il faudra faire des choix de representations de structure, avec les problemes bien connus (surtout les complexes: ferrocene,...): codes de liaisons, mesomeries/tautomeries/aromaticite, coefficients des composantes, j'en passe et des meilleures. La aussi, le choix entre format existant (et inadapte ?) et nouveau format difficile a definir et encore plus a imposer, est problematique. Quelles que soient vos motivations, je vous souhaite bonne chance et surtout beaucoup de courage. Cordialement, Michel Petitjean, Email: petitjean@itodys.jussieu.fr ITODYS (CNRS, UMR 7086) ptitjean@ccr.jussieu.fr 1 rue Guy de la Brosse Phone: +33 (0)1 44 27 48 57 75005 Paris, France. FAX : +33 (0)1 44 27 68 14 ===================================================================== Bonjour, >Je viens d'apprendre l'existence du projet COD. Je ne suis >pas producteur de donnees, mais utilisateur occasionnel de CSD >et PDB. A l'heure de l'Europe, est-il bien raisonnable de >developper un produit concurrent de CSD, qui devrait avoisiner >les 300.000 structures, et qui offre un service acceptable >pour un prix acceptable egalement ? Le seul service acceptable pour un prix acceptable est un service gratuit. La notion de prix acceptable varie considérablement d'un pays à l'autre. >Pour PDB, l'acces est >gratuit, mais base a Rutgers. Est-ce que COD doit egalement >contenir des proteines ? Pas d'objection. >Concernant le format, il me semble que >vous utilisez du CIF, qui n'est pas le meilleur type de format >pour une base de donnees. De plus, l'utilisateur a devant lui >plethore de formats cela en fera un de plus a connaitre s'il >veut extraire des fichiers de reponses pour les relire ensuite, >et cela suppose que les specifications du format soient publiques >et rigoureuses (eventuellement les routines de lecture/ecriture >devbraient etre publiques). Le format CIF est recommandé par l'IUCr. Il est publiquement documenté. Des logiciels gratuits de transformation entre PDB, SHELX (etc) et CIF existent. La base de données elle-même renverra certes des fichiers CIF en résultat, mais contiendra un strict minimum permettant les recherches sur la formulation, les paramètres de maille, les auteurs, les références, etc, etc, etc. >Qui finance le projet COD ? Les membres du comité consultatif touchent des salaires de différentes institutions nationales aux USA, en Italie, en Allemagne, en France.. >Je ne suis pas contre le projet COD, qui aurait pour merite >de briser un (ou deux) monopoles. Mais, ayant pas mal travaille >sur des bases de donnees chimiques, je considere que COD est un >projet plus difficile qu'il n'en a l'air. Visitez http//www.geo.arizona.edu/AMS/amcsd.php qui sert de modèle. >Exemple, comme il >serait bon d'avoir la formule chimique du compose cristallise >(ou allez-vous vous contenter du RN ?), il faudra faire des >choix de representations de structure, avec les problemes bien >connus (surtout les complexes ferrocene,...) codes de liaisons, >mesomeries/tautomeries/aromaticite, coefficients des composantes, >j'en passe et des meilleures. La aussi, le choix entre format >existant (et inadapte ?) et nouveau format difficile a definir >et encore plus a imposer, est problematique. La représentation graphique relève du choix de l'utilisateur de la base de données. La plupart des logiciels ont une entrée au format CIF. http//www.iucr.org/iucr-top/cif/index.html Cela dit, je n'ai que peu d'espoir dans le succès de COD. Le principal problème est de décider les cristallographes à envoyer leurs données... Bien cordialement, Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel +33 2 43 83 33 47 Fax +33 2 43 83 35 06 Email alb@cristal.org Web http//cristal.org/ ===================================================================== Subject: Re: Crystallography Open Database Date: Wed, 05 Mar 2003 10:25:18 +0100 To: afc@orthose.lmcp.jussieu.fr, Rietveld_l@ill.fr From: Alan Hewat At 09:29 05/03/2003, Armel Le Bail wrote: >A project of Crystallography Open Database (COD), accomodating >crystal structure atomic coordinates prior to their publication, is >under development. Excellent idea Armel ! But should it not be International or at least European ? Remember that there are already WWW databases such as the American Mineral Society database and ICSD-for-WWW. Scientists in some countries already have free access to the Brookhaven protein, Cambridge organic, and Karlsruhe inorganic databases via national initiatives or subscriptions. Would it not be better to build on something that already exists ? For example we will soon release new WWW software to access ICSD using modern PHP-SQL database techniques. This would also allow data input as well as data retrieval and even re-direction to on-line electronic publications. Alan. Dr Alan W. Hewat, Diffraction Group Leader. Institut Laue-Langevin, BP 156X Grenoble FRANCE 38042 fax (33)4.76.20.76.48 tel (33)4.76.20.72.13 (or .26 Mme Guillermet) ===================================================================== Subject: Re: Crystallography Open Database Date: Wed, 05 Mar 2003 13:39:14 +0100 To: rietveld_l@ill.fr, afc@orthose.lmcp.jussieu.fr From: Armel Le Bail At 10:25 05/03/2003 +0100, Alan Hewat wrote: >Excellent idea Armel ! But should it not be International or at least European ? It is Internationally open. CIF files are accepted from any country. Copies for mirrorring will be distributed to any interested parties. The main workers will be the CIF producers : those crystallographers determining crystal structures. The data upload is made in very few clicks. >Remember that there are already WWW databases such as the American >Mineral Society database and ICSD-for-WWW. Scientists in some >countries already have free access to the Brookhaven protein, >Cambridge organic, and Karlsruhe inorganic databases via national >initiatives or subscriptions. Absolutely. This is recalled in the PDF file which can be downloaded from the COD website. If you obtain more than 10 A answers to the crystallography database quiz inside of the PDF file, you may well want to become a member of the COD Advisory Board - just send an email to cod@cristal.org, and your name will be added. But the best way to contribute is to send your most recent CIF file. The main developers of the American Mineral Society database (a fully free access PHP/MySQL database which is currently growing) are members of the COD Advisory Board. >Would it not be better to build on something that already exists ? >For example we will soon release new WWW software to access ICSD >using modern PHP-SQL database techniques. This would also allow >data input as well as data retrieval and even re-direction to >on-line electronic publications. The COD will not discriminate between organic/organometallic/inorganic phases. Any "not yet published" and even "never published" dataset is welcome (as well as those yet published). Many people working on organometallic compounds have also interest, of course, in organic and inorganic phases, and need to have access to both CSD and ICSD. Unfortunately, no fusion has occurred that would have proposed a complete database. But the most unfortunate is that no open access has occurred. Armel Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel : +33 2 43 83 33 47 Fax : +33 2 43 83 35 06 Email : alb@cristal.org Web : ============================================================== Date: Wed, 05 Mar 2003 16:09:39 +0100 From: Alain Soyer To: alb@cristal.org Subject: Re: Crystallography Open Database Armel Le Bail, L'idee d'une base de donnees de structures cristallines a acces libre sur internet me semble tres seduisante. J'ai en effet de nombreuses fois passe beaucoup de temps a faire de la bibliographie pour trouver les structures de composes meme tres simples. J'ai lu rapidement le "quiz" et ai repondu plus souvent par A que par B (1 seule fois C). Un point de desaccord : il me semble que les parametres thermiques anisotropes (Uij, Bij...) doivent absolument figurer dans la base (quand ils sont connus evidemment). J'ai un peu peur que la communaute Francaise soit un peu "molle" vis a vis d'un tel projet (voir le peu de messages sur la liste afc), et que la constitution et la maintenance d'une telle base demande plus de travail que vous l'imaginez. Cela ne m'empeche pas d'approuver ce projet, et d'y participer: je viens d'envoyer une "grosse petite" structure d'antibiotique (non publiee). Bonne chance et bonne croissance a la COD ! ----------------------------------------------------------- Alain.Soyer@lmcp.jussieu.fr fax: 33.01.44.27.37.85 Laboratoire de Mineralogie Cristallographie - Case 115 Universite P. et M.Curie et Paris 7 - CNRS UMR 7590 4, place Jussieu - 75252 Paris CEDEX 05 - FRANCE URL: =============================================================== >Bonne chance et bonne croissance a la COD ! Merci ! Le plus gros travail est la finalisation des scripts PHP pour rendre le système totalement automatique. Cela me parait réalisable à terme, après une période de semi-automatisme. Le plus gros problème est en effet de convaincre les cristallographes d'envoyer leurs données. Bien cordialement, Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel +33 2 43 83 33 47 Fax +33 2 43 83 35 06 Email alb@cristal.org Web http//cristal.org/ ============================================================== Sender: toby@jazz.ncnr.nist.gov Date: Wed, 05 Mar 2003 11:53:31 -0500 From: "Brian H. Toby" Organization: NIST Center for Neutron Research To: Armel Le Bail CC: Cranswick Subject: COB Armel, I can see both sides of the picture. Comprehensive databases are expensive to run and unlike the Bio world, we do not have a generous a funding agency to pay the bills to make access free. I think we disagree on the ICSD. I have been supportive of them. They offer a good product at a reasonable price. I do not have such good things to say about the other centers, so I will shut up. On the other hand, there is a great tradition of volunteering in science and if there are enough people willing to make the effort, the COB will fly. I would be glad to have my name on the advisory board. Given that NIST has a role in the database world (though I do not), I may get in trouble for this. If so, I'll have to take my name off. From my perspective, however, the COB is exactly the sort of effort that NIST should support. I'd also like to contribute some advice and effort. One really needs to have some error checking capability, to catch the cases where origin 1 is used, etc. The IUCr uses PLATON for this, but that is not a good choice for lattice compounds. Any ideas? Also, what I would really like to see is software that can read each entry in the database and generate a powder pattern. Lachlan -- can you figure out how to do this? If that can be done, I will update & modify my software to do searching by peak position, but please do not make this particular offer public, as I would rather not alienate anyone at present. BTW, I have done a lot of work on CIF code lately. See for links to documentation and code. One last question. What if there was a web page where one could type in coordinates, etc from the literature? The web page could then generate a CIF that could go into the COD. Would this be fair use of information or a copyright violation? Probably depends on the country. On the other hand, it it is easy to do, then a small effort from a large number of people could create something massive, which is how Linux has grown to produce a product as good if not better than Microsoft. Brian ******************************************************************** Brian H. Toby, Ph.D. Leader, Crystallography Team Brian.Toby@NIST.gov NIST Center for Neutron Research, Stop 8562 voice: 301-975-4297 National Institute of Standards & Technology FAX: 301-921-9847 Gaithersburg, MD 20899-8562 ******************************************************************** ======================================================================= Date: Wed, 05 Mar 2003 19:02:49 +0100 To: "Brian H. Toby" From: Armel Le Bail Subject: Re : COD Cc: Cranswick Brian, Thank for your support ! You will be added in the Advisory Board list (tomorrow). About funding, it seems that the NSF grants some open projects that give access to knowledge. The American Mineralogist Crystal Structure Database (AMCSD) developers have obtained such funding. Amazingly, even access to AM PDF files older than 3 years will probably be given. The AMCSD will soon be much more interesting than now (they negociate with other journals for copying atomic coordinates, including for the minerals in Acta Cryst). About checking capabilities, I planned to suggest to check the CIF at the IUCr Web site... For powder patterns, this is also a project of the AMCSD by using XPOW, see : A web page that could generate a CIF by adding coordinates from the literture is a good idea. There would not be any copyright violation I believe. CSD or ICSD are doing the same. I do not think they hve negociated with hundreds of journals in which are the data. Would you like to create such a form (being CIF expert) ? Nevertheless, I think that the COD will start very slowly (only 2 CIF files only since the announce... or already 2 with an optimistic point of view ;-). Armel =================================================================== Date: Wed, 05 Mar 2003 15:08:26 -0500 To: Armel Le Bail From: Bob Downs Subject: Re: The COD Ooops, I sent it before I was finished with my message, Hareesh is more familar with the directory structure than I am, but you may be able to figure it out by going to teh web site with your browser, and then checking it out. If you run into trouble, then maybe I can help. Else we have to wait till next week. Note that this is the entire database that I sent to you. Bob _____________________________________ Dr Robert T. Downs on sabbatical at: Center for the Study of Matter at Extreme Conditions University Park Campus, Bldg. VH150 Florida International University Miami, FL 33199 Tel: (305) 348-3442 Fax: (305) 348-3070 Cell Phone: (520) 990-7367 _____________________________________ Permanent address: Dept of Geosciences 530 Gould-Simpson Building University of Arizona Tucson Arizona 85721-0077 Cell Phone: (520) 990-7367 phone:(520) 626-8092 fax: (520) 621-2672 e-mail: downs@geo.arizona.edu url: ==================================================================== ===================================================================== From: "Yokochi, Alexandre" To: "'cod@crystal.org'" Subject: COD Date: Wed, 5 Mar 2003 12:57:58 -0800 Armel, I thnk this is a fantastic idea, so sign me on for whatever tasks I can perform in aid. In addition, since I will be chairing a couple of sessions at theis year's American Crystallography Meeting, I will give the COD project a plug (unless someone else is really keen on doing it instead like Lachlan). Alex Y Dr. Alexandre F. T. Yokochi Research Assistant Professor of Chemistry Director of Diffractometry Laboratories 045 Gilbert Hall Oregon State University Corvallis, OR 97331 - 4003 PH: (541) 737-6724 FAX: (541) 737-2062 WEB: www.chem.orst.edu/yokochi/ ===================================================================== Date: Thu, 6 Mar 2003 10:14:48 +0900 Subject: Re: Crystallography Open Database From: Fujio Izumi To: alb@cristal.org Dear professor Le Bail: A project of Crystallography Open Database (COD), accomodating crystal structure atomic coordinates prior to their publication, is under development. It is intended to give faster access to the latest structure determinations, openly. Its development and success depends completely on your contributions, either by data download or/and by giving help in software improvements. Visit the COD project Web pages for more details and a crystallography database(s) quiz (PDF file): Thanks for your future help, the COD is yours, it is the right time to do something for an open database controlled by crystallographers, now or never ! I could help you by providing you with free software named VENUS for three-dimensional visualization of crystal structures and electron/nuclear densities If you are interested in VENUS, I am willing to send you a CD-R storing VENUS to you; a detailed manual written in English is included. This program can input not only CIFs but also files created by various crystallographic data bases, for example, American Mineralogist Crystal Structure Database. For details in VENUS, please refer to angle_dispersive.html#Auxiliary The latest results obtained using VENUS has just been published in K. Takada, H. Sakurai, E. Takayama-Muromachi, F. Izumi, R. A. Dilanian and T. Sasaki, "Superconductivity in two-dimensional CoO2 layers," Nature, 422 (2003) 53. ******************************************************** Fujio Izumi Advanced Materials Laboratory National Institute for Materials Science 1-1 Namiki, Tsukuba, Ibaraki 305-0044, Japan TEL: +81-29-851-3354 (ext.511) FAX: +81-29-852-7449 ******************************************************** =========================================================================== Dear Fujio Izumi, >If you are interested in VENUS, I am willing to send you a CD-R storing >VENUS to you; a detailed manual written in English is included. Yes, thanks a lot for your contribution ! Best regards, Armel Le Bail Université du Maine Laboratoire des Fluorures CNRS UMR 6010 Avenue O. Messiaen 72085 Le Mans cedex 9 FRANCE Tel +33 2 43 83 33 47 Fax +33 2 43 83 35 06 Email alb@cristal.org Web http//cristal.org/ ========================================================================= Date: Wed, 05 Mar 2003 04:59:07 -0500 From: Brian Toby Subject: Re: Re : COD To: Armel Le Bail Organization: NIST Center for Neutron Research Armel Le Bail wrote: > About checking capabilities, I planned to suggest to check > the CIF at the IUCr Web site... I have not had good luck with that beyond checking just CIF syntax. > For powder patterns, this is also a project of the AMCSD by > using XPOW, see : Ah, I see. The potential problem with that XPOW is that it does not read CIFs, also I would want something "batch" that could be run on 10**n CIFs (& preferably not in windows). One thought is -- it should not be too tough to generate Lazy-Pulverix input from CIF's I think (or is there a better choice in your code museum?). > A web page that could generate a CIF by adding coordinates > from the literture is a good idea. There would not be any copyright > violation I believe. CSD or ICSD are doing the same. I do not > think they hve negociated with hundreds of journals in which > are the data. Would you like to create such a form (being > CIF expert)? Perhaps this is a better thought -- I could add to my CIFtools a short program for creating CIFs which could then be submitted via the web or e-mail. I am too sleepy to remember why I figured out this would be better, perhaps because it is easier to create an expandable table (for atoms) in Tcl/Tk than html or perhaps because coordinate import would be possible. What CIF data names would you like to see? Brian ******************************************************************** Brian H. Toby, Ph.D. Leader, Crystallography Team Brian.Toby@NIST.gov NIST Center for Neutron Research, Stop 8562 voice: 301-975-4297 National Institute of Standards & Technology FAX: 301-921-9847 Gaithersburg, MD 20899-8562 ************************************************************ =============================================================================== Bob, >Hareesh is doing his comprehensive exams this week, so I am going to >provide you with the data, it is attached. I well received all the zip files. I was not expecting as much, thanks a lot. Only one thing is lacking now, the MySQL part of the database. This is generally inserted into a directory like ..mysql/data/basename/ where basename is the name of the database, and in that basename directory are files with extension .frm, .MYD and .MYI These files really contain the database structure (tables, fields, and all characteristics of the fields). The PHP codes are directly adressing queries to these tables and fields. The COD announce has been sent yesterday, and has received some positive answers (from Alan Hewat for instance) The number of people willing to cooperate is enlarging (Brian H. Toby, from NIST, well known as CIF expert; Alexandre Yokochi; Daniel Chateigner). But they were only two CIF files uploaded up to now... I will gather all these emails concerning the COD birth in a comprehensive document available to the COD Advisory Board. I think that we could find easier to communicate inside of this advisory board if we had a dedicated mailing list. I will open it at the Yahoo Groups system and send you an invitation to subscribe. Armel =================================================================================== Alex, Thanks a lot. I have added your name to the COD Advisory Board list. Feel free to talk about the project. Lachlan will quit CCP14 by the end of March for a permanent staff job at Chalk River - Canada - neutron powder diffraction.... The task consisting in convincing crystallographers to send their CIF files is certainly the more difficult... Otherwise, we have some PHP/MySQL programming tasks, but the main part of it is already done by the American Mineralogist guys ! So I hope that the search part of the web site will be operational fast. I will gather all these emails concerning the COD birth in a comprehensive document available to the COD Advisory Board. I think that we could find easier to communicate inside of this Advisory Board if we had a dedicated mailing list. I will open it at the Yahoo Groups system and send you an invitation to subscribe. One day after the COD announce, the statistics of connections to the Web site are 258 index.html 164 search.html 112 cod.pdf 90 upload.html 3 CIF files upload Not too bad. Armel =============================================================================== Brian, Lazy Pulverix should be able to do the job once able to read CIF files. The file names would probably be 0000001.cif, 0000002.cif, etc, quite easy for systematic buiding of a series of powder patterns. Bob Downs just sent me the 3725 CIF files in the American Mineralogist Crystal Structure Daatabase ! What CIF data names would you like to see? Well, maybe you should let the user decide for the name. Generally, chosen names are related to the chemical formula, or the author name or any complex numbering following the laboratory books classification of experiments. I will gather all these emails concerning the COD birth in a comprehensive document available to the COD Advisory Board (one more member Alexandre Yokochi). I think that we could find easier to communicate inside of this Advisory Board if we had a dedicated mailing list. I will open it at the Yahoo Groups system and send you an invitation to subscribe. One day after the COD announce, the statistics of connections to the Web site are 258 index.html 164 search.html 112 cod.pdf 90 upload.html 3 CIF files upload Not too bad. Armel PS - Lachlan is on the way to quit CCP14 by the end of March for a permanent staff job at Chalk River - Canada - neutron powder diffraction.... ================================================================================= Hi, The COD announce has been sent yesterday, and has received some positive answers (from Alan Hewat for instance) The number of people willing to cooperate is enlarging (Brian H. Toby, from NIST, well known as CIF expert; Alexandre Yokochi; Daniel Chateigner). I will gather all these emails concerning the COD birth in a comprehensive document available to the COD Advisory Board. I think that we could find easier to communicate inside of this Advisory Board if we had a dedicated mailing list. I will open it at the Yahoo Groups system and send you an invitation to subscribe. One day after the COD announce, the statistics of connections to the Web site are 258 index.html 164 search.html 112 cod.pdf 90 upload.html 3 CIF files upload Not too bad. Bob Downs just sent me the 3725 CIF files in the American Mineralogist Crystal Structure Database ! Armel ================================================================================= From: "Geert Vanhoyland" To: Subject: Crystallography Open Database Date: Thu, 6 Mar 2003 10:17:41 +0100 Dear organisers, Congratulations for this initiative! It would help the scientific community an awful lot, not only saving time and money, but even frustration. However, success will strongly depend on the community itself. Ideally, it should become custom among researchers to upload their data as they send in their manuscript to a journal. Therefore, some encouragement from editors may help? To ensure scientific quality, in my opinion, in some way distinction should be made between published and unpublished results. With regards, Geert -------------------------------------------- Dr. Vanhoyland Geert Instituut voor Materiaalonderzoek Limburgs Universitair Centrum Wetenschapspark 1, 3590 Diepenbeek (Belgium) Tel: ++32/(0)11/26.88.17 Fax: ++32/(0)11/26.88.99 ============================================================================== Geert, >To ensure scientific quality, in my opinion, in some way distinction should >be made between published and unpublished results. Well, this will be clear in the CIF file if there is a complete reference or "submitted to" or "in press" or "unpublished". Best regards, Armel ===============================================================================