Talk:General Astronomy/What People do in Astronomy

Moved original text here and gradually rewriting.

I don't think that the original text accurately describes the process of astronomy at all. Namely


 * The process of grant writing, isn't and shouldn't be a filter for scientific merit. The main criterion for whether a grant gets issued or not is whether or not the grant proposal meets the institutional priority of the grant writer. If the funding agency thinks that studying quasars is important, then you aren't going to get money to study Jupiter.


 * Most of the communications in astronomy today effectively bypasses peer review. Most of the real communications between astronomers takes place at conferences or via the Los Alamos preprint server.  By the time something is formally peer reviewed, months have passed and it is likely out of date.


 * Review articles are important but it is incorrect to assert that the secondary literature is less error-prone that the primary. Again this is  because things move fast and by the time a review article is written, it is likely out of date.

Roadrunner 00:09, 18 June 2006 (UTC)


 * I disagree with that.
 * The grant process is a filter for scientific merit. I was at the NSF town hall meeting at the last AAS meeting, and Eileen made exactly this point -- scientists shouldn't feel that the NSF isn't interested in their work just because there's no NSF program for their area. Any area of work is eligible and will be evaluated on the basis of merit.
 * It's true that communication usually bypasses the peer review system, but this has always been true. It's still considered necessary to go through the formal peer review process. Keep in mind that peer reviewed journals are as strong now as ever.
 * I'm taken aback by your claim that the secondary literature is no less error-prone than the primary literature, and I'm not really sure how to respond. Journals -- even peer reviewed journals -- are notoriously error prone. I recall reading in Nature some years ago that some 10% of 3-sigma results are eventually discredited. Review articles have the advantage of being written after some degree of consensus and concordance have developed in a field, and these types of problems have been sorted out. And that's exactly what I was getting at.
 * I think the difference in our ideas is that you think of science as a process in which old ideas become obsolete, whereas I see science as a process of iterative clarification on ideas that were previously mostly correct. I'd encourage you read Kuhn's Structure if you haven't already, as well as Bauer's Scientific Literacy and the Myth of the Scientific Method, which is where the peer review filter originates.
 * The bottom line is that the material you're questioning is all grounded in reliable sources. I'm in favor of re-incorporating this.
 * --Brian Brondel 15:44, 18 June 2006 (UTC)


 * I've read Kuhn and I'll take a look at Bauer, but most of my edits involve direct experience with how things work in the trenches. One thing that is significant is that the internet has significantly changed the practice of astronomy.  The other thing that has changed is the *massive* amount of new data that is coming in as a result of WMAP and some of the spacecraft out there.


 * NSF Grants are open to all comers, but if you are at an institution with a telescope you will have priority access to telescope time for reasons that have nothing to do with scientific merit (it's still peer reviewed, but the standard is much lower.) Also, NSF may accept grant proposals for non-program grants, but if the infrastructure isn't there, there isn't much of a chance of getting the proposal funded.


 * Also telescope time (and spacecraft budgets) is so precious that a lot of very good scientific proposals aren't funded for essentially political reasons. The situation is much less bad in astronomy than in particle physics, because the dollar amounts are lower, but I strongly disagree with the idea that the grant process is (or should be) an effective filter for scientific merit because of the high level of politics that inherent in the process.  This isn't a bad thing.  The decision as to whether to risk a space shuttle mission to fix Hubble or how much to fund planetary exploration is essentially a public policy decision, that IMHO shouldn't (and isn't) be made by scientists alone.  It's also the case that the bigger the decision, the less the decision does resemble (or should resemble) the standard peer review process, and the more the decision process resembles (and IMHO should resemble) the legislative process.


 * It is necessary to go through the peer review process for reputation sake, but the primary communications between astronomers is no longer peer reviewed. Things really changed in the 1990's with the Los Alamos Preprint Server.  People have always exchanged preprints, but the Los Alamos server changed things is that now it is impossible for someone in astronomy to do anything useful if they only thing that they read is peer-reviewed articles.


 * In astronomy, the secondary literature is notoriously unreliable because the latest information is not in the secondary literature. In the six months it takes to write a decent review article on the accelerating universe, it is very likely that there is going to be some significant piece of new data that totally invalidates much of the review article.


 * The other thing is that I can point to some review articles that clearly aren't functioning in the way Bauer mentions. There are some very nice articles in Annual Reviews which summarize "non-standard cosmologies" in either neutral or sympathetic ways.  They are excellent introductions to a different way of looking at things (which most astronomers think is nuts), but "error-correction" and "consensus-building" they are not.


 * The problem that I had with article was that it just doesn't fit with what I am seeing, and gives an impression of science that is far more authoritarian than what is actually the case. I do think that Kuhn is very valuable, and I'll need to read Bauer before commenting on his statements.

Roadrunner 16:43, 18 June 2006 (UTC)

--- Given the methods by which scientific progress happens and the potential vulnerability of science to faulty reasoning, it isn't hard to understand why scientific work has sometimes remained in error for many years &mdash; sometimes even centuries &mdash; before finally being corrected. Paradigms are very valuable to the scientists who work under them, and scientists rarely do their work with complete disinterest and neutrality, but with the specific goal of affirming their own ideas. This inherent and unavoidable bias is the reason an individual cannot successfully engage in science in isolation. The philosopher of science Thomas Kuhn, who is known for his study of the role of the paradigm in science, remarked that "once [a paradigm] has been adopted by a profession ... no theory is recognized to be testable by any quantitative tests that it has not already passed." Progress in science requires practitioners with a variety of viewpoints so that paradigms will be truly tested rather than uncritically reaffirmed.

(...)

To protect against bias in our understanding of the universe, an elaborate system of peer review has become established as a mechanism by which sloppy or incorrect work can be removed. The particular mechanisms of peer review used by modern scientists, named the "Knowledge Filter" by the chemist Henry Bauer, work by applying progressively higher standards to new research.

Unfiltered research begins as an idea. The idea is founded in previous research and the training and world-view or the researcher. The first stage of the filter is the scientist's desire for a good reputation and need for a job. Research must be approved by a panel of respected scientists in order to receive funding. In astronomy, large observatories have committees to allocate telescope time to the projects they perceive to be the best. This first step eliminates obviously wrong ideas from the body of scientific knowledge and allows other ideas to proceed towards acceptance.

Once research has progressed, it is presented to the scientific community again. This is the second filter. As the research is organized into a presentable from, errors are ironed out. The work is again subject to feedback by the community. Before a research paper can be submitted to a scientific journal for publication, it must be submitted to a knowledgeable scientist for criticism. In this step, scholarly publication is protected against poor experimental methods and incorrect conclusions. Work that progresses to this stage appears in primary literature, which reveals the very recent progress and direction of scientific work. The content of the primary literature is, to quote Bauer, "mostly not obviously wrong." It is still subject to scrutiny by the community for correctness and repeatability. The next filter ensures that research lives up to a standard of usefulness. Other scientists apply the results from publication to their own work. As they do this, they also test and extend the original work.

Research that proves itself as useful and influential will be incorporated into the secondary literature, where it will be read by a broader audience of scientists. The secondary literature is a synthesis of the results of recent research in a field. The body of secondary literature includes periodic reviews of progress, such as the Annual Review of Astronomy & Astrophysics, professional books, and other research summaries. In this filter, the opinions of many scientists about the merit of an idea are taken together. Since secondary literature is built upon research that has had time to mature and be tested, the ideas it expresses is more reliable than the primary literature.

As more time passes, the research described in the secondary literature is further corrected of errors. The work is used and reproduced by other scientists, and it begins to show concordance with other fields. When this has happened, the ideas have high credibility. This is the next filter, which brings the work into the realm of textbook science. The result of textbook science is a collection of ideas that are mostly very reliable. These ideas are slowly built up from a consensus among sciences, founded on agreement between theory and experiment and concordance with other theories.

H.H. Bauer
Looking through a summary of book, I'm pretty impressed by it, and I've got it on order. My main objection is that I'm not sure a description on how science was done in 1957 can be taken at face value in 2006. There have been some huge changes since then. There have been some pretty huge changes since 1990. Kuhn I think is different because he talked in terms of principles which have been the same since the time of the Greeks. Bauer's work seems to be less timeless.

The thing that I object the most to the original text was the statement of "textbook science." One of the reasons that I'm interested in Wikibooks is that I think it is likely to fundamentally change the way science is done from the way Bauer describes it.

Roadrunner 16:58, 18 June 2006 (UTC)


 * I hope you enjoy Bauer's book.


 * I take a different view of this whole thing. Science is, despite any pluralist or democratic ideals, in a sense an authoritarian endeavor, with peer review very much central to the whole thing. Peer review is what stands between crackpottery, silliness, individual conceit, etc., of scientists, and the popular media and textbooks. I consider this very much the leading opinion.


 * I'd also like to note that I wrote the text here on the basis of a survey of reading I undertook so that I could better understand the landscape of thought on method specifically for writing this module. This text is based on writings on method that have been very carefully thought out and well received by scholars of the philosophy of science. Those writings in turn are based on research combining the writings of a variety of scientists ranging from the Revolution to the present day on how they did their thing. And each step was filtered through the personal experiences of each writer "in the trenches." I consider this a much better way of producing knowledge. Wikibooks is just another medium.


 * --Brian Brondel 17:33, 18 June 2006 (UTC)


 * Then we have a fundamental philosophical disagreement here since I think science is one of the most anti-authoritarian institutions out there. One of the reason that I enjoy science so much is that you can go to a room full of people who are older and more senior than you are and say *YOU GUYS ARE ALL WRONG*, *THE TEXTBOOKS ARE WRONG*, *EVERYTHING YOU BELIEVE IS WRONG*, and if you have your logic and facts on your side, they'll start nodding their heads.


 * In the case of astronomy, the leading journals "Astrophysical Journal" in the US, "Astronomy and Astrophysics" in Europe, have a policy of basically publishing anything that isn't obviously stupid. ApJ has a 70% acceptance rate for manuscripts and some pretty nutty stuff makes it in there (the big bang never happened!!! black holes don't exist!!!!).  One reason that peer reviewers make their feedback transparent is so that you can judge the judges.  It has happens that a submitter looks at the referee comments, concludes that the referees are idiots and then resubmits the proposal in another journal.  You will get your work in the primary literature unless it is so bad that no one will publish it.


 * Also, my opinions on peer review are colored by the fact that I've been a referee for grant proposals. One thing that happens is that you quickly find out is that there so so many good proposals coming in, and so little money available that the grant process weeds out a lot of really good projects that you hope that someone else will fund.


 * Politics also rears its head in the process. This is a *GREAT* proposal.  Too bad our program budget doesn't cover this area, and what programs do get funded and to what amounts is generally a political decision in which there is little peer review (because it is ultimately not a scientific question).

Roadrunner 18:19, 18 June 2006 (UTC)


 * Those are good points. As you say, we're more discussing theory than book content right now, so I'll keep it brief.


 * My perspective is that textbooks should have the best science, rather than the latest science. If you go to tell a roomful of people how wrong they are, they might act polite while you give your speech, but they aren't going to go home and teach your ideas to their 101 class until you've gotten to the other side of the peer review process, which probably won't be so polite. You're certainly right about peer review of journals. That's part of Bauer's point, though -- each step in the process is successively stronger. Bauer describes primary literature as "mostly not obviously wrong." Politics does come into the review process, but I think of "we'd like to but we can't" to be separate from "we shouldn't at all." There's nothing wrong with setting an agenda when not all good science can be done.


 * It's been fun having this conversation with you. I'll let you have the last word if you like, but then I should be getting back to work. I hope to see you more at Astronomy in the future. --Brian Brondel 18:58, 18 June 2006 (UTC).

added section on peer review
I created a separate section on peer review which makes reference to Bauer's knowledge filter.

I agree that peer review is still central to science, and one thing that I wanted to do is to go into some detail about the process. One thing that I like about the peer review process is that it is very transparent. In other selection systems, you get a "NO!!!! GO AWAY" but in the peer review, the response is "well, we think that there are these problems, can you check A, B, and C, change the statement on B, and resubmit."

Roadrunner 17:45, 18 June 2006 (UTC)