Information Processing

 

Academics think of themselves as trailblazers, explorers — seekers of the truth. 

Any fundamental discovery involves a significant degree of risk. If an idea is guaranteed to work then it moves from the realm of research to engineering. Unfortunately, this also means that most research careers will invariably be failures at least if failures are measured via “objective” metrics like citations. 

Today we discuss the recent article from Mark Saroufim called Machine Learning: the great stagnation. We discuss the rise of gentleman scientists, fake rigor, incentives in ML, SOTA-chasing, "graduate student descent", distribution of talent in ML and how to learn effectively.

Machine Learning: The Great Stagnation 

The illustration above describes a global population of ~5k researchers whose papers were accepted to the leading 2019 conference in deep neural nets. To be precise they looked at ~700 authors of a randomly chosen subset of papers. There is also a more select population of individuals who gave presentations at the meeting. This is certainly not the entire field of AI, but a reasonable proxy for it. Global AI talent tracker:

For its December 2019 conference, NeurIPS saw a record-breaking 15,920 researchers submit 6,614 papers, with a paper acceptance rate of 21.6%, making it one of the largest, most popular, and most selective AI conferences on record. 

Key Takeaways 

1. The United States has a large lead over all other countries in top-tier AI research, with nearly 60% of top-tier researchers working for American universities and companies. The US lead is built on attracting international talent, with more than two-thirds of the top-tier AI researchers working in the United States having received undergraduate degrees in other countries.   

2. China is the largest source of top-tier researchers, with 29% of these researchers having received undergraduate degrees in China. But the majority of those Chinese researchers (56%) go on to study, work, and live in the United States. 

3. Over half (53%) of all the top-tier AI researchers are immigrants or foreign nationals currently working in a different country from where they received their undergraduate degrees.

Prediction: PRC share in all 3 categories will increase in coming decades as their K12, undergraduate, and graduate schools continue to improve, and their high-tech economy grows much larger. See Ditchley Foundation meeting: World Order today. 

These are excerpts from remarks I gave yesterday at a reception for new faculty.

Good afternoon and Welcome!

We are so pleased that you are here at Michigan State University. You have joined a great research university, at a very exciting time.

I’m told I only have 10 minutes in which to say something about the deep and varied research enterprise here at MSU. That’s only enough time for a high level overview, so let me start with some big picture numbers. Each year the National Science Foundation publishes its Higher Education Research and Development (or HERD) report on the total research expenditures of all US universities. MSU’s total HERD number has grown from about $500M to $700M in the last seven years. We’ve advanced faster than any other Big Ten university, and now rank 32nd in the US among all universities.

MSU is ahead of Rutgers, UT Austin, Illinois (UIUC), Purdue, Arizona, Maryland, Indiana, Iowa, ASU, Colorado (Boulder), and UVA.

Based on the HERD comparison data, MSU ranks 1st in the nation in combined Department of Energy and National Science Foundation research expenditures.

Almost all of the schools ranked above us (and many below) have major research hospitals. In those cases, the medical research component of the HERD total often exceeds the rest of campus combined. At MSU, about ~$100M of our total comes from NIH. We still have significant room to advance.

There are only a few schools without a major medical complex that rank above us -- let me mention two: UC Berkeley $771M (top public university in the US; home of Lawrence Berkeley National Lab) and MIT $952M (home of Lincoln Laboratory, a major defense research lab).

MSU, UC Berkeley, and MIT are all research powerhouses. But we are similar in another important way: all three are land-grant universities. As land-grant universities, we pride ourselves on making breakthroughs in basic research, and applying those breakthroughs to make life better for the entire world.

... MSU is home to the Facility for Rare Isotope Beams, a scientific user facility for the Office of Nuclear Physics in the Office of Science of the U.S. Department of Energy.

FRIB will be operational in 2021 and will deliver the highest intensity beams of rare isotopes available anywhere in the world. Estimates of the total investment in this project are roughly $1 billion dollars. Operated by MSU, FRIB will enable scientists to make discoveries about the properties of rare isotopes (which are unusual forms of the elements) in order to better understand the physics of nuclei, nuclear astrophysics, and the fundamental interactions of nature. It will also produce practical applications for society, including in medicine, homeland security, and industry.

... Another recent development is a new department called Computational Mathematics, Science, and Engineering or CMSE. This department was planned, authorized, and operational in only three years—quite a feat in academia. I often compare “startup time” (the fast pace at which things are accomplished in Silicon Valley) to “academic time” (i.e., nothing gets done, other than committee meetings, and a no-brainer project takes a decade to complete), but with CMSE this was a case of something on campus getting done in startup time. CMSE is one of very few such departments in the country -- it is focused on data science, machine learning, advanced computation and related applications, but is not a traditional CS department. It supports many of the new efforts on campus that require the analysis of large data sets and development of new tools and algorithms. Researchers in this department utilize datasets drawn from astrophysics, business analytics, mobile data, materials science, human and plant genomics, and many other areas. The department was conceived as fundamentally interdisciplinary -- bringing together experts in computation with subject matter experts in areas of science which are becoming increasingly reliant on data.

I can’t help mentioning a couple of big data examples related to my own research: we’ve created a compute resource with more than 500k human genomes, open to interested investigators on campus. All of the data is stored at our High Performance Computing Center or HPCC. Using this data, our collaboration demonstrated for the first time that machine learning applied to large genomic datasets could produce accurate predictors for complex human traits. We can now predict adult human height from genome alone, with accuracy roughly 1 inch. The predictor uses ~20k genetic variants distributed throughout the genome. Predictors of complex disease risk, for conditions such as heart disease, diabetes, schizophrenia, and breast cancer, have been developed and broadly replicated in out-of-sample tests. I recently participated in a meeting at No 10 Downing Street in the UK, to plan a project which will genotype 5 million individuals through their National Healthcare System. This is only the beginning for genomic Precision Medicine.

... If there is a problem -- tell us about it! -- whether it has to do with grant submissions, or startup incubation, or child care, food options on campus, your functional or dysfunctional department. We’re here to fix things, and to provide the best possible environment for your teaching, research, and creative activity.

Only one in a thousand people in our society have the privilege to engage full time in discovery -- in curiosity driven research -- for the benefit of humankind. You are part of that lucky one in a thousand, and we are here to help you succeed.

The bar has been set very high, but with the resources and new opportunities here at MSU, your potential is limitless.

My very best wishes to you all :-)

Remarks at a recent Michigan State University leadership meeting. MSU is currently #1 in the US in annual Department of Energy (DOE) and DOE + NSF (National Science Foundation) funding. There are ~30 institutions in the US with larger annual research expenditures than MSU, however in all but a few cases (e.g., MIT and UC Berkeley) this is due to a large medical research complex and significant NIH (National Institutes of Health) funding. I discuss MSU's strategy in this direction: a new biomedical research complex and new $450M McLaren hospital on our campus.

Federal R&D, which skews more toward basic research, is typically somewhat less than 1% of US GDP (~$100 billion per annum). See figure below.

WSJ: ... U.S.-based companies accounted for $329 billion of a record $781.8 billion in R&D spending tallied by PwC for the year ended June 30. While Chinese R&D investment came in at $61 billion, in 2010 that figure was just $7 billion, PwC said. Today, 145 Chinese companies are among the top 1,000 R&D spenders, up from 14 a decade ago.

... PwC’s figures don’t include private companies, however, which leaves out China’s state-owned monoliths and closely held Huawei Technologies Co., the world’s largest maker of telecommunications equipment. Huawei said it spent more than $13 billion on R&D last year.

Please help me fill this position! This search is a bit out of sync with the regular postdoc application process, so I need some help spreading the word.

Theoretical Physics Postdoc at Michigan State University 

Stephen Hsu, Vice-President for Research and Professor of Physics at MSU, anticipates filling a postdoctoral position to start in the summer or fall of 2016. The successful applicant will have broad interests in theoretical physics and good computational skills. In addition to research in particle physics and cosmology, there will be opportunities to work on problems in machine learning and computational genomics.

The High Energy Theory group at MSU currently consists of eight faculty members: Sekhar Chivukula, Jon Pumplin, Wayne Repko, Carl Schmidt, Elizabeth Simmons, Dan Stump, C.-P. Yuan and Stephen Hsu, as well as postdoctoral fellows and several graduate students. Ongoing research encompasses QCD theory and phenomenology, electroweak symmetry breaking mechanisms, supersymmetry and other beyond-the-standard-model scenarios, cosmology, and collider phenomenology. Recently, a new group of 3 theorists have been hired in the area of lattice QCD. The Physics/Astronomy Department at MSU has 60 faculty members; it has strong research programs in Condensed Matter Physics, Nuclear Physics, and Astronomy, in addition to High Energy Physics (http://www.pa.msu.edu/hep/hept.html).

See MSU Applicant Page http://www.hr.msu.edu/hiring/msujobs.htm , posting 2859 (PA). Applications should be uploaded to MSU’s online job application site, https://jobs.msu.edu and should include a CV, research plan and publication list. In addition, three letters of recommendation should be submitted electronically by the recommenders through this application system. Review of applications will begin immediately and will continue until the position is filled. MSU is an affirmative action, equal opportunity employer MSU is committed to achieving excellence through cultural diversity. The university actively encourages applications and/or nominations of women, persons of color, veterans and persons with disabilities.

MSU research ranked #6 among US universities in combined National Science Foundation + Department of Energy funding. Ahead of Michigan, Stanford, Caltech, and Berkeley. MIT is #1.

From 2015 AAU data and policy brief.

See also: US basic research down 25% in GDP terms (10 years)

"It's been only half jokingly said that today a third of GDP is attributable to quantum mechanics," -- former Lockheed CEO Norm Augustine.

I've heard the one third or 30% of GDP figure from time to time, but have never seen a detailed analysis. A list of modern technologies that arose from quantum mechanics would include: transistors, microprocessors, lasers, sophisticated chemistry and materials science, nuclear energy, memory chips, hard drive storage, LEDs, LCD displays, etc. These certainly account for a significant fraction of GDP.

Estimates of expenditures on communications and information technology alone in developed countries are typically in the 5-10% GDP range, which provides a lower bound. While the actual figure may be less than 30% it is certainly substantial. See here for a history of physics contributions to information technologies.

The huge (but poorly understood) impact of quantum mechanics on modern life is an example of the tremendous long term impact of fundamental research. There is every reason to think that increased world research expenditures would enhance productivity and quality of life, with very high ROI. However, there is little careful thinking about the "right" level of research investment as a fraction of GDP. Instead, we get:



Perhaps the most amazing aspect of the history of quantum mechanics is not its technological and practical impact, but rather how it led to deep changes in how we think about the universe. See, e.g., Two slit experiment, Bell and GHZ, Weinberg on quantum foundations, and On the origin of probability in quantum mechanics.

Early pioneers doubted whether humans were smart enough to understand quantum physics:

[Wigner] Until 1925, most great physicists, including Einstein and Planck, had doubted that man could truly grasp the deepest implications of quantum theory. They really felt that man might be too stupid to properly describe quantum phenomena. ... the men at the weekly colloquium in Berlin wondered "Is the human mind gifted enough to extend physics into the microscopic domain ...?" Many of those great men doubted that it could.

Quantum mechanics, which made possible the modern age, is nevertheless only understood by at most a fraction of a percent of the population. See also Psychometric thresholds for physics and mathematics, Chomsky: genetic barriers to scientific progress, and Beyond Human Science.

Beyond Human Science: [This Ted Chiang short story envisions a future in which science has become the province of genetically enhanced "metahumans" -- leaving non-enhanced humans to gape from the sidelines.]

... imagine if research offered hope of a different intelligence-enhancing therapy, one that would allow individuals to gradually "up-grade" their minds to a metahuman-equivalent level. Such a therapy would offer a bridge across what has become the greatest cultural divide in our species' history ...

We need not be intimidated by the accomplishments of metahuman science. We should always remember that the technologies that made metahumans possible were originally invented by humans, and they were no smarter than we.

Basic research, whose applications may be decades in the future, is an uncertain investment for any single entity (e.g., corporation), even if it is an essential public good for the long term advancement of civilization. Consequently, basic research is mostly done at universities and government labs. Indeed, the vast majority of research in the US is led by professors and takes place on campus. Unfortunately, this crucial aspect of the mission of universities is least understood by their broad constituency.

Boosters, alumni, parents, and advocates should note that the research prowess of a great university is a large component of its institutional prestige: nearly all of the most prestigious universities in the world, those that attract the brightest and most able (and, ultimately, most successful) students, are world class research institutions.

Nicholas Lemann writes in the Chronicle of Higher Education.

Chronicle: ... building on the foundation laid by the establishment of The Johns Hopkins University, in 1876, American higher education has embraced the idea of the research university as its most cherished aspiration. Today there are about 300 American universities that confer doctoral degrees, far more than envisioned by the original proselytizers for importing the research-university model from Germany to the United States. And that number understates the importance of the model, because the core members of the faculty and senior administration at hundreds more institutions hold doctoral degrees and operate within the academic tenure system that lies at the heart of the way research universities are run.

For many people who have spent their lives working in higher education, mass higher education and research universities make for a perfect fit: Together they express both the public service and the intellectual ambitions of educators. And during most of the 20th century, especially the years between 1950 and 1975, the two big ideas grew and flourished in tandem.

But they aren’t the same idea. Mass higher education, conceptually, is practical, low cost, skills oriented, and mainly concerned with teaching. It caught on because state legislatures and businesses saw it as a means of economic development and a supplier of personnel, and because families saw it as a way of ensuring a place in the middle class for their children. Research universities, on the other hand, grant extraordinary freedom and empowerment to a small, elaborately trained and selected group of people whose mission is to pursue knowledge and understanding without the constraints of immediate practical applicability under which most of the rest of the world has to operate. Some of their work is subsidized directly by the federal government and by private donors, but they also live under the economic protection that very large and successful institutions can provide to some of their component parts. ...

See you in Seattle.

Conference site. Agenda. Speakers.

This July 15 will mark the start of Microsoft Research’s fourteenth annual Faculty Summit at the Microsoft Conference Center in Redmond, Washington. More than 400 academic researchers from 200 institutions and 29 countries will join Microsoft Research to assess and explore today’s computing opportunities. This year, Bill Gates will join us to set the tone of the summit in a conversation on the topic of “Innovation and Opportunity—the Contribution of Computing to Improving Our World.”

Also delivering keynote presentations this year:

Doug Burger will discuss how changes in the hardware ecosystem will disrupt computer science.
Peter Lee and Jeannette Wing will examine how basic research helps everyone.
Clay Shirky will explore user-centric approaches to data.

Sessions covering topics ranging from “Prediction Engines” and “Big Data Platforms” to “Deep Machine Learning” and “Quantum Computing” adorn the summit agenda and will foster rich and engaging discussions.

You can watch it live.

Economics Shapes Science by Paula Stephan.

At a time when science is seen as an engine of economic growth, Paula Stephan brings a keen understanding of the ongoing cost-benefit calculations made by individuals and institutions as they compete for resources and reputation. She shows how universities offload risks by increasing the percentage of non–tenure-track faculty, requiring tenured faculty to pay salaries from outside grants, and staffing labs with foreign workers on temporary visas. With funding tight, investigators pursue safe projects rather than less fundable ones with uncertain but potentially path-breaking outcomes. Career prospects in science are increasingly dismal for the young because of ever-lengthening apprenticeships, scarcity of permanent academic positions, and the difficulty of getting funded.

Working paper version of the book.

From the abstract: Scientific research has properties of a public good; there are few monetary incentives for individuals to undertake basic research and the conventional wisdom is that the market, if left to its own devices, would under-invest in research in terms of social benefits relative to social costs. (emphasis mine)...

From the conclusions: ... In one sense, U.S. universities behave like high-end shopping malls. They are in the business of building state-of-the art facilities and a reputation that attracts good students and faculty. They then turn around and “rent” the facilities to faculty in the form of indirect costs on grants and the buy-out of salary. Faculty, in turn, create research programs, staffing them with graduate students and postdocs, who contribute to the research enterprise by their labor and the fresh ideas that they bring, but who can also be easily downsized, if and when times get tough. Universities leverage these outcomes into reputation. The amount of funding universities receive, as well as the citations and prizes awarded to their faculty, determine their peer group—the club to which they belong. They also attract donations and students and affect the university’s ranking.

See also this Nature commentary

Research efficiency: Perverse incentives

Paula Stephan
Nature 484, 29–31 (05 April 2012) doi:10.1038/484029a

Counterproductive financial incentives divert time and resources from the scientific enterprise. We should spend the money more wisely, says Paula Stephan.

Stephan notes the huge increases in biomedical research budgets in recent decades, and relative declines in funding for physical science. She wonders whether this is for the better, and so do I.