You can think about technology breakthroughs as a series of compounding tricks. Trick development is discontinuous, which is why technological progress can surprise us.
Take these quotes for example:
Anyone who expects a source of power from the transformation of these atoms is talking moonshine.
I feel absolutely sure-- well, nearly sure-- that it will not be possible to convert matter into energy for practical purposes... It is like shooting birds in the dark in a country where there only are few birds.
Just 10 years later, we had our first practical application of nuclear power (the American atomic bombs.) Does this mean Einstein or Rutherford were shortsighted? Not quite. Manipulating atoms is a very hard problem, and Einstein's analogy was accurate. There were some ways that we knew how to manipulate atoms, but never at the scale necessary to harness nuclear power.
But we discovered a trick.
In a specific isotope of uranium, a neutron collision will create two neutrons to create self-sustaining reaction. Instead of having to direct particles into specific places millions upon millions of times, you only needed to do it once! Then you could control the parameters of the reaction macroscopically using things like control rods.
This is an example of a trick. A trick is a clever application of a scientific principle in an unexpected way.
Tricks help catch up practical applications to scientific understanding. In human history, it is often the case that fundamental science has created an understanding of something, but we aren't able to apply it in practice due to engineering constraints.
Exploiting a scientific principle might look very hard or even impossible before a trick is discovered. Nuclear power is one example, but my favorite is the New York Times' prediction on flying machines. They said it'd take 1-10 million years to develop a flying machine, and the Wright brothers flew two months later!
As a footnote, these predictions aren't as bad as they look in retrospect. Flying machine skeptics and Einstein did something very rational: they took the state of the art and projected it out. Without the trick, they would have been right -- the way to flight might have been an incredibly long chain of incremental innovations, which might have taken decades to accomplish.
What correlates with finding tricks? Two things stick out to me:
Creating tricks involves mutating the system that you're working with. If you don't understand all of the parts intimately, and how they interact, you'll have a hard time finding tricks.
Historical knowledge is quite helpful. The state-of-the-art is just a snapshot that exists as a result of a history of many attempts. Without knowing the history, it's easy to walk down paths that have already been explored.
Tricks involve creativity, and conventionality tend to kill creativity. Tricks often come from places completely outside the establishment of an industry, often at intersections of all kinds.
The above criterion are not guarantees, just nudges. Tricks are finnicky business and resist guarantees.
Technology optimists project endless human greatness powered by technology development. For example, some technology optimists claim that if we escape extinction, we'll eventually become intergalactic. Unfortunately, it seems like interstellar travel will be incredibly hard, and all of the near-term technologies seem to suck.
Technology optimists will say that's OK, though! All we need to do is find a trick that'll allow us to create an Alcubierre drive, or a wormhole, or use antimatter rockets, and we'll be good. We haven't seriously been looking for these tricks, because we're too busy getting to Mars. But 400 years from now, who knows what kind of tricks we'll have found?
Well, this sounds good, but tricks are discontinuous, and no one knows when they'll come, or if they'll come at all. Maybe physicists will figure out a trick that lets us inferface with manipulate quarks, and a wave of "quark technology" will crash over society and change it entirely. Or maybe this will be impossible forever. We have no clue either way right now-- it's pure science fiction.
Even if we think an area might be “ripe” for trick-finding (e.g. not explored by the human engineering machine,) there are no guarantees we’ll find a trick. We understood the scientific principles behind superconductors and fusion for the past 40 years, and many human lifetimes were spent trying to find the trick that’d allow them to be used easily. We never accomplished cold fusion or warm superconductors, and we still have no clue if these things are possible at all (it’s assumed not.)
That's the question science fictionists have to contend with. Will the tricks run out and doom us to stagnation?
The time period that we have lived through has priviledged us. We've discovered so many damn tricks in the past 200 years that we take it for granted that there will always be a trick that exists.
But it is important for technology optimists to remember:
We don't know what tricks exist, and when we'll find them.
If your projection on progress includes humanity finding a trick, it's likely to be wrong. If your projection is on a long-enough timeframe and doesn't include humanity finding a trick somewhere, it's likely to be wrong too.
That's why fusion projections have been overeager (it's been 15-20 years away for the last 60 years). It was assumed we'd find a trick to solve certain engineering problems, and if you squinted your eyes, it seemed easy to project that the human engineering machine would figure it out. But that's the thing with tricks: they resist projections.