Sept 3, 2022

Years after shuttle, NASA rediscovers the perils of liquid hydrogen

"Every time we saw a leak, it pretty quickly exceeded our flammability limits."

NASA's Space Launch System rocket at LC-39B on September 1, 2022.

KENNEDY SPACE CENTER, Fla.—America's space agency on Saturday sought to launch a rocket largely cobbled together from the space shuttle, which itself was designed and built more than four decades ago.

As the space shuttle often was delayed due to technical problems, it therefore comes as scant surprise that the debut launch of NASA's Space Launch System rocket was scrubbed a few hours before its launch window opened. The showstopper was an 8-inch diameter line carrying liquid hydrogen into the rocket. It sprang a persistent leak at the inlet, known as a quick-disconnect, leading onboard the vehicle.

Valiantly, the launch team at Kennedy Space Center tried three different times to stanch the leak, all to no avail. Finally at 11:17 am ET, hours behind on their timeline to fuel the rocket, launch director Charlie Blackwell-Thompson called a halt.

What comes next depends on what engineers and technicians find on Monday when they inspect the vehicle at the launch pad. If the launch team decides it can replace the quick-disconnect hardware at the pad, it may be an option to perform a partial fueling test to determine the integrity of the fix. This may allow NASA to keep the vehicle on the pad ahead of the next launch. Alternatively, the engineers may decide the repairs are best performed inside the Vehicle Assembly Building and roll the rocket back inside.

Due to the orbital dynamics of the Artemis I mission to fly an uncrewed Orion spacecraft to the Moon, NASA will next have an opportunity to launch from September 19 to October 4. However, making that window would necessitate fixing the rocket at the pad and then getting a waiver from the US Space Force, which operates the launch range along the Florida coast.

At issue is the flight termination system, which is powered independently of the rocket, with batteries rated for 25 days. NASA would need to extend that battery rating to about 40 days. The space agency is expected to have those discussions with range officials soon.

If the rocket is rolled back to the Vehicle Assembly Building, which would be necessary to service the flight termination system or perform more than cursory work at the launch pad, NASA has another Artemis I launch opportunity from October 17 to October 31.

A tiny, tiny element

The space shuttle was an extremely complex vehicle, mingling the use of solid-rocket boosters—which are something akin to very, very powerful firecrackers—along with exquisitely built main engines powered by the combustion of liquid hydrogen propellant and liquid oxygen to serve as an oxidizer.

Over its lifetime, due to this complexity, the shuttle on average scrubbed nearly once every launch attempt. Some shuttle flights scrubbed as many as five times before finally lifting off. For launch controllers, it never really got a whole lot easier to manage the space shuttle's complex fueling process, and hydrogen was frequently a culprit.

Hydrogen is the most abundant element in the universe, but it is also the lightest. It takes 600 sextillion hydrogen atoms to reach the mass of a single gram. Because it is so tiny, hydrogen can squeeze through the smallest of gaps. This is not so great a problem at ambient temperatures and pressures, but at super-chilled temperatures and high pressures, hydrogen easily oozes out of any available opening.

To keep a rocket's fuel tanks topped off, propellant lines leading from ground-based systems must remain attached to the booster until the very moment of launch. In the final second, the "quick-disconnects" at the ends of these lines break away from the rocket. The difficulty is that, in order to be fail-safes in disconnecting from the rocket, this equipment cannot be bolted together tightly enough to entirely preclude the passage of hydrogen atoms—it is extremely difficult to seal these connections under high pressure and low temperatures.

NASA, therefore, has a tolerance for a small amount of hydrogen leakage. Anything above a 4 percent concentration of hydrogen in the purge area near the quick disconnect, however, is considered a flammability hazard. "We were seeing in excess of that by two or three times that," said Mike Sarafin, NASA's Artemis I mission manager, of Saturday's hydrogen leak. "It was pretty clear we weren’t going to be able to work our way through it. Every time we saw a leak, it pretty quickly exceeded our flammability limits."

Twice, launch controllers stopped the flow of hydrogen into the vehicle in hopes that the quick-disconnect would warm a little bit. They hoped that, when they restarted slowly flowing cryogenic hydrogen onboard the rocket, the quick-disconnect would find a tighter fit with the booster. It did not. Another time they tried applying a significant amount of pressure to re-seat the quick disconnect.

NASA officials are still assessing the cause of the leak, but they believe it may have been due to an errant valve being opened. This occurred during the process of chilling down the rocket prior to loading liquid hydrogen. Amid a sequence of about a dozen commands being sent to the rocket, a command was sent to a wrong valve to open. This was rectified within 3 or 4 seconds, Sarafin said. However, during this time, the hydrogen line that would develop a problematic quick-disconnect was briefly over-pressurized.

Deferring to the experts

So why does NASA use liquid hydrogen as a fuel for its rockets if it is so difficult to work with and there are easier-to-handle alternatives such as methane or kerosene? One reason is that hydrogen is a very efficient fuel, meaning that it provides better "gas mileage" when used in rocket engines. However, the real answer is that Congress mandated that NASA continue to use space shuttle main engines as part of the SLS rocket program.

In 2010, when Congress wrote the authorization bill for NASA that led to the creation of the Space Launch System, it directed the agency to "utilize existing contracts, investments, workforce, industrial base, and capabilities from the Space Shuttle and Orion and Ares 1 projects, including ... existing United States propulsion systems, including liquid fuel engines, external tank or tank related capability, and solid rocket motor engines."

During a news conference on Saturday, Ars asked NASA Administrator Bill Nelson whether it was the right decision for NASA to continue working with hydrogen after the agency's experience with the space shuttle. In 2010, Nelson was a US Senator from Florida and ringleader of the space authorization bill alongside US Sen. Kay Bailey Hutchison, of Texas. "We deferred to the experts," Nelson said.

By this Nelson meant that the Senate worked alongside some officials at NASA, and within industry, to design the SLS rocket. These industry officials, who would continue to win lucrative contracts from NASA for their work on shuttle-related hardware, were only too happy to support the new rocket design.

Among the idea's opponents was Lori Garver, who served as NASA's deputy administrator at the time. She said the decision to use space shuttle components for the agency's next-generation rocket seemed like a terrible idea, given the challenges of working with hydrogen demonstrated over the previous three decades.

"They took finicky, expensive programs that couldn't fly very often, stacked them together differently, and said now, all of a sudden, it's going to be cheap and easy," she told Ars in August. "Yeah, we've flown them before, but they've proven to be problematic and challenging. This is one of the things that boggled my mind. What about it was going to change? I attribute it to this sort of group think, the contractors and the self-licking ice cream cone."

Now, NASA faces the challenge of managing this finicky hardware through more inspections and tests after so many already. The rocket's core stage, manufactured by Boeing, was shipped from its factory in Louisiana more than two and a half years ago. It underwent nearly a year of testing in Mississippi before arriving at Kennedy Space Center in April 2021. Since then, NASA and its contractors have been assembling the complete rocket and testing it on the launch pad.

Effectively, Saturday's "launch" attempt was the sixth time NASA has tried to completely fuel the first and second stages of the rocket, and then get deep into the countdown. To date, it has not succeeded with any of these fueling tests, known as wet dress rehearsals. On Saturday, the core stage's massive liquid hydrogen tank, with a capacity of more than 500,000 gallons, was only 11 percent full when the scrub was called.

Perhaps the seventh time will be a charm.

NASA's Space Launch System Rocket at LC-39B, preparing to lift off at 8:33 am ET on August 29th, 2022.

President Eisenhower signed the law establishing the National Aeronautics and Space Administration on July 29, 1958. At the time, the United States had put about 30 kg of small satellites into orbit. Less than 11 years later, Neil Armstrong and Buzz Aldrin landed on the Moon.

President Obama signed a NASA Authorization Act on October 11, 2010. Among its provisions, the law called on NASA to create the Space Launch System rocket and have it ready for launch in 2016. It seemed reasonable. At the time, NASA had been launching rockets, including very large ones, for half a century. And in some sense, this new SLS rocket was already built.

The most challenging aspect of almost any launch vehicle is its engines. No problem—the SLS rocket would use engines left over from the space shuttle program. Its side-mounted boosters would be slightly larger versions of those that powered the shuttle for three decades. The newest part of the vehicle would be its large core stage, housing liquid hydrogen and oxygen fuel tanks to feed the rocket's four main engines. But even this component was derivative. The core stage's 8.4-meter diameter was identical to the space shuttle's external tank, which carried the same propellants for the shuttle's main engines.

Alas, construction wasn't that easy. NASA's SLS rocket program has been a hot mess almost from the beginning. It has been efficient at precisely one thing, spreading jobs around to large aerospace contractors in the states of key congressional committee leaders. Because of this, lawmakers have overlooked years of delays, a more than doubling in development costs to above $20 billion, and the availability of far cheaper and reusable rockets built by the private sector.

So here we are, nearly a dozen years after that authorization act was signed, and NASA is finally ready to launch the SLS rocket. It took the agency 11 years to go from nothing to the Moon. It has taken 12 years to go from having all the building blocks for a rocket to having it on the launch pad, ready for an uncrewed test flight.

I have decidedly mixed emotions.

With the launch just days away, I am incredibly happy for the people at NASA and the space companies that have worked hard, cut through the bureaucracy, managed thousands of requirements, and actually got this rocket built. And I'm eager to see it fly. Who doesn't want to watch a huge, Brobdingnagian rocket consume millions of kilograms of fuel and break the surly bonds of Earth's gravity?

 

On the less happy side, it remains difficult to celebrate a rocket that, in many ways, is responsible for a lost decade of US space exploration. The financial costs of the program have been enormous. Between the rocket, its ground systems, and the Orion spacecraft launching on top of the stack, NASA has spent tens of billions of dollars. But I would argue that

the opportunity costs are higher. For a decade, Congress pushed NASA's exploration focus toward an Apollo-like program, with a massive launch vehicle that is utterly expended, using 1970s technology in its engines, tanks, and boosters.

 

Effectively, NASA was told to look backward when this country's vibrant commercial space industry was ready to push toward sustainable spaceflight by building big rockets and landing them—or storing propellant in space or building reusable tugs to go back and forth between the Earth and Moon. It's as if Congress told NASA to keep printing newspapers in a world with broadband Internet.

It didn't have to be this way. In fact, a handful of visionary space policy leaders tried to stop the wastefulness but were beaten back by the defense industry and its allies in Congress.

For me personally, this is also the end of an era. In many ways, this rocket has mirrored my career as a journalist and writer covering the space industry. So as we approach this momentous launch, I want to tell the story—the real story—about where this came from and where it's going. I will make the case that the SLS rocket is the worst thing, and perhaps simultaneously the best thing, to ever happen to NASA.

I believe this story can still have a happy ending.

The side-mounted boosters on the SLS rocket are derived from the space shuttle program

Back to the beginning

I have written about the space program for two decades, dating back to the space shuttle Columbia disaster in February 2003. This tragedy forced Washington, DC-based space policymakers to reckon with the end of the space shuttle program and decide what NASA would do afterward.

The resulting search for a credible deep space exploration program has dominated NASA's human spaceflight programs for the last two decades, and it ultimately brought us to the SLS rocket and the Artemis Moon Program. There have also been two other very important macro trends. One is the rise of commercial spaceflight and its profusion of rockets and satellites. SpaceX, founded in 2002, is the exemplar of this new space movement. The other sweeping change has been the ascension of China's space program, which flew its first astronaut into orbit in 2003.

These three events—the demise of Columbia, the founding of SpaceX, and China's first human spaceflight—mark the beginning of the modern spaceflight era. I have had a privileged front-row seat to the changes wrought by these events over the last two decades, and it has been fascinating to watch the US human spaceflight enterprise finally move on from a model that was more or less established in the 1960s and 1970s into a modern, dynamic, and innovative era. But it hasn't been easy to get there.

After the Columbia disaster, President Bush set broad goals for NASA: completing the International Space Station by 2010 and retiring the aging space shuttle, flying a deep space crew vehicle (which would later be named Orion) with astronauts by 2014, and returning humans to the Moon by 2020 with the Constellation Program. NASA finished the ISS and retired the shuttle by 2011, but it whiffed on the other goals. The reasons are complicated, but I would argue that the biggest impediment came from large aerospace contractors such as Boeing, Lockheed Martin, and Northrop Grumman insisting on getting large pieces of the funding pie—and having the Congressional influence to get their way.

The contractors—it's no stretch to call them Big Aerospace, as they all rank among the top US defense contractors—won their first battle in 2005. As NASA looked for the best way to get astronauts back into deep space, the choice was between a transportation system derived from the space shuttle or working with the existing Atlas and Delta rockets used by the US military, the so-called Evolved Expendable Launch Vehicles. Ultimately, NASA decided to build rockets using its own shuttle components. Conveniently, this plan promised the most development money for the big contractors.

In an analysis of this approach, noted lunar scientist Paul Spudis wrote that while the Constellation architecture might eventually work, it required far more funding than was available. NASA's unwillingness to adopt possible alternatives, he concluded, became "a programmatic straitjacket." Spudis' preferred alternative was to use the commercially available Atlas and Delta rockets and to develop propellant depots in orbit to refuel them for lunar missions.

Spudis was right. The Bush administration never fought for the billions of additional dollars needed for Constellation, and Congress was in no hurry to see real progress. Predictably, a couple of years later, the Constellation development programs were badly behind schedule and over budget.

President Obama's election in 2008 set the stage for the second battle. He appointed aerospace executive Norm Augustine to lead a review of NASA's human exploration efforts. The first sentence of this 156-page report was succinct, and it encapsulated the problem: "The US human spaceflight program appears to be on an unsustainable trajectory."

Accordingly, the Obama administration sought a sustainable path forward and looked to the commercial space industry for help. By then, SpaceX had successfully launched its Falcon 1 rocket for the first time and was well into the development of the larger Falcon 9 rocket. Amazon CEO Jeff Bezos was investing hundreds of millions of dollars into building a large rocket at Blue Origin. And United Launch Alliance was considering options to modernize its Atlas and Delta rockets.

In its fiscal year 2011 budget request, the Obama administration sought to cancel the Ares I and Ares V rockets, as well as Orion, and instead spend $3.1 billion to fund the development of a future heavy-lift launch system. Essentially, this money would have been competed for by private industry, allowing them to perform research and development on propulsion technologies. The goal was to finalize the designs of new commercial rockets by 2015 and start to build them thereafter through a public-private partnership. Had this program happened, SpaceX Starship and Blue Origin's New Glenn vehicles might already be flying regularly.

Congress was aghast, of course, because this plan would have reduced its control of funding by letting private companies compete for contracts. The blowback to Obama's proposals, from both Republicans and Democrats, was tremendous.

"Given that we proposed terminating contracts worth billions of dollars, the negative response to the budget was not surprising," wrote NASA's deputy administrator at the time, Lori Garver, in her book Escaping Gravity. "Since NASA hadn't been a part of a larger national agenda for decades, its standard bearers included self-selected senators and representatives with existing contracts and jobs in their districts whose primary interest was often maintaining the status quo."

Congress, who had the power of the purse, struck back. It grudgingly allowed a few hundred million dollars for NASA to fund the "commercial crew program" that ultimately led to the development of SpaceX's Crew Dragon and Boeing Starliner vehicles. In turn, it got in excess of $3 billion a year for the Orion spacecraft and a new rocket, the Space Launch System. This time, Congress did not mess around. It wrote the rocket's authorization to ensure that its preferred contractors, including Boeing and Northrop Grumman, got big pieces of the action.

Big Aerospace had won its second large battle in five years over NASA's deep space exploration plans, and it has been a lasting victory. All told, NASA has now spent about $50 billion on Ares, Orion, and SLS hardware and ground systems. But ultimately, this may be a Pyrrhic victory.

“SLS is real”

In 2009, I was starting to cover space full-time for the Houston Chronicle, in addition to reporting about science and the region's myriad hurricanes. As the 40th anniversary of the Apollo 11 landing on the Moon approached, I called Chris Kraft, NASA's first and most legendary flight director, for whom Mission Control in Houston was named. I collected some quotes, and toward the end of the interview, we realized we were practically neighbors in Clear Lake.

He invited me to come by sometime, and we struck up a friendship over the next decade. (He died at the age of 95, just six days after the 50th anniversary of the Apollo 11 landing.) Every couple of months, I would drive over to his house in the afternoon, following his morning tee time, and we would sip Coca-Cola in his upstairs den. By then, Kraft had been waiting decades for something to happen with humans in deep space since Apollo, and nothing had. So he was frustrated.

Kraft liked the initial idea behind the Constellation Program but saw the problems it ran into when the promised funding ran short. By the time Congress told NASA to build the SLS rocket, he wasn't having any of it.

"It's very expensive to design, it's very expensive to develop," he told me nearly a decade ago. "When they actually begin to develop it, the budget is going to go haywire. They're going to have all kinds of technical and development issues crop up, which will drive the development costs up. Then there are the operating costs of that beast, which will eat NASA alive if they get there. They're not going to be able to fly it more than once a year, if that, because they don't have the budget to do it. So what you've got is a beast of a rocket, that would give you all of this capability, which you can't build because you don't have the money to build it in the first place, and you can't operate it if you had it."

His arguments, which have proven entirely correct, convinced me at the time. I started to make my name as a space journalist writing critically—and I would argue incisively—about the SLS program at a time when many space journalists played it more neutral. In 2014, I wrote a seven-part series in the Chronicle titled "Adrift," with the central thesis being that NASA's deep space program was off course.

To the extent that the series is remembered today, it's because of a quote given to me by then-NASA Administrator Charles Bolden. I asked him why NASA needed a heavy-lift rocket when SpaceX had started to build the Falcon Heavy, which had about 70 percent of the lift capacity of the SLS booster, at less than 10 percent the cost. (An expendable Falcon Heavy costs about $150 million. A single launch of the SLS rocket costs at least $2 billion per year.)

"Let’s be very honest,” Bolden said in response. "We don’t have a commercially available heavy-lift vehicle. The Falcon 9 Heavy may some day come about. It’s on the drawing board right now. SLS is real."

The Falcon Heavy flew for the first time in 2018 and is now commercially available. Bolden's comment has become a meme.

What happened with the SLS program in the years since "Adrift" should surprise no one. The program's funding level has increased, and its development has been stretched out. The cost-plus contracting mechanism NASA used to fund development of the vehicle incentivizes Boeing and other contractors to spend more time and money working on a vehicle because they get more fees for a longer period. The SLS was sold to the public as a rocket that would be developed on time and on budget because it was derivative of the shuttle and used heritage hardware. Its cost-plus contract ensured the opposite occurred.

For all of these reasons, but particularly because of its opportunity costs and backward looking nature, the SLS rocket is just about the worst thing to happen to NASA in the last six decades.

Rise of commercial space

For Big Aerospace, however, the party may be ending. One of the sweeping changes we've seen in the aerospace industry during the last two decades is the rise of new players. SpaceX is most notable among them, and it's certainly the most disruptive, but it's far from the only entrant.

Bolden's comment is amusing in hindsight because Falcon Heavy beat the SLS rocket to orbit by at least four and a half years. But it's also telling that SpaceX's next-generation rocket, the Starship vehicle, also very nearly beat the SLS rocket to orbit. If Starship reaches even half its potential, it will exceed the SLS rocket in every possible way. It's more powerful, far less expensive, and fully reusable, and it can launch hundreds of times a year—not once.

Those who have focused on the "space race" this year between SLS and Starship have missed the point. The real question is not which of the two super heavy-lift rockets launches first. Rather, it's "how many Starships will launch between the first and second flights of the SLS rocket?"

Nominally, the second SLS mission is due to fly in 2024, but it will probably slip into 2025. Conceivably, Starship could launch a dozen times between now and then. Maybe 30 times. Perhaps more. More than a decade ago, the Augustine commission said NASA should find a sustainable trajectory. Low-cost, reusable rockets are, quite clearly, NASA's sustainable trajectory.

And NASA is already buying into this future. Since letting the SLS and Orion contracts, it has almost exclusively awarded "fixed-price" contracts for other elements of its exploration programs. Through these contracts, NASA has moved more toward buying services from the US commercial space industry as opposed to providing a top-level design and controlling every step of the development process.

"This has been a really tough thing," said Kathy Lueders, who leads operational human exploration for NASA, at the ASCENDx Conference in Houston in April. "NASA has had a very hard time going from saying 'I'm the one doing it" to 'We are doing it.'"

SpaceX's Falcon Heavy rocket has 70 percent the lift capacity of the SLS rocket, at less than a tenth of the cost.

But that effort has been worth it. Lueders explained that NASA is working with industry to create as many types of partnerships as possible to meet the demands of its various missions. The focus is on helping the industry understand what NASA needs and then trying to buy services that those companies can also sell to other space customers. This incentivizes private industry to self-invest in these technologies and deliver low-cost, timely products.

"We do that because we feel like this is important for us as a nation to maintain our leadership in space," Lueders said. "Every nation in the world is envious of the way we have created these new relationships with our commercial industry."

NASA proved this in April 2021 when the agency selected SpaceX's Starship to serve as the "Human Landing System" for the Artemis Moon Program. This was almost unimaginable even a couple of years ago, but now the ambitious Starship vehicle is firmly on NASA's critical path back to the Moon. For now, Starship will merely ferry astronauts down to the Moon from lunar orbit and back up. But it's not too difficult to see astronauts eventually launching from Earth in Starship and returning there in the same way. If Starship can be shown to be safe and effective—still ifs, to be sure—it is far superior to SLS and Orion in cost, reusability, and cadence.

The irony is that Congress has agreed to fund Starship at the level of $2.9 billion for development and a couple of lunar missions. That's less than what NASA spends annually on SLS and Orion development costs, but it's still significant. And more importantly, in funding Starship, Congress is funding the rocket that will one day almost certainly put its beloved SLS booster out of business.

Where we go from here

The reality is that commercial space has already won the rocket wars. The industry trend appears to be irrevocably headed toward reuse. Beyond Starship, Blue Origin is building a large rocket, New Glenn, that is eventually intended to have a fully reusable first and second stage. Relativity Space is building the fully reusable large Terran R rocket. The legacy rocket company United Launch Alliance, which is co-owned by Boeing and Lockheed Martin, is studying the reuse of main engines on its new Vulcan rocket. Even Europe—stodgy old institutional Europe!—is looking at developing a reusable heavy-lift rocket in the next decade.

The reason I say that SLS was one of the best things to happen to NASA is simple. In hindsight, it's the political price the agency had to pay to bring Congress on board with a real deep space exploration program. The Artemis Program—which certainly is NASA's most "real" human deep space exploration program since Apollo—was created by Vice President Mike Pence and then-administrator Jim Bridenstine about three years ago. Congress only went along because Bridenstine promised to use the SLS rocket for all human launches to the Moon.

Since then, Congress has increasingly funded other elements needed to make Artemis real, including SpaceX's lunar lander and spacesuits for the lunar surface. For now, Congress will not buy into Artemis if NASA moves away from the SLS rocket and Orion spacecraft. So thanks to these programs, NASA has a bona fide plan to send humans back to the Moon—and possibly, someday, to Mars.

What happens next comes down to execution. If the SLS rocket works well, great. It can serve as an interim heavy lifter while SpaceX and its brethren continue to work on their large, game-changing launch systems. As those come online, the use of the SLS and eventually Orion will become obsolete. That could happen in three years. Or five. Or 10. It really doesn't matter. At some point, NASA will find itself with a full-fledged deep space program and no need for congressionally mandated rockets. It will happen—that Starship has sailed.

So thanks, big orange rocket, for greasing the skids. Good luck on climbing that gravity hill next week. We'll all be watching and cheering you on, though not all of us will mourn your eventual demise.

Green Play Ammonia™, Yielder® NFuel Energy.
Spokane, Washington. 99212
www.exactrix.com
509 995 1879 cell, Pacific.
exactrix@exactrix.com