EGS Integration Console engineers review Artemis I campaign
The NASA Exploration Ground Systems (EGS) program is continuing to review the experiences from the Artemis I launch campaign as teams plan and prepare for Artemis II in a couple of years. History suggested there would be a long ground campaign to the Space Launch System’s (SLS) first liftoff, and a series of issues — primarily with fueling the SLS Core Stage — kept Artemis I on the ground through most of 2022.
The launch team engineers staffing the Integration Console in the firing room were in the middle of the troubleshooting efforts during the three launch attempts in late August, early September, and mid-November. Resolving the issues that came up with updated procedures and updated software finally culminated with the successful first launch of SLS on Nov. 16 from Kennedy Space Center (KSC) in Florida that sent an Orion spacecraft to the Moon.
After a Long First Campaign, Artemis I Launch Demonstrates That the System Works
“I felt like we knew it was going to be challenging, knowing it was a first flow,” Phil Weber, Senior Technical Integration Manager for EGS and Lead Launch Project Engineer (LPE) for Artemis I said in a March 20 interview with NASASpaceflight. “I talked to some of my mentors that had worked back during Apollo, and they said first flows are always very challenging. You test individual pieces, but until you get it all together, you never really know for sure exactly what is going to happen.”
EGS, along with prime launch processing contractor Jacobs and their Artemis launch team partners, is responsible for the integration and launch of all the different pieces from all the different programs and projects within NASA’s Common Exploration Systems Development (CESD) division. The SLS Core Stage for Artemis I was the final piece to finish its standalone development; during the second half of 2021, EGS finished integration, or “stacking,” of the SLS and Orion vehicles on their Mobile Launcher (ML) in the Vehicle Assembly Building (VAB) and rolled Artemis I to Launch Pad 39B for the first time in mid-March of 2022.
A final Wet Dress Rehearsal (WDR) test was planned in early April, but issues with the reliability of the nitrogen purge gas supply to KSC and hydrogen leaks during loading propellant in the Core Stage forced the vehicle to make an extra round trip to the VAB and delayed completion of the test until mid-June. NASA certified the vehicle was ready for flight in August, but additional issues with loading and conditioning the Core Stage liquid hydrogen (LH2) hardware scrubbed two launch attempts in late August and early September.
Artemis I lifts off on Nov. 16. (Credit: Nathan Barker for NSF)
Another rollback to the VAB was bookended by two hurricanes, but Artemis I finally lifted off at 1:47 am EST (6:47 UTC) on Nov. 16.
Throughout the launch campaign and the offline, pre-launch processing flow before that, engineers staffing the Integration Console in firing rooms of the Launch Control Center at KSC’s Launch Complex 39 were working through issues that came up. During the countdowns for Artemis 1, the first integrated launch of NASA’s SLS launch vehicle and Orion spacecraft, launch project engineers (LPE) at the Integration Console in the middle of Firing Room 1 worked with the rest of the launch team on scripted problems during training simulations and the real ones that came up last year.
See Also
Artemis I Discussion ThreadOrion Discussion ThreadNSF StoreClick here to Join L2
“We fought through an awful lot of stuff that we learned, and to me, the success [is] that [the Orion and SLS Programs] are really happy with the performance of the launch vehicle and the spacecraft, and we’ve demonstrated now that we can get through any of these problems that come at us,” Weber said. “We know that this system works now, so to me, that’s the biggest takeaway.”
“I’m retired, so these guys are going to fight through a handful of new problems and hopefully not the same problems because they’ll get the corrective actions in place, but we know that the system works, which is huge. I mean, what a tool this country has in its toolbox now.”
The Orion and SLS Block 1 vehicles were designed to fly for the first time as largely finished products, and the programs and their prime contractors reported excellent performance during the two-hour-long launch and the four-week-long mission to a Distant Retrograde Orbit (DRO) of the Moon and return to Earth. All three programs and CESD itself continued their reviews of data and performance from Artemis I into the new year.
Reviews so far of the Ground Launch Sequencer (GLS) and Spaceport Command and Control System (SCCS) have given good grades to the computer systems that automated much of the test, checkout, and launch operations. “We were really happy with the performance of GLS for this run,” Alex Pandelos, EGS Launch Integration Operational Project Engineer (OPE) and primary GLS engineer for Artemis I, said in the March 20 interview.
Firing Room 1 in the Launch Control Center during the third Artemis I launch attempt. (Credit: NASA/Kim Shiflett)
One of the GLS responsibilities is a “red-line monitor” — the software automates simultaneous, continuous monitoring of hundreds of vehicle and ground system measurements, which have requirements defined by the Launch Commit Criteria. It will flag parameter values that are reported outside their required ranges.
There was some early concern about the capacity of the system to handle a large number of values to monitor at the required tempo, but work since then resolved concerns.
“We maintained significant margin in all of our time checks, both when we looked at GLS and SCCS as a whole,” Pandelos said. “Some of those studies we did years back said we needed to enhance the ability to support higher data rates than maybe we thought [when] we originally started blank-sheet-of-paper designing it.”
“But, it paid off well; it wasn’t just that we were barely getting by, but that we have room to grow for future missions, and so that was important to see.”
Liftoff on Nov. 16 After Riding Out Hurricane Nicole on the Pad
After issues with Core Stage LH2 propellant loading and conditioning on the first two launch attempts delayed the countdown timeline and ultimately scrubbed those attempts, the launch team spent a few weeks revising software and procedures for tanking. A Sept. 21 tanking test helped validate the overall propellant loading process, but before the team could recycle for a launch attempt in late September, Hurricane Ian forced NASA to roll the vehicle back to the VAB and reset for a mid-November launch opportunity.
The vehicle and Mobile Launcher were already back on Pad 39B getting ready for a Nov. 14 launch attempt when Hurricane Nicole made a surprise late-season landfall on Nov. 10. After getting back to work, the Integration Console engineers tackled issues from Nicole, as they do for other severe weather events that are seen at KSC.
“From our perspective, we treat all of those as individual NCs, non-conformances, that come up,” Anton Kiriwas, EGS Senior Technical Integration Manager and Senior Launch Project Engineer (LPE), said in the interview. “As soon as it was safe to have people out at the pad [after the hurricane], we began inspections immediately. Every single subsystem went out and inspected their systems.”
“We needed to make sure we were safe to get back into a powered configuration, [and] we wanted to make sure that we understood things that were out of configuration. We just methodologically worked through each of [the issues] as either a constraint or no constraint towards the upcoming launch attempt.”
The view from the KSC Press Site as Artemis I launched on Nov. 16. (Credit: Michael Baylor for NSF)
“So, for us, it was probably a larger number than we were used to going into a launch attempt, but same exact process,” Kiriwas noted.
“We deal with significant weather here routinely,” Tony Bartolone, EGS Launch Project Engineer, added. “Thunderstorms, especially in the summertime, can bring high winds [and] can bring lightning.
“We have weather procedures to force the evaluation that Anton was talking about to have our subsystems go out and do inspections and verify their system data. We don’t have a procedure for hurricanes, but we leverage what we already had on the books. We worked through them and were able to get everything resolved.”
What may have been a more challenging effort was for the Orion and SLS Programs to verify that the forces from Nicole’s wind field were within the design loads for the vehicle. “The other piece of it was gathering up the data to provide back to the design center for the analysis on the SLS Program side,” Pandelos said. “For us, that was really to gather up the data and provide it back [to the SLS Program] and be on call if they need additional support.”
The storm made landfall faster than the vehicle could be safely returned to the VAB. In addition to preparing to ride out the storm at Pad 39B, EGS set up additional instrumentation to help provide additional data for SLS. “[NASA SLS Chief Engineer Dr. John] Blevins had asked us if we had any capability to try to measure the wind directly at the 60-foot level, and the weather towers don’t have one at that position,” Weber said.
“That’s kind of the baseline requirement that they use for their limits is at [the] 60-foot level, so our instrumentation guys came in with a portable anemometer and mounted it on the handrails and talked to the SLS loads team on exactly where they wanted it. [They talked to SLS about] where they thought the peak winds would be coming from, so it wouldn’t be shielded or shadowed, and that was the measurement that they used to [have] complete confidence that there was no structural issue to the vehicle at all.”
The updates to tanking procedures exercised in the Sept. 21 tanking test led to a smoother countdown on the third attempt. “I think the first half of that entire launch countdown went incredibly smooth. We mentioned several times that’s kind of what we think a launch attempt should look like,” Kiriwas said.
“There were very minor issues, [and] each one was worked as no constraint [to launch]. It really wasn’t until we had the issue with the [Mobile Launcher] replenish valve that we had any major issues, and that fortunately is a ground issue, and it was one that we were able to go work. It always makes us more comfortable to have a [ground systems] issue than something that makes us question the flight hardware.”
The replenish valve that began leaking is in the Core Stage LH2 valve complex, or “skid,” located in the base of the Mobile Launcher platform. The Core Stage was already loaded for launch, in its “stable replenish” phase, and much of the way through the LH2 “bleed” period for thermal conditioning of its four RS-25 engines when the valve began to leak. The Integration Console engineers talked about different options for dealing with the leak, including bypassing the replenish valve and using a different one to control the flow of LH2.
“We talked the replenish valve quite a lot, and the (cryogenic) loading team was looking at different options,” Weber said. “Those guys are just incredible at trying to think ahead for contingency scenarios, and so they had already created a procedure that they had documented, but it had actually never been tested in a cryo environment.”
(Credit: NASA)
(Photo Caption: Seen near the top center of the image, the three members of the red crew climb stairs to exit the base of the Mobile Launcher platform during the third Artemis I launch attempt that later resulted in a successful launch. The red crew fixed a leaky replenish valve in the Core Stage LH2 “skid,” a valve complex that helps control the flow of cryogenic fuel from the LH2 storage sphere at the perimeter of the pad through plumbing to the LH2 Tail Service Mast Umbilical and into the Core Stage.)
“[The procedure was designed for a situation where] the replenish valve is leaking badly. You isolate that, close the replenish valve, and then try to use the main fill and drain valve to regulate and serve that replenish function when you’re up in the 100% full [range].
“The downside of that [contingency procedure] is they hadn’t tested it, and my view was that it kind of contradicted the ‘kinder, gentler’ loading because that main fill valve isn’t a metering valve; it’s a bang-bang, open-close [valve], and so we ran the risk of potentially creating a pressure/temperature spike that would create a bigger leak than the limits would allow for,” Weber added.
“So, [we were] weighing the pros and cons, and again we had a pretty lengthy discussion over on the anomaly loop about that, to the point where [NASA Artemis Launch Director Charlie Blackwell-Thompson] came over and asked us to kind of hurry up and cut to the chase, so we came back and said we think the right technical answer is to send the red team, the red crew, out and have them tighten up the nuts on the packing gland, and so that’s what they did.”
Although the replenish valve only affected the Core Stage propellant lines and loading of LH2 into the SLS second stage, the Interim Cryogenic Propulsion Stage (ICPS) continued during the discussion of what to do. Ultimately, all LH2 propellant loading had to be stopped for the red team to repair the valve.
“In general, a stop flow in one does not imply a stop flow in the other, but for sending a red crew out, we do have a constraint that we cannot be flowing hydrogen at all,” Kiriwas noted. “Technically, the hardware would allow it, but procedurally and safety-wise, we can’t put personnel at risk with hydrogen actively flowing through the Mobile Launcher,” Bartolone added.
Both stages lost some propellant to boiloff while the flow of liquid hydrogen was stopped, but the Core Stage was already full when that started, whereas the ICPS was not. Getting the ICPS LH2 tank fully loaded and ready for terminal count was one of the items that delayed liftoff about 45 minutes into the two-hour long launch window.
The other issue was with a part of the networking infrastructure the Eastern Test Range uses to communicate with the flight termination system on the rocket. “We have no requirements on our end, it’s all Range requirements, and so when they were able to get it resolved and they were in a configuration to allow us to continue, they notified us, and we were able to pick up at that point,” Kiriwas said.
Hydrogen Leaks and Hydrogen Bleed Sensors
Core Stage liquid hydrogen propellant loading operations got a lot of attention during the Artemis I launch campaign. Weber was working at KSC in 1990 when the Space Shuttle Program had hydrogen leaks that delayed launches and missions and became known as the “summer of hydrogen.”
“Summer of hydrogen in 1990 was six months. This ended up being about eight months, so this was even a little bit [longer], and it was really different because we were chasing a lot of different leaks at different locations,” he noted. “We knew exactly where they were; we were just trying to mitigate them, whereas back in ‘summer of hydrogen,’ we were just looking for the leak for a long time before we could actually isolate where it was.”
(Credit: Nathan Barker for NSF)
(Photo Caption: EGS launch pad engineers work on securing connections in the liquid hydrogen (LH2) tail service mast umbilical (TSMU) prior to the third Artemis I launch attempt. Persistent leaks with LH2 quick disconnects in the umbilical plate ground-to-flight connection extended the launch campaign through most of 2022.)
The concentration of gaseous hydrogen picked up by a hazardous gas detection system in the cavity between a ground-side tail service mast (TSM) umbilical plate and the companion vehicle-side liquid hydrogen plate on the Core Stage engine section repeatedly exceeded the upper limit of four percent during the transition from a “slow fill” flow rate to “fast fill.”
The LH2 fill and drain system is a pressure-fed system, and one change in the overall procedure was to use a kinder, gentler, and slower rise in pressure during that transition. “This was a change to the loading profile that we had done to minimize the number of temperature and pressure changes across that seal,” Kiriwas explained.
“So, we took that original profile that we had planned, and we did extend it from a timeline perspective, and then we modified it from a procedural and a software standpoint. The first time that we did that, we had to go utilize a number of manual operations within the software, in some cases actually bypassing some of the automated parts of that software to enable the kinds of changes [we wanted to make].”
“It’s one of the downsides with automation is when you automate it, it can be difficult to ‘un-automate it,’ so the team spent a lot of time within our development lab prototyping how they would do that, and then we did it for the first attempt,” Kiriwas noted. “Going into the tanking test on [Sept. 21], though, we wanted to make sure that we were in the best situation from a team perspective, software perspective, procedure perspective.”
“[After the second launch attempt], we took about three weeks to go rewrite the procedures [and] go actually update the software, finding the best balance between not wanting to change too much of the software and at the same time keep the most amount of automation we could for those sorts of changes that we were doing, and then get the team retrained. We actually ran several days of just non-stop training with the team, making sure we went through multiple [propellant] loads against a simulated environment.”
“We got all the team familiar with the updates to the procedure, familiar with the updates to the software, and that was what we carried into the tanking test,” Kiriwas added.
After the issues with hydrogen leaks that popped up during tankings for the WDR and launch, the engineering community also revisited hydrogen flammability tests performed early in the last decade at the Launch Equipment Test Facility. “They came across some testing that they did back in 2011, [and] I believe that testing proved that you actually have to get up to about 16% [hydrogen concentration] before you start getting into a flammable condition,” Tom Clark, Manager of Propulsion/Avionics Engineering with ERC, said in a previous interview last Nov. 11.
“That was in an environment that kind of mimicked our disconnect area — those areas between the plates. We used the 4% [limit, but we] knew that was a very, very conservative number because to do that, you almost have to be in a lab environment and under perfect conditions.” The umbilical cavity is the small, enclosed space between the ground and flight plates and is also continuously purged with inert helium gas during tanking to increase safety margins.
The updated Core Stage LH2 loading and thermal conditioning process was fully demonstrated in the Sept. 21 tanking test. After the successful tanking test, the maximum allowable limit for hydrogen concentration was raised from 4% to 10% for subsequent launch attempts to maintain a factor of safety below 16% — in addition to the helium purge.
The Artemis I vehicle in the Launch Pad 39B lights as NASA proceeded into the first launch attempt on Aug. 29. (Credit: Stephen Marr for NSF)
The technical issue that scrubbed the first launch attempt on Aug. 29 was a suspect reading from a Core Stage Main Propulsion System (MPS) sensor in the line that provides a liquid hydrogen bleed flow to engine three. All four engines have their own bleed line that runs from the stage into the engine, but the temperature reading for the MPS sensor in the engine three line violated a complicated launch rule.
“They had seen similar behavior [from the sensor] before; they had seen it at Green Run,” Kiriwas noted. “At that point, they said they’re going to try to see [if] we can institute some changes procedurally to get that flowing. They didn’t want to assume that it was flowing and that was an erroneous measurement.”
The Aug. 29 launch attempt was the first opportunity to see the behavior of the LH2 bleed system at KSC because a hydrogen leak during the last Wet Dress Rehearsal in mid-June had prevented exercising the hydrogen engine bleed. In the mid-June WDR test, the LH2 bleed engine conditioning was not performed, and related measurements and launch commit criteria were excluded.
“We had what we jokingly called ‘probably the most complicated launch commit criteria in the history of humanity’ for that engine conditioning, and it was really to try to mitigate the risk associated with the sensors on engine three not being trustworthy,” Weber said. When the launch team finally started the LH2 bleed in the Aug. 29 launch attempt, the sensor in the engine three bleed line showed a higher temperature than the other three engines, as if LH2 was not properly flowing past the sensor.
The reading was contradicted by other temperature and pressure sensors indicating that LH2 was flowing through the engine’s bleed circuit and properly chilling down the engine equipment. “Under the launch attempt umbrella that we were in, we didn’t have the ‘luxury,’ I’ll say, of just doing a whole bunch of testing, so we instituted a whole bunch of troubleshooting steps during that launch attempt,” Kiriwas said.
“I think this was a really major one for us. We actually shut off the bleed to three of the engines and had all of it flowing through engine 3, so we knew a hundred percent that it was definitely flowing liquid in and liquid out, and that measurement still wasn’t showing the temperatures we expected it to, so that data got brought back to the SLS team. Their analysts took a look at it and said we think we understand this behavior now. We are flowing liquid; there’s no reason to believe we’re not.”
The troubleshooting during the launch countdown took a long time, and it was decided to scrub the first attempt, let everyone get some rest from the overnight operations, and take a longer look at the data.
“In hindsight, after going through all the data from launch attempt 1, the consensus across the whole technical community here at Kennedy and the Marshall team and all their contractors was that we were probably OK, but we didn’t know that [during the launch attempt],” Weber said. “By isolating and only flowing through engine 3 and demonstrating the Core Stage delta-P (pressure) sensors showing flow was occurring — and we had some downstream GSE temperature measurements that showed liquid temperatures — everyone was able to get comfortable, so for launch attempt 2, we were able to back out that really complicated LCC and use a much more straightforward method to verify conditioning of the engines.”
The second launch attempt was scrubbed before the start of LH2 engine bleed, so the next time the LH2 engine bleed procedure was executed was on the Nov. 16 attempt when Artemis I finally launched.
“I think John Blevins likes to say a direct witness is best, but if you can’t get it, we certainly have other indicators that we can rely on, and then we trusted the process,” Kiriwas said. “We knew how to go and initiate that bleed; we knew that once it was initiated, it would continue to flow; and we knew that we needed to get it flowing for long enough to satisfy the requirement, and then we could fly from there; and that was really what we updated that LCC to say, rather than the complex kind of analysis that we had originally been trying to do with it.”
(Lead image: The Artemis I vehicle climbs away from Launch Pad 39B on Nov. 16, beginning its inaugural flight. Credit: Nathan Barker for NSF)
The post EGS Integration Console engineers review Artemis I campaign appeared first on NASASpaceFlight.com.
Continue Reading
