Kampflybloggen (The Combat Aircraft Blog) is the official blog of the Norwegian F-35 Program Office within the Norwegian Ministry of Defence. The author of this post, Major Morten «Dolby» Hanche, has more than 2200 hours behind him in the F-16, he is a U.S. Navy Test Pilot School graduate, and on 10 November 2015 he became the first Norwegian to fly the F-35. He now serves as an instructor pilot with the 62nd Fighter Squadron at Luke Air Force Base in Arizona. This post is an updated version of an article .
Yet again, information from the «Director Operational Test & Evaluation» (DOT&E) has stirred critics into a frenzy over the F-35. The fact that the information was leaked seems to have agitated people even more. (We have our hands on classified documents! Now we know it all!) Yet again, the leaked memo described aspects of the F-35 which need improvement. Yet again, the report resulted in press articles which painted a pretty sinister picture of the F-35. The article featured in POGO
serves as one such example.
I finished up writing this article before getting ready to fly another sortie in the F-35. Based on my own experiences flying the F-35A, I feel that the media´s interpretation of the previous DOT&E report is influenced heavily by unrealistic expectations – something which seems to be a trend. I don´t see the point in countering every claim that´s being brought up. First off, it´d make for a very long article. Secondly, I would not be dealing with the bigger problem, which in my mind is a lack of understanding.
I fully expect the F-35´s most hardened critics to discount this article, regardless of what I write. However, some may choose to believe my story, based on the fact that I know the airplane and its capabilities as a pilot. I don´t make my claims based on bits and pieces of information, derived from potentially unreliable sources. They are based on experience actually flying and training with the jet for nearly a year
My goal is to shed some light on airplane development and testing; why we test, what we discover in testing and what a test report may result in. I write this based on my own experience, both through education at the US Naval Test Pilot School, but more importantly through working with the F-16 and the F-35, both operationally and in test settings.
What smartphones tell us about technology development
I´ll start with smartphones, as another example of technology development. Admittedly, phones are somewhat different from a fighter airplane, but there are similarities. A smartphone is a complex system of systems – just like a fighter jet. The phones keep evolving with both new hard- and software. It is not unheard of therefore that the manufacturers issue updates. Updates which provide new capabilities, but which also aim to correct previous errors.
According to Wikipedia, Apple released its iOS 9.0 operating system to their iPhones and iPads on 16 September 2015. The 9.0.1 update was issued already on 23 September, followed closely by the 9.0.2 update on 30 September. Then 9.1 on 21 October and 9.2 on 8 December 2015.
Such a frequent update rate might indicate that not everything worked perfectly from the start. Still, wouldn´t it be a bit harsh to claim that the phones didn´t work with the first four software versions? Might the truth be a little more nuanced? Can a smartphone be a good product, even if it doesn´t work 100% from day one? Does a smartphone ever work 100%? I have experienced various strange occurences with my phones over the years. Still, for me, having a phone with all its peculiarities has been more useful than the alternative – not having a phone.
This isn´t an article about phones. The point I´m trying to make is that technology development and testing is a series of compromises; compromises in reliability, in performance and in quality. Only rarely is the world black or white. A machine may work well, even if it doesn´t fulfill all specifications. I´ll go on with a brief intro to how we typically test.
…technology development and testing is a series of compromises; compromise in reliability, in performance and in quality. Only rarely is the world black or white. A machine may work well, even if it doesn´t fulfill all specifications.
How we test a fighter jet
Testing of combat aircraft typically sees a disctinction between Developmental Test (DT) and Operational Test (OT). In short we can say that DT seeks to answer whether the machine works according to the design specifications, whether the machine is safe to operate and what its safe operating limits end up being. OT on the other hand seeks to find out whether the machine can solve a particular task, like: «Is the X-YZ able to provide effective Close Air Support, in the presence of threat A, B and C?»
The test program for a machine like the F-35 is an enormous undertaking. The contours of the F-35´s test program are described top-level in the Test and Evaluation Master Plan (TEMP), totaling 1400 pages. Each sub-test in the TEMP results in a detailed test plan for that event. Especially in DT, a test flight is literally planned down to the minute, in order to accomplish as many test points as quickly and safely as possible. Flight testing is an expensive undertaking.
A test program should discover most important errors and flaws. However, time and resources available make it unrealistic to uncover every single issue. Risk is mitigated by testing the most critical components, like the engine in a single-engined fighter, to stricter tolerances. The amount of testing is a statistically driven decision. We know that there are things we don´t know, even at the completion of testing. We also know that there are likely few gross or dangerous errors which haven´t been found.
Each error we find during testing is documented and characterized. The language and format used is to the point. The test engineer and test pilot type up their findings and typically describe the situation «in a vacuum» – without regard for how costly or difficult it might be to address the issue. Each issue is then related to the mission – how will this quality or problem affect the given task?
Such a test report might read something like: «The SuperToaster 3000 was evaluated for uniform heat distribution and time to crispy toast, at the National Toast Center of Excellence, with room temperatures varying between 65 and 75 deg F. The toasting temperature was selected by turning a dial on the front of the toaster. Even with full crispyness selected, the toaster´s maximum temperature was low, and toasting of even the thinnest slices of white bread took more than 10 minutes. During early morning breakfasts, the time consuming toasting process will result in cranky parents, the kids being dropped off late for school and correspondingly negative effects on their grades and later career opportunities.»
This mission relation was probably a little over-the-top – a little like how some media articles relate its tidbits of information to an imagined F-35 mission. In isolation, a system may not work as advertised, but could there be a workaround? (In the toaster-case, maybe cereal for breakfast?)
Anyway, after the issue is documented, the errors are then catalogued, debated over and prioritized. Test engineers, test pilots, design engineers and customer representatives are often involved in the dialogue that follows when something undesirable is discovered. Together, these will have to agree on a path forward. Completely understanding the issue is crucial. Alternatives could be a re-design, accepting the flaw, mitigating the flaw procedurally or compensating by documenting the issue better. The team will have to compromise when prioritizing. Even when developing a new fighter jet, there are limits to what can be fixed, based on cost, time available, test resources available and also the complexity of the problem. Altogether, development and testing is an iterative process, where adjustments may have to take place during DT, OT or after the system is put into operational service.
...
there's just
