Featured Interview: Professor Julian Scott Yeomans
“SimDec” is a new and novel approach that significantly enhances the analytical capabilities of users by readily exposing counterintuitive behaviours so that they can be readily understood by technical specialists and non-technical users alike.
Sounds confusing? It might be if you’re not a specialist or statistician. But, with SimDec, you don’t have to be an expert to use it effectively.
Find out more about this amazing new approach to sensitivity analysis in our in-depth interview with one of its creators, Schulich professor Julian Scott Yeomans.
You have a very interesting background – in addition to being a Professor of Operations Management & Information Systems (OMIS), you are also the Program Director for both the Master of Management in Artificial Intelligence (MMAI) and the Master of Business Analytics (MBAN) as Schulich. What was it that drew you to this field?
Well, in addition to those somewhat pretentiously-sounding titles, I also have degrees in four different fields. I have always resided somewhere along the quantitative spectrum and have been a bit of a roving academic interest-wise – wandering from place-to-place to seek out things that hold my attention. Perhaps a type of punctuated evolution where I will work on something intensely for a time, then move onto something else as it strikes my fancy. The ability to remain devoted to any one topic over an entire career has always escaped me – I prefer doing the equivalent of cannonballs into the pool of ideas to see what comes out. Hence, being able to work in the multidisciplinary areas of operations, information systems, analytics, and AI provides perfect cover for shifting allegiance from one topic to another.
As to being the Director of both the MMAI and MBAN Programs … I actually missed the meeting where this decision was taking place, so was informed that I held those positions the next time I appeared on campus *laughs*. More seriously, I was perhaps the OMIS faculty member with a profile most-aligned with those fields, while also being “senior” enough such that, should the administrative burden detract from research output, it would not negatively impact my career-trajectory nearly as much as for one of my more up-and-coming colleagues. Academic life is still a publish-or-perish environment.
We’re here today to talk about your new book Sensitivity Analysis for Business, Technology and Policymaking Made Easy with Simulation Decomposition (SimDec), which was recently released in e-book format and is scheduled for a more formal release in November 2024. Long title, and a seemingly complex topic – so we might need a bit of background here. So let me ask: what exactly is Simulation Decomposition and Sensitivity Analysis?
A very good question which, unfortunately, does not have a 30-second elevator-pitch response, so bear with me.
In general, for anyone that has data, the most basic investigations involve:
- Finding which factors have the most influence,
- Assessing what exactly their actual influence is, and,
- Determining what happens if the factors change.
In a broad sense, addressing all three of these questions simultaneously is the key component of sensitivity analysis. These tasks represent the fundamentals of data analysis in science, engineering, computing, analytics, AI, etc. So, in a certain sense, sensitivity analysis can be thought of as a comprehensive analysis of data. A bit more specifically, sensitivity analysis, itself, studies how uncertainty in the outputs of data/models/systems can be “attributed” to the uncertainties in the inputs. This attribution involves calculating sensitivity indices that quantify the influence of an input (or group of inputs) on the output. There are methods to accomplish this task but they tend to be somewhat … “inaccessible”.
A related field is uncertainty analysis, which focuses greater attention onto uncertainty quantification and the propagation of those uncertainties. In an ideal world, sensitivity analysis and uncertainty analysis should be performed together seamlessly and in tandem.
That would be in an ideal world.
However, the practice of addressing these requirements, either individually or in combination, is extremely difficult – often involving impenetrable mathematical procedures which are completely beyond the grasp of us mere mortals. Even when implemented, they require considerable quantities of data and tend to be extremely computationally intensive. So much so that, even amongst academics, engineers, and scientists who should be sophisticated enough to conduct such analyses, 95% of their studies do not! But that result becomes even worse when it turns out that, of those 5% that have performed a sensitivity analysis, 4 out of 5 have been done incorrectly.
So, the reality is that only 1% of data analysis studies actually conduct a meaningful sensitivity analysis. Just think of the staggering consequences of that deficiency on “real life”!!!
Interesting. So, when you were faced with this problem, how did you decide to approach it? And did you have any collaborators?
Enter Mariia Kozlova, my co-author and research partner. Mariia is young, driven, and brilliant – all attributes that I am not encumbered with.
The genesis of Simulation Decomposition (SimDec) occurred 5 years ago. Our original version of SimDec was a visualization approach for the uncertainty analysis of output distributions, decomposed and colour-coded by a selection of variables chosen by the user. Basically, it was a what-if tool for performing a visual exploratory analysis of data. It really was a great analytical device, but its uptake was rather tepid. A recurring criticism was that, because the variables selected were user-driven, it essentially amounted to a type of visual p-hacking (i.e. if you play around with any data for long-enough, eventually you will “uncover” a pattern simply by chance).
After looking around at the “more scientific” variable identification options available, we decided to counter that criticism by identifying the most influential variables by calculating their sensitivity indices – our introduction to the world sensitivity analysis. Have I mentioned mathematically impenetrable? No matter how much effort we dedicated to the cause, comprehension of the “industry standard” methods remained inaccessible. We are intellectually slow! To counter this, we created our own indices that required far less data, orders of magnitude fewer calculations, and – most importantly – determined values that perform at least as well as the other methods. Think of it – indices that are good, easy to calculate, and understandable all at the same time – the heresy!
Now sensitivity indices alone can never tell the full story of the data, no matter how good or reliable they may seem. Sensitivity indices can only describe the relative strength of an effect, not its shape. The actual shape of an effect is often crucial for understanding underlying behaviour and any decision-making surrounding it. So, if you combine identifying the decomposition variables based on the strength of their calculated sensitivity indices with the visual analytics from our original procedure … voila!
This updated SimDec inherently amalgamated uncertainty analysis directly into sensitivity analysis. Done in tandem, this approach exposes the strength and character of the data effects, thereby enabling deeper insights into its fundamental behaviours.
We appear to have found the somewhat aspirational Holy Grail for practically applying sensitivity analysis to the “real world”.
Who would make use of Sensitivity Analysis and why?
I believe the question would be better phrased replacing “would” with “should”. There is a clear gap between “should” and “would” that must be filled.
Ha, fair enough – who “should” make use of Sensitivity Analysis and why?
If you are performing any type of analysis on data in your possession, then essentially everybody should be performing a sensitivity analysis on it. Absolutely everybody!
This all comes back to some combination of the three key questions about your data:
- Which are the most influential factors,
- What is their influence, and
- What if they change?
If you have data but don’t want to know these things about it, what is the point of having that data? Extensive searches have indicated that virtually nobody in industry and academia that should be performing sensitivity analyses on their data, is actually doing so. Nobody!
Might a state-of-the-art book bridging that gap not prove extremely beneficial?
A very modest response that leads nicely into my next question. So, tell us about your research and your new book!
Everybody who uses data (from analysts to policymakers) needs to conduct sensitivity analyses on it. Sensitivity analyses can be difficult to do because the existing techniques are extremely complex. Consequently, virtually nobody at all performs sensitivity analyses, even though they should.
In contrast, we have created a new combined sensitivity-uncertainty approach that we believe is both very easy to implement “in practice” (& have created the necessary computational tools to achieve this) and at least as good as any other existing technique currently available (we actually consider it better but that reflects a certain bias).
The overall purpose of the book is to test SimDec as a method for “shaking the foundations” of “real world” data analysis. Specifically, to explore how well SimDec performs as a sensitivity analysis tool when tested on a wide spectrum of application cases.
To achieve this, we needed numerous “real world” sites to experiment with. Quite surprisingly, many of the organizations that we cold-contacted accepted our invitations with remarkable enthusiasm for such an intensive project – and we selected 10 of them.
As a result, the book contains 13 chapters. The first 3 chapters provide the necessary background details to understand sensitivity analysis, SimDec, and how to use the accompanying software. The remaining 10 chapters discuss the various “real world” applications studied.
So, what are the real-world examples of how sensitivity analysis is applied that you cover in the book?
The application chapters each cover a specific project in detail with these topics ranging from: corporate finance, public support, 3D manufacturing in construction, deep tech entrepreneurship, carbon footprint analysis, geology and model fidelity, P2X fuels, structural reliability, superconducting magnets, and personal decision-making. The SimDec method can be considered quite application agnostic – all it requires is the data.
While the analysis method was our creation and interpreting its output fell directly at our feet, our knowledge of topics in (say) high-energy physics and structural integrity is, and remains, rather … limited at best. Consequently, Mariia and I co-wrote each application chapter with the support from our various colleagues in each of the respective organizations.
In almost all cases, the sensitivity analysis results that were produced turned out to be quite remarkable (the details are in the book) in that they uncovered outcomes that had not been considered, a priori. In fact, the results were so beneficial that everybody planned to keep using SimDec as part of their standard organizational practice and the vast majority wanted to undertake further studies with us – which is always a good seal of approval.
Building on that, what are the broader implications of your work?
The general implications from this study are that, if you can provide an accessible easy-to-use mechanism for conducting a meaningful sensitivity analysis of data, not only will people do so, but they will also uncover very useful information that they were not previously aware of. There are many broad technological and strategic benefits to be gained by analyzing the results from performing this task. SimDec provides its necessary vehicle.
How do you think this will impact the field of analytics (and don’t be afraid to be immodest!)
There are two parts to my answer here, depending upon whether you are an academic or in the “real world”:
First, for academics: A sensitivity analysis should be done whenever you are analyzing data. Clearly, this has not been the case previously because either people are not familiar with the concept or find the options available to perform it too computationally daunting. Therefore, irrespective of field, if you are an academic, nobody has been doing sensitivity analyses and would definitely not have used SimDec. So, for the next year or so, if you use it on your data (and uncover some cool relationships) you can publish a paper on “An Application of SimDec to Field/Topic XXX”.
So, if you are a graduate student, an up-and-coming junior faculty member who must publish for tenure, a tenured faculty member looking to put out something new, a “mature” scholar searching for an outlet (i.e. everyone!), SimDec provides an immediate option for a publication (& academe is a publish-or-perish world). Longer term, once the requirement for performing a straightforward sensitivity analysis becomes “main stream”, academics should be including it automatically in the data analysis section of their research – as part of the already-standard statistical-type component (hopefully boosting my citation counts dramatically by always referring to this book as its guiding source).
Secondly for industry settings and “real world” applications: If you can derive more information from analyzing your existing data, from an economic and/or practical sense, why on earth would you ever not do it? SimDec provides a now readily-accessible straightforward technique to accomplish the task. What is the worst that could happen from using it? If its findings are viewed as completely worthless, then simply don’t use them. But I have a sneaking suspicion that applying any form of sensitivity analysis would always be beneficial. Whether this analysis be done with SimDec or (God forbid) some better method that might emerge, “real life” applications will never be worse off from receiving such scrutiny. Of course, from a purely egocentric standpoint, the best course of action would be to make SimDec the industry standard!
Are there any new trends or emerging technologies in sensitivity analysis that you find particularly exciting or promising?
Other than SimDec? Clearly not!
Actually, our hopes are that this, and perhaps related work, will spur more attention toward the “real world” practical application of sensitivity analysis. That there is more extensive consideration for advancing ease-of-use methods to make the approach more universally accessible to all. Our fingers are firmly crossed!
What areas of SimDec and sensitivity analysis do you believe are ripe for further exploration?
Advancing “theory” is always something that can be attacked with impunity as an area for further exploration – at least from an academic perspective. Theory always leads to more theory. And this mathematical elegance is what gets widely published. Unfortunately, from a practical standpoint, the current theory is already computationally impenetrable. Right now, people are staying away from conducting sensitivity analyses in droves. Creating more theory that makes it even more mathematically inaccessible won’t break down any of the barriers to entry.
So, for me, it is the application of SimDec to “real world” settings and actually using it for the sensitivity analysis of results & modelling that holds the most promise. Widespread performance of sensitivity analysis is sorely lacking in all applied & academic settings, mainly because existing methods are much too impenetrable and inaccessible. SimDec changes this complexity dynamic, dramatically. So, we need to be rather evangelical with everyone who uses data, to get them to perform sensitivity analyses on it … and we just happen to have a method that they can easily employ. (Though if you can find a better method, please use it!)
Are there any upcoming projects or research directions you’re particularly excited about?
There are a TON of directions that we are being pulled in, at the moment, so it is hard to provide a comprehensive answer to this question. Off the top of my head, a few of my favourite current “real world” studies are:
- The impact of the nitrogen-rich, food-water-energy agricultural system in the US Midwest on the resulting low levels of dissolved oxygen (hypoxia) in the Gulf of Mexico (with two researchers at Purdue University);
- A sensitivity analysis of the heat exchangers in the Finnish nuclear district heating reactors (with four nuclear engineers at LUT University);
- Using SimDec to enhance the sustainability and efficiency in the 3D-printing of concrete for the construction industry (with a construction engineer from the Netherlands, a concrete researcher at ETH Switzerland, and the CTO of a Finnish robotics company);
- Analysis of cancer data – breast cancer in humans, hemangiosarcoma in dogs (with a local veterinary colleague); and,
- Enhancing our computational algorithms by borrowing efficiencies from structural chemistry methods (yes, I had to Google what “structural chemistry” was, too) – perhaps doing our part to contribute to the overall impenetrability of the subject via this study (with a German chemist at the U of Bielefeld).
So, clearly just your standard, run-of-the-mill set of business stuff.
What advice do you give to students interested in pursuing research in this area?
There is really a multi-pronged response to this question depending upon the type and scale of interest.
For those that “simply” want to use SimDec and/or sensitivity analysis on their own data, you can get a good introduction by reading our first 3 chapters and whatever selected application chapters that interest you – to get an idea as to how run and interpret the output. The computational tools are readily available and the code can be modified to suit your purposes. You can even pull the uncertainty and sensitivity components apart to use them separately. If you join our Discord forum, there is a readily available Q&A forum.
For those that might want to advance SimDec research and/or work on the applied side of sensitivity analysis, it would be best to get in touch with me (syeomans@yorku.ca) or Mariia (Mariia.Kozlova@lut.fi), directly, to determine what-is-what.
And if you want to do a full-on, deep-dive into the theory of sensitivity analysis, it would probably be best to establish contact via our Discord community – as most of the established “big names” in the field are members. Mariia and I can provide an appropriate introduction, if desired.
Note that all of these resources are completely open to anyone beyond the walls of the academic world, too!
How can people find out about your work?
Absolutely everything we have done is completely Open Science, so is freely available and accessible to anyone. In addition to all published research papers, the links shown below can also be found on my faculty webpage (should you happen to mislay this valuable newsletter).
An open access (OA) version of the book is free-to-download from: https://doi.org/10.4324/9781003453789
(you can also purchase a $250 hard-copy through the publisher via that link, or from sellers like Amazon as well)
And while I encourage you all to read the book, I should note that we’re making some “minor” changes to it – despite us talking extensively about colour in our SimDec outputs, it seems like our publisher accidentally printed several of the figures grey scale (whoops). So, if you’ve downloaded it, I’d encourage you to redownload it again with the corrected colors at a later date- but, I assure you, the data and analysis is still correct. So feel free to read at your leisure (and check back a bit later in September for a more ‘final’ version).
All computer codes (Python, R, Julia, Matlab, VBA) are available open source from:
https://github.com/Simulation-Decomposition
If you are not a programmer, the interface for our no-coding-required web dashboard can be found at: https://simdec.io/.
You can join our SimDec discussion forum at: https://discord.gg/8jkEyqXu2W. Our Discord community essentially contains a “who’s who” of the “big names” in sensitivity analysis and provides a great networking resource for aspiring users and researchers. Of course, you can also read movie and music reviews, general bonhomie, Q&A, football results, etc. etc.
And we also maintain an earlier, more general SimDec website, from when we anticipated becoming software billionaires, while living off the royalties of a certain book: www.SimDec.Fi
And, mentioned above, if you have an interest in advancing SimDec research and/or work on the applied side of sensitivity, it would be best to get in touch with me (syeomans@yorku.ca) or Mariia (Mariia.Kozlova@lut.fi), directly, to determine what-is-what.
Any closing remarks?
SimDec and sensitivity analysis should not be considered “spectator sports”.
If you possess data, you need to actively participate and get “stuck in”. SimDec is completely application agnostic and, clearly, I want you to use it on whatever application of interest you might have. I can make all kinds of claims as to its benefits, but I am not an academic snake-oil salesman and SimDec should not be promoted as cold fusion.
Try it for yourself to determine if it could really be useful for you. There is no downside to experimenting with it.