Close
result
How do we know which information to trust?
Dr Charlotte Farmer
8 min read
April 12
Dr Charlotte Farmer describes her personal strategy for ensuring that organisational decisions are based on reliable evidence
The most trusted information is what we know from firsthand experience. But as chiefs of staff or chief operating officers (COOs), we have to rely on information from sources other than ourselves all the time. We are Jacks-of-all-trades, specialists in perhaps a handful of skills that we honed at the beginning of our careers but have allowed to lay fallow as we become experts in managing experts. We cannot be the subject matter experts of all the organisation does, but our principals expect us to speak and advise them as though we were. Through years of consulting to leaders, senior executives, and heads of organisations, I have developed a process for discovery or ‘triangulation’, screening or ‘synthesis’, and application of trusted information to effectively support the sound decision-making needs of my principals and colleagues, and ultimately, the organisations.
Source triangulation – gathering trusted information
Source triangulation – gathering trusted information
In the vast majority of cases, I have had one chance to present my data, make my case, and provide recommendations and next steps. And in most cases, I have not had the luxury of time to conduct exhaustive or comprehensive research. Yet the assumption of leadership is that my findings, justifications, and conclusions are based on trusted information.
To arrive at trusted information, I start with trusted sources. These fall into three categories: 1. stakeholders, 2. organisational purpose, mission and internal records, and 3. external expertise or third-party information generators. Typical information-gathering efforts involve all three of these categories, which can be tracked to the corners of a triangle not only to ensure thoroughness but also to gain an array of perspectives.
… in most cases, I have not had the luxury of time to conduct exhaustive or comprehensive research. Yet the assumption of leadership is that my findings, justifications, and conclusions are based on trusted information.
Stakeholders
Stakeholders are those who are influenced by or who influence the mission or corporate purpose. They might be employees, customers, general counsels and inspectors general, other corporate functions (e.g., Finance, Human Resources, and Technology), end users, shareholders, legislators, and often institutional or business partners.
At UL Research Institutes (ULRI), where I am currently chief operating officer, internal stakeholders comprise employees, customers and consumers (end users), and business partners who help us develop, test, and promulgate our safety standards. We also count academics and research centres: we look to universities and laboratories to scale and accelerate safety science discovery.
Objective partners are important stakeholders; I solicit their thoughts and ideas frequently. Just as important are the end users—they drive the need for standards, government services, and production. When I was COO to the Chief Information Officer at The MITRE Corporation, for example, I constantly asked government clients how well we were doing and what they needed us to do to prepare for the next 10 years. While at Eastman, a business-to-business enterprise, we conducted buyers’ focus groups in addition to ad hoc discussions with people embedded along the supply chain from raw materials to manufacturing.
Third-party observers
Third-party observers
The last category of information sources is third-party information producers whose content influences large populations and constituencies at large. I am always interested in knowing what they have to say about us – my organisation at the time and the industries in which it has equities, my competitors, and ancillary industries. Many of these sources would be academic and professional publications such as Harvard Business Review, Yale Law Review, and the American Medical Association Journal (and this journal of the Chief of Staff Association). Respected business intelligence generators are also a key component of this category. Several comb, consolidate, and digest tremendous amounts of reliable data from corporate and government reports and market statistics to provide objective information. Dun & Bradstreet and S&P may be counted among them. News media are considerable influencers, though their editorial lines must be acknowledged, and citations given. Fortune, Forbes, and Bloomberg come readily to mind, and there are several others.
Organisational records
Organisational records
The second category of information is derived from organisations’ corporate purpose, vision, mission, priorities, and strategic goals as captured in formal documents, materials, and files. Almost all of this information is derived from primary sources—corporate records and official reporting.
They might include, for publicly traded companies, annual and quarterly reports (10-Ks and 10-Qs) required by the Securities and Exchange Commission to provide statistics and activity data that indicate corporate health; or credit ratings and profiles in well-respected directories such as Dun & Bradstreet and Standard and Poor’s.
Objective partners are important stakeholders; I solicit their thoughts and ideas frequently.
Synthesis
Synthesis
Having assembled my information, I need to analyse it and decide how to use it to support the story I am telling.
To do this, I follow a process that I think of as ‘synthesis’: weighing and ordering the gathered data, then applying and merging multiple data points. The use of decision analysis techniques promotes improved critical thinking about strategies, alternatives, risks, and systematic decision making. It also helps me know what information to trust. Analytical techniques vary from simple tabulations to multivariate statistical methods and econometric models. Which technique to use depends on the data and its purpose. Generally, my analysis consists of looking at the data as a whole to build the story I want to tell. The story describes a critical path forward that takes into account alternatives and their pros and cons; risks and their management; key stakeholders and relationship building; and projected and/or desired outcomes with recommendations. To deal with any data gaps, I identify and acknowledge them, then apply a system for moving forward in light of those gaps. Knowing what stopgap measures and information to use is situational: it depends on the question to be answered or the problem to be solved. The end drives the means.
To deal with any data gaps, I identify and acknowledge them, then apply a system for moving forward in light of those gaps.
Review
Review
Typically, leadership assumes that my analyses are sound. Knowing this, I leave little to chance. I have a group of colleagues and specialists on whom I can call to critically evaluate my work. I also invite my peers to review the material, including the information upon which I base my analyses, the analyses themselves, and the recommendations. In all cases, I ask for candour and accept all comments. I have found that the more accepting I am of their input, the more honest and thorough the reviews are. Incorporating these comments strengthens my deliverables. US Supreme Court Justice Ruth Bader Ginsburg routinely asked Justice Antonin Scalia, her philosophical sceptic, for his review of her written opinions. Revising her material based on his comments resulted in opinions that were impervious to much other criticism. As I have enjoyed success with each ensuing effort, leadership’s trust in my use of the right information and analyses has grown. The successes and votes of confidence underscore the quality of the data.
I have a group of colleagues and specialists on whom I can call to critically evaluate my work.
The process in practice: a case study of ULRI
The process in practice: a case study of ULRI
I joined ULRI when it was launching a massive transformation and revitalisation effort. As part of the push, we formulated a global impact strategy to grow talent, capacity, and scope. My role was to conduct a rigorous multi-front scan covering internal and external indicators to discover what was going to enable and/or inhibit achievement of our goals and objectives. On the political and legal fronts, we wanted to know about any changes in salient national and local policies and regulations, as well as international treaties with which we interface because our services are offered in 100 countries. In addition to government sources such as legislative activity and auditors’ reports, were Executive Orders signed by the President of the United States and analysts’ reports on Brexit.
On the economic and market fronts, we examined worldwide indicators in tariffs, fluctuations in GDPs and exchange rates, unemployment, and costs of living. We looked at government reports published by the US Department of Commerce and the Federal Reserve Board, as well as periodicals such as The Economist, and annual reports of banks and corporations. On the technological front, we looked into new and emerging technologies such as robotics and artificial intelligence including increments of speed of evolution; and on the operational front, my focus was on progress toward organisational missions and strategic plans based on key performance indicators, and employee engagement and satisfaction from internal surveys. Lastly, we considered sociological and environmental factors, deriving information from The Aspen Institute and World Economic Forum studies to address culture, health and safety, demographics, trends in climate change, sustainability, and ethical sourcing.
We mobilised several groups in ULRI to gather and digest the data to provide us summaries and material findings. I suggested sources from my list, especially from my experience working in the government space for more than 20 years. I was also interested in speaking to organisations engaged in breakthrough investigations. We hired a consultant to benchmark institutions that were doing scientific research similar to ours. The consultant looked at our most daunting challenges, and sought areas where we could contribute the greatest value in our research. The consultant identified four organisations. We visited the Janelia Research Campus of the Howard Hughes Medical Institute in Ashburn, Virginia, where they are conducting research in neurosciences that is not being done in academia and industry. We went to the Center for Firefighter Safety Research and Development of the University of Maryland, College Park, Maryland. It is responding to the need for significant improvements in safety for first responders in emergencies and homeland security threats identified in 9/11 events.
Turning my lens on internal considerations, I collaborated with peers, scientists, and human resources specialists to understand the talents, skillsets, and degree of expertise that will be needed to achieve ULRI’s new strategic goals. The findings in this regard pointed clearly to the shortfalls in staffing that must be addressed in the near term for us to be prepared for the mid- and long-term goals and objectives. We also explored the types of laboratories we would need and where they should be located. Development of automobile batteries for example, should be near manufacturers for efficiencies in the testing phase. And we looked for new organisations with which to partner, to build alliances, to collaborate, to give us an edge in making progress.
The credible data focused the scan, enabled better analysis, informed forecasting, and generally strengthened the plan.
This seven-point scan helped identify the critical path elements of the five-year strategic plan. The credible data focused the scan, enabled better analysis, informed forecasting, and generally strengthened the plan. The Board’s ‘red team’ review helped model the organisation and provided guidance and insight to further refine the plan’s topical areas, objectives and milestones. As the plan evolves, we are preparing to implement it, knowing that the roadmap is based on sound information.
Concluding Observations
Concluding Observations
In today’s world of research and development, information is being generated at staggering rates. One source (Social Media Today) estimated that, ‘In 2018, more than 2.5 quintillion bytes of data were created every day’. In spite of ‒ or perhaps because of ‒ this volume of information, we chiefs of staff sometimes wonder whether we have gathered enough information, whether we have checked enough sources, whether we have the right information, whether we have left out any crucial information. How do we know when we have enough? I also think about prioritising information, giving some more weight than others. Am I assigning the correct values to the different information? And in so doing, am I imposing my own bias? Often, deadlines take care of these questions for me. I must be satisfied in knowing that I have leveraged the triangulation and synthesis process, relied on my list and staff, engaged the review of experts, experienced successes that confirm my judgment is reliable, and enjoyed the votes of confidence of my leadership.
Author Bio
Dr Charlotte M. Farmer
Chief Operating Officer of UL Research Institutes
Certified Chief of Staff®
Dr Charlotte M. Farmer is Chief Operating Officer of UL Research Institutes (ULRI), Chicago, Illinois, where she has been charged with facilitating the growth of its strategic initiatives globally, and bringing about transformation through innovations in safety science. Before she joined ULRI in November 2021, she was the COO to the Chief Information Officer at The MITRE Corporation, Washington, District of Columbia, a premier research and development organisation supporting the United States Federal Government. There for more than a decade, she worked directly with government departments to drive modernisation and speed ideas to market. She built coalitions of industry leaders to address national issues in areas such as cyber security and pandemic response. For 10 years preceding MITRE, Dr Farmer was an executive at Booz Allen Hamilton, Inc., specializing in transforming organisations through streamlining, improved staffing, and enhanced decision-making. From 1993 to 2000, she wore three hats at Eastman Chemical Company: Finance Director, Innovation Manager, and Operations Manager. She was also a Program Manager at Eastman Kodak Company from 1987 to 1993.
Dr Farmer graduated magna cum laude with a bachelor’s degree in chemical engineering from Tennessee Technological University, Cookeville, Tennessee. She holds a Master of Science in chemical engineering from the University of Tennessee, Knoxville, an MBA from the University of North Carolina Kenan-Flagler Business School at Chapel Hill, and a doctorate in Engineering from George Washington University in Washington, D.C. Among many professional honours, Dr Farmer was named a 2021 Exceptional Women Awardees Foundation inductee and included in Savoy Magazine’s listing of Most Influential Black Executives in Corporate America in 2020. She currently serves as board chair of Mobility Unlimited Technology Worldwide, a nonprofit startup that performs research and data collection to safely adapt spaces for wheelchair users, blind citizens, and mobility aid users. She also holds board memberships or advisory roles with the National GEM (Graduate Education for Minorities) Consortium, HireVue, Crittenton Services of Greater Washington, and the International Council on Systems Engineering.
She also completed The Chief of Staff Association's Chief of Staff Executive Certification programme at Saïd Business School, University of Oxford.
