On the evolution of mechanical pencils

When I was a physics student, my father started giving me mechanical pencils. You’d think all possibilities would have been invented by the 1980s. But some of these pencils were incredible feats of engineering with fun new ways to click out and retract the pencil graphite. I think my father had a point that there was always more to be invented, even for something so simple. We'd frequently look at bridges and discuss ways that they could be designed differently to distribute the weight and talk through the engineers' decisions around the common designs around us in the real world. Every time I'd invent something new, I'd diagram it for him with my pencils and he'd ask probing questions about the design choices I had made. One day, for instance, I’d invented a "Runner’s Ratchet Shoe." I was an avid runner in those days and would get sore knees and shin splints. The Runner's Ratchet was a set of levers attaching to the runner's ankle that would put the downward motion of a runner's shoe into softened impact, cushioning the shock to the knees while redirecting that downward force into a forward springing action propelling the runner forward. He looked at my drawing and exclaimed, “Congratulations for re-inventing the bicycle!” It took me a moment to see that the motion of the foot between my invention and the bicycle was the same, while the muscle stress of my design was probably greater, obviating the benefit I was pursuing in the first place.

When I went to college. He started giving me fountain pens. I asked him why he was giving me all these. I could just use a biro pen after all. He said, “If you have a better pen, you’ll write better thoughts.” I could feel the effect of the instrument on the way I framed my thoughts. I was more careful and considered about what I wrote and how. Fountain pens slowed me down a little bit. They force you to write differently and sometimes they alter the pace of your writing and therefore the way you see the initial thought. It feels as if you’re committing something weighty to paper when you write with quill ink. I enjoyed the complexity over time. I took away the lesson that the instrument shapes the experience, emphasis of the path over the goal. Complexity, challenge and adversity in any process can make the end product more refined.

One day, I was working on a new invention that, yet again, had complex moving gears and levers. It was another piece of sports equipment I'd named, Ski Pole Leg Support. This invention was again to address knee soreness, this time from hours of sitting on ski lifts with dangling legs tugged on by heavy skis and ski boots. The device would allow skiers like me to suspend the weight of the legs onto a chair lift through the length of the ski-pole which would suspend a retractable support for the base of the ski boot. As I was visualizing the motion of the device in my head, I thought that what I really needed was a pen that could draw in 3 dimensions directly, the way I saw the device in my mind's eye. That way I could demonstrate the machine in spatial rendering rather than asking my professors to look at my 2D drawings and then asking them to to re-imagine them as 3 dimensional objects, Da Vinci style. 

With this new inspiration, I designed a concept of just such a drawing tool. It looked like a fishbowl that had a pen-like proboscis which would move through a viscous solution to draw the lines which would suspend in location supported by the viscosity of the medium. I realized that a fountain pen moving through the solution would disturb the suspension solution itself through friction and therefore damage the rendered image as the drawing got more complex, unless the user drew from the bottom up. So, as an alternative to that approach, I imagined using an array of piezoelectric ultrasound speakers laid out in a grid in the base of the globe to direct shock waves through the solution, converging shock waves on certain points. At the locations where the shock waves would intersect, they would cause destructive interference and therefore increase the fluid pressure at the drawing coordinates desired. The solution at those shock-points would form a visible distillate from the solution's chemicals at these points, which would allow the drawing to persist. (The same way that a cathode ray tube uses a single string of electrons to paint a picture over a grid where the electrons strike the luminescent screen. But I'd use sound instead of electrons.) When the drawing was finished, you could preserve it temporarily, then erase it by spinning the globe. Centripetal rotation would pull the distillate apart so it could settle in the base to dissolve again and be reused, like the way a lava lamp uses its molten wax in solution, re-melting when it falls close to the heated base. I thought of it like a 3D version of the popular 2D Etch-a-sketch toy which was popular during my childhood. Might this drawing globe have market potential I wondered?

Before I shared the concept with toy manufacturers, I met an inventor named Michael Zane who had graduated from my college, Franklin and Marshall. He said he was willing to look over my product concepts and give me some advice. After he stared at my drawings a bit he gave me an approving glance. He said he liked my ideas. But he then commented “If you have any interest or ability outside of inventing, pursue that with a passion!” He thought his career path was not something that he’d wish on anybody. It was incredibly difficult to file and protect patents as you tried to sell your products in a fiercely competitive international market, he explained. He told me stories of many inventors whose lives were consumed and hopes dashed by throwing too much of their lives into one idea. So his advice was to live a different life than he saw on paper as my future. I did go to the US Patent and Trademark Office to research the ideas in the sector of several of my future patent ideas. But over the years I let my dreams my physical hardware inventions trickle out of my mind and focused my inventions on new problems and opportunities in the digital technology and internet space.

Looking back on my 3D Etch-a-sketch concept 30 years later, I see how a fountain pen for aquariums wasn’t going to find a mass market fit, even if I'd thrown all my gusto behind it. Mr. Zane had saved me lots of frustration and door knocking. I’m very glad I pursued those other interests I had at the time. Mechanical pencils and fountain pens are cool. But your life should be about more than something that has been reinvented 1000 times since the dawn of art. The inventions I focused on over the last three decades were team efforts, not lone entrepreneur stories. Coordinating groups of people in a company to build something new is the way to go these days. As the oft quoted adage goes: If you want to go fast, go alone; If you want to go far, go together. My teams have won patents, changed industries and impacted the lives of millions of people. It would have been a different life story if I'd just pursued selling plastic drawing toys at the start of my career.

I say all this because I have been toying around with VR and AR products for the last 8 years since my company decided to leap into the new industry. I’m starting to see echoes of what I’d wanted decades ago, now implemented in products. My colleagues and I go into VR to discuss technology and new product ideas. We tend to use Spatial.io, a virtual reality conferencing platform. One day I drew a diagram in the air with my fingers. I described a product concept I’d debated with one of my friends, Paul Douriaguine, who was a work colleague from my time working at a startup in Sydney. We discussed the concept of using aerial photography and photogrammetry to assemble a virtual reproduction of an oil refinery or other physical facility or factory. We discussed using automated time-lapse captured images from multiple drone flights around the facility to watch for areas of discoloration that might indicate mold, rust or oil leaks which could be used to prevent physical structure damage. 

My rendering, pictured above, showed how a drone flight path, conducted autonomously or crowd-sourced, to capture structural images for analysis. Flight path was portrayed in green on the camera angles a drone would follow around the facility depicted in blue. Then my friend John P Joseph, who had actually worked on oil facilities with his own AR company jumped in and diagrammed how his team looked at the problem for long-distance pipeline maintenance and function monitoring.  

Then my other friend, Olivier Yiptong, jumped in to talk about how to establish the server architecture to achieve the service we were describing mechanically across pipes, facilities and flying devices. 

It was an amazing thing to watch. Three people with entirely different backgrounds (business, product and engineering) had assembled sporadically. In the span of about 15 minutes, all of us were able to rapidly discuss different layers of a product and service initiative to achieve understanding of a range of opportunities and limitations through a process that might have taken hours of preparation and presentation time in any other medium.

The experience made me reflect back in time. My first conception on the best way to draw an invention 3 decades ago was to make a product leap from paper to fish bowl globes. Here I was today, inside the globe with other clever folk inventing in a shared virtualized space. In Spatial, I was able to diagram the concept in a fun and effective way, even if a bit sloppy because I didn't have a 3D ruler and protractor yet! (Wait a tick... what if... oh never mind...)

VR is an old idea receiving heaps of attention and investment right now. Just as some say that the Apollo missions were something the past could afford that the present could not, I think that VR is an idea that couldn’t have found a market when I was young, but that actually can address new use cases and interesting applications now. Perhaps it doesn’t pass the Larry Page toothbrush test: something you’d want to use 2 times a day. But it is significantly valuable in what it can convey experientially. I find myself preferring it over the camera-gaze experience of pandemic life video conferencing platforms. Now when my engineering and product expert friends want to meet, we typically opt for the VR meeting as it feels more human to move and gesture in a space rather than sit transfixed, staring into a camera lens. Perhaps the recent pandemic has created a great enough frustration in general society that we yearn to get back to a 3D experience, even in remote conference meetings. Seeing people through a flat screen while posing to be rendered in 2D is an artifice of the past times. I suspect that eventually people will come to prefer 3D to meet when they can’t meet in person more broadly. 2D seems to be an awkward adolescent phase for our industry in my view.

These VR designing pens are similar to the fountain pens and mechanical pencils of yesteryear, but in a medium that is just being created for our next generation of inventors and future physicists now. Our tools are imprecise at present. But in the coming years they will be honed because of the obvious benefit and efficiency they bring in facilitating social connections and collaboration. Over the pandemic years I've gained a new form of conversational venue that has caused more discussions to happen in better ways than the technologies I'd become used to before. I will continue these brainstorms in virtual 3D environments because, when separate from my team, I still want to communicate and share the way we are used to in physical space. But there, we obviate separations in space while still keeping the robust density of media that we can share in our globe-like environment, unlimited by the restrictions of what can be crammed through a small camera aperture.

The emerging technology for stereoscopic media in the home

A decade ago, I followed the emergence of affordable 3D televisions at the Consumer Electronics Show for experiencing movies at home. One of the problems of adoption of the technology was the lack of media that mainstream audiences could view on the devices. If you were to have one of these advanced TVs, there was a lack of viewable content streamed or sold to those same households. It was just too much work to pair the content with the device. Filming in stereoscope is a complex process that isn't well supported by the commercial media channels to the home as it is for the cinema.

While stereoscopic headsets are being released in significant volumes following the wave of VR as a mainstream consumer experience, the content availability challenge still looms. (IDC projects 30 million headsets a year to be released by 2026 across multiple vendors. Meta claims 15 Million sold to date.) This time the gaming sector is leading the charge in new media creation with 3D virtual environment simulation using world building software platforms distributed by Unity & Epic Games. The video gaming industry dwarfs the scale of cinema in terms of media spend with US gaming sector alone totaling over $60 Billion annually in contrast to cinema at $37 Billion. So this time around the 3D media story may be different. With lower production cost of using software media creation and a higher per-customer revenue stream of game sales, there will be more options than I had with my 3D Vizio TV.

I recently discovered the artistry of Luke Ross, an engineer who is bringing realistic 3D depth perception to legacy video games originally rendered in 2D. His technique is currently applied to allow 3 dimensional "parallax" depth to a 2D scene by having the computer render parallel images of the scene, depicted to each eye in a head-mounted-display sequentially. Leveraging the way that our brains perceive depth in the real world, his technique persuades us that typically flat-perspective scenes actually are deep landscapes, receding into the distance. Filming the recent Disney series The Mandalorian was conducted using the same world building programs used to make video game simulations of spacious environments. Jon Favreau, the show's director, chose to film in studio using Unreal Engine instead of George Lucas style on-scene filming because it drastically extended the world landscapes he could reproduce on his limited budget. Converting The Mandalorian into Avatar-like 3D rendering for Vizio TVs or VR head mounted displays would still be a huge leap for a studio to make because of the complexity of fusing of simulated and real sets. But when live action goes a step deeper to simulate the actors movements directly into 3D models, such as the approach of Peter Jackson's Lord of the Rings series, rapid rollouts to 2D and 3D markets simultaneously becomes far more feasible using Luke Ross "alternate-eye-rendering" (abbreviated AER).

Stereoscopic cameras have been around for a long time. Capturing parallax perspective and rendering that same two camera input to two display outputs is the relatively straightforward way to achieve 3D media. What is so compelling about the concept of AER is that the technique achieves depth perception through the use of a kind of illusion which occurs in the brain's perception of synthesized frames. Having a stereoscopic play-through of every perspective a player/actor might navigate in a game or movie is exceedingly complex. So instead, Luke moves a single perspective through the trajectory, then having the display output jitter the camera slightly to the right and left in sequence. When right glimpse happens, input to the left eye pauses. Then the alternate glimpse is shown to the left eye while right eye output pauses. You can envision this by blinking your right, then left eye while looking at your finger in front of your face. Each eye sees more behind the close object's edges than the other eye in that instant. So objects near appear to hover close to you against the background, which barely moves at all.

Vast landscapes of Final Fantasy VII appear more realistic with parallax depth rendering. 
https://www.theverge.com/2022/8/10/23300463/ffvii-remake-intergrade-pc-vr-luke-ross-mod

The effect, when you perceive it for the first time, astounds you with how realistic the portrayed landscape becomes. It's like having a 3D IMAX in your home to experience this with a VR headset. The exciting thing is that game designers and directors don't have to rework their entire product to allow this to be possible. AER can be done entirely post-production. It is still a fair bit of work. But much more feasible to achieve on grand scale than rendering all legacy media anew in 3D VR stereoscopic view. This makes me believe that it will be a short matter of time before this will be commonly available to most readers of my blog. (Especially if I have anything to do with this process.)

You may not yet have a consumer VR headset at your disposal yet. But currently HP Reverb, Pico, Meta Quest, and HTC Vive are all cheaper than my 3D Vizio TV. The rendered experience of a 65 inch TV in your living room is still typically smaller in your field of view than a wide field of vision VR headset. So over coming years, many more people may opt for the nearer screen over the larger screen. When they do, more people will start seeking access 3D content which now, thanks to Luke, has a more scalable way to reach the market for this emerging audience.

Going into the cave to see the world differently

Growing up in Oregon, where there are many volcanoes, my family would often go spelunking in the lava tubes left over from past eruptions. These caves formed where the outside of a river of lava cooled, letting the inside of the lava flow further downhill, leaving a hollow husk of the river's rock shape to climb in. Caves always invoke a kind of supernatural sense of awe for me. The sensory deprivation of the darkness gives a sense of humility we feel when we take ourselves out of societal context, bringing about a sense of ascending out of our ordinary self-awareness. When I travel abroad, I enjoy exploring caves and seeing the lore that has sprung up about them.

In Bali, Indonesia are a series of caves that one can explore at the basin of rivers flowing down the sides of the volcanic island. However, the caves of Bali are man-made. Many centuries prior, people used tools to carve out the caves at the base of the river into a series of sheltered rooms. When the monsoon rains came, all the caves would be submerged, only to be navigable again the following year.

The islands of Indonesia used to be home to a different species of hominid prior to Homo Sapiens' arrival. Homo Floresiensis were a smaller hominid who also resided in caves like the Denisovans of Siberia. Homo Floresiensis is suspected to have evolved differently from Homo Sapiens and Denisovans, to be smaller in stature because of the constrained natural resources of the Indonesian archipelago. Many other species of animal discovered there were also diminished in stature in contrast to their respective ancestors on the Asian continent. When I climbed into the cave shelters on Bali, which likely were carved many thousands of years later than Floresiensis, I pondered the significance of caves as protective and spiritual sanctuaries for our Hominid species through the millennia. Echoes of these other hominids live on in our DNA as some of them cohabitated and inter-bred with Homo Sapiens as the latter spread around the globe.

In Sri Lanka, caves served as monastic retreats for monks and mystics as Buddhism swept through the region, encouraging a spiritual path of asceticism and meditation away from society's hubs of bustle and industry. The giant mesa of Sigiriya was one such retreat with ornate frescoes painted on the inside of the caves from a period of time that a local king sought to fashion the mesa into a paradisiacal castle fortress. The inside of the cave is a representation of the ideas of beauty of the time, the way a camera obscura projects the outside world through a pinhole lens or mirror into a dark chamber, now frozen in time for next era's visitors to witness.

In Thailand, the lineage of Rama kings had summer retreats near the caves under the country's mountainous southern coast. A few hours hike from the beaches you can visit a vast underground cavern replete with an underground temple pavilion and many images of the pre-enlightenment Siddhartha. He sits as if frozen in time in the gesture of pointing at the ground, where he resolved to stay until he crossed the precipice of Nirvana. (Siddhartha was one of many Buddhas in the broader tradition. But in the Thai tradition, this particular Buddha is important to the culture because he represented the path of independent transcendence for all beings, an ideal for everyday community to pursue in emulation.) Outside the national palace of Bangkok, there is a small pavilion that looks exactly like the pavilion Rama V had built in Phraya Nakhon cave. 

Perhaps the king would use that pavilion outside his window as a mental reminder of the symbolic connection to the cave where the monks would meditate for the transcendence of Samsara for all of civilization. Venturing to the north through Laos, there are hundreds of caves to explore where local citizens create temples of reclining Buddhas deep in the karst mountains to revere the monastic traditions that spread through the country as Buddhism spread north from India to China.

In Jordan, the Nabatean people used caves as tombs and fashioned majestic facades in the style of architecture they'd seen elsewhere by chiseling and polishing limestone like a sculptor. Allegedly, once their city of caves became a place for mystics and pilgrims to visit, they came up with an economic scheme to require all visitors to exchange their gold coin for lesser value coins they would mint locally, leading Petra to become a very wealthy city offering its citizens relative prosperity based on the city they'd constructed out of sand with deft artifice of the chisel. While the caves started out as a form of palace-like grandeur, the interiors just provide shelter from the elements and are unadorned. Walking into them is a transformative journey from outward splendor to a sense of inner awe. You experience the awe of natural places with the human artifice dropped. To this day the Bedouin people there welcome tourists to explore the caves with concessions and hiking provisions provided at tents set up to give you respite from the sun while you enjoy tea and refreshment.

In Egypt, caves were tombs for royalty which provided a path to eternal life. Excavated only recently, these caves portray amazing mystical traditions and understanding of the cycle of life, death and rebirth enacted through the worship of human representatives of archaic gods. Many of the gods of Egypt are depicted as animal and as chimeric human forms. The mortals who were revered as leaders of the society may have been seen as temporal representations and sporadic embodiments of the deities in ephemeral time. Spirits of the underworld would gift life to humans through the the Ankh symbol. When the kings and queens of Egypt died, they were entombed in ornately decorated caves that remain well preserved to this day near Luxor and Aswan.

In Mexico, caves are considered the domain of the rain god Chaac. The many cenotes (limestone caves) of the Yucatan peninsula formed in part due to their being on the edge of an impact crater of the Chicxulub meteor, which led to the extinction of the dinosaurs 66 million years ago. Maya priests would create altars at the bottoms of the caves in supplication to Chaac to bring rains for the sustenance of the maize crops. These fresh water cenotes served the flatland jungle areas of Yucatan with fresh drinking water that sustained the inland population until the civilization's collapse in 900 AD, after which the Maya transitioned to building their cultural hubs on the coast of Yucatan rather than the jungles.

Not all caves can be visited by avid tourists like myself. Because of the significance of certain caves to our cultural lore, archeologists and artists are making them discoverable via facsimile copies so that we can all share our fascination with this part of our history and culture. One such archeological effort was captured in the Werner Herzog film, Cave of Forgotten Dreams. In this movie, he filmed the Chauvet-Pont d’Arc cave in France with a group of anthropologists as his guide. The cavern is covered with Paleolithic art depicting animals and an image of a human hand outlined as if it were a signature by the artist. Herzog and the anthropologists reveal the story of our human ancestors' intentions in decorating this cave and speculate on the daily life and potential spiritual aspects of the people who had made the cave paintings. What was so unique about this film beyond the site itself was that Herzog filmed it in 3D, allowing the viewer to get a sense of depth that is usually not depicted in cinema. Looking into the dark recesses of the cave as they entered feels a bit like the suspense of going into a cave in the first person experience. 

(Chauvet Cave Drawings. Source: https://whc.unesco.org/en/list/1426/)

Seeing pictures and movies about caves can convey a lot of cultural context. But stepping into them is an altogether different experience that is hard to simulate through media. One of my most touching experiences I remember was a trip to Okinawa, where my grandfather had served in the military during WWII. He had told our family stories of the horrors of past wars he'd had to endure. I rented a car to drive to the sites I'd heard about from my grandfather's stories. One of the most touching experiences was visiting the Himeyuri Peace Memorial. This was the site of a cave where a group of young students had been conscripted into serving as a makeshift hospital for wounded soldiers. When the military advanced here, they were instructed to throw tear-gas into the cave to get any civilians to surrender. The children were too afraid to come out of the cave and suffocated there. The people of Okinawa built the museum on the site to commemorate the lives of the students who perished there and to chronicle the suffering of all people during the war, with a hope to see the end of wars globally. 

Particularly touching for me was a dark room where each student was introduced by name with a placard of what was known about their lives. I saw the faces of youth and read the chronicles of their last days trying to save the lives of as many wounded people as they could. I came to understand the sense of fear they must have had when they crawled into the cave, never to come out. The final exhibit was a physical reconstruction of the cave that memorial visitors could go into to see what the mouth of the cave they stared up at must have looked like. This facsimile cave where I stood, looking out at the green trees in the sun conveyed something that reading and hearing stories never could.

 

(Himeyuri Peace Museum Cave. Source: https://jinotourblog.weebly.com/himeyuri-peace-museum)

So why all this talk of international travel, human heritage, history and venturing into caves you may wonder? I see it as metaphorically similar to the concept of emerging VR technology. In this new context, we put on a visor to give an ornate expression of something elsewhere (or elsewhen) by temporarily silencing the environment immediately around us. One of my earliest experiences of VR's potential was a tour of an Egyptian tomb for Nefertari. The simulated experience stitches together thousands of scaled images to give the viewer a sense of being present in the physical tomb in Luxor, Egypt. While being in a place physically has a visceral impact that is hard to describe, being able to represent an historic location through media gives the opportunity to have a glimpse of the same space such that many more people, who may not have the opportunity to travel, can experience it.

Now that LIDAR scanning has emerged as an accessible mainstream technology, we are able to create more of these realistic captures of caves and world monuments in ways that people anywhere can experience. A decade ago, I was able to experience the Keck Caves archeology project which seeks to capture underground archeological digs using LIDAR scanning with precise depth measurement which can be rendered as a point-cloud of map data to track physical attributes of a site during an excavation. Once the site is scanned, it can be studied by an unlimited number of later archaeologists in digital renderings without having any impact on the physical site itself by donning a set of glasses that reconstruct the parallax effect for depth perception. I was able to put on some early prototypes of VR 3D glasses then which allowed me to see depth perspective of the caves by rendering different images to my right and left eyes. 

I was delighted to discover that archeologists are already publishing VR renderings of historical sites for the general consumer to experience the Egyptian tombs without having to fly around the world. When I planned my recent visit to Angkor Wat temple complex in Cambodia, I donned my VR headset to fly over the landscape in the Google Earth (former Keyhole acquisition) with pictures of the layout of the historic site. It's very exciting to see artists and game developers embracing 3D medium as a new means for artistic expression. But as a traveler, I want to see more world heritage sites. I tremendously admire the work of Zamani Project and Cyark to preserve more of our historical landmarks for future generations to discover via new media interfaces.

When I introduced my mother into the experience of VR, she made an astute observation. She didn't want to see synthetic renderings of artistic spaces. She wanted to use the lenses to see the astronomical photographs from the James Web Space Telescope, to see the edges of our universe in greater detail like a planetarium would show. Being able to fly to the edges of the observable cosmos based on the data we have on the structure of our tiny 13.8 billion year old spacetime bubble is not an incomprehensible challenge for the educators and developers to bring about in the coming decade. We have only just learned to see that far. Now making that vision accessible to all is within our technical grasp. 

While the artistic opportunities of expression of VR are profound alone, it also allows us to see tremendous breadth of human history and culture by capturing the physical world for educational and tourist exploration. While it may strike some people as odd to wear glasses or head-mounted displays, the vision we can obtain by going into the dark helps us to see further than we otherwise might.

Approaches to enhancing cartography and location-discovery over the web

A French postal worker, who used to fly the coast of Africa delivering postal dispatches, wrote a delightfully witty book I read long ago. In the story of Le Petit Prince, the author, Antoine de Saint-Exupéry, talks about how his plane broke down in the desert one time. He narrates the tale as his pilot self, recounting the story of a small person who approached him while trying to repair the plane. This character claimed to be an alien who had been hopping planets prior to landing on Earth, which had such tremendous gravity that it couldn't be escaped. Being stuck on Earth didn't seem to be a particular bother other than the fact that it meant he could never return to his origin planet, where his favorite flower lived. The story turns into a narrative on the nature of love and loss as the alien prince tries to adapt to the ways of Earth and comments on all the strange characters he'd met on the various planets prior to getting marooned in the desert with the pilot.

I recount this tale because I've been thinking about cartography a good deal recently. The efforts of various companies to create proprietary maps remind me of the "businessman" character in Antoine's story. (See chapter 13.) He was too busy to talk to our protagonist because he was tabulating all the stars. He asserted that by keeping record of the stars he was owning them. The prince channels the perspective of Chief Seattle that it is preposterous to assert that one can own nature. (In Chief Seattle's case it was native tribal land he was being ask to sell to the US government.) The augmented reality mapping community isn't trying to own spaces, but rather create a utilitarian linking model, metaphorically similar to the Global Positioning System (GPS) but specifically for the shared context between people for use in location-based content discovery and information sharing. 

You might think we could just have one app to rule them all, and that all location-based content should be visible within that. But this limits utility. It calls to mind the parable of Śāntideva, the Tibetan monk who mused:

“Where would I find enough leather
To cover the entire surface of the earth?
But with leather soles beneath my feet,
It’s as if the whole world has been covered.” *

Getting everyone on Earth to use one tool in order to benefit from location-based data isn't feasible nor optimal, any more than covering the earth with leather. It would spur a new digital divide problem on top of the one we already have, and become too cumbersome to maintain and update securely. The internet is a more scalable approach than an app-siloed approach too. The web's pooled effort of millions of developers, curators and contributing users are already adept at working on rich content generation, with adaptable sharing, access and security already taken care of. So in the location-based web, we have to think about renderings of web content in a way that can be shared by thousands of browsers, apps and location-"aware" tools that don't have screens. This will allow individuals worldwide to put soles beneath their feet to navigate this information terrain in diverse and dynamic ways.

While one can't directly own locations on a map, we can indeed create facsimile versions of real space and thereafter build multiple virtual content layers atop that. In Mozilla (a non-profit open source web development company) we had a mapping initiative that allowed location based radio wave triangulation to be charted across multiple wavelengths (cell-tower for long range, wifi for short range) stored in a digital repository for developers to build upon. Called Mozilla Location Services, we were able to leverage public frequency radio triangulation for any place in the world that had telephony coverage to be a navigable utility layer for posting or pulling web content. We could then build our own content layers and mapping tools that could pin to locations in the physical world. 

What is the best user experience to navigate this “augmented” coordinate space? Unless a browser had a way of querying for location and rendering the pinned object or content, nobody would see it. My friend Ben Morrow, who has been studying the AR space for years, explained, “This is a search and discovery problem. You have to ensure that whatever space you're in isn’t overburdened by irrelevant AR content in your space, which would result in an ‘AR Garbage World,’ or if you’re in a sparsely curated space you don’t experience the inverse, an ‘AR Desert World,’ where there is nothing of value for you to draw from.” So to deliver this, there needs to be a public-visibility of assets that are available in a specific location through a passive or direct querying utility, but with acceptable level of user control and preferences to give the user filtering control of what needs to be seen, when. (Imagine strolling by a "discoverable object" at a coordinate near you. You might look for specific content, like a location-based list of places to eat near you. But in this context, can content surface to you without downloading a specific app bespoke for that content?) TV and terrestrial radio is a content layer of real-time streaming video/radio abstracted away from specific coordinate locations. The spatial web content layer would be overlapping directly in places that are shared by millions of people daily. Navigating those layers should be easier than downloading a unique viewer for each channel. We have thousands of individual apps right now that render specific views of locally posted data. But we don't have a standardized approach such that any one of those app viewers could tune into all the channels of the others the way a TV or radio can easily switch between radio wavelengths.

To build this shared-content layer, we need engineers, artists, business teams and shared standards for engagement and architecture across millions of personal navigation devices. Rather than the celestial businessman who seeks to chart the stars alone, we need creators with the philosophy of an open-source cartographer. When the world-wide-web of hypertext documents charted the course for our internet of today, it started with a small interlinked corpus of nodes, which then expanded exponentially in the 1990s as millions of contributors posted and linked their own creations with others. I remember early approaches for content sharing and navigation depended on web-rings where site hosts would link their content to other sites of known related content. Bloggers created their own specialized hubs of niche content. Companies like Yahoo, Excite and LookSmart hired hundreds of content curators to pull in sources of new content as the web grew. Netscape’s Open Directory Project and Wikimedia achieved similar content mapping databases for multiple languages around the world even without a corporate sponsored business model. Then once, the scale of web content grew faster than could be aggregated or charted efficiently a purely algorithmic model was needed to scale the corpus of the web efficiently, leading to the web crawlers and Boolean web query models we have for sifting content today.

Web search engines of today didn’t yield their value from a top-down approach, but rather a bottom up. They gained their utility for us by mimicking the way that web developers inter-link and label their content for discovery. Early algorithmic search engines (Altavista, Hotbot, Inktomi, Fast, Bing, Google for instance) would crawl the web much like a spider, following the way each thread of the web linked to every other and ascribing weighted value to different kinds of connections site hosts would paste into their html by way of “anchor-text links.” It was crowd-sourced in a way that made it more dynamic and resilient than any single company or group could create. 

The spatial web will be woven similarly, in a crowd-sourced way with location reviews of restaurant goers, measured velocity of sharing actions and graffiti-style content postings that happen in the physical world. The search engines of this cartographic space will prove their value by filtering millions of tread paths of those wandering its trails and leaving notations for others, reinforcing the well-trod paths as potentially of utility to others just like the keyword searching model we’ve become accustomed to, which replaced the editorially curated/recommended web indexes of the past.

While the location-based internet has had many exciting phases of evolution over the past decades, it is rising to a fever pitch these days as a broad array of well-funded companies are starting to plan new utilities to map digital services over our shared physical space. The current pace of technology advancement here is a fascinating rabbit hole of strategies for cartography enthusiasts like me.

I became particularly interested in cartography during my international travels (50 countries to date) when I would use tools derived from the Open Street Map's open source database of physical locations to plan, journal and photographically document my journeys. Sometimes I was able to find my way about with digital maps in a way that was not possible with market-available paper maps that I'd bought. (In many cases, the places I'd go were not mapped precisely by Nokia Here, Bing nor Google Maps in a useful way. For dense cities, those services are great. But for the digitally disconnected regions of the world, they don't provide the anticipated utility.) 

I care about expanding this portion of our technology because I believe there is tremendous good that we can do for our community, and those who come after us, by facilitating greater levels of information layering on our physical world and the chronicling of information that can be available to people anywhere specific at a moment's search. There are tremendous resources that we've created in the corpus of the web that are not directly accessible in a given location without particular forward-leaning effort with keyword querying of web content. That's something we can address with good tools and intuitive user experience models that make the web generally more applicable to the daily lives of people in specific places.

Now that Virtual Reality gaming has advanced to a relatively mature market, several companies are now pushing the 3D models of gaming engines (such as Unity and Unreal) to create simulated virtual overlays of the shared space we live in. You may be concerned about these advancements from a safety perspective, with good cause. Distractions can be a danger in any context. The hazards of digital overlays in the real world are already being tested by pioneering companies who will be mindful of this. With Niantic's gaming layer (discovered through mobile phone screens, and eventually digital glasses) there can be risks of people being distracted from real-world safety issues. Niantic themselves have taken care to ensure that users don't engage with their games in ways that would put them in physical danger. With the release of more eyeglass products from Snapchat, Nreal and Facebook Aria, we'll start to see more social utility, services and commerce advance this trend beyond the well-established use of Augmented Reality in the workplace, which has been the focus of earlier efforts of Microsoft, Google and Magic Leap. Just as the mobile phone and automobile industry have taken measures to reduce distracted driving, caused by the proliferation of mobile phone usage in the wrong contexts, we can anticipate the industry addressing our concerns about such distractions outside of the driving context as well as these "mobile glasses browsers" become common in public.

It is very easy to have content layers that trigger in-app content specific to only one developer, the way that Yelp Monocle showed AR overlays of the real world camera view, but only showed Yelp content. But what will be the next giant leap is to see when content can be app-agnostic, the way web pages are. We already have the file formatting for representing spatial objects in a portable way. (kml and glb are 3D equivalents of jpeg/mpeg for instance) We next need to implement a shared/public hosting repository where digital assets can then be queried in a location by a means contextually relevant to the need of the user. Due to the cost of infrastructure necessary to support location content discovery, there needs to be an appropriate business model for sustaining the underlying architecture.

Before departing the Google Maps division, the Niantic project had an interesting angle for this. During the development of their first game, Ingress, I met the team at the San Francisco Game Developers' Conference. They explained that there was a commercial value to moving people around in places. If they were to place an Ingress portal in a location that happened to be a store, those people playing Ingress nearby might eventually walk into that store to buy something. So location-based incentives may be the economic value to a cartographic platform that could influence where people decide to go in a specific location. This is also of great value for travelers, who are actively looking for tips on how to discover a location and which places to visit, book lodging with, or discover culinary and cultural opportunities.

Around the same time as that conference, I'd learned of a company that was creating a real-world Pac Man game that would drop little digital treats on their driving game to encourage people to create a digital layer of mapping that they could claim to be their proprietary travel-based index of the world. At a TEDx talk in Silicon Valley their product manager described the perils map routing. She suggested that the Waze app could leverage the willingness to explore of their users, somewhat like a group of foraging ants, to "random-walk" the Earth until they knew the real-time travel velocity of every navigable path on the planet. It struck me that this was a slight ecological nightmare if it was gamifying the concept of driving the roundabout way, just for a digital treat, instead of navigating in a straight path, which would require the least expenditure of resources, even if not time. Perhaps over time the initial waste of the indirect-route incentive would lead to traffic reduction for subsequent users who might not have known about an alternative efficient route that was better than the route that a larger map service would recommend. Ultimately, Google decided that the gamified approach of Waze and the active "contributor" community of people reporting road hazards and route feedback was enough incremental value over Google Maps alone to merit an acquisition of the company, while the decided to spin off the Niantic project as a stand-alone company.

If you happen to be a VR enthusiast, you probably have seen the rich 3D environments that Google Earth have rendered specifically for VR-ready devices. You can fly though the streets of New York, Chicago or San Francisco seeing a spatially accurate depiction of these major cities. You may then be surprised to see that a similar level of effort has not been put into cultural or historically significant world locations. The reasons may seem obvious. The city-scapes are assembled using Google's lidar photography (Such as the cameras on Waymo cars) combined with known information about the buildings that make up the city's topographical 3D structure. Obviously you can't drive a Waymo car through Chichén Itzá. An altogether different approach needs to be applied. One of Google's contributors explained that for areas that are not navigable by car to map with lidar, you must use aerial photography with photogrammetry (stitching of photos from different angles) to create depth maps. Putting San Francisco on the map is therefore, very easy. Putting Chichén Itzá, Petra or Angkor Wat on the map is stunningly hard. It's a finite expense, but the benefit from doing it doesn't yet outweigh the cost of doing it. It's just a matter of time and logistics though.

Considering the economic problem that leaves so much of Google Earth unmapped in 3D relief, you might wonder how AR is going to be any different. This is an issue that a bunch of people are trying to solve actively right now. The good news is we can use the hive-like behaviors of humans to accomplish it, just like the Waze team did for the streets of the US. But there needs to be some benefit or value of participating that incentivizes people to do the contribution work necessary to make a good map. That's where people like me come in. As a mapping contributor, uploading photos, making reviews and leaving tips for travelers, I'm doing the necessary work to create the content rich location layer across that will eventually populate our smart-glasses, automobile dashboard screens and social traveling apps. 

If it all the work had to be done by advertising companies, then every location you go to you'd be hearing navigation tips like "Turn left at the Wendy's then go straight to the 7-Eleven, where you bear a right, you're at your destination when you can see a Home Depot on the opposite side of the street." All that overt advertising would probably drive attrition of the navigation tools because the heavily-sponsored environment is annoying and distracting over time. It reminds me of the skyscraper in Hong Kong that for years had the giant text "AD HERE" on its Kowloon-facing side. The incentives need to be subtle, like the "Pokémon Go Incense" and "Poké-lures" that Niantic provides for individuals to draw customers to the proximity of commercial businesses when they need foot traffic. It's more appropriate to leverage context without being so overt as to say, "To get your next Poké Monster, walk into the store next to you and buy a soda, scanning the QR code on its side." You may in fact do those things of your own volition. But being instructed to to them in order to unlock a level of a game is annoying.

While these incentive schemes work well for large companies with a broad user base, smaller developers and content creators will have a challenge trying to create real-world content publishing. For instance, if I had a large photogrammetry map of Chichén Itzá, how could I get that content to someone who could benefit from it? Generally, I'd have to have a content marketing strategy to try to find the person who needs it when they need it. I'd have to place marketing materials at the locations where the need for that 3D map might arise, in the real world, or do marketing campaigns in search engines or travel portals to let people discover it. So not only do I have to pay all the money necessary to accomplish the 3D map generation, I have to ensure people can find it. That's a particularly baroque undertaking. There are indeed people who are working on the first part of the problem, making the 3D models with lidar and photogrammetry stitching for the benefit of posterity. Yet people with those talents don't typically have the marketing savvy to address the discovery side of the user experience. That's where the free-to-index search architecture that popularized sites like Yahoo!, Bing and Google come in. But the mechanisms that made those companies possible can't be directly extended to solve this search challenge. A new means of sifting data needs to be deployed for this case.

Fortunately the market is evolving significantly to the point that the fundamental tools of AR app development are available cheaply to any developer. But the free-to-index public repository for location-based content is not yet solved. A few candidate approaches have been proposed in different countries. In Japan, Tonchidot suggested the approach of a generally discoverable "air tag" with coordinates and content types that could be visited by a user with idle time using an app capable of posting and rendering the content. 

In the US, QR-codes are typically used to engage location or topic-specific content queries from a physical sticker or billboard. The team at Verses Labs proposes a global domain registry approach for location mapping that is similar to the DNS lookup table approach of the Internet domain registry legacy model. (The administrative body for DNS registrations is called ICANN.) Recently, an Estonian company, Over Holding, has proposed a concept whereby developers share a blockchain record of location assets with a tenancy privilege for developers who publish to or buy the location "hexagon" where the AR content will be placed. This isn't meant as NFT land grab hype. They envision a model whereby virtual estate needs to be able to fluctuate up and down in price in a means similar to the open competitive market that defines price of physical real estate that it is emulating. What Clear Channel (US) and Ströer (EU) is to the physical advertising world in billboard marketing, they'll mimic with digital equivalent of "rights to display" in the shared space that developers leverage on their digital monetization architecture. 

While the ICANN domain registry approach yielded many free market search engines in the past, this could be an exceedingly complex centralization effort to run for the entire globe. Whichever approach for location-based posting/indexing emerges, it will need to develop defenses just like the web-index techniques for spam prevention, content preferences, filtering and ephemerality/freshness if it is to become valuable and beneficial to us on a broad scale.

It will be interesting over the next few years to see how the discoverability and publishing rights for location advance. It's too early to tell whether we'll stay in-app for the next decade or go toward one of these more decentralized models for content sharing. A few more companies need to jump into the pool before a good standardization effort for cross-platform content visibility and share-ability emerges. While it may seem a Sisyphean effort to chronicle and map the world when we don't know yet what the eventual shared standard is going to be, I think it's a valuable expense of resources for the web of tomorrow, which should ideally be specifically relevant to us based on where we are, not just who we are.

Bridging the uncanny chasm

I remember my first experience visiting the Oregon Museum of Science and Industry in Portland when I was a teenager. I had always been fascinated with sciences. So a playground of interactive exhibits with hands-on experiments was just my kind of thing. I particularly recall my first meeting a conversational chat-bot that they had installed which could output considerate responses to questions the user typed in. Dr. Know, as it was affectionately named, was an instantiation of the Eliza program from MIT Artificial Intelligence Laboratory. It was an example of a Turing test, an idea put forward by Alan Turing, an early pioneer in computer programming. He theorized that computers would eventually be indistinguishable from humans once their logic structures matched those we form through socialization processes. The Turing test is a process of inquiry followed between a human and a robot that would be used to figure out if the robot was sentient or not. (Ironically, on the internet, servers spend a considerable amount of time giving internet users Turing tests before they allow us to view websites. These "Completely automated public Turing tests to tell computers and humans apart," CAPTCHAs for short, are meant to lessen the amount of time that web servers spend dialoguing amongst themselves without benefiting humanity in some way.)

These days, a majority of American households have at least one device that is capable of representing an interface to a conversational virtual assistant. Computer vendors have embedded speech-to-text inputs in hardware they vend to encourage us to speak questions to their respective cloud service virtual assistants. (Cortana, Siri, Alexa, Alice, Google Assistant) Their goal is to eventually obviate the need for keyboard entry for the next generation brain-computer interfaces. Staring at phone screens can be fun. But it can distract humans from leading normal lives rich with interpersonal social interaction, with humans. A lot of our time each day we spend talking through our fingers using secondary representations of language. If enough people offer up their voice patterns, computers can learn all accents to thereafter bypass the alphabetic language we typically type at them. Once they speak the same language as us, without the distance of the plastic/silicon intermediary of the computer screen, our near and distant relationships will return to a more normal means of communicating, and we’ll spend far less of our lives communicating through finger gestures.

Each of us have had our own experiences conducting Turing tests with machines in the home or on phone lines, trying to navigate their logic structures and capabilities. I’ve seen lots of failed attempts to bridge what Masahiro Mori called “The Uncanny Valley” of foreignness that divides humans and machines, preventing them from forming comfortable trust-based relationships. I’m fascinated to watch the emotions people use to express themselves when they know their counterpart is not human. I’m impressed with the tricks the tech companies use to embed implied emotion through tone of voice their virtual assistants speak at us. Ironically, it’s usually the humans that sound like robots in these interactions, with flat matter-of-fact insisting rather than sing-song tone of voice typically used among fellow humans we seek to convince, sway or implore. 

Some people bristle at the idea that we would refer to a machine-learning algorithm plus a database as an artificial intelligence. It is as if the term intelligence needs to be an honorary title conferred only on those of us that have and express emotion, beyond vocal inflections. When I was testing the Turing computer in OMSI (or it was testing me) I was enthralled to see that the machine could grapple with the rules engine we use to socialize. The size of its knowledge database didn’t matter to me so much as its deft capability to generate an acceptable output response to well-framed questions. It took me about 30 minutes before I had mapped out the logic frameworks the programmers had used in the conversational flows. I could eventually predict how it was going to answer any question I posed. After that happened, I became satiated that I understood this mechanical friend well enough and I could go on with my day.

As children, we interface with the world intensely to find the external connections that give us pleasurable or negative reactions. I walked away from Dr. Know feeling like it had dead-ended too many of my questions. I'd gotten to the end of the dialog maze. It's a similar experience to what a lot of us have with smart speakers these days. They sometimes can’t progress in a dialog unless we buy something or augment them with some third party “skill” not readily on hand. As we come to depend more on virtual companions we don't want to hear, "I'm sorry Dave, I'm afraid I can't do that" when we're facing time-critical challenges. Human patience has a much shorter fuse than bot patience.

 

I recently joined a company, Akin, that is developing AI for use in assisting people with exactly those time-critical decisions and actions. My team, who formerly worked on the Watson platform at IBM, are extending the use of AI technology to help engineers working in complex assembly contexts and in families coordinating inter-dependencies across the family unit. Watson is the conversational AI that was designed, like Eliza, to banter back-and-forth in dialog with random questions. It was famous for beating Jeopardy champion Ken Jennings at his own game. Knowledge games are an area where AI can outmaneuver humans increasingly over time as Moore’s Law of increasing computer power favors machines. Humans can’t scale at the same rate unless they augment their capacity with external data sources, other people, or data from the internet.

AI platforms can be specifically good at repetitive tasks, (Set a timer; Turn on lights; Turn up the volume) state monitoring (Weather forecasts; There's somebody at the door; Your package will arrive tomorrow). AI assistants can enhance our effectiveness by helping to stay on task and focused to avoid distraction. Beyond just assisting, we are learning more creative applications of AI where they are able to significantly advance in fields of inference such as pattern recognition, protein folding and vaccine creation. The more we can delegate tasks to AI, the more we free the human mind from burdensome task management so that our minds can exercise their own strengths.

I look forward to the point that we can engage and collaborate more with our in-home AIs. Relationships thrive when the output is more than just the sum of inputs. We want to have robotic assistants that do far better than just what they’re told, or respond factually to what they’re asked. Just as friends propel us to maximize our own personal potential, AI assistants should be able to amplify our efforts toward a goal. To do this they have to get beyond our human tendency to repel things that are uncannily familiar. We need to find ways to let them draw closer, allow us to invest more of ourselves and rely more on them. In order to build trust with Akin’s AI assistants, the founders have incorporated as a public benefit corporation and are working with universities and health researchers to measure the quality of life benefit that results in using Akin AI in the home. 

We look forward to sharing more of our advances over the coming years.

Defending against quantum-computer hacking using biometrics

In 1978, BBC radio ran a satirical exposé about a group of hyper-intelligent pan-dimensional beings who were trying to get inside the brain of a human named Arthur Dent, who was a fugitive from the planet Earth. The radio show was so popular that it went on to become a book, a TV series, a movie and recently has been re-released on CD for future generations in its original audio format. 

The story, called The Hitchhiker's Guide to the Galaxy, was told from the perspective of Arthur, who had no idea the importance of his former home, Earth, which had been destroyed shortly after he escaped. Neither did Arthur have an understanding of the tremendous significance of his own brain. The pan-dimensional beings, which appeared to him as mice, were actually the administrators of a massive planet-sized computer. They regarded Arthur as just a circuit in the computer that they managed. The mice had been conducting experiments on humans from inside Skinner boxes in the laboratories of human psychologists for many years. (Meanwhile, the psychologists believed it was they who were experimenting on the mice!)

Deep Thought, on the future computer, Earth

The Earth had been planned by another more primitive computer, called Deep Thought, which argued that life forms would be the circuits of this great future computer that could come up with the great question of "life the universe and everything." Humans would therefore wander its, staring at Brownian motion in their tea cups, pondering the existential question for the benefit of all sentient beings in the galaxy, who eagerly awaited the outcome. Deep Thought had already revealed that the ultimate answer was 42. But determining the ultimate question would require millennia of pondering. The story alleges that finally a woman in a cafe in London had come up with the ultimate question while staring at her tea. This made the unfortunate destruction of the Earth very frustrating to the mice who'd been working on the surface of the planet for millennia prior. The mystery of the ultimate question was to remain hidden until the Earth could be replaced.

Martin Freeman as Arthur, facing interrogation by mice

However, the mice had reason to believe Arthur might have remnants of the great question inside his mind, as he was the last vestige of the original Earth program. From their perspective this secret question was much more valuable to the galaxy than his brain was of value to Arthur. Yet Arthur was reluctant to yield his brain. He was ultimately able to avoid being diced up by the mice. But he did end up marooned on the surface of Earth Mark II, a rebuilt computer based on the original Earth blueprints after the mice decided to start again from scratch.  (Find out what happened to Arthur on Earth Mark II, starting at Episode 6 of the BBC radio show to hear of his adventures thereafter.)

Rising up from the perspective of the radio series, we may assume that we humans are actually on the second coming of Earth. In our own narrative, humans have just built their own hyper-intelligent computers which we call quantum computers. These crafty computers have circuits that can think three-thoughts instead of just two as with their predecessors, binary computers. A great deal can be achieved by allowing a circuit to go from thinking yes/no (0 or 1 in a logic gateway) to thinking yes/no/maybe! Just two years ago in Nature magazine we read pronouncements that a group of scientists had used a quantum array of circuits to demonstrate “Quantum Supremacy” for their particular computer in terms of a high-speed of calculation. While this is great news for anyone with a quantum computer, it was suddenly bad news for everyone else's non-quantum computers, as it implied that the rest of us would have to go back to the drawing board to try to figure out how to secure our binary-logic computers and computer networks that were suddenly deemed less supreme.

There may be no hyper-intelligent pan-dimensional beings trying to hack our skulls. But there are a bunch of ordinary folk who plan to use these computers, like those pesky mice, to peer into our networks and steal our secret questions, as they've been doing with binary computers and phishing exploits for decades. Our legacy means of encrypting networks have been based on introducing complex hashes of data. Introducing hashes with mathematical complexity, referred to as "introducing entropy" or "cryptographic salts," makes the decryption of such data without access to keys too complex for a binary computer. As we saw with the Sycamore quantum array, a process that could take 10,000 years for a binary super-computer, takes mere 200 seconds for a quantum array. (This was followed by the Jiuzhang computer which claimed to be even faster.) Theoretically such a fast computation process could be used to apply Shor's algorithm to factor RSA-level crypto-keys while the keys were still in use.  This implies that we need to introduce greater algorithmic complexity to eliminate this vulnerability should such computers be used for decryption in the future.

PQ Solutions, has been working on this challenge of protecting legacy networks and software from threats emerging in the post-quantum era with a means that is both backward compatible with RSA networks, yet future-proof against decryption attacks regardless of computer speed. While we're currently standardizing this cryptographic approach with the US Department of Commerce NIST working group, we are also introducing products in the market today to allow other companies to have cryptographic-agility to layer in this new standard once the NIST process is complete. (Final Post-quantum cryptographic standards will be announced next year and required for all government service providers thereafter.)

Our identity validation platform Nomidio, allows companies to ensure that they only authenticate users for access to their secure/private networks after they've been biometrically proven to be who they say they are. You may wonder why we think biometrics are the key to quantum-safe encryption. We will be presenting on this topic in the upcoming conference Quantum Business Europe. Please join us if you're available to hear about our products and our philosophy of networks protected against such threats. But for those who can't join the conference demos, I'll elaborate on our approach.

Computers will be better over time at factoring large numbers which we use in defense against binary computers for RSA encryption. So we need to change the game with something that computers can't factor or decrypt. We can borrow a concept from Deep Thought, that humans are the answer for the challenge. Computer-stored passwords are a vulnerability we all know because they are static in time and typically stored partially in the clear through a process called public-key cryptography. We are among a broad consensus of security companies that advocate for transitioning to passwordless network protection. Just like the increasing incidence of car theft by capturing radio signals from the key fobs, we now have to ensure our keys are not left in a place where their signals can be captured. 

PQ Solutions' approach to securing network end-points is by introducing live performance of biometric proofs into the encryption process. Quantum computers can be used to simulate incredibly complex mathematical equations and physical systems. But a quantum computer wouldn't be able to simulate a human. By sampling behavioral elements of a live authentication flow we can ensure machine-based intrusions are not able to access a network or breach static encrypted files signed with the biometric hash. Unlike car keys and their RFID signals, your identity can't be stolen from you.

The benefit of using an "Identity as a Service" (IDaaS) platform is that companies don't themselves have to hold any biometric data on their servers. Remember the European privacy regulation GDPR which tightly regulates data collection and protection? That's why a company's chief technical officer does not want to build an in-house biometric database of their users.  Nomido IDaaS provides a zero-knowledge cloud-based solution for identity validation so CTOs can delegate access for proven individuals internally, while outsourcing identity proofing in their access management technology stack. Our goal with Nomidio is to give companies vault-like biometric authenticity checks without causing a large data footprint for our relying parties and partners.

A secure network is like the hull of a submarine. Deep underwater the hull of a submarine is hardened against leaks. If you wanted to put a window in a sumbarine, you'd have to ensure that it was as pressure-tight as the hull of the submarine itself. Nomidio does just that. As a user is granted access into a network they have to provide a multi-factor proof that they are who they say they are. Their biometrics are then woven into the encrypted session access key used to grant visibility while certifying that their access token and cannot be duplicated, captured or recreated by any person who is not them. This is beyond just proving they have the phone or the RSA key-fob of the formerly-approved employee, as in the case with 2-factor apps or SMS based systems. With Nomidio, a user must match the live facial likeness of an authenticating user, along with an authenticity check of their biometric voiceprint as they log in. Both of these, as separate factors in the multi-factor authentication, cannot be attached to each other and cannot pass based on past recordings or images of the same person.

If you're interested to learn more, attend our free-pass demonstrations at Quantum Business Europe, or visit nomidio.com to learn how to integrate using open protocols OpenID or SAML. We provide a 1 month free access demo account in Amazon Web Services and Azure Marketplace.

Please enjoy our videos from the Post Quantum Europe conference. (n.b. PQ Solutions is a platinum sponsor of this conference.)

• Christopher's speech on leveraging biometrics in network defense: Link to Youtube
• Andersen Cheng's summary of the timing of the quantum hacking threat: Link to Youtube
• CJ Tjhai's presentation on hybrid encryption with post-quantum cryptography: Link to Youtube
  

For more information on Hitchhiker's Guide to the Galaxy visit:
2005 Cinema version: https://www.imdb.com/title/tt0371724/ BBC TV version: https://www.justwatch.com/us/tv-show/the-hitchhikers-guide-to-the-galaxy BBC HHGTTG Legacy Link: https://www.bbc.co.uk/programmes/b03v379k/episodes/guide Recently re-published CD collections:  https://www.amazon.com/Hitchhikers-Guide-Galaxy-Primary-Phase/dp/1787533204

 

From cosmology to quantum computers

Over the past few decades, the concepts of quantum mechanics have been sprinkled into our common lingo and news headlines in a gradual crescendo. Concepts that were mere theories at the beginning of the last century are now demonstrable facts. As uneasily as we sit with their implications, we carry about their applied technologies daily in our pockets. These concepts have been fascinating for me to study. Though they may seem a bit esoteric to most people, they are becoming ever more personally relevant to our common experiences. 

Some concepts that are astounding, but that are increasingly ordinary in our daily lives:

• We control the quantum jumps of electrons to emit photons of the exact spectrum we want to light our living rooms with. Vice-versa, we use photons of specific spectra to control where we want electrons to go and how we want them to behave. (Photovoltaic effects, common in our technology as the light-emitting diode, also used to detect perturbations caused by the presence of a finger on a cellphone screen through electromagnetic repulsion, which sends signals to our computers chips.)
  
• We've mastered material science to force electrons into super-chilled wave-crystal states that allow us to research un-Earthly states of matter that exist in few places in our universe at its current temperature. (Bose-Einstein condensates, which have bearing on superconductors and may at some point also be applied to our technology decades from now.)
  
• We have been able to entangle wave-states of paired quantum particles, then beam them over distances in a vacuum, to then read the paired state information at a significant distance from the split. (Quantum teleportation, which may someday lead to entangled-particle communication many decades from now.)
  
• We've built computers that can write to and read from single quantum particles in a super-imposed wave state of two spins. (Quantum computing, just entering into application recently and soon to be used at a greater scale.) 
  

It's the last item I'd like to address in this blog post. Just last year, a quantum computer proved "supremacy" over classical computer circuits in calculation speed. While our ability to harness and control quantum particles is fantastic news for the advancement of our technology. It's a bit of bad news for our soon-to-be legacy computer encryption approaches. Specifically meaning, its has implications for the continuity of software and internet industries as we've come to depend on them. So it's worthy of attention. Over the next two years you may see or hear a lot more about it.

It takes a bit of time to explain why one would need to be "quantum-safe" in the context of these advances. The US government standards body, NIST, is currently in an open call for proposals to determine the new encryption standards for this "post-quantum" era, the way we speak of post-modern art. The goal of our government and contributing engineering teams is to protect our future software industry to prepare for the proliferation of quantum computers and a greater risk to decryption of secure data we use day to day. If you had awoken thinking we were just starting the quantum era now, you may wonder what is the nature of this era we're moving beyond, and wonder what are these computers are that use quantum effects in their operation. As is usual in my blog, I'd like to start the story with a digression about why I'm fascinated with this topic. If you haven't followed the emergence of this field, it may be interesting.

I remember as a teenager listening to a satirical BBC radio show called The Hitchhiker’s Guide to the Galaxy. In this show, Douglas Adams postulated a spaceship that could leap through spacetime to different locations simply by calculating the precise "improbability" of the spaceship being at any specific location. This concept was inspired by the idea of the “quantum wave function,” of quantum field theory. The wave function is a probabilistic formula for modeling wave-particle trajectories, applied to interpretation of the debris coming out of atom smashers. (See Feynman Diagrams for more on this.) We know that we can’t pin down the precise location of a particle, due to the Heisenberg uncertainty principle, so the wave function is an approximating model using statistics to refer to the locations a particle is most likely to be found. In his narrative, Douglas Adams reversed the concept of the quantum wave function such that a probabilistic calculation caused the effect of making the spaceship appear in random places in the universe. (Description of his hypothetical “improbability drive” is here.) The idea that particles could hop through spacetime outside of a linear trajectory was my first introduction to what people called the “weirdness” of quantum mechanics. These hops are referred to as quantum jumps, or in some specific cases quantum tunneling.  

(Side note: I tend to say “particle-waves” because the term particle is loaded as a concept implying that matter is tangible. Our current approach to describing sub-atomic components is to acknowledge that they behave as both particles and waves depending on how they are measured. We could say, “waves that were formerly known as particles” because we've learned from the last 100 years that our world is non-tangible in the particulate sense. What we generally see behave as object-like particles around us are actually energetic waves that appear as point particles when disturbed or obliterated. That’s the most peculiar thing about quantum mechanics that has fascinated me. What we think of as tangible is actually an emergent property of various energetic fields that react strongly against each other in spacetime. As atoms are mostly empty beyond these energetic wave interactions, it is an odd conception to think that everything around us that we interact with in tangible ways is mostly empty vacuum in a field of tightly-bound highly-energetic ripples of force. Another term that took getting used to is the idea of fused time+space that we just call spacetime now because Einstein's theories of relativity demonstrated the inter-connectedness of the dimensions of space with time.)

When I was studying physics in high school, I remember the day that my professor was discussing the Bohr model of atomic structure. After lab was done, I went to ask professor Rolfe what the other diagrams were on the back pages of the charts as I paged through them. They were peculiar-looking globular shapes that depicted the electron configurations of various elements in 3D. These maps showed the probabilistic location of electrons on the outer shells of atoms, where their negative charges bulged away from the positively-charged proton. Bohr diagrams depict the atom as a 2 dimensional disk, like the model of planets in our solar system, which makes it very easy to envision their chemical combinations with other atoms. Yet in experimental demonstrations we find that atoms are like bumpy balls of positive and negative charge with multiple poles. Professor Rolfe explained, "We refer to electron positions as 'orbitals', but they actually don't orbit the proton. Rather they buzz around in a field of space at specific areas on the outermost edges of the proton's positive charge field." "Wow! Can we study that next class?" I asked. "No. We have to focus on the curriculum, which is specific to chemical bonds. You'll get to quantum mechanics when you get to college," he explained.

One thing parents and teachers learn about kids is that the best way to challenge and inspire them is to tell them they can't do something! Professor Rolfe saw the twinkle in my eye and knew he was sending me off to the races. I talked with my father about it. He in turn suggested I delve into General Relativity first, and started me on the path toward my fascination with cosmology with a book called "Relativity Visualized." Einstein had been an early contributor to quantum mechanics. But to understand why Einstein was provoked into his study of particle-entanglement, it was important to understand the ideas behind his general and special theories of relativity. From there, I branched out to read more about quantum field theory and the new concepts of how relativity and quantum mechanics should eventually dovetail in cosmology as part of a unified theory of the four fundamental forces of nature. Nuclear strong force (which binds atoms together), nuclear weak force (which causes atomic radiation), electromagnetic force (governing electron and photon behaviors) and gravity (which is currently best described by the geometry of spacetime and has evaded an elegant tie into quantum field theory.)

Relativity was particularly strange for me to grasp: the equivalence of mass and energy, the inter-connectedness of space and time, the limits of light speed and the warping of the spacetime continuum described by the presence of mass/energy. While it didn't sit well in my Newtonian-focused understanding of space and causality, I could grasp the ideas of relativity's predictions, which are being confirmed on every astronomical observation that has happened since. If you've watched Nova or any other science shows about relativity you may have seen demonstrations of the spacetime continuum as a 2D surface that creates indentations where stars are present. That's indeed one great way to visualize gravitational distortions of spacetime. But when you consider the idea of time dilation for objects moving at near light speed, a different aspect of relativity from the mere presence of matter, you get a slightly more peculiar view of the implications of the nature of our spacetime. You get a sense of our universe being made of a kind of taffy-like constituency. 

The large scale spatial-temporal view of our cosmos is best grasped by thinking of the behaviors of gravity and light acting against the background tapestry of spacetime. Yet light waves, when studied on the microscopic scale behave much like the particles that we accept that we're bodily made of. One way to bridge our thinking from the cosmic scale down to the scale of our substantive selves is to focus on the similarities between electrons and photons. These two wave-particles that make up the dual components of the photoelectric effect are tightly bound to each other and yet seem so vastly different in their natures and observable behaviors. I'm calling it dual components because photons (light waves) are emitted/generated by the hop an electron makes when it leaps from a high-energy orbital to a lower resting state closer to the protons in the atomic nucleus. Conversely if an atom is struck by a photon and its energy absorbed, the electron hops to a higher orbital.

Photons have the distinction that they can move faster than all other known quantum wave-particles such as electrons and quarks. But the head-scratching really starts once you begin to delve into those specific behaviors of non-photon quanta. Why should the photon be able to travel at the fastest allowable speed known in our universe, but the other quanta are not? The photon is not a charged particle like the electron or quark. That's one clue. Why does not having charge imply that it gets to travel in the fast lane while other quantum waves can't? This puzzle continues to riddle the theorists who've been investigating the explanations such as (currently theoretical) non-space dimensions that we can't perceive directly which bridge our thinking to a hidden mechanics of our cosmos. But I shouldn't get too far down the rabbit hole here. There are great publications to follow on quantum-gravity theories, string theory, black holes and the holographic principle that I encourage interested folk to look up. (Greene, Hawking, Susskind and Scharf are good authors for the enthusiastic.) Though I find the cosmology topics fascinating, I'd like to put those concepts aside for the moment and to drill down into the aspects of quantum field theory that specifically apply to our emerging technology applications brought about by harnessing these behaviors for practical purposes by tweaking these tiny waves in our machinery. So leaving the "spacey" aspects of this topic there, I'll progress now the specifics of where we are applying these new capabilities to our machines in the imminent future.

Progress harnessing the weirdness of quantum wave/particles in computers

IQM Quantum Computer

Digital technology for the mass consumer market of the last century has been dependent on our exquisite mastery of photo-electric effects in conductive metals, vacuum tubes and silicon wafers. Moore’s law has predicted continual leaps in the processing power of binary computer chips over the past decades based on our ability to pack logic gateways densely together on a chip. But at this point the advancement we are making with quantum computers is based on an entirely different nature of the logic gateway itself. It’s not because of just the smallness of our computer circuits that will result in the next leap in computing power. Rather, it’s because we’re jumping beyond the concept of a binary circuit completely. Put most simply, instead of creating a logic gateway of 0 or 1, we can now create a triple logic gateway that consists of 0, 1 and 1⁄2. (Not literally 1⁄2, but a juxtaposition representing 0 and 1, which I'll get to later.) That’s what all the buzz is about. This may seem like a small incremental jump for a single circuit. But on a meta-layer, if every logic gateway of a computer were a 3-bit instead of a 2-bit circuit, the advancement of an array of circuits in terms of computing power would be absolutely tremendous. The quantum computer is just that, an array of quantum bits that can coordinate calculations incredibly rapidly. 

In the United States, we had a moment of transition where the FCC and all TV manufacturers made a switch over to preferring digital over analog radio transmissions. All old TVs had to use a new digital converter to receive the new spectrum. Nothing like that is going to happen in this case. We’ll certainly always be using binary computers. They’re so useful. And we’re adept at producing them with low resource cost now. The new computer circuits we’re discussing don’t obviate the need for earlier technology. They’re just useful for entirely different purposes. The emergence of the calculator (Professor Rolfe used to call them "calcu-sooners") didn’t mean the abacus and slide-rule were no longer useful. Calculators are preferred because of their higher rate of performing calculations. Same will be so of quantum computers as they become more common. Anyone with a really powerful calcu-sooner will beat anyone who tries to conduct the same calculation with a contemporary binary calculator. For instance, the alleged speed of the computer cited in the quantum supremacy proof above is such that a calculation which would take a traditional binary super-computer 10,000 years to complete could be completed in 200 seconds with a quantum computer.

"How are they creating these cool calcu-sooners?" you may ask. For this we have to go back into the somewhat spacey stuff again. This time turning from the telescope to the microscope, we adjust the focus of our lens to peer at the nature of space and energy within atoms. It may sound strange to say we need to look at space here. But remember Einstein's direct correlation between energy's presence in spacetime warping it's fabric. Energy can't exist in space without altering it. Theories emerging since relativity have explored and tested this concept of the bindings of wave-particle motion to the underlying spacetime. It is at this sub-atomic scale that we see the peculiar behaviors of matter that yield the opportunity of manipulating the quantum bits in a near-frozen state. 

At the smallest scale, during the “physical” interactions we cause in atom smashers, we observe peculiar states of fermions and force-particles called bosons that spring out of ordinary atoms. We can observe, at some energy densities, exotic wave-particles springing into existence and nearly immediately disappearing as if they emerged from a sub-strata underneath our visible cosmos that is filled with something else entirely, summoned only by the energy density of these small but intense particle collisions. These exotic states of matter can only have a fleeting existence at our world's temperatures and densities. (Our universe formerly had much higher temperature and density 13.8 billion years ago. So these high-temperature interactions give us a way to glimpse the nature of wave-particles in spacetime from a different perspective, as they were more commonly once long ago.)

Going in the opposite temperature direction, when we chill particles in an area of vacuum as cold as is possible for humans, we see other exotic behaviors such as quantum particles existing in super-positions of overlapping opposite states of spin in a single moment. We also observe harmonized in ice-like "condensate" states, where a bunch of disparate particles behave in a uniform super-fluid. Why should matter be able to harmonize and be in super-positions of two states at once? Why this is, or the cause of this phenomena, is the subject of fascinating explorations and debate. We don’t precisely know yet. But suffice it to say that we can now super-chill atoms to the point that they start behaving in this exotic way in a way that is useful in a machine. We can now write and read using their hovering in this super-imposed state between two absolute values. As long as we keep them in this low temperature state we can harness their fluctuating state as a computer logic gateway. This is where the concept of spin-up/spin-down/both-at-once comes from in the 1/2 value mentioned above. By leveraging this "shimmer" between the spin states, we get our third state for the logic gateway that we need to create the quantum computer.

Were temperature to be slightly higher, altering and measuring the spin of a quantum bit would not be possible. The more bits we add into the array, the more chaotically the machine behaves, and the less utility we can derive from the system from the perspective of a useful computer. Achieving a reasonably robust array of quantum bits is incredibly challenging due to the temperatures needed to keep the array stable. The Sycamore quantum computer used in the above demonstration had only 54 bits. Lots of work goes into making these sensitive machines. So we can somewhat infer that they are not going to proliferate particularly rapidly in the commercial world due to economic constraints. Yet in a few years there will likely be thousands of them.

I'll leave it to the scholars and inventors of our next generation to talk about the wonderful things we'll be able to do with this advanced computing power. It is good news that humans are now on the brink of this great new capability. My near-term focus, as with much of the software industry's security engineers in the coming years, will be to ensure our legacy computer industry will be able to isolate quantum computers from the rest of our legacy networks and software. The way we build secure networks today is by leveraging mathematically complex hashing of data that is transmitted over the internet. By sprinkling in "RSA" level encryption hashes (our legacy standard), you can feel reasonably secure in using your home internet connection to log into your bank account with the assurance that no mid-stream interference will make your account vulnerable. So long as the encryption hash you use can't be factored by a computer during the time that your login credential is in use, you're safe. That's where the concern arises for the hypothesized concept of a decryption attempt leveraging a quantum computer. Most hashes take mere years to calculate on an ordinary computer. So if the factoring capability were greatly enhanced, even the currently most-secure hashes could be vulnerable.

Most of us don't need to be too concerned about this. Only the companies we rely on to secure our communications and web services do. That's why our industry is transitioning to post-quantum cryptography over the next few years. I hope by now this all makes sense and you can understand why the US Department of Commerce is doing this. Next time you read of post-quantum (insert-noun-here) in the news, you'll be generally up to speed on what they are talking about. It is simply trying to save our past and present from vulnerabilities that could emerge in the future. Therefore it's referred to as "future-proofing" so that we can go on using our legacy know-how and open web protocols safely in the future. 

We have seen ordinary citizens regularly be targeted by hackers no matter how obscure they may think they are. No certain person is likely to be the target of decryption attacks. If such attacks happen with a quantum computer, they will likely be focused on networks of information that are deemed valuable. The economics of cost implied by the complexity of quantum computers could lead us to conclude that in general they will be used to only solve very interesting opportunities and problems. But some assert that state secrets and financial institutions are so interesting that they will likely be the first vectors of exposure for the decryption experiments of state-sponsored actors who would gain from such access or information. Internet and software companies don't sit around thinking that their customers are too obscure to be interesting targets. So many of the companies we rely on daily will be implementing these new encryption standards when they are finalized. At some point you'll be asked to upgrade your software with a new set of tools and protocols that provide this new standard for security.

Entangled-particle cryptography

Post-quantum encryption that the NIST team at US Department of Commerce is researching doesn't involve quantum field theory specifically, as it is a defense against quantum computers. One of the most exciting concepts I'd enjoyed reading about is quantum effects of nonlocality and quantum entanglement, which lead up to what we've been hearing about in the media as "quantum teleportation." Brian Greene explained at length in his book, The Elegant Universe, how this idea of teleported information through the manipulation of particles, which were paired earlier in time, could create the potential for a perfect cryptography. 

The mechanism of using paired particles involves a process of first creating an entangled wave state between two particles, or splitting a single quantum wave in two, which can then separate spatially over time to transmit information when those particles are then read in remote locations. The concept stems from a prediction of Einstein, Podolsky and Rosen, abbreviated as the EPR Paradox. This process of causing interaction between two particles, inherently connected while spatially separated, was what Einstein termed "freaky action at a distance." We are now relatively proficient at using beam splitters or other light-control means to cause the entangled wave states needed to do this kind of rendering, sending and capturing of particle signatures at short distances. But they can't be used in long-distance telephony at present because the wave state involved is too fragile. You'd typically need a vacuum to utilize the signals, which are not easy to create on the surface of Earth over long distances.

The broad application of this kind of reliable stream of paired-particle plumbing for the purpose of messaging is many orders of magnitude more complex in scale than even the work involved in creating a quantum computer. So the practical application of the benefits is many decades further in the future than where we find ourselves today. So our best hope for a reliable encryption for our internet and software industries is to rely on means of encryption that are designed to be too-highly complex for a computer, of any type, to be able to decrypt. The methods to create this kind of incredibly strong encryption are themselves fascinating. So I'll get to those in a subsequent post.

While it will be hard to determine when quantum computers will be turned toward exploiting vulnerabilities in legacy encryption, there are many who would agree that it's time to start battening down the hatches now.