Right and Wrong in AI


The DARPA Cyber Grand Challenge (CGC) 2016 competition has captured the imagination of many with its AI challenge. In a nutshell, it is a contest where seven highly capable computers compete, and a team owns each computer. Each team creates a piece of software which can autonomously identify flaws in their computer and fix them and identify flaws in the other six computers and hack them. A game inspired by the Catch The Flag (CTF) game which is played by real teams protecting their computer and hacking into others aiming to capture a digital asset which is the flag. In the CGC challenge, the goal is to build an offensive and defensive AI bot that follows the CTF rules.

In recent five years, AI has become a highly popular topic discussed both in the corridors of tech companies as well as outside of it where the amount of money invested in the development of AI aimed at different applications is tremendous and growing. Use cases of industrial and personal robotics, smart human to machine interactions, predictive algorithms of all different sorts, autonomous driving, face and voice recognition and others fantastic use cases. AI as a field in computer science has always sparked the imagination which also resulted in some great sci-fi movies. Recently we hear a growing list of few high-profile thought leaders such as Bill Gates, Stephen Hawking and Elon Musk raising concerns about the risks involved in developing AI. The dreaded nightmare of machines taking over our lives and furthermore aiming to harm us or even worse, annihilate us is always there.

The DARPA CGC competition which is a challenge born out of good intentions aiming to close the ever growing gap between attackers sophistication and defenders toolset has raised concerns from Elon Musk fearing that it can lead to Skynet. Skynet from the Terminator movie as a metaphor for a destructive and malicious AI haunting mankind. Indeed the CGC challenge has set the high bar for AI and one can imagine how a smart software that knows how to attack and defend itself will turn into a malicious and uncontrollable machine driven force. On the other hand, there seems to be a long way until a self-aware mechanical enemy can be created. How long will it take and if at all is the main question that stands in the air. This article is aiming to dissect the underlying risks posed by the CGC contest which is of a real concern and in general contemplates on what is right and wrong in AI.

Dissecting Skynet

AI history has parts which are publicly available such as work done in academia as well as parts that are hidden and take place at the labs of many private companies and individuals. The ordinary people outsiders of the industry are exposed only to the effects of AI such as using a smart chat bot that can speak to you intelligently. One way to approach the dissection of the impact of CGC is to track it bottom up and understand how each new concept in the program can lead to a new step in the evolution of AI and imagining future possible steps. The other way which I choose for this article is to start at the end and go backward.

To start at Skynet.

Skynet is defined by Wikipedia as Rarely depicted visually in any of the Terminator media, Skynet gained self-awareness after it had spread into millions of computer servers all across the world; realising the extent of its abilities, its creators tried to deactivate it. In the interest of self-preservation, Skynet concluded that all of humanity would attempt to destroy it and impede its capability in safeguarding the world. Its operations are almost exclusively performed by servers, mobile devices, drones, military satellites, war-machines, androids and cyborgs (usually a Terminator), and other computer systems. As a programming directive, Skynet's manifestation is that of an overarching, global, artificial intelligence hierarchy (AI takeover), which seeks to exterminate the human race in order to fulfil the mandates of its original coding.”.  The definition of Skynet discusses several core capabilities which it has acquired and seem to be a firm basis for its power and behaviour:

Self Awareness

A rather vague skill which is borrowed from humans where in translation to machines it may mean the ability to identify its own form, weaknesses, strengths, risks posed by its environment as well as opportunities.

Self Defence

Capacity to identify its shortcomings, awareness to risks, categorizing the actors as agents of risk and take different risk mitigation measures to protect itself. Protect first from destruction and later on from losing territories under control.

Self Preservation

The ability to set a goal of protecting its existence’ applying self-defence to survive and adapt to a changing environment.

Auto Spreading

Capacity to spread its presence into other computing devices which have enough computing power and resources to support it and to allows a method of synchronization among those devices forming a single entity. Sync seems to be obviously implemented via data communications methods, but it is not limited to that. These vague capabilities are interwoven with each other, and there seem to be other more primitive conditions which are required for an active Skynet to emerge.

The following are more atomic principles which are not overlapping with each other:


The ability to recognize its form including recognizing its software components and algorithms as inseparable part of its existence. Following the identification of the elements that comprise the bot then there is a recursive process of learning what the conditions that are required for each element to properly run . For example understanding that a particular OS is required for its SW components to run and that a specific processor is needed for the OS to run and that a specific type of electricity source is required for the processor to work appropriately and on and on. Eventually, the bot should be able to acquire all this knowledge where its boundaries are set in the digital world, and the second principle is extending this knowledge.

Environment Recognition

The ability to identify objects, conditions and intentions arising from the reality to achieve two things: To broaden the process of self-recognition so for example if the bot understands that it requires an electrical source then identifying the available electrical sources in a particular geographical location is an extension of the physical world. The second goal is to understand the environment in terms of general and specific conditions that have an impact on itself and what are the implications. For example weather or stock markets. Also an understanding of the real life actors which can affect its integrity and these are the humans (or other bots). Machines needs to understand humans in two aspects: their capabilities and their intentions and both eventually are based on a historic view of the digital trails people leave and the ability to predict future behaviour based on the history. If we imagine a logical flow of a machine trying to understand relevant humans following the chain of its self-recognition process then it will identify whom are the people operating the electrical grid that supplies the power to the machine and identifying weaknesses and behavioural patterns of them and then predicting their intentions which eventually may bring the machine to a conclusion that a specific person is posing too much risk on its existence.

Goal Setting

The equivalent of human desire in machines is the ability to set a specific goal that is based on knowledge of the environment and itself and then to establish a nonlinear milestone to be achieved. An example goal can be to have a replica of its presence on multiple computers in different geographical locations to reduce the risk of shutdown. Setting a goal and investing efforts towards achieving it also requires the ability to craft strategies and refine them on the fly where strategies here mean a sequence of actions which will get the bot closer to its goal. The machine needs to be pre-seeded with at least one apriori goal which is survival and to apply a top level strategy which continuously aspires for the continuation of operation and reduction of risk.

Humans are the most unpredictable factor for machines to comprehend and as such, they would probably be deemed as enemies very fast in the case of the existence of such intelligent machine. Assuming the technical difficulties standing in front of such intelligent machine such as roaming across different computers, learning the digital and physical environment and gaining the long-term thinking are solved the uncontrolled variable which is humans, people with their own desires and control on the system and free will, would logically be identified as a serious risk to the top level goal of survivability.

What We Have Today

The following is an analysis of the state of the development of AI in light of these three principles with specific commentary on the risks that are induced from the CGC competition:

Self Recognition

Today the leading development of AI in that area is in the form of different models which can acquire knowledge and can be used for decision making. Starting from decision trees, machine learning clusters up to deep learning neural networks. These are all models that are specially designed for specific use cases such as face recognition or stock market prediction. The evolution of models, especially in the non supervised field of research, is fast paced and the level of broadness in the perception of models grows as well. The second part that is required to achieve this capability is exploration, discovery and new information understanding where today all models are being fed by humans with specific data sources and significant portions of the knowledge about its form are undocumented and not accessible. Having said that learning machines are gaining access to more and more data sources including the ability to autonomously select access to information sources available via APIs. We can definitely foresee that machines will evolve towards owning significant part of the required capabilities to achieve Self Recognition. In the CGC contest the bots were indeed needed to defend themselves and as such to identify security holes in the software they were running in which is equivalent to recognising themselves. Still it was a very narrowed down application of discovery and exploration with limited and structured models and data sources designed for the particular problem. It seems more as a composition of ready made technologies which were customised towards the particular issue posed by CGC vs. a real non-linear jump in the evolution of AI.

Environment Recognition

Here there are many trends which help the machines become more aware of their surroundings. Starting from IoT which is wiring the physical world up to digitisation of many aspects of the physical world including human behaviour such as Facebook profiles and Fitbit heart monitors. The data today is not accessible easily to machines since it is distributed and highly variant in its data formats and meaning. Still it exists which is a good start in this direction. Humans on the other hand are again the most difficult nut to crack for machines as well as to other people as we know. Still understanding people may not be that critical for machines since they can be risk averse and not necessarily go too deep to understand humans and just decide to eliminate the risk factor. In the CGC contest understanding the environment did not pose a great challenge as the environment was highly controlled and documented so it was again reusing tools needed for solving the particular problem of how to make sure security holes are not been exposed by others as well as trying to penetrate the same or other security holes in other similar machines. On top of that CGC have created an artificial environment of a new unique OS which was set up in order to make sure vulnerabilities uncovered in the competition are not being used in the wild on real life computers and the side effect of that was the fact that the environment the machines needed to learn was not the real life environment.

Goal Setting

Goal setting and strategy crafting are something machines already do in many specific use-case driven products. For example setting the goal of maximizing revenues of a stocks portfolio and then creating and employing different strategies to reach that - goals that are designed and controlled by humans. We did not see yet a machine which has been given a top level goal of survival. There are many developments in the area of business continuation, but still, it is limited to tools aimed to achieve tactical goals and not a grand goal of survivability. The goal of survival is fascinating in the fact that it serves the interest of the machine and in the case it is the only or primary goal then this is when it becomes problematic. The CGC contest was new in the aspect of setting the underlying goal of survivability into the bots, and although the implementation in the competition was narrowed down to the very particular use case, still it made many people think about what survivability may mean to machines.

Final Note

The real risk posed by CGC was by sparking the thought of how can we teach a machine to survive and once it is reached then Skynet can be closer than ever. Of course no one can control or restrict the imagination of other people and survivability has been on the mind of many before the challenge but still this time it was sponsored by DARPA. It is not new that certain plans to achieve something eventually lead to whole different results and we will see within time whether the CGC contest started a fire in the wrong direction. In a way today we are like the people in Zion as depicted in the Matrix movie where the machines in Zion do not control the people but on the other hand, the people are entirely dependent on them and shutting them down becomes out of the question. In this fragile duo, it is indeed wise to understand where AI research goes and which ways are available to mitigate certain risks. The same as the line of thought being applied to nuclear bombs technology. One approach for risk mitigation is to think about more resilient infrastructure for the next centuries where it won’t be easy for a machine to seize control of critical infrastructure and enslave us.

Now it is 5th of August 2016, few hours after the competition ended and it seems that humanity is intact. As far as we see.

The article will be published as part of the book of TIP16 Program (Trans-disciplinary Innovation Program at Hebrew University) where I had the pleasure and privilege to lead the Cyber and Big Data track. 

Here’s How to Feed Your App Jealousy

This whole story is under the assumption that some developer is jealous with the app success of another guygirl. Jealousy is not the only reason to try to disrupt the success of another appdeveloper - direct competition can also be a "bad" reason to do so.   It all started one day when I saw that Look & Cook daily reviews report arrived to my inbox with something around thirty 5 star reviews. Cool? No. Each review was very short and lacked a personal touch. Of course it looked strange to me. It is not that I don't think Look & Cook does not deserve such daily treatment from its users:) but still my cynical sense got alerted. Judge it for yourself: 6a010536b66d71970c019b0046b284970d-800wi   So I checked out with my friends at Look & Cook to see whether someone decided to spend a hundred bucks on buying some good reviews just for fun and of course the answer was no. Look & Cook received naturally very good and authentic reviews from hundreds of users worldwide and we never had the urge to beef it up needless to say on the "lame" quality of such reviews.   So I had to do something about it since I knew deep inside how Apple's fraud detection algorithm can work and spot such anomaly. So I went to their developer portal and it happens that they have a fraud reporting form which I guess was meant for other type of fraud and not for a developer reporting about a fraud on his own app. Anyway, I reported there the incident assuming this at least would be an evidence that I am innocent. Someone could go into an argument that in the dynamics of conflict resolution this could be also a tactic on my end to take the blame off my shoulders. Still that was the only thing I could doat that time. I got the usual auto reply from Apple that "we got your report and we will take care of this" although I never got until now any human response. At that night our app was probably detected by Apple's algorithm and we got dropped from almost every country ranking where we used to rank high on almost every country on the globe. This got me sad. Our app climbed on the countries charts not because we bought downloads or something like that. People really like it and use it and tell their friends and it took us a long time to get there. Still I understood the limitations of an automatic algorithm which decides such things blindly and I waited patiently for someone on Apple's side to read my report. And then I got this frightening email with the dreadful subject of:

Notification of iOS Developer Program License Agreement (PLA) violation

Let me explain this - for a developer this is a point of crisis. It is like your mam and dad will get a long letter from the school administrator that you cheated on a test while you were the best pupil in the world and never done that in your life. It was actually even worse. I will not quote the email body but it had all kind of scary words like fraud and "terms of service" and I think you get it. So I got myself together and replied harshly, a bit, that I spotted that as well and I actually reported on this and never got an answer. Well, I did not get an answer on this mail too. Eventually the situation did not escalate and the app is slowly climbing the charts (since users like it of course:) all the way up back again. Also I am not getting anymore scary machine mails about me being a crook or something like that.   And now to the hypothesis on how this happened. If you'l google for "buy reviews" then you will notice that there is a whole industry behind it. You can buy reviews by the meter, quality, language and target store. So the most obvious line of thought I had, in my conspires mind I have to admit, was that someone who doesn't like us/our success/our app/food in general decided to spend a hundred bucks to buy some good reviews. But then my "trusting" side thought that it is probably a mistake and some 20 cents an hour worker just got the whole iTunes link wrong. So to the conclusions: To Me - Stop bragging too much about the app's success. No can do - I enjoy that. To My Close Friends @ Look & Cook - Sorry for being suspicious:) To The One Whom Did it By Mistake - Please pay attention next time. To The One Whom Did it By Intention - The Good Guys Rule!!! To Apple Developer Support - Please reply in a humanly voice to the developers' scared emails, someone can get a heart attack and the liability will be on you. To Apple's Anomaly Detection Algorithm Developer - Why can't you attribute some credit to an app based on its good history so such anomalies will not put it immediately in the same category of apps like "playing chess with one finger in my nose". Now I feel better:)    

The web is changing

I have been reading about the whereabouts of News Corp., Google and Microsoft in recent two weeks and I noticed something weird happening here about but could not put my finger on it. To those who do not know the storyline here is a short description posted on Hitwise today:

Two weeks ago we posted on Rupert Murdoch's threat to block Google from Indexing News Corp. content. While at first it seemed as though Murdoch was merely posturing with hypotheticals, reports continue to indicate that News Corp. is seriously considering choosing Bing as the exclusive "indexer" of their news content.

via weblogs.hitwise.com

At first I thought Mr. Murdoch was playing tricks on Google but when Microsoft entered the picture with their proposal to News Corp. to exclusively allow indexing of their sites on Bing only things got clearer. I am not talking about the not surprising tactic from Microsoft's arsenal but on a total different thing.

The new phenomena here is the change of balance between publishers and Google. The status que until today was that everyone just wished Google would index their websites and the more the merrier. Indexing meant traffic which summed up to more revenues from advertising. Industries has been created on this raw desire to be indexed on Google, for example SEO and SEM were millions of dollars have been poured into it. News Corp. as a big website with big assets understood that Google is no less important to them then the opposite.

I am not sure whether Bill Gates got it and talked to News Corp or the other way around (though it smells like Bill's way of thinking) but something has changed here.

Now that this has happened we can contemplate on few directions. For example what will happen if other websites will follow through and will deindex themselves from Google? And also is this happening because Google is no longer the main hub for getting users to websites where social bookmarking and tiny urls on twitter fill the gap? Is this the reason Google is developing operating systems to grab hold on users while they know they loose ground in the pure web market?

I think this is a serious topic for Google to think on.


What is wrong with Microsoft? Adobe Air vs. Silverlight

The new war on the rich desktop client is being won easily! by Adobe Air. You would expect Microsoft to win it swiftly  with all their expertise and real estate on desktops but somehow it is being played by them so cumbersomely. Microsoft first tried to chase the flash in-browser capabilities with their own version of Silverlight while at the same time Adobe was already crawling into the desktop. While still being stuck at solving the rich in-browser puzzle Adobe is becoming a very cool first mover to the rich desktop market. 

Few weeks ago Microsoft launched something called Out of Browser experience which looks like an unplanned feature aiming to strike back. SSB or Single Site Browsers are nothing new (see Fluid, Bubbles and Prism as examples) and a shortcut to a website is nothing even close to really installed client.
Maybe it is the cross platform portability aspect that holds things back, an issue within Microsoft that is not solved strategically yet (and should have been a long time ago after noticing that Windows is not the only operating system in the world). 

Actually their platform conflict raises an interesting strategic problem: on one hand their milking cow which is MS-Windows has to be advocated ferociously, on the other hand they look into other areas for growth and these areas potentially address users that are not feeding their cow but their rivals. And now to the big question: should I admit the existence of other users and support them and by that indirectly affect badly my cows condition in comparison to others. Or should I ignore them in hopes that others' cows will die without our support or at least not get stronger. I think it is a question of balance. First, how old is your cow? Every cow will die at the end and knowing when it is over the top is a good thing. Second, Is your cow really threatened by others and how the threat affects it in terms of making her life shorter. Third, has the business changed to chickes and eggs and you didn't notice it?

Sorry for the animals metaphor but I got into it:)

Blogging – A Competitive Double-edged Sword

 You've got a new direction for your company - should you blog about it or not? Few years ago many web 2.0 entrepreneurs were blogging quite openly on their plans and product roadmaps, right before web 2.0 has become a real competitive landscape. Nowadays, since there are already giants playing in the web 2.0 battlefield, blogging instinctively on your plans does not come automatically anymore (at least not for me:). It happens many times when an emerging market is created, entrepreneurs who deep inside are not sure in 100% that their new identified turf will become a viable market, are usually open about their inner thoughts and plans also as part of evangelism and for conducting some market response validation. My personal conflict now is whether this open "policy" on revealing your plans is a real strategy (as it was defined in the early days) or is it just something that was OK during the early days of web 2.0 but it is not so anymore. Dudu

Site Footer