{"id":1429,"date":"2024-03-10T18:46:02","date_gmt":"2024-03-11T01:46:02","guid":{"rendered":"https:\/\/kihbernetics.org\/?p=1429"},"modified":"2024-03-10T20:01:44","modified_gmt":"2024-03-11T03:01:44","slug":"the-real-world-model","status":"publish","type":"post","link":"https:\/\/kihbernetics.org\/?p=1429","title":{"rendered":"The Real World Model"},"content":{"rendered":"\n<p>A couple of weeks ago, Yann LeCun posted his definition of a &#8220;world model&#8221; as used in AI on Twitter (<a href=\"https:\/\/x.com\/ylecun\/status\/1759933365241921817?s=20\" data-type=\"link\" data-id=\"https:\/\/x.com\/ylecun\/status\/1759933365241921817?s=20\" target=\"_blank\" rel=\"noreferrer noopener\">@ylecun<\/a>). The post had more than 400 K views, was reposted around 500 times, and got more than 150 comments. One of the comments was mine in which I asked if there is a block diagram explaining how everything works together and if it resembles the attached picture of my dynamical (<strong><em>kihbernetic<\/em><\/strong>) system model. My post was viewed a mere 35 times, and had no responses.<\/p>\n\n\n\n<p>I went through the whole thread a couple of times trying to understand the matter, and the more I read about it, the more I was frustrated with the apparent contradictions that no one seemed to be bothered with. Anyway, one of the comments I found most useful was from <a href=\"https:\/\/x.com\/ezamora1981\/status\/1759966822202233140?s=20\" target=\"_blank\" rel=\"noreferrer noopener\">Erik Zamora<\/a> and it contained this block diagram: <\/p>\n\n\n\n<div class=\"wp-block-media-text is-stacked-on-mobile is-vertically-aligned-center\" style=\"grid-template-columns:46% auto\"><figure class=\"wp-block-media-text__media\"><img decoding=\"async\" width=\"1024\" height=\"966\" src=\"https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/GGymx3pXMAAkK4V-1024x966.jpg\" alt=\"\" class=\"wp-image-1432 size-full\" srcset=\"https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/GGymx3pXMAAkK4V-1024x966.jpg 1024w, https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/GGymx3pXMAAkK4V-300x283.jpg 300w, https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/GGymx3pXMAAkK4V-768x725.jpg 768w, https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/GGymx3pXMAAkK4V.jpg 1530w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><div class=\"wp-block-media-text__content\">\n<p>In short, LeCun&#8217;s model is an <strong><em>observer<\/em><\/strong> that <em>encodes<\/em> <strong><em>observations<\/em><\/strong> (<em>x<sub>t<\/sub><\/em>) into <strong><em>representations <\/em><\/strong>(<em>h<sub>t<\/sub><\/em>) which are then fed into a <em>predictor<\/em>,<em> <\/em>along with all other inputs, such as an <strong><em>action proposal<\/em><\/strong> (<em>a<sub>t<\/sub><\/em>), a <strong><em>previous estimate of the state of the world<\/em><\/strong> (<em>s<sub>t<\/sub><\/em>), and a <em><strong>latent variable proposal<\/strong><\/em> (<em>z<sub>t<\/sub><\/em>) which according to LeCun &#8220;<em>represents the <strong>unknown information<\/strong> that would allow us to predict exactly what happens, and <em>parameterizes the set (or distribution) of plausible predictions<\/em>.&#8221;<\/em><br><br>Both <em><strong>Enc()<\/strong><\/em> and <strong><em>Pred()<\/em><\/strong> are trainable deterministic functions e.g. neural nets, and <em>z<sub>t<\/sub><\/em> must be sampled from a distribution or varied over a set.<br><br>LeCun says &#8220;<em>The trick is to train the entire thing from observation triplets (x(t),a(t),x(t+1)) while preventing the Encoder from collapsing to a trivial solution on which it ignores the input.<\/em>&#8220;<br><br>Zamora asked some very interesting questions that LeCun graciously answered as follows:<\/p>\n<\/div><\/div>\n\n\n\n<ol class=\"wp-block-list\">\n<li><em>Why s(t) and h(t) are not the same variables?<\/em>\n<ul class=\"wp-block-list\">\n<li><em>Because s(t) may contain information about the state of the world that is not contained in the current observation x(t). s(t) may contain information accumulated over time. For example. it could be a window of past h(t), or the content of a memory updated with h(t).<\/em><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><em>Can P(z(t)) be conditioned by actions, observations and s<\/em>tates?\n<ul class=\"wp-block-list\">\n<li><em>Yes, P(z(t)) can be conditioned on h(t), a(t), and s(t).<\/em><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><em>Why must Enc() and Pred() be deterministic?<\/em>\n<ul class=\"wp-block-list\">\n<li><em>The cleanest way to represent non-deterministic functions is to make a deterministic function depend on a latent variable (which can come from a distribution). This parameterization makes everything much clearer and non ambiguous.<\/em><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><em>Do we use Enc() to encode x(t+1)?<\/em>\n<ul class=\"wp-block-list\">\n<li><em>yes, Enc() is used to encode all x(t) as soon as they are available. When training the system, we observe a sequence of x(t)s and a(t)s and train Enc() and Pred() to minimize the prediction error of h(t+1). This implies that Pred() predicts h(t+1) as part of predicting s(t+1)<\/em>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><em>How to train P(z)?<\/em>\n<ul class=\"wp-block-list\">\n<li><em>During training, we observe a sequence of x(t) and a(t), so we can infer the best value of z(t) that minimizes the prediction error of h(t+1). We can train a system to model the distribution of z(t) thereby obtained, possibly conditioned on h(t), a(t), and s(t).<\/em><\/li>\n\n\n\n<li><em>Alternatively, we can infer a distribution q(z) over z(t) using Bayesian inference to minimize the free energy<br>F(q) = \\int_z q(z) E(z) + 1\/b \\int_z q(z) \\log(q(z))<br>where E(z) is the prediction error of h(t+1).<br>This trades the average energy for the entropy of q(z).<br>q(z) should be chosen in a family of distributions that makes this problem tractable, e.g. Gaussian (as in variational inference for VAE and other methods).<\/em><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p>Now, to give you an idea of my &#8220;proficiency&#8221; with AI, let&#8217;s just say that this last paragraph in LeCun&#8217;s explanation starting with &#8220;Alternatively&#8221; is way above my head. I am no software engineer and my average knowledge of math is enough to vaguely understand the mechanics of it all, but I can&#8217;t say anything about the complicated physical calculations and computational &#8220;tricks&#8221; SW engineers regularly use to do their &#8220;magic&#8221; in solving problems. However, having spent most of my career working along with SW developers, I know that sometimes they may be carried away, and try to implement &#8220;the right solution for the wrong problem&#8221;. So, it is the &#8220;mechanics&#8221; of the AI problem I want to talk about next, not the underlying &#8220;physics&#8221; and algorithms used to implement it which was the main topic of discussion in this Twitter thread that I had trouble following.<\/p>\n\n\n\n<p>The mechanics I know best how it works is the one of <strong><em>control<\/em><\/strong> in <strong><em>dynamical systems<\/em><\/strong>. An AI is a <em>dynamical <strong>learning <\/strong>system<\/em> and all such systems have similar structures made with some mandatory components. Of course, the details of the underlying &#8220;algorithms&#8221; will be different for various physical substrates that the components (natural and\/or artificial) are made of, and the capabilities that each one of those systems can support. The mandatory components of any dynamical (kihbernetic) system and their relationships that I identify in my <strong><em>Kihbernetic system model<\/em><\/strong> are as follows: <\/p>\n\n\n\n<div class=\"wp-block-media-text is-stacked-on-mobile\"><figure class=\"wp-block-media-text__media\"><img decoding=\"async\" width=\"735\" height=\"618\" src=\"https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/DSM-State-Space-Complete.png\" alt=\"\" class=\"wp-image-1436 size-full\" srcset=\"https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/DSM-State-Space-Complete.png 735w, https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/DSM-State-Space-Complete-300x252.png 300w\" sizes=\"(max-width: 735px) 100vw, 735px\" \/><\/figure><div class=\"wp-block-media-text__content\">\n<p>1\ufe0f\u20e3 <strong>System <\/strong>with a <strong><em>unique identity<\/em><\/strong>. Learning systems will individually <em>grow <\/em>through a unique sequence of interactions with their immediate <em>environment<\/em> and thus have a unique <em>history <\/em>of experiences that cannot simply be duplicated into another, <em>identical <\/em>system.<br><strong>2\ufe0f\u20e3Processes<\/strong> &#8211; a recursive internal <strong><em>autopoiesis<\/em><\/strong> of <em>learning <\/em>and <em>growth<\/em>, and the linear <strong><em>allopoiesis<\/em><\/strong> dealing sequentially with external environmental issues such as collecting resources and getting rid of waste, tracking the state of the environment and reacting appropriately, and cooperating with other similar entities. These processes are distributed on<br>3\ufe0f\u20e3<strong>Control levels<\/strong> &#8211; The immediate <strong><em>regulation<\/em><\/strong> of a large number of variables for maintaining the overall stability (homeostasis) of the system; The <strong><em>control <\/em><\/strong>of the regulators by tracking their performance and spreading their burden, as well as the implementation of &#8220;higher goals&#8221; by planning, and scheduling necessary regulators&#8217; engagement, and finally the <strong><em>guidance <\/em><\/strong>level for maintaining the identity, the long term goals and character of the system.<br>4\ufe0f\u20e3<strong>Variables<\/strong> &#8211; <strong><em>Input <\/em><\/strong>(material, energy, and data), <strong><em>Output <\/em><\/strong>(waste, action\/behavior, data), <strong><em>Information <\/em><\/strong>(&#8220;the difference that makes a difference&#8221;), and the <strong><em>Knowledge <\/em><\/strong>state of the system.<br>5\ufe0f\u20e3<strong>Functions<\/strong> &#8211; <strong><em>Reaction<\/em><\/strong>, <strong><em>Perception<\/em><\/strong>, <strong><em>Integration<\/em><\/strong> <strong><em>Prediction<\/em><\/strong>, and <strong><em>Control<\/em><\/strong><\/p>\n<\/div><\/div>\n\n\n\n<p>It is these four variables and five functions that I wish to discuss in more detail related to the &#8220;World Model&#8221; proposal from above.<\/p>\n\n\n\n<p>LeCun&#8217;s model specifies only two functions: <strong><em>Encoding Enc()<\/em><\/strong> and <strong><em>Prediction Pred()<\/em><\/strong> but as Zamora correctly identifies (and LeCun seems to agree) it should have at least another <strong><em>Delay <\/em><\/strong>function that serves as a kind of <em>memory <\/em>for preserving the previous state(s) of the model, and a <strong><em>Generator <\/em><\/strong><em>(<strong>P(z)<\/strong><\/em> in Zamora&#8217;s depiction) of the <em>latent variable proposal<\/em> or <em>unknown information<\/em> (<em>z<sub>t<\/sub><\/em>). However, it is still unclear (to me) how (where) the <em>action proposal<\/em> (<em>a<sub>t<\/sub><\/em>) may be generated. LeCun explains the meaning of <em>a<sub>t<\/sub><\/em> with this example that does not really help in identifying the originator:<\/p>\n\n\n\n<p><em>x(t): a glass is sitting on a table<br>a(t): a hand pushes the glass by 10cm<br>x(t+1): the glass is now 10cm away from where it was<\/em>.<\/p>\n\n\n\n<p>Obviously, <em>a<sub>t<\/sub><\/em> and <em>z<sub>t<\/sub><\/em> are the variables I have the most &#8220;problems&#8221; with. I can understand the concept of <em>observation <\/em>(<em>x<sub>t<\/sub><\/em>) and of <em>state <\/em>(<em>s<sub>t<\/sub><\/em>) which would correspond to the <em>Input <\/em>and <em>Knowledge <\/em>variables in my model. I can even understand the concept of internal <em>representation <\/em>(<em>h<sub>t<\/sub><\/em>) of the observed data (<em>x<sub>t<\/sub><\/em>), which would correspond to the arrow pointing out from the <em>Perception <\/em>function (<em>B<\/em>) in my kihbernetic model, which is the function responsible for converting (transducing) <em>external <\/em>physical perturbances of the &#8220;sensory apparatus&#8221; into <em>internal <\/em>low-level &#8220;neural states&#8221; of the observer system, which are more suitable for further neural &#8220;processing&#8221;. However, simply selecting <em>a<sub>t<\/sub><\/em> and <em>z<sub>t<\/sub><\/em> by sampling from a distribution, or varying over some prepared set seems (to me) like a &#8220;cheap trick&#8221; for making a deterministic function look like it is not deterministic.<\/p>\n\n\n\n<p>The variable <em>a<sub>t<\/sub><\/em> (the <em>action proposal<\/em>) is ambiguous. It can represent either one proposal (<em>intention<\/em>) among many, of the observer <em>internally<\/em> running a <em>simulation <\/em>to identify the best option for a future action. This proposal(s) affect (refine) only the observer&#8217;s <em>knowledge <\/em>of the world in the form of the state <em>s<sub>t<\/sub><\/em> unlike the <em>final <\/em>proposal selected for a real action (<em>experiment<\/em>) that may change things in the real world, and the consequences of which may become known only after making the next observation <em>x<sub>t<\/sub><\/em>.<\/p>\n\n\n\n<p>The variable <em>z<sub>t<\/sub><\/em> which as LeCuns says &#8220;<em>represents the unknown information that would allow us to predict exactly what happens<\/em>&#8221; is exactly that: <em>information<\/em>. The only thing is that, by definition, <em>all information is unknown<\/em>. If it was known it would be <em>knowledge<\/em> rather than information. In the kihbernetic model, we use <strong><em>knowledge <\/em><\/strong>to extract <strong><em>information <\/em><\/strong>from <strong><em>data<\/em><\/strong>. Two different observer systems (or even the same system at different points in time) will extract <em>different <\/em>information from the same set of data, depending on their current <em>knowledge <strong>state<\/strong><\/em>.<\/p>\n\n\n\n<p>To better explain my point of view for the appropriate representation of an &#8220;AI world model&#8221; I produced the following <em>mapping <\/em>of LeCun&#8217;s world model onto my <em>kihbernetic <\/em>system model. LeCun&#8217;s <strong><em>Pred()<\/em><\/strong> function combines my <strong><em>Perception <\/em><\/strong>and <strong><em>Prediction<\/em><\/strong> while <strong><em>Enc() <\/em><\/strong>maps to my <strong><em>Control <\/em><\/strong>&amp; <strong><em>Reaction<\/em><\/strong>:<\/p>\n\n\n\n<div class=\"wp-block-media-text is-stacked-on-mobile is-vertically-aligned-center\" style=\"grid-template-columns:41% auto\"><figure class=\"wp-block-media-text__media\"><img decoding=\"async\" width=\"856\" height=\"1024\" src=\"https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/World-Model-1-856x1024.png\" alt=\"\" class=\"wp-image-1448 size-full\" srcset=\"https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/World-Model-1-856x1024.png 856w, https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/World-Model-1-251x300.png 251w, https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/World-Model-1-768x919.png 768w, https:\/\/kihbernetics.org\/wp-content\/uploads\/2024\/02\/World-Model-1.png 1097w\" sizes=\"(max-width: 856px) 100vw, 856px\" \/><\/figure><div class=\"wp-block-media-text__content\">\n<p><em>Pred()<\/em> and <em>Enc()<\/em> are still deterministic algorithmic functions but their original meaning is somehow different in my interpretation. The thing that introduces a degree of uncertainty is the (also deterministic) <em>memory <\/em>function on the top which can range from a simple <em>delay <\/em>function to a more complex <em>integrator <\/em>maintaining the <em>knowledge state<\/em> of the whole system.<\/p>\n\n\n\n<p>The <em>Enc()<\/em> (encoding) function instead of just doing the trivial transformation of <strong><em>external <\/em><\/strong>observations (measurements) (<em>x<sub>t<\/sub><\/em>) into <strong><em>internal <\/em><\/strong>representations (<em>h<sub>t<\/sub><\/em>) is now encoding the <strong><em>final <\/em><\/strong>action proposal (<em>a<sub>t<\/sub><\/em>) selected as the most appropriate (best) response to the input (prompt) that will induce (or not) some reaction in the real world that may be witnessed in some subsequent input (<em>x<sub>t<\/sub><\/em>). Obviously, the real world acts here as the &#8220;reward system&#8221;, which is how it should be if we want our AI tools to align<strong><em> <\/em><\/strong>with our values. The internal variables <em>s<sub>t<\/sub><\/em> and <em>a<sub>t<\/sub><\/em> coming out of the &#8220;memory function&#8221; represent the <em>knowledge state<\/em> of the system which is fed to the other two functions as a <strong><em>parameter <\/em><\/strong>that changes their &#8220;behavior&#8221; so that, depending on the state they are in, they may produce a different (better) output for the same input, which is, if you think about it, the definition of <strong><em>learning<\/em><\/strong>.    <\/p>\n\n\n\n<p>The <em>Pred()<\/em> (prediction) function is the central tenet of every learning system. To learn, the system must have some <strong><em>expectations <\/em><\/strong>about the future generated from the <em><strong>knowledge <\/strong>state<\/em> discussed above. An important part of this <em>state vector<\/em> is also the <em>preliminary action proposals<\/em> fed to the prediction function as &#8220;what if&#8221; inputs for internal &#8220;simulation&#8221; and refinement of possible real-world scenarios <em>before <\/em>selecting the one that will actually be encoded. This internal <em>autopoietic <\/em>(learning) loop &#8220;runs&#8221; with a frequency that is an order of magnitude higher than the external action loop through the environment, and the main principle is the minimization of the <strong><em>information<\/em> <\/strong>variables (<em>z<sub>t<\/sub><\/em> or <em>s<sub>t+1<\/sub><\/em>) that represent the <em>difference<\/em> between the expected state (predicted by the world model simulation) and the actual (observed) one.<\/p>\n<\/div><\/div>\n\n\n\n<p>So, to conclude let&#8217;s describe the proposed &#8220;mechanics&#8221; of all of this. An observation, input, or prompt is available as <em>input <\/em>to the system where it is encoded into an <em>action response output <\/em>based on the current <em><strong>knowledge <\/strong>state<\/em> of the system. If the input describes a situation in the real world that is already known to the system because it dealt with it in the past that&#8217;s all that is needed for a proper response. The system can detect this is a known problem because the predictions it made in the previous step correspond to the observed input in this step, so the output can be encoded with the existing knowledge. <\/p>\n\n\n\n<p>If the response delivered into the real world is not appropriate for the current situation, the world will respond differently than expected and the observation will not match the prediction, so there will be a <em>difference <\/em>between the two (<strong><em>information<\/em><\/strong>) that will have to be integrated with the existing knowledge into a new (upgraded) knowledge state that will hopefully allow for a better prediction and response. The dynamic of this internal <em>learning results <\/em>cycle does not depend on the dynamic of the external &#8220;reward&#8221; loop. The system is free to learn at any pace with an arbitrary precision of how predictions should match observations. The system can also test multiple &#8220;what if&#8221; scenarios within these learning cycles and even generate <em>test action proposals<\/em> to experiment with different scenarios in the real world.<\/p>\n\n\n\n<p>In the end, I want to draw attention to the work of researchers like <a href=\"https:\/\/twitter.com\/_albertgu\" target=\"_blank\" rel=\"noreferrer noopener\">Albert Gu<\/a> using <strong><em>structured state space models<\/em><\/strong> (SSM) as the base for developing AI agents (see for example <a href=\"https:\/\/arxiv.org\/abs\/2111.00396\">here<\/a>). This work seems to me very promising with lots of potential and similar to the ideas I&#8217;ve presented in this post.<\/p>\n\n\n\n<p>  <\/p>\n\n\n\n<p>  <\/p>\n","protected":false},"excerpt":{"rendered":"<p>A couple of weeks ago, Yann LeCun posted his definition of a &#8220;world model&#8221; as used in AI on Twitter (@ylecun). The post had more than 400 K views, was reposted around 500 times, and got more than 150 comments. One of the comments was mine in which I asked if there is a block [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1462,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[16],"tags":[],"class_list":["post-1429","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-modeling"],"modified_by":"py","_links":{"self":[{"href":"https:\/\/kihbernetics.org\/index.php?rest_route=\/wp\/v2\/posts\/1429","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kihbernetics.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kihbernetics.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kihbernetics.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kihbernetics.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1429"}],"version-history":[{"count":27,"href":"https:\/\/kihbernetics.org\/index.php?rest_route=\/wp\/v2\/posts\/1429\/revisions"}],"predecessor-version":[{"id":1461,"href":"https:\/\/kihbernetics.org\/index.php?rest_route=\/wp\/v2\/posts\/1429\/revisions\/1461"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kihbernetics.org\/index.php?rest_route=\/wp\/v2\/media\/1462"}],"wp:attachment":[{"href":"https:\/\/kihbernetics.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1429"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kihbernetics.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1429"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kihbernetics.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1429"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}