Jekyll2021-10-01T19:03:35+00:00https://mehrsoft.com/MyBlog/feed.xmlContinuous Exploration & ExploitationDocumenting my journey getting deeper into software and and learning technologies.
Daryoush MehrtashKernels and Dot product2021-03-07T00:00:00+00:002021-03-07T00:00:00+00:00https://mehrsoft.com/MyBlog/2021/03/07/kernels-and-dotprodcut<blockquote>
<p>Kernels and Deep learning.</p>
</blockquote>
<!--more-->
<p>Following comment in</p>
<blockquote>
<p>The one thing that irks me is kernels keep being referred to as “mapping to infinite dimensional space” in these vids. While true, it’s kind of unhelpful. It’s probably more helpful to think of it as mapping to a (potentially infinite) space where the (potentially infinite) bases aren’t restricted to be vectors. The whole trick about the Hilbert space is it’s just a space with a well-defined inner product, and that inner product can be between functions etc. You could have a function mapping to a space where the bases are the sin and cosine functions, for example, and you use that space to take the inner product between different sine/cosine wave combinations. Typically the kind of space you want to map to will express something about the problem you’re trying to solve.</p>
<p>I bring this up because I’ve seen kernels mentioned twice now, and both times the explanations are kind of lacking. Not Yannic’s fault, I was lucky enough to spend a bunch of time around kernel gurus that cleared up a bunch of misconceptions for me…before that, I was making similar mistakes. Content otherwise very solid, as usual!</p>
</blockquote>
<p>in</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/hAooAOFRsYc" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>is informative.</p>
<p>The author “Alex Stenlake” has interesting talk in the following Machine Learning Street Talk:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/y_RjsDHl5Y4" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>Daryoush MehrtashKernels and Deep learning.Operational vs Denotational2018-10-16T00:00:00+00:002018-10-16T00:00:00+00:00https://mehrsoft.com/MyBlog/2018/10/16/Opperational-Denotational<blockquote>
<p>The question of language is not functional vs imperative. Rather Operational Semantic vs Denotational Semantic.</p>
</blockquote>
<!--more-->
<p>I am trying to evaluate GO language for a project I am interested in. It appears that there are some “go gotcha” constructs that one needs to be aware of when programming in go language.</p>
<p>A useful blog post on the topic is: <a href="https://divan.github.io/posts/avoid_gotchas/">How to avoid Go gotchas</a>. The intersting part is:</p>
<blockquote>
<p>Remember, “gotchas are .. valid constructs .. but is counter-intuitive”? That is it. You have only two options:</p>
<p>“fix” the language
fix the intuition
Second is actually would be better seen as build the intuition. Once you have a clear mental image of how slices or interfaces work under the hood, it’s almost impossible to run into those mistakes.</p>
<p>So, it worked for me and probably will work for others. That’s why I decided to gather that basic knowledge of some Go internals in this post and help people to build the intuition about an in-memory representation of different things.</p>
</blockquote>
<p>This demonstrates the fundamental problem with programming languages that are defined by their operational semantic. To make sense of a code you need to have a “clear mental” image of <strong>HOW</strong>
the program runs not <strong>WHAT</strong>, or the meaning of the instruction.</p>
<p>Conal Elliot’s <a href="https://www.youtube.com/embed/teRC_Lf61Gw">presentation</a> covers the topic more in depth.</p>Daryoush MehrtashThe question of language is not functional vs imperative. Rather Operational Semantic vs Denotational Semantic.Reproducing Kernel Hilbert Space explained2018-10-15T00:00:00+00:002018-10-15T00:00:00+00:00https://mehrsoft.com/MyBlog/2018/10/15/RKHS<blockquote>
<p>Couple of simple yet in-depth resources to learn about Reproducing Kernel Hilbert Spaces (RKHS).</p>
</blockquote>
<!--more-->
<p>I am trying to learn application of <a href="http://www.ong-home.my/papers/krause11cgp-ucb.pdf"> Gaussian process to Contextual Bandit</a> problem in re-reinforcement learning. Needed a refresher on the RKHS and found this write up: <a href="http://users.umiacs.umd.edu/~hal/docs/daume04rkhs.pdf">From Zero to Reproducing Kernel Hilbert Spaces in Twelve Pages or Less</a>
and <a href="https://youtu.be/gb25CfVZm3Q">RKHS Lecture on YouTube</a> useful.</p>Daryoush MehrtashCouple of simple yet in-depth resources to learn about Reproducing Kernel Hilbert Spaces (RKHS).Differential Form2018-09-25T00:00:00+00:002018-09-25T00:00:00+00:00https://mehrsoft.com/MyBlog/2018/09/25/differential-form<blockquote>
<p>idea of differential form has interesting connection with lambda expression’s abstraction and application.</p>
</blockquote>
<!--more-->
<p>This video does a nice job defining differnetial form in concepts that are also in Lambda Calculus.</p>
<p><a href="https://www.youtube.com/watch?v=Fy9GRN02I1A">Introduction to Differential Form</a></p>Daryoush Mehrtashidea of differential form has interesting connection with lambda expression’s abstraction and application.Dealing with Unknown2018-08-30T18:57:00+00:002018-08-30T18:57:00+00:00https://mehrsoft.com/MyBlog/2018/08/30/dealing-with-unknownThe 2nd paragraph of the Shannon's <a href="http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf">A Mathematical Theory of Communication paper</a> says: <blockquote>The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.</blockquote> BACHELIER's The <a href="http://http://www.radio.goldseek.com/bachelier-thesis-theory-of-speculation-en.pdf">THE THEORY OF SPECULATION</a> paper that started the Finance theory starts off with: <blockquote>The determination of these fluctuations is subject to an infinite number of factors: it is therefore impossible to expect a mathematically exact forecast. Contradictory opinions in regard to these fluctuations are so divided that at the same instant buyers believe the market is rising and sellers that it is falling. Undoubtedly, the Theory of Probability will never be applicable to the movements of quoted prices and the dynamics of the Stock Exchange will never be an exact science. However, it is possible to study mathematically the static state of the market at a given instant, that is to say, to establish the probability law for the price fluctuations that the market admits at this instant. Indeed, while the market does not foresee fluctuations, it considers which of them are more or less probable, and this probability can be evaluated mathematically.</blockquote>daryoushThe 2nd paragraph of the Shannon's A Mathematical Theory of Communication paper says: The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design. BACHELIER's The THE THEORY OF SPECULATION paper that started the Finance theory starts off with: The determination of these fluctuations is subject to an infinite number of factors: it is therefore impossible to expect a mathematically exact forecast. Contradictory opinions in regard to these fluctuations are so divided that at the same instant buyers believe the market is rising and sellers that it is falling. Undoubtedly, the Theory of Probability will never be applicable to the movements of quoted prices and the dynamics of the Stock Exchange will never be an exact science. However, it is possible to study mathematically the static state of the market at a given instant, that is to say, to establish the probability law for the price fluctuations that the market admits at this instant. Indeed, while the market does not foresee fluctuations, it considers which of them are more or less probable, and this probability can be evaluated mathematically.Fractional Derivatives and Robot Swarms2018-08-23T18:39:00+00:002018-08-23T18:39:00+00:00https://mehrsoft.com/MyBlog/2018/08/23/fractional-derivatives-and-robot-swarmsAn interesting idea... Note to self: read the <a href="https://curate.nd.edu/show/37720c50p2z">thesis</a><br /><br /><br /><br /><br /><br /><iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/AkS1TCAZCMw" width="480"></iframe><br /><br /><br /> This is related to the observation that the intelligent system's interaction is more complicated than simple processes that model the interaction as a random and uncorrelated bouncing of particles (Brownian motion). <iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/QWvDPe6GaSA" width="480"></iframe> <br/>Which in turn may be related to <br/><iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/gB9n2gHsHN4" width="480"></iframe> <br/>And more on fractional thinking: <a href="https://secure.inl.gov/ISRCS2010/docs/abstracts/Chen.pdf">Fractional Calculus, Delay Dynamics and Networked Control Systems</a>daryoushAn interesting idea... Note to self: read the thesis This is related to the observation that the intelligent system's interaction is more complicated than simple processes that model the interaction as a random and uncorrelated bouncing of particles (Brownian motion). Which in turn may be related to And more on fractional thinking: Fractional Calculus, Delay Dynamics and Networked Control SystemsFunctional Programming and Denotation Semantic2018-08-22T22:36:00+00:002018-08-22T22:36:00+00:00https://mehrsoft.com/MyBlog/2018/08/22/functional-programming-and-denotationIn <a href="https://www.cs.cmu.edu/~crary/819-f09/Landin66.pdf">The Next 700 Programming Languages</a> paper Peter Ladin define the key feature of functional programming. He says: <blockquote>The commonplace expressions of arithmetic and algebra have a certain simplicity that most communications to computers lack. In particular, (a) each expression has a nesting subexpression structure, (b) each subexpression denotes something (usually a number, truth value or numerical function), (c) the thing an expression denotes, i.e., its "value", depends only on the values of its subexpressions, not on other properties of them. It is these properties, and crucially (c), that explains why such expressions are easier to construct and understand. Thus it is (c) that lies behind the evolutionary trend towards "bigger righthand sides" in place of strings of small, explicitly sequenced assignments and jumps. When faced with a new notation that borrows the functional appearance of everyday algebra, it is (c) that gives us a test for whether the notation is genuinely functional or merely masquerading. </blockquote><br/>It all starts with a functional language. Following lecture shows Denotational Semantic of an expression in Lambda Calculus to Set. <iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/EI5i54VrAbc" width="480"></iframe> <br/>Once the meaning of an expression is knowns we need the meaning of the values that the expression reduces to. This is where as a designer you define the meaning of the values in your specific domain. <br/>In the following lecture, Conal Elliot shows how you extend the denotational semantic to values. He contrasts this approach to the standard imperative API model. In imperative API the interface is about creating the state, initializing it and defining functions to side effect the state. <br/>Functional programming, on the other hand, is about APIs to define things that "ARE" in the domain. There is a clear distinction between values and evaluation. So you have base values, transformation, and composition operators. In all, you need to make sure there is a clear, and precise meaning to the values and that it can lead to an efficient implementation. The distinction is sometimes hard to get. For me an analogy is SQL. An SQL expression defines that property of the result your are interested in, the database engine then turns that into an execution plan and evaluates it to give you the results. <iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/teRC_Lf61Gw" width="480"></iframe><br/>The key take away from the talk is that it is not about Functional vs Imperative or if a particular language does or doesn't have lambda expression. It is about Denotational vs Operational semantic. Another word, do you think about your program in term of its execution model or in term of the meaning of the expressions. <br/>An interesting concept that shows up in Conal's work is that the semantic function is a natural transformation. Bartosz Milewski does a nice job of explaining what the natural transformation is. <iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/2LJC-XD5Ffo" width="480"></iframe>daryoushIn The Next 700 Programming Languages paper Peter Ladin define the key feature of functional programming. He says: The commonplace expressions of arithmetic and algebra have a certain simplicity that most communications to computers lack. In particular, (a) each expression has a nesting subexpression structure, (b) each subexpression denotes something (usually a number, truth value or numerical function), (c) the thing an expression denotes, i.e., its "value", depends only on the values of its subexpressions, not on other properties of them. It is these properties, and crucially (c), that explains why such expressions are easier to construct and understand. Thus it is (c) that lies behind the evolutionary trend towards "bigger righthand sides" in place of strings of small, explicitly sequenced assignments and jumps. When faced with a new notation that borrows the functional appearance of everyday algebra, it is (c) that gives us a test for whether the notation is genuinely functional or merely masquerading. It all starts with a functional language. Following lecture shows Denotational Semantic of an expression in Lambda Calculus to Set. Once the meaning of an expression is knowns we need the meaning of the values that the expression reduces to. This is where as a designer you define the meaning of the values in your specific domain. In the following lecture, Conal Elliot shows how you extend the denotational semantic to values. He contrasts this approach to the standard imperative API model. In imperative API the interface is about creating the state, initializing it and defining functions to side effect the state. Functional programming, on the other hand, is about APIs to define things that "ARE" in the domain. There is a clear distinction between values and evaluation. So you have base values, transformation, and composition operators. In all, you need to make sure there is a clear, and precise meaning to the values and that it can lead to an efficient implementation. The distinction is sometimes hard to get. For me an analogy is SQL. An SQL expression defines that property of the result your are interested in, the database engine then turns that into an execution plan and evaluates it to give you the results. The key take away from the talk is that it is not about Functional vs Imperative or if a particular language does or doesn't have lambda expression. It is about Denotational vs Operational semantic. Another word, do you think about your program in term of its execution model or in term of the meaning of the expressions. An interesting concept that shows up in Conal's work is that the semantic function is a natural transformation. Bartosz Milewski does a nice job of explaining what the natural transformation is.In Opening Black Box Of Deep Neural2018-08-13T22:40:00+00:002018-08-13T22:40:00+00:00https://mehrsoft.com/MyBlog/2018/08/13/in-opening-black-box-of-deep-neuralThe <a href="https://arxiv.org/abs/1703.00810">Opening the Black Box of Deep Neural Networks via Information</a> uses the diffusion process and heat equation as intuition for gradient flow between the layers of the deep network. I needed a refresher on the underlying math. Found some very good sources: <br/><br/>In "<a href="https://youtu.be/4J74tquQ7jU">What is Laplacian</a>" and subsequent video you get a great insightful introduction of Laplacian, heat and wave equations. <br/><br/> In the <a href="https://ocw.mit.edu/resources/res-18-009-learn-differential-equations-up-close-with-gilbert-strang-and-cleve-moler-fall-2015/differential-equations-and-linear-algebra/applied-mathematics-and-ata/">applied math section</a> which continues to <a href="https://ocw.mit.edu/resources/res-18-009-learn-differential-equations-up-close-with-gilbert-strang-and-cleve-moler-fall-2015/differential-equations-and-linear-algebra/fourier-and-laplace-transforms/">"Fourier and Laplace Transformation"</a> section of the differential equation lecture series, there is also a shorter explanation of diffusion. <br/><br/> There is also stochastic nature to the whole thing that is nicely covered in the Stochastic process, Ito calculus , and stochastic differential equation section of the Topics in Mathematics with Applications in Finance lectures with <a href="https://ocw.mit.edu/courses/mathematics/18-s096-topics-in-mathematics-with-applications-in-finance-fall-2013/lecture-notes/">Notes</a> and <a href="https://ocw.mit.edu/courses/mathematics/18-s096-topics-in-mathematics-with-applications-in-finance-fall-2013/video-lectures/">Vidoes</a>.daryoushThe Opening the Black Box of Deep Neural Networks via Information uses the diffusion process and heat equation as intuition for gradient flow between the layers of the deep network. I needed a refresher on the underlying math. Found some very good sources: In "What is Laplacian" and subsequent video you get a great insightful introduction of Laplacian, heat and wave equations. In the applied math section which continues to "Fourier and Laplace Transformation" section of the differential equation lecture series, there is also a shorter explanation of diffusion. There is also stochastic nature to the whole thing that is nicely covered in the Stochastic process, Ito calculus , and stochastic differential equation section of the Topics in Mathematics with Applications in Finance lectures with Notes and Vidoes.From General Solution to Specific2018-07-18T21:08:00+00:002018-07-18T21:08:00+00:00https://mehrsoft.com/MyBlog/2018/07/18/from-general-solution-to-specific<blockquote>One of the most important skills in mathematical problem solving is the ability to generalize. A given problem, such as the integral we just computed, may appear to be intractable on its own. However, by stepping back and considering the problem not in isolation but as an individual member of an entire class of related problems, we can discern facts that were previously hidden from us. Trying to compute the integral for the particular value a = 1 was too difficult, so instead we calculated the value of the integral for every possibly value of a. Paradoxically, there are many situations where it is actually easier to solve a general problem than it is to solve a specific.</blockquote> <a href="https://medium.com/@jackebersole/richard-feynmans-integral-trick-e7afae85e25c">Richard Feynman’s integral trick </a>daryoushOne of the most important skills in mathematical problem solving is the ability to generalize. A given problem, such as the integral we just computed, may appear to be intractable on its own. However, by stepping back and considering the problem not in isolation but as an individual member of an entire class of related problems, we can discern facts that were previously hidden from us. Trying to compute the integral for the particular value a = 1 was too difficult, so instead we calculated the value of the integral for every possibly value of a. Paradoxically, there are many situations where it is actually easier to solve a general problem than it is to solve a specific. Richard Feynman’s integral trickCommunication and Learning2018-07-18T18:10:00+00:002018-07-18T18:10:00+00:00https://mehrsoft.com/MyBlog/2018/07/18/communication-and-learning<blockquote>The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design. </blockquote><a href="http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf">A Mathematical Theory of Communication</a>daryoushThe fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design. A Mathematical Theory of Communication